We’re back! The entire OSAF/Chandler world was offline for about 24 hours from April 27th at 2pm to April 28th. This outage has been fixed with no permanent damage.

The root cause was “sparking” (ouch) on the power lines reported by PG&E.  Our colo hosts, Hosted @ ISC, switched over to their generator as a precaution but the generator failed.  ISC started shutting down machines when they knew the outage would exceed their UPS power.

Here’s where our outage stretched out longer than it needed to. ISC didn’t let us know they were powering down the machines, or that power was back up a couple hours later. They responded to my 3pm inquiry to their ticketing system late on the 27th, saying that everything should have been fine hours ago. Unfortunately, I had already gone to bed, so the OSAF/Chandler outage had to wait out the night.

In the morning of the 28th, I asked ISC to hit the power buttons on the machines for me, but nothing happened. I packed up quick tech kit plus some spare machines and hopped in the car for the hour drive to Redwood City. On-site around noon, I confirmed the machines appeared dead. Weird. Hopefully the power issues hadn’t killed all 4 physical machines’ power supplies or motherboards at once, right?

Turns out, our managed power device that lets us turn machines on and off remotely had taken a header. I started moving machines’ power around, finding out by the end which ports on the power switch were dead. The Hub machine moved off the power switch entirely.

That would have all be relatively simple, except for the extended duration, but there were other post-shutdown details that smacked me around for a while. I found that the tightened DNS configuration implemented back when DNS security went crazy kept our DNS machines from answering questions from themselves. I kept production machines like Hub off until I had DNS sorted out.

But far worse was a surprise related to the Debian Etch to Lenny upgrade. Since Lenny was released a few months ago, I’ve been upgrading OSAF machines in the background. I don’t know how I missed this, but the upgrade can remove the package used to bring up networking (ifupdown). I had been breaking the cardinal sysadmin rule of always rebooting machines after upgrading the OS, so I was very very confused when some machines came back up without any networking. Including our primary DNS server. And the secondary DNS server seemed broken because of the DNS config error I mentioned above.

I shot myself in the foot even more because before I realized all this, I had decided that the already-extended outage “seemed a good time” to upgrade the Hub from etch to lenny. After the reboot, no networking! Gah, it was just working fine before the upgrade!

So, after getting a grip on what was going on, tracing through networking startup scripts, and tracking down the missing ifupdown package issue, I go bumming through the ISC office for a USB key. I use one of the existing machines to pull the ifupdown package (both i386 for the virtual machines and amd64 for the Hub) using a command-line web browser, troll through syslog to figure out what device to mount, and get these packages onto their needed machines. Luckily, this all works and all machines come back up.

All’s well that ends well I suppose, but this was a tough outage to swallow after the 9 hours from the big fiber cut less than a month ago. Natural questions like “is our hosting good enough?” and “should we move?” come up.

My view is that weird things happen in every hosting environment; moving is not a cure-all for reliability issues. Our reliability isn’t as good as we’d like or as good as could be achieved, but I feel it’s still better to sit tight. The main reason is cost: hosting the Hub consumes a good amount of bandwidth (about 8Mbps) and ISC is providing all services (space, power, bandwidth, remote hands) for free for 9 rack units worth of equipment. It would be possible to move some services like mailing lists, code repositories, and wikis to other free services and someday the community may choose to go that route.  But it’s nice to not have restrictions on capabilities or capacity that we get from hosting almost all of our own services; many open source projects would love to have access to the resources and flexibility that OSAF enjoys.

Overall I think we’ve just had a spate of bad luck and while a very rare 24-hour outage might be unacceptable to a commercial venture, as long as they stay very rare, they are acceptable to OSAF and Chandler communities and very worth the tradeoffs.

There are three items I want to undertake as a result of the outage:

• Place a DNS secondary outside of ISC
• Print out some phone numbers, IP ranges, and other “might need it offline” info
• Talk to ISC about coordinating during outages

As a final note, it turns out I could have determined that “everything should be fine now” by going to status.isc.org where there was some information about the outage, including when it ended. I hadn’t been checking that page during the outage but I certainly will (as well as using the phone as needed) during any future outages.

A new Chandler Desktop release, version 1.0.3, is now available.

Highlights include better support for Google’s CalDAV server,
the addition of a menu for choosing the calendar’s first day of the week
and, lastly, we’ve built .deb packages — for both 32-bit
and 64-bit processors — for Ubuntu 8.04 (“Hardy”) and
8.10 “Intrepid”).

Here is the full list of bugs fixed in 1.0.3:

• 11038 Feature request: a UI affordance for selecting the starting day of the week
• 12228 Get Mac native spell checker working again
• 12253 Sy&nc Mana&ger… (note two ampersands) in menus.py:223
• 12274 Deprecated libiuc36 dependency
• 12278 Support Google’s CalDAV implementation
• 12328 make all binaries in internal fails with: NameError: global name ‘log’ is not defined
• 12472 ValueError: too many values to unpack (selected message has angle brackets within the name as well as the SMTP address)
• 12484 Won’t start (Mandriva and possibly other Linuxes)

You can download the app here,
or by using the “Check for Updates” feature in Chandler.

Things recently released their 1.0 and there’s been some interesting discussion about it vis-a-vis Chandler on the Users-List.

Some interesting analysis of relative pros and cons include:

- Things’ more native Mac OSX experience
- New ability to sync Things with Apple iCal
- Chandler’s ability to share and sync across multiple computers
- Always present notes pane (Chandler) versus separate pop-up (Things)
- Lack of printing in Chandler

Have you tried out Things? What has your experience been? What do you like about Things that you would like to see incorporated in Chandler?

Screenshot of Things was taken from the Culture Code website.

I have been fairly silent on the blog and mailing lists so I thought I would follow Eugene’s lead and post an update. My role over the past several months has been largely behind-the-scenes to tackle administrative and operational issues that support our transition to an all volunteer organization. Those activities are soon coming to a close and I look forward to resuming more outward participation in the project and engaging with our growing community. It has been great to see all the activity on the users list particularly from newcomers stepping up to help answer questions, log bugs and engage in discussions.

As Eugene mentioned, the new board met for the first time in mid-November. Also in attendance were the remaining OSAF staffers, Mimi, Grant and Jeffrey. Katie was also on hand to help provide some context for the new board. This meeting was an opportunity for all the new board members to get to know each other and meet in person. We also talked about where the project is today, the assets we have and what we might want to achieve in new year. For me, it was a forum for soliciting feedback on wide range of administrative issues that needed to be taken care of. Both Grant and Jeffrey provided an update on the architecture project and we agreed to let them continue to build out the data and interactions models and we would access our progress in early 2009. Last week Mimi posted our plan through the end of Feb 2009 on the dev list.

Our next board meeting is scheduled for February 6th, 2009. Our agenda hasn’t been finalized but will likely focus on the state of the architecture work and nailing down a plan for evangelizing the progress we have made so far. Both Grant and Jeffrey will be transitioning to volunteers at the end of February but will remain active members of our community, joining many former OSAF staffers.

I am still using Chandler on a daily basis to manage my kids lives. I blogged about this earlier in the year. Since then, my collections have expanded to include a number of other personal projects. I am very proud of what we accomplished in 2008 and am energized to help further build out our community in 2009.

Late last year, I had the distinct pleasure of joining the OSAF board. I had gotten to know Mimi, Katie, and several other OSAF staffers over the years and had watched the project evolve with some interest. Chandler had gone through a very public awkward period, but it had also gone through a less public maturation period. Chandler had become something that was very cool, something that I’ve been using every day for almost a year now. More importantly, I found the larger vision exciting and the unique development process compelling.

All that said, I had a bit of trepidation when Sheila and Mimi approached me about joining the OSAF board. OSAF was in the process of transitioning from an organization with paid staff to an unfunded steward of the Chandler code. The Chandler code was in the process of being rearchitected, something that was much needed, but wouldn’t be complete before the OSAF transition. I wasn’t sure what the future held for the project.

The more I talked to the Chandler team, the more those fears began to dissipate. Everyone in the community — developers, designers, and users alike — was excited about the project and its future. So were Andre Mueninghoff and Alex Russell, who ended up joining the board with me.

Last November, the new board spent two days together in San Francisco, where my enthusiasm reached a new high. Chandler has a ton of things going for it:

• Chandler is unique in that it’s a task-oriented tool for managing your information
• You can access Chandler anywhere, thanks to the lesser-known Chandler Server and its superior support of standards, such as CalDAV.
• Chandler works with the iPhone. It plays nicely with Apple iCal, Google Calendar, and Evolution. And again, because of its support of open standards, the mashup possibilities are endless.
• The Chandler community is active. New versions of the Server and Desktop were released last September and October. The user community is active, friendly, and growing.
• The Desktop rearchitecture is going swimmingly, and it’s going to get Python developers excited, not to mention users who will benefit from its new design.
• The people are great.

I can’t reiterate that last point enough. OSAF’s transition to an all-volunteer organization will soon be complete, thanks to thebehind-the-scenes effort of Sheila and Jared. The future of the project rests not only on the power of the ideas, but on the passion of the participants.

With this in mind, for those of you already participating, thank you! For those of you thinking about participating, please engage with us! Join our mailing lists or come on our IRC channel, and find out how you can help!

Happy New Year!

Next week, Jeffrey and I will be presenting on the Chandler Re-architecture Project at the 2008 Nonprofit Software Development Summit in Oakland, CA, hosted by Aspiration.

If you are in the area, we’d love to see you at the conference! (I will be updating this post with details as to which day we will be presenting.)

Update We will be presenting at 3PM on Tuesday Nov 18th. Here is the full schedule.

Below is the description for our sessions.

Chandler is an open source Note-to-Self Organizer designed for personal and small-group task management and calendaring, created by the Open Source Application Foundation.

Since our Preview release, we’ve had a number of users from a wide range of non-profits (ie. universities, other open source projects, summer camps, green technology, etc). Some are using Chandler individually, others have looped spouses and co-workers in to collaborate on projects and day-to-day tasks. You can read some of their stories here.

We will begin with a quick introduction to Chandler and demo what it is capable of today. However, the focus of the talk will be on newly started re-architecture work that will enable Chandler’s core note, task and calendar management workflows to become the anchor for a fully extensible and customizable information management platform.

To appeal to both everyday users of the software and developers with a technical interest in the project, our presentation will be two-part. First, we will walk through some user scenarios and then we will provide a technical overview of the re-architecture work we’ve been doing.

*****
Session 1: Extending Chandler: Chandler as a Confluence of Workflows

You can try out Chandler’s core information management workflows in the current 1.0 release: Collecting new information. Triaging tasks. Scheduling and reminders. List management and note-taking. Sharing and collaboration. Still, it was always our intention to build on these workflows to include document management, issue-tracking, and CRM (or to be even more specific, grant proposals, newsletters, and donor profiles).

In this session, we will walk through one scenario that demonstrates how Chandler’s workflow can be:

1. Extended to integrate new kinds of information management into the core workflows mentioned above.
2. Customized to meet the specific needs of your organization.

Session 2: Extending Chandler: Technical Overview of the Re-Architecture

Chandler’s re-architecture is focused on providing a framework with:

1. Events driven by Trellis
2. Strict separation of domain model, interaction model, and UI
3. Thorough testing of each layer without depending on higher layers
4. Thoroughly documented entry_points to make extension writing approachable
5. Doctesting so documentation is accurate and isn’t left for the last minute

The re-architecture project is still in its early days. The current documentation is available, and of course so is the source code.

Presenters: Jeffrey Harris, Developer and Mimi Yin, Product Designer.

On Wednesday Nov 12th, our final OSAF/Chandler Project servers will migrate to ISC.  The affected services include chandlerproject.org web and wiki, subversion, and downloads.  Mail and Chandler Hub will not be affected.  Overall downtime should be from about 1pm Pacific to about 4pm.
Your patience is appreciated.  Please report any problems you have after the outage to the chandler-users mailing list and IRC channel.

Chris Kelly recently blogged about his use of Chandler. He uses it for personal to-dos and reference notes and collaborates with a team of 5 people on software development for Gallery.

Here are some choice quotes:

On personal use:

On August 8, Chandler 1.0 was announced and I decided to give it a shot. First I just used it for my personal tasks related to Gallery, which I stored in Chandler Hub, but in less than a week I realized how powerful of a tool Chandler really was…I stopped using my paid Remember The Milk account and Google Calendar, and over the next few days moved everything into Chandler.

On collaborating with others:

…it’s taken a little bit of effort to get everyone using it the same way but the results have been an even bigger deal here…Chandler is a very low overhead way for developers to keep track of what they are doing in such a way that anyone can easily get the big picture in one place. Chandler has gotten rid of the need for weekly status reports to our mailing list…Everyone knows who is responsible for things and I can finally spend a lot less time harassing people…This frees up a lot of time so that developers can get back to developing and our meetings are back to lively discussion and decision making instead of boring project overhead.

Six months ago, I had to step away from the project unexpectedly due to a family crisis. I’ve really missed working with the team — I’m very proud of them for shipping a solid Chandler 1.0, web and desktop. I’m also so heartened to watch the successful transition to an all volunteer organization. OSAF is lucky to have such great people join the new board and user advisory group — I’m excited to see new energy come to the project. Thank you, Sheila, for stepping in when I needed to be away and keeping things on track!

I have been following the project even as I’ve been away, keeping up with the lists and the blog. When I have a little more free time and energy I’ll be looking for ways to help out — perhaps a bit of coding on Chandler 2.0. Looking forward to seeing what happens in the next phase of the project…

On Wednesday Nov 5th, I’ll be moving two OSAF/Chandler Project servers to our good friends at Hosted @ ISC.  The affected services include Chandler Hub, OSAF mail (such as mailing lists), and the iGoogle widget.  These servers will be on ISC’s gigabit connectivity and large UPSes so we’ll continue to get great infrastructure and ability to grow (particularly in bandwidth costs related to Chandler Hub usage) for free.  This is hugely important for our mission of delivering free and open productivity tools for groups.  Our thanks to ISC for their ongoing support, going on 8 years now.

The outage will last from about 9am to perhaps 2pm or so.  There won’t be any changes other than moving cities and IP addresses, but there may continue to be small problems after that that get fixed over a couple days.  Your patience is appreciated.  Please report any problems you have after the outage to the chandler-users mailing list and IRC channel.