The root cause was “sparking” (ouch) on the power lines reported by PG&E.  Our colo hosts, Hosted @ ISC, switched over to their generator as a precaution but the generator failed.  ISC started shutting down machines when they knew the outage would exceed their UPS power.

Here’s where our outage stretched out longer than it needed to. ISC didn’t let us know they were powering down the machines, or that power was back up a couple hours later. They responded to my 3pm inquiry to their ticketing system late on the 27th, saying that everything should have been fine hours ago. Unfortunately, I had already gone to bed, so the OSAF/Chandler outage had to wait out the night.

In the morning of the 28th, I asked ISC to hit the power buttons on the machines for me, but nothing happened. I packed up quick tech kit plus some spare machines and hopped in the car for the hour drive to Redwood City. On-site around noon, I confirmed the machines appeared dead. Weird. Hopefully the power issues hadn’t killed all 4 physical machines’ power supplies or motherboards at once, right?

Turns out, our managed power device that lets us turn machines on and off remotely had taken a header. I started moving machines’ power around, finding out by the end which ports on the power switch were dead. The Hub machine moved off the power switch entirely.

That would have all be relatively simple, except for the extended duration, but there were other post-shutdown details that smacked me around for a while. I found that the tightened DNS configuration implemented back when DNS security went crazy kept our DNS machines from answering questions from themselves. I kept production machines like Hub off until I had DNS sorted out.

But far worse was a surprise related to the Debian Etch to Lenny upgrade. Since Lenny was released a few months ago, I’ve been upgrading OSAF machines in the background. I don’t know how I missed this, but the upgrade can remove the package used to bring up networking (ifupdown). I had been breaking the cardinal sysadmin rule of always rebooting machines after upgrading the OS, so I was very very confused when some machines came back up without any networking. Including our primary DNS server. And the secondary DNS server seemed broken because of the DNS config error I mentioned above.

I shot myself in the foot even more because before I realized all this, I had decided that the already-extended outage “seemed a good time” to upgrade the Hub from etch to lenny. After the reboot, no networking! Gah, it was just working fine before the upgrade!

So, after getting a grip on what was going on, tracing through networking startup scripts, and tracking down the missing ifupdown package issue, I go bumming through the ISC office for a USB key. I use one of the existing machines to pull the ifupdown package (both i386 for the virtual machines and amd64 for the Hub) using a command-line web browser, troll through syslog to figure out what device to mount, and get these packages onto their needed machines. Luckily, this all works and all machines come back up.

All’s well that ends well I suppose, but this was a tough outage to swallow after the 9 hours from the big fiber cut less than a month ago. Natural questions like “is our hosting good enough?” and “should we move?” come up.

My view is that weird things happen in every hosting environment; moving is not a cure-all for reliability issues. Our reliability isn’t as good as we’d like or as good as could be achieved, but I feel it’s still better to sit tight. The main reason is cost: hosting the Hub consumes a good amount of bandwidth (about 8Mbps) and ISC is providing all services (space, power, bandwidth, remote hands) for free for 9 rack units worth of equipment. It would be possible to move some services like mailing lists, code repositories, and wikis to other free services and someday the community may choose to go that route.  But it’s nice to not have restrictions on capabilities or capacity that we get from hosting almost all of our own services; many open source projects would love to have access to the resources and flexibility that OSAF enjoys.

Overall I think we’ve just had a spate of bad luck and while a very rare 24-hour outage might be unacceptable to a commercial venture, as long as they stay very rare, they are acceptable to OSAF and Chandler communities and very worth the tradeoffs.

There are three items I want to undertake as a result of the outage:

• Place a DNS secondary outside of ISC
• Print out some phone numbers, IP ranges, and other “might need it offline” info
• Talk to ISC about coordinating during outages

As a final note, it turns out I could have determined that “everything should be fine now” by going to status.isc.org where there was some information about the outage, including when it ended. I hadn’t been checking that page during the outage but I certainly will (as well as using the phone as needed) during any future outages.

Highlights include better support for Google’s CalDAV server,
the addition of a menu for choosing the calendar’s first day of the week
and, lastly, we’ve built .deb packages — for both 32-bit
and 64-bit processors — for Ubuntu 8.04 (“Hardy”) and
8.10 “Intrepid”).

Here is the full list of bugs fixed in 1.0.3:

• 11038 Feature request: a UI affordance for selecting the starting day of the week
• 12228 Get Mac native spell checker working again
• 12253 Sy&nc Mana&ger… (note two ampersands) in menus.py:223
• 12274 Deprecated libiuc36 dependency
• 12278 Support Google’s CalDAV implementation
• 12328 make all binaries in internal fails with: NameError: global name ‘log’ is not defined
• 12472 ValueError: too many values to unpack (selected message has angle brackets within the name as well as the SMTP address)
• 12484 Won’t start (Mandriva and possibly other Linuxes)

You can download the app here,
or by using the “Check for Updates” feature in Chandler.

The outage will last from about 9am to perhaps 2pm or so.  There won’t be any changes other than moving cities and IP addresses, but there may continue to be small problems after that that get fixed over a couple days.  Your patience is appreciated.  Please report any problems you have after the outage to the chandler-users mailing list and IRC channel.

One of Chandler’s appeals for me is the ability to manage lots of
different ideas, but only focus on a few at a time. The way focus
works in Chandler is that items are “triaged” to NOW, LATER, or DONE.

DONE is just like filing an email. NOW is like leaving the
email in my inbox. The big enhancement, for me, is LATER.

With the LATER section, you can mark a message as important to pay
attention to, but not right now. This risks becoming a gargantuan pit
of “someday, maybe” items, though, so you can also assign an alarm,
and that item will pop back to NOW in the future.

Now I can choose just a few things to keep in my NOW section, and
collapse the LATER and DONE sections so they don’t distract me.

One big problem with the LATER section, until recently, was that it
was sorted pretty arbitrarily. I have dozens of LATER items.
Birthdays, events months from now, reminders to do something next
month, and more. What I (and most users, from what we’ve heard on our
mailing lists) want is for the LATER section to sort according to
when an item will become NOW again.

I’m happy to announce that in our latest release (1.0rc1), the LATER
section sorts appropriately! For all you non-Chandler users reading
this, that may not seem like a big deal. But it has a surprisingly
big impact on my use of Chandler.

When LATER sorted arbitrarily, I spent a lot of my time going and looking at my calendar to
remind myself if I had upcoming events. Really, I don’t have enough
events to want a calendar view very often. Mostly, I just want to
know what’s coming soon. Now the LATER section gives me that.

I used to not bother assigning alarms to events. I’m not sure why, it
just felt like too much effort for not enough payoff. Being able to
see a prioritized-by-date list of LATER items pushes the
effort-to-payoff ratio to a point where I’m always assigning alarms to
anything I don’t want to think about right now. As a result, my NOW
section is much more manageable and I’m calmer about what I need to be
thinking about.

1. Clickable links in the item notes field
2. Support more custom recurrence rules.

So please help us test by trying out these 2 new features! As usual, send feedback and report problems to The Chandler-Users mailing list.

The build is available here, or,
if you’re running 0.7.7 or later, via the Tools >> Check for Test Updates menu item.

Note for Mac and Linux: At this time, there is minimal support for visual feedback that links are clickable on Mac and Linux. Meaning, links will not be blue or underlined. However, if you mouse over a valid URL, your cursor should switch to the hand, and clicks should be handled correctly.

As we have in the past, OSAF will be participating in SoC as a mentoring organization this
year. A full list of projects we believe will be both useful to the project in the long term
and stand a high chance of being included in our code base in the short run can be found
here, but
as in past years outside proposals are more than welcome.

Comprehensive information about this year’s program, including important bits like
eligibility
and application instructions
can be found on the the official
FAQ. Student
applications will be accepted from March 24 – March 31, so now is the time to find
a project and put together a solid proposal!

As always, feel free to ask on one of the Chandler Project
mailing lists
or in #cosmo or #chandler on irc.freenode.net for more information.

Chandler is about trying to match the way people really work. And everyone uses lots of tools to get their job done. One of the first things people want to know when considering trying out Chandler software is “Will it work with what I already use? Can I switch back if I don’t like it? Will it work with the tools that friends of mine use?”

We believe the answer to all these questions is “YES”! You can safely and productively start using one or more of the Chandler Project components on top of your existing toolset. Go ahead, try it out! Read more below to learn the details.

For a bullet-list summary of our best-available notes on specific applications and which features are supported with each, see our interop overview.

Import/export

The gold standard of calendar transfer is the “ICS” file (in iCalendar format). Most calendar and task list applications support both import and export of ICS files.

You can try out Chandler Desktop without switching from your current setup. Just export one or more ICS files from your current application, then import those files into Chandler Desktop (continuing to use your current app). To switch over permanently, just export+import again a final time!

If you later decide you’d like to change again, you can export ICS files from Chandler Desktop or Chandler Hub, using those files for import into a wide variety of applications.

We’ve seen import and export work for Outlook 2003/2007, Mozilla Lightning/Sunbird, Apple iCal, and others; it should work with a great many apps, probably yours included.

Note that Outlook doesn’t export full information by default; we’ve found this $10 application from littlemachines produces high-quality exports from Outlook that work well with Chandler Desktop.

In practice, doing ICS import/export can have gotachas. Not all application combinations/roundtrips are 100% perfect. We urge you to keep backups and try out import/export before committing your important data to any application. In Chandler Desktop, we’ve spent a lot of time tuning our import/export routines to handle as many variants and details as we can. Chandler Desktop properly handles events, tasks, timezones, recurrence, and other details. Please report any import/export problems you encounter.

We’ve put together some additional information about import/export with Chandler Desktop specifically, so check that for additional hints and notes.

Synchronization

ICS import/export is great for transferring your data between apps, but it’s a manual process not suited to keeping multiple applications in sync. Usually when you import a data set, your app will overwrite changes you may have in your local copies of those events. It’s hard to make changes in two separate apps.

Chandler Desktop and Chandler Hub both support multiple network sync protocols. Where other applications (Outlook, iCal, etc) overlap at least one of these protocols, interoperability is possible on at least some level.

One main idea to keep in mind when thinking about these various systems is whether a scheme is “read-only” or “read-write” (ie, bidirectional). It seems like read-only (or 1-way) interoperability works more reliably today, but protocols like CalDAV promise a new era of real-time, 2-way synchronization of calendar data between lots of free and for-pay applications and web services.

The big news is that you can do 2-way/read-write calendar and task synchronization today, both privately and shared with other people. Here’s a list of the main ways to do that, based on Chandler Project software.

Webcal, 1-way sync

The most simple network protocol is to take an ICS file (see above in import/export) and post it to the web, so various apps can download it (redownloading to check for changes periodically). This system is called webcal.

Chandler Desktop works great for subscribing to a number of public webcal URLs and overlaying them all on one canvas. This is a great way to keep track of lots of calendars.

If you store any events/tasks on Chandler Hub, then you can login to get a URL that you can enter into the right spot in Outlook, Apple iCal, Google Calendar, Lightning/Sunbird, Evolution, Zimbra, and many other apps to synchronize that Hub calendar with your app. This is always a read-only/1-way procedure.

Using webcal, in Outlook 2007, you can overlay say personal or family Hub calendars on top of your Exchange/Outlook calendars you use at work. (Look for “Internet Calendar” features in Outlook’s help.) If you make a change on the Hub or Chandler Desktop and then synchronize, you’ll see that change in your work Outlook’s display.

You can also subscribe to Hub calendars in Outlook 2003, but only view the calendars side-by-side. Other apps like iCal, Lightning/Sunbird, Google Calendar, and Evolution all support overlaying the Hub calendar with other calendars.

Many applications, Outlook included, can also publish a webcal calendar to a web server. You can use Chandler Hub as a destination server for most of these webcal-publishing apps. This works, but please note this does not provide a web UI for that calendar, and it’s again a 1-way publication. The original application will very likely not detect any changes made to this webcal file on the server.

Webcal, 2-way sync

Chandler Desktop can also do 2-way synchronization via webcal. Most applications treat a webcal file as read-only or write-only, but Chandler Desktop will check for changes in a webcal file it is monitoring and integrate those changes. If used with another application that also checks for changes, you get 2-way synchronization. We know Lightning/Sunbird does this (though you might chose to use CalDAV to synchronize instead).

CalDAV, 2-way sync

CalDAV is an emerging standard protocol for open calendar exchange. It’s not a protocol that’s used directly between two clients (like ICS files are), but rather defines a calendar server to which multiple clients can subscribe and synchronize. Chandler Hub also provides a read-write web UI to any calendar you store or use in your account.

The Open Source Applications Foundation via the Chandler Project was an early supporter of CalDAV. Together, our Chandler Desktop application and Chandler Server server product are some of the oldest and most mature implementations of the CalDAV standard and we plan to continue that support.

Chandler Hub is, as far as we know, essentially the leading free CalDAV service offered to the public. Given a fully-cooperating CalDAV client (Lightning/Sunbird, iCal 3.x, and Evolution all cooperate to various degrees), you can use these other clients regularly or occassionally and even use Chandler Desktop for advanced work (like sharing a single item between multiple calendars).

Chandler Desktop can subscribe to and publish a collection (calendar+events) to any CalDAV server (Apple Calendar Server, RSCDS, Bedework), or actually any WebDAV server (Apache mod_dav, .Mac, etc). Both of these mechanisms support bidirectional (read-write) synchronization, so multiple applications or people can all create, edit, and delete events and tasks any time they want, using the application of their choice.

iCal 3 (in Apple 10.5 “Leopard”) is a great new CalDAV-using PIM client. You can use it to make changes to your Hub collection, and still be able to use the Hub web UI to make changes from anywhere. Note that iCal 3 supports read-write calendars only to calendars in your account. Chandler Desktop and Chandler Hub let you subscribe to shared collections owned by other users with full read-write access.

Email integration

Chandler Desktop is not a complete email client; it is rather intended to complement your existing email client. The mechanism we use is to create dedicated Chandler folders on your IMAP server. Using your regular email client (Outlook, Thunderbird, Mail.app, Evolution, etc), you just drag an email from your inbox into a Chandler folder, where the message will be parsed for event, task, and other information.

You should also be able to send email update of items from Chandler Desktop using just about any outgoing mail server available. Events emailed this way appear as ICS attachments. We’ve tested Exchange, Postfix, Gmail, Yahoo mail, and Hotmail/MSN among others.

1. User Interface changes

   The trunk currently has implemented a fair deal of the Task → Star and
   Pared
   down Desktop UI proposals. Also, as my personal scratchy itch (and with some UI
   help from Mimi), I’ve made user collections in the sidebar re-orderable via drag-and-drop.

2. Cosmo security fixes

   Randy Letness has been the driver for addressing a security issue on
   the server side, where it’s relatively easy to give yourself read-write
   access to items that have been shared with you read-only. 0.7.5 will
   have some fixes to make things work more smoothly on the client
   side (see this thread
   for some of the discussion that went into this).

3. Printing

   Reid Ellis did some good work to get basic printing support in on
   a branch. Printing month views is working well for me, so I will
   be integrating his code into the trunk (possibly with some minor
   polish).

Here are a couple of linkies for your mouse clicking convenience:

• Bugs
  resolved so far in 0.7.5.
• The Desktop Work Queue, which I’m using as a guide for what to code next.

If there are fixes you would dearly love to see in 0.7.5, feel free to comment in bugs (or file new ones): I’ll do my best to respond. Patches are welcome, as is testing help. I will be releasing some checkpoint builds in the next week or two as we lead up to the actual release.

Chandler Desktop is an open source, standards-based personal information manager (PIM) built around small group collaboration and a core set of information management workflows modelled on Inbox usage patterns.

The 0.7.4.1 release contains the fix for a bug that severely complicated upgrades from version 0.7.3; see

#11823 Couldn’t migrate 0.7.3 to 0.7.4 on MacOSX

for details.

The 0.7.4.1 release is a single bug-fix release to 0.7.4, which was the fourth in a series of ongoing releases since Chandler Preview 0.7.0 intended to respond to the ongoing feedback received from users. Additional releases improving Chandler Desktop are planned and ongoing.

Chandler Desktop 0.7.4.1 is available for download for Windows, Mac, and Linux here.

Additional information is available from the Chandler Project homepage .

Thanks for your interest in Chandler Desktop!