The root cause was “sparking” (ouch) on the power lines reported by PG&E.  Our colo hosts, Hosted @ ISC, switched over to their generator as a precaution but the generator failed.  ISC started shutting down machines when they knew the outage would exceed their UPS power.

Here’s where our outage stretched out longer than it needed to. ISC didn’t let us know they were powering down the machines, or that power was back up a couple hours later. They responded to my 3pm inquiry to their ticketing system late on the 27th, saying that everything should have been fine hours ago. Unfortunately, I had already gone to bed, so the OSAF/Chandler outage had to wait out the night.

In the morning of the 28th, I asked ISC to hit the power buttons on the machines for me, but nothing happened. I packed up quick tech kit plus some spare machines and hopped in the car for the hour drive to Redwood City. On-site around noon, I confirmed the machines appeared dead. Weird. Hopefully the power issues hadn’t killed all 4 physical machines’ power supplies or motherboards at once, right?

Turns out, our managed power device that lets us turn machines on and off remotely had taken a header. I started moving machines’ power around, finding out by the end which ports on the power switch were dead. The Hub machine moved off the power switch entirely.

That would have all be relatively simple, except for the extended duration, but there were other post-shutdown details that smacked me around for a while. I found that the tightened DNS configuration implemented back when DNS security went crazy kept our DNS machines from answering questions from themselves. I kept production machines like Hub off until I had DNS sorted out.

But far worse was a surprise related to the Debian Etch to Lenny upgrade. Since Lenny was released a few months ago, I’ve been upgrading OSAF machines in the background. I don’t know how I missed this, but the upgrade can remove the package used to bring up networking (ifupdown). I had been breaking the cardinal sysadmin rule of always rebooting machines after upgrading the OS, so I was very very confused when some machines came back up without any networking. Including our primary DNS server. And the secondary DNS server seemed broken because of the DNS config error I mentioned above.

I shot myself in the foot even more because before I realized all this, I had decided that the already-extended outage “seemed a good time” to upgrade the Hub from etch to lenny. After the reboot, no networking! Gah, it was just working fine before the upgrade!

So, after getting a grip on what was going on, tracing through networking startup scripts, and tracking down the missing ifupdown package issue, I go bumming through the ISC office for a USB key. I use one of the existing machines to pull the ifupdown package (both i386 for the virtual machines and amd64 for the Hub) using a command-line web browser, troll through syslog to figure out what device to mount, and get these packages onto their needed machines. Luckily, this all works and all machines come back up.

All’s well that ends well I suppose, but this was a tough outage to swallow after the 9 hours from the big fiber cut less than a month ago. Natural questions like “is our hosting good enough?” and “should we move?” come up.

My view is that weird things happen in every hosting environment; moving is not a cure-all for reliability issues. Our reliability isn’t as good as we’d like or as good as could be achieved, but I feel it’s still better to sit tight. The main reason is cost: hosting the Hub consumes a good amount of bandwidth (about 8Mbps) and ISC is providing all services (space, power, bandwidth, remote hands) for free for 9 rack units worth of equipment. It would be possible to move some services like mailing lists, code repositories, and wikis to other free services and someday the community may choose to go that route.  But it’s nice to not have restrictions on capabilities or capacity that we get from hosting almost all of our own services; many open source projects would love to have access to the resources and flexibility that OSAF enjoys.

Overall I think we’ve just had a spate of bad luck and while a very rare 24-hour outage might be unacceptable to a commercial venture, as long as they stay very rare, they are acceptable to OSAF and Chandler communities and very worth the tradeoffs.

There are three items I want to undertake as a result of the outage:

• Place a DNS secondary outside of ISC
• Print out some phone numbers, IP ranges, and other “might need it offline” info
• Talk to ISC about coordinating during outages

As a final note, it turns out I could have determined that “everything should be fine now” by going to status.isc.org where there was some information about the outage, including when it ended. I hadn’t been checking that page during the outage but I certainly will (as well as using the phone as needed) during any future outages.

The outage will last from about 9am to perhaps 2pm or so.  There won’t be any changes other than moving cities and IP addresses, but there may continue to be small problems after that that get fixed over a couple days.  Your patience is appreciated.  Please report any problems you have after the outage to the chandler-users mailing list and IRC channel.

Chandler Server is a server and Ajax web UI for managing and sharing calendars, events, and tasks. It implements open data standards including CalDAV, WebDAV, Atom, and Atompub.

This release contains two significant features and four bug fixes. Any user can now delete their own account and data by using the settings dialog. Interoperability with some CalDAV implementations including iCal 3 should be improved by support for the CTAG draft standard.

The Chandler Hub has been updated and is now running the 1.1.0 version as well.

Chandler Server 1.1.0 is available for download as a ready-to-run bundle at:

http://chandlerproject.org/serverdownload

and the source code is available from subversion at:

http://svn.osafoundation.org/server/cosmo/tags/rel_1.1.0

Send us feedback at the open mailing list (no subscription required):

chandler-users@osafoundation.org

We look forward to hearing from you!

The issues fixed in this release include:

• [Bug 12283] Users should be able to delete their own Hub accounts
• [Bug 12279] Add support for CTag, Apple’s extension of ETag to a Calendar Collection
• [Bug 11851] Sort triage status by TriageStatusChangedDate
• [Bug 12334] Fix recovery pages
• [Bug 12349] In IE7, in DV, long notes obscure DV header section
• [Bug 12357] Account browser help link broken

The detail view effort aimed to simplify what users see as they create and edit their data. Now, a simple note looks like this:

We’ve moved the Remove and Save buttons to the top, so they don’t shift around as the detail view size changes. We’ve also simplified the UI for adding the “star” stamp (which used to be called task) and adding to the calendar.

Throughout the new UI, we make use of dojo’s elaborate support for fades, which help make expand and disappear changes easier to follow.

One of these fade-ins happens if you click on “ADD TO CALENDAR” or the event icon.

To reduce clutter, we’ve used more hint-text in widgets and fewer permanent labels. We also hide event related widgets that don’t apply to the current event. So, for example, “anytime” events which don’t have a start time don’t need a timezone, so we don’t display a timezone picker until you choose a start time for your event.

We’ve also made a much-clamored for addition for events, a date picker. Happily, dojo gave us this for free.

Once you fill in a start-date, end-date defaults to start-date.

Finally, we’ve moved our notes field to dojo’s expanding text area. So if you’ve got three pages of notes, the notes field (and the entire detail view pane) will automatically expand to give you room to type everything.

For reference, here’s a snapshot of the old detail view.

The solution to this problem, the collection sharing dialog you can find on
Chandler Hub today, solves this problem
and more. Any time you’d like to share your Chandler Hub data with other people or
even just other applications you use, this starting point will make everything
else easier.

Today we’ll two follow users, Adam and Zed, as they run through a sharing
worflow we think you’ll find useful.

Invite Collaborators

Adam and Zed are coaching their sons’ soccer team, the Beagles. Adam has been using Chandler
at work, and creates a new collection to help with the mountain of organization that
will need to take place. He has published this collection to
http://hub.chandlerproject.org and would like Zed
to give him some feedback on a proposed practice schedule.

After logging into the Hub Web UI, Adam clicks on the pulldown arrow next to “Soccer” to
bring up the collection sharing dialog.

He then clicks on the “invite” button to generate two sharing links.

Finally, he right clicks on the “View and Edit” sharing link and selects Send link….
He could also just copy the link location and paste it manually into the email or instant message
program of his choice.

When Zed gets the link he can paste it into his favorite web browser and instantly review and edit
Adam’s practice schedule. What’s more, if he wants to start using the Chandler Desktop client, he
can use this link to subscribe to this collection there.

Collaborate with iCal

Zed’s not ready to take the Chandler Desktop plunge. It’s not that he doesn’t like Chandler, but he’s
been a die-hard Apple iCal fan for years. Fortunately, iCal and Chandler work great together. Once Zed
pastes the link he received from Adam into his web browser, he’s excited to see a big green button
on the left side of the screen labeled “Apple iCal, Google.” He clicks it and follows the instructions
in the dialog that appears to add this collection to iCal. Unfortunately, due to underlying technical restrictions
he won’t be able to edit the collection or add new events from iCal. Fortunately, Zed has bookmarked the link
Adam sent him, and can do all the editing he needs there.

Sky’s the Limit

Hub interoperability
doesn’t end with support for Apple iCal. Several other clients, including the Lightning
Calendaring Extension for Mozilla Thunderbird, have built in support for CalDAV, a calendar sharing standard.
Hundreds of feed readers, usually used for keeping track of blogs and news, can subscribe to the Atom based
collection feeds provided by Chandler Hub. Instructions for each of these methods can be found in the collection
sharing dialog and the Chandler Project wiki.

1. Log in to Chandler Hub and open collection details for collection you want to receive notifications for (click on down arrow next to collection name in collection list):

   

2. If it hasn’t been done already, click [Invite] to generate sharing links.
3. Right-click or Cmd-Click (Mac) on either one of the [Invite] link to Copy Link. (Note Collections you subscribed to will only display 1 “Invite” link, the link you used to subscribe.)
4. Log in to your Google Calendar account
5. In your calendar list, click on [Add] pulldown and select Add by URL

   

6. Paste the URL obtained in step 2 into Public Calendar Address field, changing “pim” to “webcal”. The URL should look something like: https://hub.chandlerproject.org/pim/collection/215715b0-5a0b-11dd-92a1-e12efc62301d?ticket=26htadc0
7. Click [Add]
8. In the calendar list, click on the down arrow and select Notifications

   

9. Setup daily agenda email or SMS

You are done! Google Calendar will automatically refresh the URL every few hours and will send a daily agenda to you at 5am every morning.

Of course there are some limitations. The email will go to your gmail account, the agenda only includes events, there is a possibility of data being out of date, there is no triage status information, and no way to link back to the original items on Chandler Hub. Of course, this is why we are currently developing these features for the server!

Publishing using WebDAV definitely works, but as Jon points out, it’s hard to find a free web host that will give you a calendar view of your events. It also tends to be slow for large, frequently updated calendars, because the whole calendar has to be sent every time something changes.

Happily, for Mac OS X Leopard users, there’s a better option. Leopard shipped with iCal3, which includes CalDAV support. CalDAV is a protocol that allows fast (well, faster than WebDAV) multi-user access to calendars.

Chandler Hub was designed to make Chandler Desktop sharing easy, but it’s also a CalDAV server. So if you’d like to publish public calendars and edit private calendars with friends and colleagues, Chandler Hub may be a good choice for you.

Chandler Hub is a free service that, among other things, gives you shareable web-based calendars, editable using the web or a CalDAV client.

So, here’s how to share a public calendar using CalDAV and iCal3.

Sign up for a Chandler Hub account.

1. When you sign up you’ll need to provide an email address
   

   

2. Activate the account by clicking on the emailed link. Once the account is activated, log in. Initially, your account won’t have any calendars
   

   

3. You’ll need to create at least one calendar (which Chandler calls a collection) in the Hub, or iCal won’t let you work with the account
   

   

Configure Apple iCal to sync with Chandler Hub

1. In iCal Preferences, create a new CalDAV account by selecting the Accounts tab and clicking the plus at the bottom left
   

   

2. You’ll need to expand the Server Options pane so you can enter your Account URL. The URL will be https://hub.chandlerproject.org/dav/users/your_username
   

   

3. Once you click Add, you may need to refresh by going to the Calendar > Refresh menu. Within a few seconds, you should see your calendars
   

   

4. [Optional] Now you can add new calendars directly from iCal, but it’s a little tricky. If you’ve already created all the calendars you need, you can skip this step.

   iCal won’t let you drag and drop an old calendar onto your CalDAV account, you can only create a new calendar. To create one, first select one of your CalDAV calendars, then click the plus at the bottom left.

   Unfortunately, when you create the calendar, you’ll get an iCal error message. The error message is mysterious, but happily it doesn’t prevent the new calendar from working. Just click OK and name your calendar.

   

   (Note) Don’t try to delete a Hub calendar from within iCal, delete calendars from the Hub’s web interface. If you accidentally do try deleting from iCal, it’ll make the Hub account unusable. To recover from that, you’d need to delete the account in iCal’s preferences and recreate it. No data should be lost, but it’s a hassle.

5. You should now have a few calendars to work with in iCal.
   

   

View and share your calendars on Chandler Hub

1. To share a calendar with the world, go back to Chandler Hub and confirm that all your calendars appear in both iCal and the Hub.

   

2. Click the white “i” in a colored box, to the right of the calendar name. You’ll see the calendar info screen.
   

   

3. To share the calendar with the wider public, click the Invite button
   

   

4. You can just send these URLs as is, but there’s one last detail.

   Chandler has two main views, a week view and a list view. The default view is list, but that’s probably not what you want if you’re sharing a calendar.

   To change views, look at the small icons near the top left of the screen. The calendar icon is on the right.

   

   If you’d like people viewing your calendar to always see the week view, you can add

   &view=cal

   to the end of your sharing URL.

That’s it! Here’s a link to the example calendar I used to make this blog post, feel free to play with it if you’d like to see how it looks without creating an account.

The first changes I’m excited about are, like our latest release, less wholesale
modifications than improvements and commitments to stable APIs with performance
enhancement sugar to sweeten the deal. Dojo’s internationalization (i18n) and event
APIs have matured to the point where developers can expect to rely on them without
fearing a future change like the one we’ve just experienced. As a result we’ve begun the
process of migrating our custom i18n code to Dojo’s API, away from the custom, backend
dependent code we’ve used in the past. We’ve also started streamlining our use of Dojo’s
topic APIs to make our code easier to read and understand. Both these processes are works in
progress, so keep an eye on this space for more detailed information in the future.

Several components have also seen essentially complete overhauls, most prominently
the XMLHttpRequest wrappers and Dijit, the full featured HTML/CSS UI toolkit built on
Dojo Core. Instead of using dojo.io.bind and passing callback functions,
method, and header information dojo.xhrGet, dojo.xhrPost and a
handful of other methods accept a variety of arguments, make HTTP requests and return
dojo.Deferred objects. This return value, a port of the asynchronous
task management API introduced by Python’s
Twisted networking library, provides an easy way for developers to manage complex
sets of asynchronous actions like server requests. Since our Web UI data APIs already used
dojo.Deferred internally, this change led to a very nice code reduction.

Dojo’s user interface building API, Dijit, has been greatly improved since Dojo 0.4.
In addition to moving to its own namespace as part of Dojo’s overall API flattening,
Dijit is better streamlined, better tested, and easier to use. A number of Chandler Server
UI components have been ported to the Dijit APIs, and are, as a result, better tested,
more modular, and closer to being embeddable outside of our Web UI.

The Dojo team has also been hard at work putting together the next generation of
Javascript tools to support high performance rich applications on top of the
Open Web.
Two of these tools are already finding their way into heavy use within our code base,
and are poised to become critical pieces of our infrastructure over the next year.

The first, dojo.data,
is “uniform data access layer” that allows UI components to be built without worrying
about backend data formats. Our user administration interface has been essentially
completely rewritten, but required almost no new UI code. All we had to do
was implement a dojo.data store on top of the Cosmo Management Protocol
(CMP)
and hook it up to Dojo’s
Grid widget
to get full in-place user field editing, “infinite scrolling” for handling
large numbers of users and a handful of other goodies. While our end-user calendar and
item list UIs have not been moved to this API, the ongoing
web widget project
is being built on a new dojo.data store that we hope to eventually
integrate into our current UI.

The second piece of new functionality that I’m excited about is
dojo.query. This do-it-all CSS query function
is the go-to guy for finding pieces of DOM to manipulate. The beauty of this and other
query functions is that they are based on features most web developers eventually expect
to be supported natively by all major web browsers. By allowing developers to start using
these features now we can build advanced web applications that will get trivially more
performant with time, and motivate browser developers to continue implementing this
critical functionality.

In addition to improving the tools we use to build our Web UI, Dojo’s 1.0 line has
introduced some major infrastructure improvements in the form of a
DOH,
a new Dojo-independent testing harness, and a from-scratch rewrite of the
Dojo build system.
We’ve managed to port all of our tests to DOH by writing some wrappers to avoid
a completely rewrite, which has allowed us to take advantage of the very nice
test harness bundled with Dojo. Nearly as important as this test framework is the build
system that plays a key role in transforming over 1MB of Javascript into a much more
digestible 138K loaded in several different stages. The Dojo 1.0 line makes this process
much cleaner and easy to understand, as well as offering advanced functionality like
layering, which allows us to break our Javascript into large chunks
appropriate to different pieces of our UI.

Dojo has been an integral part of our project to build a new kind of information and
knowledge management ecosystem and we are lucky to be able to rely on a vibrant community
of developers producing a first class piece of software. If you’re interested in digging
deeper and helping us integrate even more of the exciting new functionality provided
in its latest release, Dojo 1.1, please don’t hesitate to ask questions on our
development list,
or via IRC on irc.freenode.net in the #cosmo channel.