Common to all kinds of repository notification, a method on a subscribing
item is invoked when changes take place in an item. The subscriber may be
notified immediately after the change happened or the notification may be
queued for later delivery. The subscription may be persisted in the
repository and eventually exist in all repository views or may be kept
transient in a view until that view is closed.

The use cases below describe the various kinds of repository notifications
and how to set them up. There is overlap between them, read on
carefully. Collection item notifications are based off some of these use
cases and are described last.

• How to receive a notification when an attribute of a given name
  changes on any item.
  
  This is done with href="http://blogs.osafoundation.org/chandlerdb/2005/02/using_attribute.html">attribute
  monitors.
  • a monitor is always a persistent subscription
  • monitor methods are always invoked immediately after a monitored
    attribute changes in the view of the subscriber items

  • monitor methods are not invoked during view refresh

• How to receive a notification when a given item changes
  A item is said to change when one or several of its attribute values are
  changed, added or removed, or when its kind changes during item creation,
  stamping or deletion. This style of notification is setup with the
  watchItem API.
  • an item watcher may be persistent by calling watchItem on the
    subscriber:
    subscriber.watchItem(item, methodName)

  • an item watcher may be kept transient to a view by calling
    watchItem on the view instead:
    subscriber.itsView.watchItem(subscriber, item, methodName)

  • an item watcher method is invoked immediately after an item changed in the
    view of the subscriber items

  • an item watcher method is invoked during view refresh when a watched item is
    refreshed to changes the occurred earlier in other views

  The method name passed to watchItem must correspond to a
  method defined on the subscriber item’s implementation class:

  def watchedItemChanged(self, op, item, names):
      # self is the subscriber item
      # op is one of 'set', 'remove' or 'refresh'
      #   'set' means that an attribute value was set
      #   'remove' means that an attribute value removed
      #   'refresh' means that the item was changed because the view refreshed
      # item is the uuid of the item that changed
      # names is a tuple naming all the attributes that changed
      #    it normally contains one name except when op is 'refresh'
      #    if the item's kind changed, then it also contains 'itsKind'
      ...
  

• How to receive a notification when any item of a given kind is created,
  deleted or stamped.
  This style of notification is setup with the
  watchKind API.
  • a kind watcher may be persistent by calling watchKind on the
    subscriber:
    subscriber.watchKind(kind, methodName)

  • a kind watcher may be kept transient to a view by calling
    watchKind on the view instead:
    subscriber.itsView.watchKind(subscriber, kind, methodName)

  • a kind watcher method is invoked immediately after any item’s watched kind
    changed in the view of the subscriber items

  • a kind watcher method is not invoked during view refresh

  The method name passed to watchKind must correspond to a
  method defined on the subscriber item’s implementation class:

  def watchedKindChanged(self, op, kind, item):
      # self is the subscriber item
      # op is one of 'add' or 'remove'
      #   'add' means that the item is now of that kind (or a subkind thereof)
      #   'remove' means that the item is no longer of that kind
      # item is the item or the uuid of the item that changed
      ...
  

• How to receive a notification when any item is added to or removed from
  an href="http://blogs.osafoundation.org/chandlerdb/2005/07/abstract_sets_o.html">abstract
  set or a bi-directional ref collection.
  This style of notification is setup with the
  watchCollection API.
  • a collection watcher may be persistent by calling
    watchCollection on the subscriber:
    subscriber.watchCollection(item, attribute, methodName)

  • a collection watcher may be kept transient to a view by calling
    watchCollection on the view instead:
    subscriber.itsView.watchCollection(subscriber, item, attribute,
methodName)

  • a collection watcher method is invoked immediately after any item’s
    watched collection changed in the view of the subscriber items

  • a collection watcher method is invoked during view refresh

  The method name passed to watchCollection must correspond to a
  method defined on the subscriber item’s implementation class:

  def watchedCollectionChanged(self, op, item, attribute, other):
      # self is the subscriber item
      # op is one of 'add', 'remove', 'refresh' or 'changed'
      # item is the item or the uuid of the item that owns
      #   the collection that changed
      # attribute is the name of the attribute that contains the collection
      #   value that changed.
      # other is the item or the uuid of the item that was added to, removed
      #   from, refreshed or otherwise changed while it belonged to the collection
      ...
  

• How to receive a notification when any item is added to or removed from
  a Collection item or when an item belonging to a Collection item is
  changed
  This style of notification is setup with the
  notificationQueueSubscribe API.
  • a collection item subscription may be persistent by calling
    notificationQueueSubscribe on the collection item:
    collection.notificationQueueSubscribe(subscriber)

  • a collection item subscription may be kept transient to a view by calling
    notificationQueueSubscribe on the view instead:
    collection.itsView.notificationQueueSubscribe(subscriber)

  • collection item notifications are queued and are dispatched during view
    refresh or when view.dispatchNotifications() is called
    explicitely.

  A href="http://blogs.osafoundation.org/chandlerdb/2006/03/collections_ite.html">collection
  item notification subscriber must implement a method called
  onCollectionNotification:

  def onCollectionNotification(self, op, collection, attribute, other):
      # self is the subscriber item
      # op is one of 'add', 'remove', 'refresh' or 'changed'
      # collection is the collection item that changed
      # attribute is the name of the attribute that contains the collection
      #   value that changed.
      # other is the item or the uuid of the item that was added to, removed
      #   from, refreshed or otherwise changed while it belonged to the collection
      ...
  

A collection item is said to wrap a set of items because it delegates
the API of that set value. All methods defined on this value’s implementation
class that aren’t also defined on the collection item’s implementation class
are available directly on the collection item.

The collection item’s set value is stored in an attribute whose name is
declared on its implementation class via the __collection__
attribute. That attribute is declared like any other on the collection
item’s kind or via Chandler’s href="http://chandler.osafoundation.org/docs/0.6/parcel-schema-guide.html">schema
API.

The collection item’s implementation class must at least be a subclass of
the repository’s Collection class and must be setup with the
CollectionClass metaclass.
When the set of items is intended to be an href="http://blogs.osafoundation.org/chandlerdb/2005/07/abstract_sets_o.html">abstract
set it must be initialized in the collection item’s
implementation class’ constructor.
For example, using the href="http://chandler.osafoundation.org/docs/0.6/parcel-schema-guide.html">schema
API:

• Defining a collection item class wrapping a simple bi-directional item reference collection.

  from repository.item.Collection import Collection
  from application import schema
  
  class GreenCollection(Collection):
    __metaclass__ = schema.CollectionClass
    __collection__ = 'bunch'
  
    # declare a collection attribute of bi-refs of cardinality 'list'
    bunch = schema.Sequence(otherName='collections', initialValue=[])
    ...
  

• Defining a collection item class wrapping an abstract union set.

  from repository.item.Collection import Collection
  from repository.item.Sets import Union
  from application import schema
  
  class BlueCollection(Collection):
    __metaclass__ = schema.CollectionClass
    __collection__ = 'plastic'
  
    # declare a simple abstract set of items attribute
    plastic = schema.One(schema.TypeReference('//Schema/Core/AbstractSet'))
  
    # in this example, a and b are collection items
    def __init__(self, a, b, *args, **kwds):
        super(BlueCollection, self).__init__(*args, **kwds)
        self.plastic = Union((a, a.__collection__), (b, b.__collection__))
    ...
  

• Defining a collection item wrapping class an abstract intersection set of
  bi-directional item references.

  from repository.item.Collection import Collection
  from repository.item.Sets import Intersection
  from application import schema
  
  class RedCollection(Collection):
    __metaclass__ = schema.CollectionClass
    __collection__ = 'paper'
  
    # declare a collection attribute of bi-refs of cardinality 'set'
    paper = schema.Many(otherName='inPaper')
  
    # in this example, a and b are collection items
    def __init__(self, a, b, *args, **kwds):
        super(RefCollection, self).__init__(*args, **kwds)
        self.paper = Intersection((a, a.__collection__), (b, b.__collection__))
    ...
  

  This class wraps a bi-directional set. When an item appears in the
  intersection set, the collection item will also appear in the item’s
  inPaper attribute which, like any other attribute, must be
  declared on the item’s kind. A collection item may be added or set
  explicitely to inPaper if and only if the collection item supports
  such explicit addition by implementing an add(item) method.

Unexpectedly, importing mail messages was very slow however, and was getting worse as more messages were added to the repository. The first cause of slowness was easily resolved by turning off ‘bz2′ compression of all the text LOBs being created.

The second cause of slowness was more surprising though. It had to do with looking up existing email address items from the email addresses occurring in the headers of the messages being imported. The EmailAddress item class defined here
has a class method called getEmailAddress() that will linearly iterate all EmailAddress instances every time such an item is sought. The mailbox I was trying to import contains about 6,500 mails with 800 unique email addresses. Linearly searching them won’t scale.

In the early days of the repository we had planned on having a query system and Ted even implemented a query language not unlike one you’d encounter in an SQL database. Last year, while working on Chandler 0.6, it occurred to us that a formal query language in a Python-based system was somewhat redundant and that we should be using computed collections instead of queries altogether to accomplish the same goals . Hence abstract sets and their item wrappers were implemented.

By combining abstract sets, collections and indexes – something I had added to bi-directional reference collections during the Chandler 0.5 release – we realized we could get very decent item look-up performance as well as cache the computed aspect of abstract sets making membership tests and iteration also considerably faster.

This technique can be illustrated with the example below taken from the EmailAddress index I added during last week’s sprints.

• First, a collection needs to be setup. This can be as simple as a bi-directional reference collection, an abstract collection of all items of a given kind or a more complicated computed collection combining or filtering other collections. In the EmailAddress example, I just created a KindCollection instance based on the EmailAddress kind. In the app parcel I added yet another collection called emailAddressCollection.

  emailAddressCollection = \
      KindCollection.update(parcel, 'emailAddressCollection',
                            kind=pim.mail.EmailAddress.getKind(view),
                            recursive=True)
  

• Then, a sorted index needs be added to the collection. I wanted to be able to find existing email addresses such that no two email address items in the collection have the same lowercase internet email address string. For this purpose, I added the _compareAddr() method to the EmailAddress class in the mail parcel:

   def _compareAddr(self, other):
       return cmp(self.emailAddress.lower(), other.emailAddress.lower())
  

  and the index creation call after the code creating the collection:

  emailAddressCollection.rep.addIndex('emailAddress', 'compare',
                                      compare='_compareAddr')
  

  which creates a ‘compare’ index called ‘emailAddress’, an index calling a method called ‘_compareAddr’
  on the items it is comparing.

• And finally to use this collection and the index to find items I added a findEmailAddress() class method to the EmailAddress class:

      @classmethod
      def findEmailAddress(cls, view, emailAddress):
  
          collection = schema.ns("osaf.app", view).emailAddressCollection.rep
          emailAddress = emailAddress.lower()
  
          def compareAddr(key):
              return cmp(emailAddress, view[key].emailAddress.lower())
  
          uuid = collection.findInIndex('emailAddress', 'exact', compareAddr)
          if uuid is None:
              return None
  
          return view[uuid]
  

  The findInIndex() call looks for an exact match as per the compareAddr local function which compares lowercase versions of internet address strings.

The technique above replaces the linear search with a binary search yielding a very noticeable performance improvement in the overall importing of email messages into the repository.

After a while, however, older versions as well as deleted items end up using up a significant amount of space in the repository. Recently, Sleepycat released a new version of Berkeley DB making it possible to compact a set of database files thereby reclaiming unused disk space.

A new repository API, compact() deletes all old versions of items and their related data before calling the Berkeley DB compact() API. This new functionality is available through Chandler’s “Test” menu.

Purging old item versions out of a repository is irreversible. Backing up the repository first is recommended.

Using Python’s hotshot profiler to identify heavily used code and then measuring the performance of individual code snippets with the timeit module a number of significant performance improvements were made in the area of bi-directional reference collections, collection indexes, item loading and item saving.

• A lot of heavily used code was moved to C using Python’s C API. For example, the base class for a bi-directional reference collection element, LinkedMap was moved to C.
  Similarly, the base class for collection indexes, SkipList was moved to C.

• Many Berkeley DB calls were made directly from C, eventually bypassing Python’s bsddb module completely. In particular, moving the secondary DB indexing calls to C yielded some nice commit improvements.
• Major performance improvements were made possible by the increased use of collection indexes to locate items of interest. For example, finding all calendar events relevant to a given week is now done by computing an intersection of two collection indexes.

Performance is an on-going task one is never really done with. So far, these performance improvements were not achieved at the expense of maintainability or flexibility. The underlying repository format has seen many changes but the public repository APIs have remained relatively stable.

Chandler’s repository is backed by a Berkeley DB environment, a set of files store in a directory called __repository__, itself stored in a somewhat obscure location described here.
This directory contains three kinds of files:

• Berkeley DB environment files: these files are specific to a given machine and are necessary for Berkeley DB to function. They contain no Chandler information, can be re-created at will with no loss of data and are not backed up.
• Berkeley DB transactional log files: these are specific to a given hardware architecture and contain one or more recent transactions. They can contain Chandler data and are backed up. Before a backup is made, the repository is checkpointed, all but the latest log file are flushed into the database files.
• Berkeley DB database files: these are the files with the actual Chandler repository data. Backing them up goes a long way in preventing catastrophic data loss.

When making a backup, the repository’s backup() method will look for a non-existent sibling to the dbHome directory path given to it for storing the backed up files. This new directory may later be used with the repository’s open() method to restore repository files from.

Once a backup is completed the directory where it was made is never written into again. Instead, a restore is done by copying the backup into the dbHome directory argument passed to the open() method, deleting and replacing its content with the files files from the backup.

A backed up repository’s files may be copied and used on another machine. If the target machine’s CPU endianness is not the same as the source machine’s – for example, when going to a PowerPC Macintosh from a Wintel PC – the log files should be deleted first.

Because it doesn’t actually store any items, an abstract set is considered to be a regular ‘literal’ value owned by an attribute on an item, expressed by the set owner tuple (item, attribute).

An abstract set is constructed around other sets and ultimately around concrete item sets, that is, ref collections. An abstract set can be nested in another set by value (same set owners) or by reference (different set owners).

For example, the Chandler content model item collection is defined as the combination of three ref collections: results, inclusions and exclusions. Let ic be an ItemCollection item, a functionally equivalent set can be constructed as follows:

    Difference(Union((ic, 'results'), (ic, 'inclusions')), (ic, 'exclusions'))

The outer set – the difference – wraps the inner set – the union – by value and the ‘exclusions’ ref collection by reference. The inner set wraps the ‘results’ and ‘inclusions’ ref collections by reference as well.

The outer set can then be stored in an attribute on an item, which becomes its owner. From that point on, whenever an item is added to or removed from any of the concrete collections backing the set, a notification is sent to its owner item by invoking its collectionChanged() method. As a notification travels up the set tree from its source, it is evaluated against the node’s operator and is either discarded, inverted or passed on.

For example, in the above set, if an item is added to ic.results and it already belongs to ic.inclusions, the corresponding ‘add’ notification is discarded. Similarly, if an item is added to ic.exclusions and it belongs to the union, the corresponding ‘add’ notification is changed into a ‘remove’ notification.

As with ref collections, one or more indexes can be added to abstract sets so that they can be iterated over in a particular order or so that item references can be retrieved by a numeric location.

All abstract set implementations are located in the repository.item.Sets module:

• Union, wraps two sets or collections and represents the set of items that belong to one or the other, inclusively.
• Intersection, wraps two sets or collections and represents the set of items that belong to both.
• Difference, wraps two sets or collections and represents the set of items that belong to the first but not to the second.
• KindSet, wraps no set or collection but represents the set of all items of a given kind, exactly or recursively. Whenever an item is created, deleted or stamped to that kind, the corresponding ‘add’ or ‘remove’ notification is sent to the owner item’s collectionChanged() method.
• FilterSet, wraps another set or collection and represents the set of all the items of that set that pass the filter test.

As with the ‘list’ or ‘dict’ cardinalities, the ’set’ cardinality allows to describe an attribute as a set of literal values of a given type, such as a set of strings, for example.

The ‘Set’ type can be used for values meant to be sets of literal values with attributes of arbitrary cardinality or type.

The repository’s data model now implements date and time support around Python’s datetime module. Support for time zones was added by leveraging IBM’s ICU library which is wrapped for use from Python by the PyICU extension library. PyICU exposes ICU time zones via the ICUtzinfo class which implements the datetime module’s tzinfo protocol for use with datetime and time types.

The data model supports the following types:

• Date
  for objects containing date information.
• Time
  for objects containing time information, with or without a time zone.
• DateTime
  for objects containing both date and time information, this type can be also be used with or without a time zone.
• TimeDelta
  for representing a time difference.