Qt Mobility Tech Preview - Organizer API

kschultz

This seems good work. I can't play with the API, I have to wait for the SIS-Files, but I can study it. From this a few remarks:

The following details should also be included:

alarm
class (public/private)
categories

These are important features of iCalendar, and I'm sure, you wouldn't forget e.g. alarm features, they are not now in the API because it's work in progress.

The lastmodified-Attribute is already included in the API, which is quite important for synchronizing. There is also a question "what about last accessed?" near by. In my opinion, "last accessed" is not so usefull. Say, the organizer engine is implemented by a database. If the user simply looks at a calendar month overview with e.g. 20 events, this would mean 20 updates on the database. I can't imagine a use case for "last accessed" which justifies such overhead.

The priority-attribute is also already included in the API, with a scale from 1-9 as iCalendar-RFC defines it. Many calendars have only a 3 scale (e.g. my Nokia phone displays only 3 different priorities, though it has internal a range 1-255), perhaps the QOrganizerItemPriority class should have some methods to convert between a three-level priority scheme and the nine-level priority scheme.

chriadam

Hi kschultz,

Thanks for your feedback. In regards to alarms and categories, I can tell you that we already have drafts of API for these; they are certainly not forgotten. That said, the scope of support for them is still under discussion; we'll keep everyone in the loop regarding these topics.

LastAccessed is probably less useful, although some backends may want to store this information, in order to allow clients to write context-aware applications ("hottest" items, etc). Also, some backends might aggregate multiple online stores, and could use last accessed as a "freshness" measure (as the device goes in and out of wifi, for example). To be honest, these semantics still need to be discussed and decided upon.

Priority is a troubling one; it seems that most different calendar and scheduling backends/APIs offer different priority measures. We decided to go with iCal-spec priority for this reason. But yes, it's possible that we will add some convenience functions to convert between common priority levels, if it's a use case that pops up with some regularity.

Cheers,
Chris.

chriadam

Hi kschultz,

You can see some API to support alarms and categories (reminders and collections) in the mobility repository now: http://qt.gitorious.org/qt-mobility/qt-mobility/trees/master/src/organizer

We're still considering the semantics and possibilities of timestamps, and other areas of the API also; so if you (or anyone else) has any more feedback on the API (especially the newly added things like reminders/collections) we'd certainly be interested in hearing your thoughts.

Cheers,
Chris.

chriadam

Hello everyone,

We're looking for some feedback and discussion on an important aspect of the API: multiple calendar support. I'll briefly explain precisely what I mean when talking about this term, and what the "problem" is, and then I'll lay out our proposed solutions for discussion.

Definition of "Multiple Calendar Support":
We have the semantic that a QOrganizerItemManager has a schema (or set of QOrganizerItemDetailDefinitions) which defines which pieces of data are valid to be stored in items which are stored in that manager. Thus, one way to think of the QOrganizerItemManager is to consider it as a "datastore" (although in practice, it could aggegrate or span several datastores, so long as they all support the same schema).

When we talk about "multiple calendars", we mean (very specifically) a "collection of items". An individual calendar might be user-defined (eg, a "social events" calendar, a "football season" calendar, etc) or might be manager-defined (eg, the system might have built in support for "work" and "personal" calendars), depending on the QOrganizerItemManager. How or where each individual calendar is stored is entirely manager specific, but every calendar in a particular QOrganizerItemManager must support the same schema (ie, schema is per-manager, NOT per-calendar). In our API, we refer to a "calendar" as a "collection" - and a collection can have an id, properties (like a name, an icon, a "colour", some tags, etc).

A given device might have several QOrganizerItemManagers available - one might interface with the System Calendar, one might interface with an online calendar service (Ovi, for example), and another might be a read-only, aggregating calendar which aggregates both of the above managers. This has NOTHING to do with "multiple calendar support". This is (conceptually speaking) "multiple datastore support" and is not the subject of this discussion.

So, when we talk about "multiple calendar support" we simply mean: the ability for the API to support multiple collections of items in a single manager, where each collection has properties, and collections can be added/removed/modified by clients (if such operations are supported by the manager).

(continued)

chriadam

The "Problem"
The problem is that different systems have different levels of support for the concept, or have different semantics for the concept. Most of these differences can be abstracted away, however there are costs associated with such abstractions. Even worse, sometimes the way in which they support the concept, can have a radical impact on other areas of the API. In particular, one area of major difference between different solutions is how individual items are given an id. Some have calendar-unique ids, while others have datastore-unique ids.

Currently, in the repository, our API assumes that a QOrganizerItem is identified by a QOrganizerItemId (which consists of a URI which identifies a QOrganizerItemManager, and a QOrganizerItemLocalId which is a manager-unique id for that item). That is, we currently assume datastore-unique ids. This means that to fetch an item, you only need to know its QOrganizerItemLocalId, since it uniquely identifies an item in a manager, regardless of which collection it is saved in. Similarly, to remove an item you need only specify its local id.

On platforms where item ids are only calendar-unique, however, the QOrganizerItemManagerEngine plugin would have to store some extra data in order to support the current model: it would have to create a map from QOrganizerItemLocalId to QPair<QOrganizerCollectionId, calendar_unique_id> -- which is prone to problems (performance, memory overhead, persistence of collection ids, and has a higher maintenance burden).

If we changed our API's semantics so that the QOrganizerItemLocalId is per-collection (that is, calendar-unique rather than manager-unique), we'd have to change the semantics of the fetch and remove functions (since a QOrganizerItemLocalId no longer uniquely identifies an item in a manager; but rather it only uniquely identifies an item in a collection in a manager which may have multiple collections). This in turn would be less consistent with other APIs in mobility (such as the Contacts API, the Landmarks API, and so on).

So, that's the "problem".

Proposed Solutions
a) Current solution: QOrganizerItemLocalId uniquely identifies an item in a manager.
(plus) consistent with Contacts API
(plus) simple semantics for developers
(plus) QOrganizerItemLocalId is very light-weight (just an int)
(minus) greatly increased maintenance burden on backend developers
(minus) reduced performance (speed and memory) of applications using most common backends

b) Modified current solution: QOrganizerItemLocalId becomes a compound type which consists of both the CollectionId and a CollectionLocalId. Thus, a QOrganizerItemLocalId still uniquely identifies an item in a manager.
(plus) similar semantics to Contacts API
(plus) simple semantics for developers, with some caveats
(minus) QOrganizerItemLocalId is not light weight (class with setters/accessors, compound type)
(minus) slightly increased maintenance burden on backend developers (persistence of collectionIds, etc)
(minus) reduced performance (speed and memory) of some operations due to the heavier nature of the local id

c) Alternative solution: change the semantic of QOrganizerItemLocalId so that it is Collection-unique rather than manager-unique
(plus) reduced maintenance burden on backend developers
(plus) performance will not be negatively impacted
(plus) QOrganizerItemLocalId remains light-weight, but in order for this positive to be realised, the API will have to be changed so that all operations take a single collectionId along with a list of collection-local QOrganizerItemLocalIds.
(minus) Not consistent with Contacts API
(minus) May be slightly confusing semantics for developers if they're used to Contacts/Landmarks

Obviously, we want clients to be able to persistently store an id (or a <collectionid, itemid> pair) which uniquely identifies an item in a manager, and have that id be valid after a reboot. We also want the best performance and lowest maintenance burden possible. And of course, we want the APIs to be as consistent and intuitive as possible.

In this particular case, these goals are in conflict, so we thought we'd open it up to the community for discussion and feedback, before making a decision (which, by necessity, must be made shortly - so don't delay).

Cheers,
Chris.

chriadam

Some extra food for thought on the three options:

Solution (1) requires the map to be built up at run time (since it cannot be persisted), either on instantiation of the manager, or the first time the client attempts to perform an operation on the manager (fetch, save or remove). Thus, for a manager which manages a large number of calendar entries (several hundred spread across multiple physical database files, or several thousand in a single file), this cost could be very prohibitive (possibly several seconds just to start an application!)

Solution (2) does not suffer the above cost, but passing around (and using) a list of QOrganizerItemLocalIds becomes much more expensive. This could be a problem in the case where (for example) the client fetches the entire list of ids which matches a filter, then fetches "pages" of Items from the manager by specifying smaller lists of the ids (eg, as the user scrolls around a list). This cost can be "hidden" from the user to some extent with tricks like preloading a larger window size etc, but only by using more memory or other such tradeoffs. We believe that QList of QOrganizerItemLocalId will use 3 times as much memory with solution (2) compared to solution (1), with one extra malloc (and if QOrganizerItemLocalId is implemented is a class with a d-pointer, it would use 4 times as much memory, with 2 extra mallocs) per item in the list.

Solution (3) does not suffer from either of the above costs, but is not as consistent with other Qt Mobility APIs as either of the above two options. It should be noted that some backends may natively store collection ids as strings rather than integers, and so some form of mapping must be stored in any event.

At this point, given the prohibitive startup cost of option (1), we really are looking at implementing either option (2) or option (3).

Cheers,
Chris.

chriadam

More food for thought:

Solution (2) allows a single operation (eg, QOIM::removeItems(list of ids)) to change data in multiple calendars, since each id could have a different collectionId part. This is both a positive and a negative: it's more flexible for client programmers (they don't have to manually batch up operations per collection), but it also means that it's impossible to guarantee transaction support at the API level. It also means that two seemingly similar requests (eg, remove 5 items) could have entirely different performance (since they might touch different calendars).

Solution (3) has no such capability: for every remove request, you'd have to specify a single collection id, plus a list of collection-local-ids. This is the inverse of the situation above; with solution (3), it's less flexible for client programmers, but you get more consistent performance characteristics (because it's explicitly part of the operation which collection you're operating on), and individual managers could (if they wanted to) provide metadata per collection to describe transaction support for that collection.

Any comments or other feedback, or votes (and your reasons)?

Cheers,
Chris.

chriadam

Some other possibilities have been suggested:

Solution (4): simply have QOrganizerItemLocalId be a quint64, where it's specified that the first 32 bits identify the collection, and the last 32 bits identify the item in the collection. This is (roughly) the same as solution (2) but without the large performance overhead (in terms of mallocs required, etc). However, it does have some problems: it limits the collection id to an integer (and the item id also, however the item id is assumed to be an integer in all solutions so far). Note that solution (2) allows the individual parts of the QOrganizerItemLocalId to have different datatypes (since it's a class with accessors/mutators), which means a nicer API and greater extensibility, at the cost of run-time performance for certain use-cases.

Solution (5): simply have QOrganizerItemLocalId be a QString, and define a format (eg, "collectionId::itemId") which backends must conform to. This allows managers who interface with stores which use integers for ids to merely use conversion functions offered by QString to convert between native ids and QOrganizerItemLocalId; and allows managers who interface with stores which use strings for ids, to merely concatenate them in some predefined manner. Also, it allows us to provide a nice API for clients (if required) to return the segments of the local id. But, it does have some downsides: mainly, performance. If everything is done as a string, then there are conversion overheads on every call, and of course a QString is at least 20 bytes (plus whatever is required to store the data in the string). A list of such strings is therefore more costly than solution (2), but it's more flexible and doesn't require managers to store or generate mappings between native vs Qt Mobility ids (since the conversion is complete at run-time).

Cheers,
Chris.

kschultz

You should stuck to the basic concept, which is: a QOrganizerItemManager manages items, it represents a logical datastore of items. The items have an unique id in the logical datastore, the QOrganizerItemManager manages the namespace of the id. In using the API we don't know the content and structure of the id. We only use the id to put it in find(..) or remove(..)- methods, which should be at level of the QOrganizerItemManager's namespace.

The problem you are describing is that some systems have only a unique id in relation to a collection of items. Well, this must not show up in the API. The QOrganizerItemManager implementation of those systems would have to combine the namespace of a "collection" with the namespace of his assemblage of collections.

I see your problem description not so as question about the design of the API - this should be driven by the basic logic as outlined above - , but "what implementation should we choose for the QOrganizerItemLocalId, so that the different Manager Engines can use it effectivly?"

So in terms of your solution proposals, I will only look at those which support "QOrganizerItemLocalId uniquely identifies an item in a manager."

solution 1: implementation as int restricts too much, if the key of a physical datastore is already an int, one need a little more space to build a compound id, a map is not a good implementation solution.

solution 2: if the implementation is a compound type in the sense of C++, this can still be a light weight thing. That is the advantage of C++.
I mean: only the specific QOrganizerItemManager knows how he builds the id out of the id's of his physical datastores. In the public API there should be no access to parts of the id.

solution 3: is an other API with very special semantics, which I don't consider

solution 4: implementation as int64: possible but difficult to handle for the implementors. I think the impl of the QOrganizerItemLocalId should have 64 bits at minimum, so a simple combination of two int32 is possible.

solution 5: implementation as QString: the easiest way to build a compound id, but may be too memory intensive.

If all participants can easyly work with an int64, go with int64. But if one needs more flexibility,
I think something like a union or like a QVariant would be usefull. Something like an encapsulated null terminated memory chunk. Those who need only an int or two int's can directly put this in, a null termination is appended, no conversion routines. Those who need a string or a :: compound string use it like a latin string, knowing that the clients will have lot of memory consumption.
To repeat: this is the API for those who write the QOrganizerItemManager, not the public API.

chriadam

Hi kschultz, thanks for that feedback.

You're right, solution (2) allows the specific manager to build the id manually. We have prototyped up a (mostly complete, but untested; prototype) version of this API, available at:
http://qt.gitorious.org/qt-mobility/contacts/trees/multical-custom/src/organizer
Especially, see the qorganizeritem.h/cpp, qorganizeritemenginelocalid.h/cpp, and for an example implementation, the engines/qorganizeritemmemorybackend.cpp/_p.h.

Solution (4) with quint64 was investigated more thoroughly, and it doesn't seem to provide any advantages over the quint32 version if the nativeId could possibly be a string (that is, on those platforms, a full database read is required to build up the mapping on instantiation or on first read), so this solution is not optimal.

Solution (5) is nice, but apart from the memory-usage, there is the added problem that whenever the engine receives on of these, it must parse the string to break out the bits that it needs, serially, for every id in the list (eg for a fetch request taking a list of ids). This cost seems prohibitive, since it affects (pretty much) every operation.

So, we think you're right -- solution (2) seems the nicest. If you (or anyone else) do (does) get a chance to review the proposed API, that'd be greatly appreciated.

Cheers,
Chris.

edit: there's currently one major missing piece of functionality from that branch: converting to/from a QVariant isn't implemented (needs to serialize/deserialize the private implementation in that conversion process). Just to be clear, it IS implementable (although may have to instantiate the engine factory and request the deserialization during the process..) but just not implemented yet.