 This session is culling during standards for free software. Our speaker has been programming for the last 30 years with a particular interest in free software for the last 15. In 1907, he started a company with friends called Catalyst IT, which worked with free software. He quit that in 1908 and started working on DAV iCal. Can we please welcome our speaker, Andrew McMillan? Yeah, that was a Y2K bug there I quit in 2008, of course. But these things happen. Some software just can't cope, particularly some wetware. So yeah, I'm going to talk about calendaring standards for free software. And the first question when you talk about that is why do we want standards? So, you know, if we don't have standards, we can't inter-operate. Well, we can have de facto standards. But particularly, I think, with calendaring, we want to not just coordinate our own work, but to coordinate our work and other people's work. So the part of the point of having a calendar is to know when a shared event is going to happen. So we need standards. So during this presentation, I'm going to talk about... Firstly, I'm going to talk about various standards used in calendaring. And then I'm going to give some sort of light review of the various open source software that's available for calendaring as well. Now, if you've got any questions, I'd normally love to have them during the presentation. But in this case, we want to run around with microphones and so on. So if we can save them for the end. And I'm also committing another sin of presentation here, being an organiser of the conference. And as well as speaking at the conference makes it a little bit disjoint. And I haven't practised this as much as I should have, so... But maybe I'll get excited as I once heard someone say, before a very, very fine presentation, so we can hope. So yes, along the way, current standards software will mostly be covered at the end. First of all, the standards. And the main standard that is used in calendaring currently is defined in RFC 5545 or which replaces RFC 2445. And this is the standard that defines the iCalendar format that contains a vCalendar block and then vTimeZone and vEvent and vTudu and vJournal. You know, people look at it and... It was invented quite a long time ago, which is often the case with standards. Email looks pretty clunky these days. And iCalendar looks fairly clunky these days. And people want to make it, you know, have all little tags and things inside it. But when you actually dig down into that standard, it's not as bad as you might think. It's containers within containers and it actually maps pretty well to a structured, you know, a modern sort of structured standard. And it's supported very widely. It works well for interchange. Everything does it. And so it's gonna be around for quite a while yet, although there are moves to replace it with something else. Alongside RFC 5545, we get RFC 5546, which is the iCalendar Transport Interoperability Protocol. It's a standard for moving iCalendar data around sending messages that are iCalendar data. And this is what your email client sees when you get invited to a meeting if you're using a mail as a transport for calendar data. Again, it's maybe even a little more clunky because it's trying to do things remotely, sort of like a protocol that copies a payload across and then runs in batch. Did you want to say something, Ben? So, Outlook, yeah. So Ben's asking whether this is related to what Outlook and Exchange use for messaging events. And Outlook will send emails containing ITIP messages. So if somebody using Outlook invites you to a meeting and you get it via email, where that transformation happens internally, I don't know, it might happen in the Exchange server. It might not happen if they're using an exchange, you know, if it's an exchange to exchange communication, but certainly if it goes outside the organization, that's what you'd expect to see. There are definitely interoperability issues with Microsoft products. And in particular, the open source software uses Olson, Timezone name convention, and Microsoft use different time zones. So maybe we can talk about that a bit later, but... So this is supported widely via email. I don't think there are really any other transport mechanisms that are heavily used for ITIP. Another format that's sort of more recent design is Colab format. This is used in a number of open source products, some of which interoperate with Exchange in particular. It's an XML-based format, which is all fine and good, but it was sort of invented without the benefit of a lot of the experience that went into the iCalendar format over many years, so it misses out on, you know, they made some mistakes in this. In particular, scheduling repeating events that happen in a particular time zone does not work. It's a particular problem with repeating events. If you have an event that is scheduled weekly at 10 a.m. in Berlin, and that crosses a daylight saving boundary, you can't simply say, oh, we'll just store all the times as UTC. It's, you have to actually retain that time zone information, so. And this standard doesn't do it. And I don't think it's incredibly heavily used. Another RFC that you'll find is 3283, which is a guide to internet calendaring. I really don't think this is a great use of RFCs, but it sort of gets outdated too quickly, so I wouldn't really bother reading that one. Other things that people work on are free busy time, so knowing when people have an appointment, how you can work that out, there is a standard for free busy within the iCalendar standard, and that is used for querying free busy regular but the ways of inquiring free busy for people who are not using the same server as you, the standards are sort of limited availability, but there are some moves to improve it as we'll see shortly. So one of the standards for sharing calendars that sort of appeared in the 90s and early this millennium is using web dev to store calendars. And whoever did this first probably did it the wrong way because they just said, there's a web dev server, I'll put my iCalendar file on there, and it's a vCalendar file with lots and lots of events. And if you've used a calendar for a couple of years, one of these files gets to be two megs, three megs, five megs and so it starts significantly impacting the network and you run into major contention issues with who can modify the calendar, you haven't got control over who can modify individual events and so you get a lot of contention problems and it, Microsoft must love it. After that, somebody came up with an improvement on that which is WCAP, this is a lot better. This is nearly good enough I think. And what this does is instead of storing all of the events in one iCalendar file, it stores a whole lot of little files, each one with an event with all of the necessary metadata associated with it like time zone and so on. And this is basically, it requires the client to do a bit more work but it, so I mean, you can't use this in the same way that you can use webdav calendars which effectively is just adding a, oh goody we can fetch the file from another server layer onto having it on a local file system. But it really does have a lot of advantages in terms of bandwidth use and so on. Yeah and CalDAV is a much newer protocol released in 2006 or 2007 I think. And this is like WCAP, it has some additional report functionality so clients can request events across a certain range. So it's got some extensions to DAV that apply specifically to calendaring. And this is, I guess this is what is reasonably widely used. There are quite a number of servers that support CalDAV, exchanges noticeably absent from that of course but many of the other servers do support it and it's starting to get better over time so CalDAV appears to be the standard that people will be using going forward and we're adding extensions to it to improve it to add sort of new functionality to it but CalDAV is the basis that those things are being built on and in particular the extensions that we're adding are extensions for coordinating meetings among work group calendars so adding the ability to straightforwardly query free busy for other members of the work group to be able to delegate access to other calendars within the work group and to use the ITIP format to send messages from calendar to calendar without using an email transport underneath so that the CalDAV server actually does the communication with itself effectively. So you end up with calendar inbox, calendar out box and some of the calendar clients are starting to support this, this is a RFC draft, I hope it's gonna be out this year. It's been quite a long time in the making and I think it's getting very close now. Certainly the Apple clients support it, Mozilla Lightning supports it and there's been a little bit of rumbling about maybe adding some support to evolution for this so it'll be good to see those things. Moving on from that, the next standard under development is iSchedule so this is taking those scheduling extensions and then allowing communication from one calendar server to another so this provides a protocol which is quite similar in a lot of ways to email and given that it's 2011 now, we hope we don't wanna get spam calendar invitations to meet me and get two million US dollars or something like the text message I got yesterday and so this includes quite a lot of reference to newer email standards as well so a lot of the iSchedule work which is still early days really I think it'll be a year or two but before the standard is finalized. It's using Decom for sort of strong authentication and it mandates TLS for transport and those sorts of things and ways of approved senders, receivers for calendars. So it does get a lot more complicated though. So along with calendaring, the standards that you need when you start scheduling the events for other people are address books. I guess that's why they've sort of gone in the same way but VCard is the standard for address book information much like VCalendar is for event information and so CardDAV is a standard for address books that is very similar to CalDAV in a lot of ways. Similar in the way that CardDAV is much closer to, well the storage of VCards on a web drive address book is much more similar to WCAP and then CardDAV reports to that and that is a nearly finalized standard which is waiting for finalization of VCard version four before it will be released. So another issue with these calendar standards is people seem to want to transform them to XML. So there is an XML XCalendar standard which is very close to development, to finalization and it is essentially an XSLT kind of transform of the iCalendar format and that's the intention is that it should be completely compatible with iCalendar. With CalDAV now actively in use in quite a few places is starting to see performance issues that get highlighted once your beautiful design meets the real world. So a new proposed standard recently is web dev synchronization which CalDAV is still a pull protocol effectively you have to query the server to see if there are changes to a calendar and so in order to minimize the amount of traffic that happens in that situation there's web dev synchronization which simply maintains a version number effectively for each calendar and you can request is there a new version of this calendar and if there is then you'll get a bunch of data about it. So that's a lot of CalDAV servers are keen to use this and I expect that'll be finalized in the next two or three months because it's not a particularly complicated standard that one but it's still pull synchronization and it would be nice to get push synchronization. So no particularly strong standards for push synchronization yet. There's kind of been some work by Apple to do push synchronization. So the Apple client does implement an internal sort of pre-RFC document and I think we'll see something come out of this using XMPP seems likely to be the way things will go. It may just be that more clients will start to implement the Apple version which is not an uncommon approach and my own CalDAV server does the same supports the push that way. So in order to make it easy for people to use things like CalDAV we want to encourage setup of a client to be easy so that you can just open your device and type in maybe your email address and your username and password and you'll be able to connect your calendar program will be able to figure it all out. So some of the newer things that are being used in CalDAV and CalDAV in order to achieve that goal there's been some SRV records defined for CalDAV and CardDAV and there's also use of a relatively new RFC which specifies the use of slash dot well-known slash something or other for what's called well-known URLs so that that can be a standard place on servers that you can hit and you'll get redirected to wherever the real thing is. And that's only very recently been finalized but pretty much everybody is starting to use that now. The SRV records of course define a server in a port once you've got a domain name and then once you've got the server in the port you kind of need to find the home URL so this is how you do that. There was a lot of debate about whether you should need to that SRV records were supposed to let you find the server and you don't need a URL then you just look at the server but a lot of people don't want to run their calendar and the root of the server or I don't know seems silly to me but then anyway there's lots of free software clients for particularly for CalDAV Lightning is the Mozilla plugin for Thunderbird and there was a standalone calendar client for Mozilla but the reason we recently stopped working on that that was called Sunbird but it's kind of died now so there is no standalone calendar that's lightning is it and you get your email at the same time. Evolution of course is the GNOME one likewise that's everything at once and both of these support events and to-dos Evolution also supports V-Journal for notes being stored on the CalDAV server. There's various options in KDE latest versions of KDE I believe Akanadi has a plugin for the does CalDAV there's also a K-CalDAV and a K-CardDAV kind of resource something or other but I've never managed to build those so I can't say too much about them. Mulberry is a project by Sirus Dabou and he's one of the main authors of a lot of these calendaring specifications he works at Apple on their calendar server project that's pretty clunky on Linux but looks a lot prettier on OS X Horde has some support for CalDAV now I believe RoundCube has patches available I don't think it's in the trunk of it there's a thing called A-Cal which is not quite released yet that's been my project for the last three months is writing an Android CalDAV client and there's a project called Chandler which is an interesting project from the Open Source Applications Foundation and they have a companion server to that so Chandler works with CalDAV but it's sort of designed to work with their companion server that's pretty good and then we have other client software so people are often interested in what proprietary software is available that supports this so EM Client is a Windows software that does CalDAV, CardDAV and that's pretty full function Apple, iCal does all of the CalDAV, CardDAV stuff well, Apple address book does the CardDAV bit and Microsoft Outlook has a free software plugin which has about 40 pages of configuration for it making it rather difficult for people to use in an enterprise environment as I understand it but I haven't got a copy of Microsoft Outlook so I can't say how easy it is to use the server status is kind of similar I guess, free software Apple Calendar server is probably the biggest, most fully featured CalDAV, CardDAV server it's obviously an Apple project but very much open source it's BSD licensed I think and it works well, it's written in Python it uses extended attributes in the file system to store metadata I believe they are switching that to using Postgres for a database because of performance issues with it there's DaviCal which is my main project I've been working on since 2005 that's Postgres back end database and PHP application not really a web application, more of an HTTP application Sogo is a project out of a company in Canada and Zimbra is a, I think they are owned by Yahoo or something but it's a free software, VMware, right Zimbra, there's a commercial version and there's an open source version So Zimbra have what they call an open source version but all the useful bits, also known as the bits which are hideously painful and broken are not open source they have their patched, Postfix their patched, MySQL their patched everything but there's no real reason for those the bits that are Zimbra are all in the giant Java blob that they don't give you source for Right, okay, there's Radicale which is a Python based project I believe it uses MySQL back end or does now that's still very early days for that one I think it's under development Cosmo, which is the companion to Chandler Cosmo is a Java program which is I guess it was very early Kaldiv server and unfortunately I think it has suffered from quite a lot of code rot and it's Kaldiv support it implements various old sort of standards and it gets a few things wrong and it does a lot of its communication with Chandler it's not done by Kaldiv and so that particular path doesn't get tested very well There's Mod Kaldiv for Apache that is a very lightweight Kaldiv server I think it's got some significant limitations as to what it can do in terms of Kaldiv and there's an interesting one called DAV Mail which is a single user Kaldiv server which allows you to use evolution via Kaldiv and on your local machine to to a exchange server in the back end or something peculiar along those lines There's also been some work in the SAMBA project with the Open Change to do an implementation of the exchange exchange interface in free software I don't know how well it's working has anybody tried that? It implements mapping which Microsoft declared was obsolete or obsolescent as of exchange 2007 they've now moved to a new protocol which is HTTP and SOPA based called Exchange Web Services we are currently putting together an evolution back end which supports that So there's various other server software, Microsoft Exchange which we've talked about a little bit that's obviously a very, very widely deployed solution Sun Calendar Server which is now owned by Oracle Oracle Beehive which is another So Sun Calendar Server and Oracle Beehive both do support Kaldiv support significant Kaldiv and Lotus Notes which does interoperate with iCalendar but it doesn't do Kaldiv or anything like that There's no doubt thousands of others out there The EM client people also have a server that supports Kaldiv for example So yes, further reading I don't have any of that Yes, if you have any questions I'd be happy to share with you Yes, if you have any questions I'd be happy to talk about them Do you know if Nervol Group Wise does iCal? I mean, I should know this given that you work with Nervol products but Group Wise I'm sure you can interchange iCalendar with it, but Yeah, I'm not sure And just on DevMail It's basically an exchange gateway, so from Exchange 2003, they're working on making it work on 2007 plus Yeah, and it's pretty awesome But it's a personal gateway, right? Yeah, it's a personal gateway Or you can actually set it up to be a bit of a server for multiple people, I think I haven't done it that way I usually just run it on the desktop Thanks I run a mail server and it's got a group web server as well The biggest problem that I find is clients not interoperating with each other they supply date formats incorrectly and do a whole lot of horrible things and then, so if you read it with one and write it with another you get very bad results Do you see that changing? It's I guess you know, it's always going to be a problem I'm a member of the of the calendar and scheduling consortium and as part of that I get involved in interoperability testing for various bits and pieces of calendar software, I try and put whatever interoperability testing things I can so I know, for example, that Oracle do test interoperability with and and I certainly test against all of the so with my android client I've tested so far against Beehive Sogo, you know pretty much all of the ones I have access to which is about seven different test servers and then you know, I run a lot of testing for client software so you do what you can, but you know, there's always going to be some interoperability issues it just takes time to get rid of those, I think but I mean, if you speak to Mr. Allman, I think he'll suggest that they still happen 30 years on with email. I believe that Chandler and Cosmo are fading out of the way, there's no more funding for Mitch Kapoor, no expert maybe somebody knows more than I do, but I think I read a report of that recently The source code is free and in the community so I think there isn't funding but there isn't funding for a lot of these projects Chandler I think probably will survive much better than Cosmo will because there are much better alternatives to Cosmo you know, it's a resource hog, it's kind of old it doesn't implement modern standards and I haven't looked at the code but I think it's a long way from really being a good thing to use in production I was involved in one site where Cosmo was replaced with Davacal and there was 1000% or so performance improvement I think you mentioned there was like an Android app are other mobile phone manufacturers and operating system producers actually following the standards process as far as you can see well, the iPhone has support for Caldev and CardDev built in, Apple are very keen on these particular standards so the Apple products do support Caldev, CardDev and often support interesting extensions to Caldev and CardDev as well and it's not an embrace and extend strategy so much as these protocols are still under development strategy so the people involved in some of the standards work are particularly on the calendar server team so they are quite involved in this and they do want to see these as standards and they do want to see adoption of them so that's a good thing in terms of the Caldev stuff, Apple is the most important client in existence it seems but open source clients are also getting more and more use particularly cross-platform ones so, yeah Are they doing much towards standardizing the date format you mentioned before that there's a lot of problems with that? Somebody mentioned before there were a lot of problems with that the date format is part of the iCalendar standard and rfc5545 perhaps clarifies that a bit further but I can tell you exactly what dates should look like in an iCalendar file perhaps there's some software that doesn't always get that right What about programming, I don't know if this is mentioned programming libraries or languages in which there's strong support for this kind of thing Yep so there's a Lib iCal which is a C library which is pretty extensively used the source of that is on Sourceforge and it's got some weird name like Free Radical or something like that I forget the exact name of it but it's Free Association thank you there's a there's a Java iCalendar library iCalfulJ that's pretty comprehensive we looked we used that for a little while in the android project but ended up ripping it out it's target was it's target seemed to be files that contained many events and we were writing a Caldav client where we had many individual blobs and the performance didn't seem to be too good from our point of view but we may just have been in too much of a hurry and not able to learn it so we ended up writing our own so I guess that will be available reasonably soon when we put iCal out there's libraries in Python and Perl and that are reasonably comprehensive I believe I was looking at some code the other day using the Python library I'm not a Python coder so I can't say exactly but seeing the Python code that was dealing with the iCalendaring stuff the vEvent I think it was using a vEvent library that seemed to be very sane and once it was being driven correctly it started producing the right code so that was actually the code that's in ZooKeeper for the conference system so hopefully the iCalendar feed of the conference schedule is now valid which it wasn't on Monday you mentioned that Apple is using extended attributes for their server Radicale uses I think it's just flat iCal files they were using extended attributes for the metadata so they were using flat files they were using flat files for the calendar event data but when you're processing the calendar event data for reports and so on you often need to have metadata about it for example the calendar object itself doesn't say what its permissions are necessarily so you have to have various metadata and the file system metadata typically is not sufficient to define the metadata that you need to respond to various report requests and so on that CalDav defines I think that's where Radicale might run some trouble in the future because it wasn't made to use a database several passes of that and one thing that they did was to put SQLite databases in each folder and use those for storing the metadata and that's a faster answer than using extended attributes and then beyond SQLite databases looking at Postgre as far as I understand it and that again is going to offer bigger performance improvements can you quickly describe the implementation of Davicale like is PHP and PostgreSQL yeah okay so Davicale is a PHP application um I don't know essentially it goes big in PHP at the top and code happens and all of the data is stored in the database with as little massaging as possible so it doesn't use file system at all for data storage the blobs are stored in the database the metadata is stored in the database and I use PostgreSQL essentially because I like it I like its date handling capability it's very strong in that area I used PHP just because I know it um and uh yeah I don't know I can I can go chapter and verse on it if you want but it's not really it's not really the thing to do in front of a room full of people I think to analyze the code it's probably better sit down with it and point it but all the documentation is online PHP doc kind of formatted I try and maintain all that stuff as well and some people are scared of PHP I have had it audited by somebody who does that for a living so that's good too any more questions okay let's put our hands together for our speaker on behalf of LCA I'd like to present our speaker with this little present I think he probably knows what's in there by now thank you thank you