User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.3a5pre) Gecko/20100424 Firefox/3.7a5pre Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.5pre) Gecko/20100424 Lightning/1.0b2pre Thunderbird/3.1b2pre All events and tasks created in a local calendar (SQLite3 format) prior to the landing of Bug 479867 are not displayed with Lightning builds after the database scheme change. This is especially bad as they can't be recovered by reinstalling an older build anymore. Events and tasks have to be exported e.g. to ics file prior to installing a new Lightning and imported thereafter. If a user didn't make backups of the database, the Lightning update results in a total dataloss. Reproducible: Always Steps to Reproduce: 1. Install Lightning build from a hg pull older than 2010-04-09 02:15 +0200 2. Create an event in the default local calendar 3. Install Lightning build newer than 2010-04-09 02:15 +0200 4. Open the Calendar tab. Actual Results: The event created in Step 2 has gone. It is still visible when opening local.sqlite in a hex viewer, but not displayed in Lightning. Expected Results: All events and tasks are converted to the new scheme. Tested with with Thunderbird and Lightning builds from <http://hg.mozilla.org/comm-central/rev/bdca8b735557> and <http://hg.mozilla.org/comm-central/rev/f89c1ff316dc>

The correct landing time of Bug 479867 was Wed Apr 21 11:51:22 2010 -0700, not 2010-04-09 02:15 +0200, my apologies.

Do you see any error messages during or after the upgrade in console or error console?

(In reply to comment #2) > Do you see any error messages during or after the upgrade in console or error > console? No, no errors and no warnings, complete silence in EC.

Sorry, with calendar.debug.log and calendar.debug.log.verbose set to 'true' I do get messages Storage: Preparing to upgrade v18 to v19 Storage: Upgrading to v19 both in the EC and in the console. Still no errors.

Did some testing with Sunbird and this seems to affect only the 1st local calendar with URL "moz-profile-calendar://". For the 2nd and 3rd local calendar the events and tasks are displayed after the upgrade.

Thunderbird 3.1 Beta 2 is coming soon. If users and testers update and install a corresponding nightly build they will loose their calendars. So either we need a fix soon or backout Bug 479867.

Wow, what an ugly bug! It seems that even before the v19 upgrade, the first storage calendar created has an uri moz-profile-calendar://, without an ?id= parameter. I went through numerous approaches, changing the uri to a new scheme seems to be the best. This way we can easily detect the right calendar: the pure moz-profile-calendar:// must be the one with id=0. The others have an explicit id parameter and can be normally migrated. Downsides to this patch: * If the user has already migrated to v19, then the first storage calendar in his list will contain the events from the id=0 calendar. We should blog about this, but I think we can get away with this for nightly users * On the first start after this upgrade, all old calendars will still have moz-profile-calendar://... as an uri. This will go away after the second restart. I don't think this has other implications, but you never know.

Comment on attachment 442991 [details] [diff] [review] Fix - v1 Tb 3.1 Beta 2 is scheduled in 2 days, we're going to need mass testing of this bug quickly! The alternative would be to back out, but this will upset existing nightly users that have upgraded to v19 already.

Remember to backup your profile before you try this patch!

A build with the patch <https://bugzilla.mozilla.org/attachment.cgi?id=442991> works if both the database and the uri were not migrated. If the migrated database is overwritten by a backup, done prior to scheme upgrade, events are not shown.

(In reply to comment #10) > A build with the patch <https://bugzilla.mozilla.org/attachment.cgi?id=442991> > works if both the database and the uri were not migrated. If the migrated > database is overwritten by a backup, done prior to scheme upgrade, events are > not shown. Not sure I understand. Prior to which schema upgrade? The following points are important. Does the patch fix the situation: * User has previously upgraded v18->v19 and now has broken calendars, then starts Lightning with the patch applied. * User has a database v18 and starts Lightning with the patch applied. Please note that also the prefs are changed, so if you restore a previous backup, you need to also copy the prefs.js (or just copy the whole profile).

(In reply to comment #11) > Not sure I understand. Prior to which schema upgrade? Sorry for a hurried comment. I meant a local.sqlite backup made before the database scheme upgrade to v19. > The following points are important. Does the patch fix the situation: > > * User has previously upgraded v18->v19 and now has broken calendars, then > starts Lightning with the patch applied. A broken local.sqlite gets repaired when the calendar URI in prefs.js is of "moz-profile-calendar://" type. If it is already "moz-storage-calendar://", calendar remains broken. > * User has a database v18 and starts Lightning with the patch applied. Yes, this works fine, if the calendar URI in prefs.js is of "moz-profile-calendar://" type. > Please note that also the prefs are changed, so if you restore a previous > backup, you need to also copy the prefs.js (or just copy the whole profile). Yes, I saw this, but IMHO it would be great if Lightning could catch the case when the user ignores this requirement (e.g. because restoring a backup of prefs.js would wipe out important changes).

(In reply to comment #12) > > Please note that also the prefs are changed, so if you restore a previous > > backup, you need to also copy the prefs.js (or just copy the whole profile). > > Yes, I saw this, but IMHO it would be great if Lightning could catch the case > when the user ignores this requirement (e.g. because restoring a backup of > prefs.js would wipe out important changes). I see your reasoning, but since prefs.js contains the full calendar registry (i.e all calendar properties, the ids, ...) its not possible without creating other problems. If all calendars have the moz-storage-calendar:// uri and the local.sqlite still has a calendar with cal_id = 0, then its unclear to which calendar these events belong. At the time of that migration code, I don't have access to the other calendars, so I can't just check which calendar doesn't have events (this will also fail if the user has deliberately created a calendar without events). What I could do is add some code to my fix database extension that just creates a new calendar from the calendar with cal_id = 0, I'd like to postpone this to a bit later though.

Oh, and thank you for your valuable feedback! Glad to hear the patch works for you!

If I migrate the Database from V18 to V19 with the patch applied averything goes well. I also noticed the first downside mentioned by Philipp in Comment #7. It means in practice that if I have two calendars, the items from the first calendar are merged into the second and the original first one ends up empty. Maybe I find time to try with some more complex setups.

I tried if having remote calendars in lightning does affect this somehow --> It doesn't. If I have one local and one remote calendar, everything works perfect. If I have two local and one remote calendar, the same happens as if I had two local calendars. (see my previous comment.) If I delete my original first calendar and have 2 local calendars left thereafter, everything works perfect. If I disable all local calendars except the one that had dataloss, the events are nevertheless moved to the next local calendar in the list - even though it's disabled.

(In reply to comment #16) > If I disable all local calendars except the one that had dataloss, the events > are nevertheless moved to the next local calendar in the list - even though > it's disabled. Another downside we need to live with. Looks like all is well, I'd appreciate a code-level review from anyone, so we can get this checked in. Thanks!

Comment on attachment 442991 [details] [diff] [review] Fix - v1 Just one comment on migrateTables, I know it's not new code but the statement should really be in a try {}finally{stmt.reset();} block. otherwise r=simon

Pushed to comm-central <http://hg.mozilla.org/comm-central/rev/a7440b911d9c> -> FIXED

Pushed to comm-1.9.2 <http://hg.mozilla.org/releases/comm-1.9.2/rev/3845d050e376> -> FIXED