[4.0.1.45] Hundreds of contacts lost after update

[Ygriega]

REPRODUCIBILITY (% or how often): 100%
BUILD ID = OS VERSION (Settings > About product): 4.0.1.45
HARDWARE (Jolla1, Tablet, XA2,…): Xperia X, XA2 Ultra & Gemini
UI LANGUAGE: German
REGRESSION: (compared to previous public release: Yes, No, ?): Yes

DESCRIPTION:

After the upgrade from 3.4.0 to 4.0.1 all of my upgraded devices (Xperia X, XA2 Ultra and Gemini) each lost hundreds of contacts - all had more or less the same contacts database (though some individual contacts were different as last sync was some time back). So far I couldn’t really identify a pattern why certain contacts made it successfully to 4.0.1 and others (the vast majority) not.

I saw that this also happened to others in the 4.0.1 release notes thread, but also several restarts (which solved it for the others) didn’t help here…

PRECONDITIONS:

None

STEPS TO REPRODUCE:

1. Check contacts database under 3.4.0
2. Upgrade to 4.0.1
3. Check contacts database again

EXPECTED RESULT:

Contacts remain the same under 4.0.1

ACTUAL RESULT:

Several hundreds of contacts lost in 4.0.1

ADDITIONAL INFORMATION:

Could be related to my other contacts-related bug report at [4.0.1.45] Non-jailed contacts SQLite database no longer updated
Update 1: Just also checked the content of the two different databases I mentioned in the other bug report (privileged vs. unprivileged): The unprivileged one still contains all entries (well, I had a backup, but it’s good to see that the stuff is still on the device), but the privileged one has much less entries and also most of the still existing ones as duplicates. So something might have gone wrong during the conversion.

[Thaodan]

Can you look which kind of contacts are lost e.g. local or remote contacts (caldav, google contacts etc.)?

[Ygriega]

I only have local contacts and I don’t use any automated synchronization options (no Exchange, no Google, no Cardav no nothing).

[ljo]

For me it seemed to mainly depend on origin being sms/mms. Not sure if it is a helpful reflection. But right now I can’t see any contacts at all after 4 days of only minor problems. Today I basically got hit by all jail related issues reported in the forum.

[jovirkku]

@Ygriega: Did you see a NOTE here: [release notes] Koli 4.0.1? Would that do the trick?

[Ygriega]

As said above - I don’t use any synchronization. The only thing that remotely goes into this direction is that the contacts database was imported from an Xperia, the Xperia got it from a Jolla and the Jolla got its entries from an N9…

[Ygriega]

But maybe I can dig deeper in the DB itself. Which flag do you actually use to ignore the “synchronized” contacts that don’t make it to the new SFOS4 database?

[gabrielg]

@Ygriega

But maybe I can dig deeper in the DB itself.

You should be able to open the DB from the CLI with sqlite - if you are comfortable doing that, at least you could check whether the contacts are still there or not.
I would also check for errors/warnings in the journal (has to be from the time to boot as the process in charge of converting the schema may trigger way before you launch People).

[Ygriega]

Please see the “Update 1” in “Additional Information”

[gabrielg]

@Ygriega

Please see the “Update 1”

OK - and what about the journal? TBH, I could never make sense of the privileged vs unprivileged concept. One of them surely has all my contacts, the other I don’t know.
Perhaps you could share the schema that you ended up with - I would be happy to compare it to mine. You shouldn’t have any personal information there, hopefully!

[Ygriega]

As long as we’re still in some kind of transition phase where Sailjail apps and Non-Sailjail apps coexist, I expect that both databases contain the same information (they don’t, see my other bug report). So privileged vs. unprivileged contacts shouldn’t make a difference.
And is the conversion really happening at each reboot? I thought it’s done during upgrade. However, if it’s done during boot, I’m happy to do so, but help me with some hints which journal to look at and how. Thanks!

Of course, I could post the schema of both privileged and unprivileged databases, schemas don’t contain any personal information. But isn’t the schema the same everywhere? As it’s mentioned in the release notes also @jovirkku referenced - it’s probably because the system treats several entries as synchronized ones and therefore didn’t convert them. So it’s the content and that’s certainly something I won’t post here.

So any hints especially from sailors about the conversion procedure and the attributes that potentially make a difference would be the first thing I would wait for…

[pmelas]

I’ve lost my contacts too, but all my contacts come from a local nextcloud service, and synchronising the account I got the contacts back, however incoming calls are not recognised from stored contacts.
It only can match incoming calls to a contact only if the contact number is stored without its international country code prefix.

[gabrielg]

@Ygriega

But isn’t the schema the same everywhere?

No, the schema gets upgraded on the same DB, in my experience anyway.

@Ygriega

but help me with some hints which journal to look at and how.

Basically run journalctl as root - Logs do get quite long and by default they are only kept in memory. You could, if needed, store them to disk and set an upper storage limit, e.g., in /etc/systemd/journald.conf set Storage=persistent and SystemMaxUse=50M, then you may as well restart to hopefully catch that schema error, if that’s the problem.

[Ygriega]

Mhmm, why should the schema be different on several devices if they run the same OS version?

Anyway, I have definitely no time to tinker with journalctl… Therefore, I imported my backup after I cleaned suspicious attributes. Everything is back now.

However, I’d still be interested how Jolla determined if a contact was synchronized or not… Other people might face the same issue.

[jovirkku]

@Ygriega

if a contact was synchronized or not

On Sailfish 4, each contact (viewed in People app) has the address book indicated with a colourful icon under the name of the contact when it is a synchronised contact (e.g.Google, Exchange, etc.). If a contact is local (i.e., not synced with any external service) then there is a colourless phone icon or SIM card icon and the words “Address book”.

If the contact is viewed in the editor, then the choice of address book appears in the “Saved to” field.

Note that it is not possible to edit/change the address book yet but this should become possible later when the work in this area proceeds.

Only local contacts are included in Sailfish backups.

[peterleinchen]

@jovirkku

If a contact is local (i.e., not synced with any external service) then there is a colourless phone icon or SIM card icon and the words “Address book”.

@jovirkku
ould it be possible to show this information also in the main view (right now you need to go to the detail of each contact to get to this information)?

[jovirkku]

@peterleinchen

Would it be possible to show this information also in the main view (right now you need to go to the detail of each contact to get to this information)?

I do not know. Probably, the designers want to keep the main view clear and non-crowded. Nothing more than one tap on a name is enough to see the address book, then swipe to the right to get back to the main view. I would think checking the address book is needed only rarely.

[chris.adams]

@Ygriega

However, I’d still be interested how Jolla determined if a contact was synchronized or not… Other people might face the same issue.

In the 3.x.x database, synchronised constituent contacts had a specific syncTarget field (and most will also have an associated accountId although this is usually stored in out-of-band data rather than easily queryable). In 4.x.x we now support addressbooks as a top-level concept, so each contact is associated with a particular addressbook (collection). See https://git.sailfishos.org/mer-core/qtcontacts-sqlite/ for details. For the backup/restore code, see https://git.sailfishos.org/mer-core/nemo-qml-plugin-contacts/blob/master/tools/vcardconverter/main.cpp but you’ll have to spelunk through the commit history a bit to see the old syncTarget-based filtering.

The fact that your local contacts were clobbered by the update is very troubling - I’m glad it was resolved via restoring from the backup. I wish I had a copy of a database which failed to migrate properly so that I could investigate further, but I suppose it is too late at this point in any case, as most people will have long since upgraded their device. I can only hope that not too many people hit this same issue, and that all of those who did, had a backup which they could restore their contact data from.

[nas]

ahoy @chris.adams ,

~2 months ago I updated my phone to sailfish 4 (XperiaX). Soon I realized I lost many contacts. These days I investigate how many contacts I lost, and I compared the contacts .sqlite file I backuped before the update with the one after. I lost 306 contacts. Still not sure what is the pattern, why these contacts didn’t copied to the new database. Right now I write a php script to generate a .vcf file with all these contacts and the information they had, so I suppose I will be able to import to the phonebook again - I don’t use any on-line contacts service (like goοgle / nextcloud etc).

Let me know if you are interested on any details from the “old” contacts.db Vs the “new”.

[chris.adams]

Yes - I am interested, but I don’t know precisely what to look for. The upgrade steps which are run, can be seen in the qtcontacts-sqlite repository. If you have development skills, potentially you could build the older version and the newer version, with some extra debug logging perhaps, and run the upgrade manually on a copy of the backed-up sqlite db (e.g. within scratchbox or something) to see what is happening…