How are the avatars connected into the People application?

[NodaUK]

I’m posting here, rather than in Bugs, as I’m looking for information. I’ve looked in the Sailfish OS Cheat Sheet.

I just upgraded to 4.1.0.24 (Kvarken) and needed to free some space on the phone as pre-upgrade backups were failing due to lack of internal storage space (the backup device was a large, empty microSD card). So, I deleted a couple of years worth of photos by doing a simple rm YYYY* at the command line for the relevant years.

After the upgrade to 4.1.0.24 (which hung at the end of the Sailfish progress bar and needed a forced reboot (Volume Up+Power), I’ve discovered a number of avatar images in the People app are missing.

So, what I’d like to know is where the People App stores the paths to the avatar images so I can read them and locate the files in my separate backup of the photos and restore them. It’s a bit of a pain to go through all my contacts one by one looking for missing avatars, then search for the corresponding image, when the filename will be stored somewhere, making the process a great deal more efficient.

I’d also like to understand how the Gallery app works and where/how it looks for images to display. Being able to organise the images in folders would be good, even if it has to be done outside of the app itself.

It would be good if avatar images used in the People app were stored separately, so they don’t vanish when the original is removed.

Phone is a Sony Xperia XA2 - Dual SIM

[atlochowski]

@NodaUK

I’d also like to understand how the Gallery app works and where/how it looks for images to display. Being able to organise the images in folders would be good, even if it has to be done outside of the app itself.

It’s not Gallery app looking for images but app called Tracker.

[NodaUK]

Replying to my own post.

1. The ‘People’ app stores its data in an SQLite database.
2. The SQLite database is found at:

/home/nemo/.local/share/system/privileged/Contacts/qtcontacts-sqlite

where there are three files:

contacts.db contacts.db-shm contacts.db-wal

You need to be in devel-su to access this directory

The .db is the database, the db-wal file is a write-ahead log, and the db-shm file is an mmaped shared memory file for coordinating access to the database and cacheing the wal-file.

So, lets open the database read-only. I change directory to where the database is located first.

#cd /home/nemo/.local/share/system/privileged/Contacts/qtcontacts-sqlite
#sqlite3 -cmd ".timeout 5000" "file:contacts.db?mode=ro"

The timeout will wait for up to 5000 milliseconds if the database is exclusively locked by someone else. The ?mode=ro means we open the database read-only.

So lets take a look:
# sqlite3 -cmd ".timeout 5000" "file:contacts.db?mode=ro"
SQLite version 3.33.0 2020-08-14 13:23:32
Enter ".help" for usage hints.
sqlite> .tables
Addresses Families OnlineAccounts
Anniversaries Favorites Organizations
Avatars Genders OriginMetadata
Birthdays GeoLocations PhoneNumbers
Collections GlobalPresences Presences
CollectionsMetadata Guids Relationships
Contacts Hobbies Ringtones
DbSettings Identities SyncTargets
Details Names Tags
DisplayLabels Nicknames Urls
EmailAddresses Notes
ExtendedDetails OOB

Aha! An Avatars table.
Let’s see.
sqlite> .schema Avatars
CREATE TABLE Avatars (
detailId INTEGER PRIMARY KEY ASC REFERENCES Details (detailId),
contactId INTEGER KEY,
imageUrl TEXT,
videoUrl TEXT,
avatarMetadata TEXT);

So the images are linked to via an image URL.
Lets look:
sqlite> SELECT * FROM Avatars LIMIT 2;
60|21|file:///home/nemo/.local/share/system/Contacts/avatars/439c35f67445ddff7126e30ac75683c7.jpg||
63|22|file:///home/nemo/.local/share/system/Contacts/avatars/439c35f67445ddff7126e30ac75683c7.jpg||

Bingo! So Avatars are stored in
/home/nemo/.local/share/system/Contacts/avatars

…but it turns out, on looking at all the database that the images can be stored elsewhere: some are stored in:
/home/nemo/.local/share/data/avatars

And it turns out that the ones listed as being in /home/nemo/.local/share/data/avatars are all present and correct, but all the ones in /home/nemo/.local/share/system/Contacts/avatars are missing.

Is this an upgrade bug, a consequence of the upgrade hanging, a consequence of me deleting pictures from the gallery or what. Any ideas?

[atlochowski]

@NodaUK

Is this an upgrade bug, a consequence of the upgrade hanging, a consequence of me deleting pictures from the gallery or what. Any ideas?

It’s a known upgrade bug.

[NodaUK]

That’s somewhat irritating*. I have 96 Avatars to recreate.

The backup I took just before upgrading is … different. The ‘People’ data is stored as a large file of ‘vcf’ data in the tar, with pictures encoded in what looks like base64. I’m not quite sure** how to restore the Avatars from that. Sigh.

*I thought I had read the release notes. Oh well. There is a link https://forum.sailfishos.org/t/lost-all-avatars-on-upgrade/5536, which doesn’t work (missing or private), under the title Sailfish OS Forum and T.J.C: Issues reported by the community and fixed in this update, but since it says the issue was fixed, I wasn’t worried. My optimism was unjustified.

**For ‘not quite sure’ read have ‘almost no idea’. Backup restores everything, whereas I’d quite like to choose to restore only People/(Contacts). I’m sure I can hack something up to extract the .jpgs from the vcf data and put them in the appropriate place, but it would be nice if I didn’t have to.

[NodaUK]

Another reply to myself.

It’s easy enough to split out the ‘People’ data from the backup. The backup is just a tar archive, so you extract the file /People/data/all.vcf . This is just a text file consisting of concatenated vcf files, with avatar images base64-encoded in each card. I did this on the sd card.

Restoring from this file is simple if you have the File Browser App installed. You just navigate to the file and open it in the File Browser, and automagically, all the People/Contacts are imported, but it looks like they are not duplicated, which is helpful.

As it happened, I hacked out a bash script to split out all.vcf into individual vcf/vcd files so I could import individual details before I tried the approach of opening the concatenated all.vcf. I had thought I might need to extract the individual avatar images and restore them one at a time, but that (thankfully) turned out not to be necessary.

Looking at the People App, it would be interesting to understand why it doesn’t store the images in the SQLite database itself, rather than just pointers to the images stored elsewhere in the file system. I’m sure there’s probably a good reason.

Similarly, I miss the ‘Data’ app on my old Psion, as it allowed me to set up arbitrary flexible databases with a simple (text mode) user interface. Many people miss Apple’s Hypercard for similar reasons. (In fact, the whole Psion suite of software (Agenda, Data, Sheet, Word) was/is excellent)

[rgrnetalk]

Gallery uses three documentgallerymodels with the property standardPaths/pictures or screenshots or filter on type video, see /usr/share/jolla-gallery/gallery.qml. On tjc folders for gallery was requested a long time ago: https://together.jolla.com/question/1948/feature-request-folderalbum-view-in-gallery/?sort=votes&page=1

On openrepos, there’s sailphoto (some marketing here, I have developed it

[NodaUK]

Thank you for that.

I should state my question more precisely: what are the paths described by Standardpaths.pictures? Or, if you want to teach me how to fish, rather than feeding me fish: where can I find out how/where Standardpaths.pictures is defined?

As for folders in Gallery, I would like two things:

1. The ability to manually put images in separate folders but still have them appear in the Gallery - perhaps by extending the standard search path
2. The ability to filter by folder, so I can temporarily exclude certain folders and/or folder trees.

It would also be helpful if a filter option could filter by ctime (which I know is not creation date) and date-times (if any) in EXIF information; and within a certain radius of a location (assuming recorded in EXIF data). I’m lazy, so unlikely to tag my images.

I’ll take a look at sailphoto.

[rgrnetalk]

If you want to fish, you should know where to get the bait: https://sailfishos.org/develop/docs/silica/qml-sailfishsilica-sailfish-silica-standardpaths.html

[NodaUK]

Thank you for the clues.

That link didn’t actually tell me a lot, but inspired me to kick off a search that found this document:

Qt Documentation : Qt 5.15 : Qt Core : C++ Classes : QStandardPaths

in which it says that the actual path (under Linux) is
PicturesLocation : "~/Pictures"

Other standard paths can be more complicated, e.g.:
FontsLocation : "~/.fonts", "~/.local/share/fonts", "/usr/local/share/fonts", "/usr/share/fonts"
AppLocalDataLocation : ~/.local/share/<APPNAME>, /usr/local/share/<APPNAME>, /usr/share/<APPNAME>
AppConfigLocation : ~/.config/<APPNAME>, /etc/xdg/<APPNAME>

What that document doesn’t say is that folders created in `~/Pictures’ are also searched for image files.

Curating images is a difficult problem as there are many different opinions on how best to do it, some of which are mutually exclusive. Much like many human endeavours.