`Sailfish.Pickers` file pickers are abysmally slow

Bug Reports
#3

Nope. The ImagePickerPage uses qtdocgallery for fetching content, and that’s having a tracker backend as you also speculated. Though not sure could one say it’s specifically tracker that’s causing slowdowns or rather the queries made to it. Might be also some internal handling qtdocgallery. Or even something in the UI layer. But if the gallery app is equally(?) bad then it might be in the middleware side.

For quick reference, run the image picker with dbus-monitor on to get the query:

method call time=1729577276.694994 sender=:1.393 -> destination=org.freedesktop.Tracker3.Miner.Files serial=110 path=/org/freedesktop/Tracker3/Endpoint; interface=org.freedesktop.Tracker3.Endpoint; member=Query
   string "SELECT ?p0 ?p1 ?p2 ?p3 ?p4 ?p5 ?p6 ?p7 ?p8 ?p9 ?p10 ?p11  WHERE { GRAPH tracker:Pictures { SELECT ?x as ?p0 nie:isStoredAs(?x) as ?p1 rdf:type(?x) as ?p2 nie:isStoredAs(?x) as ?p3 nie:title(?x) as ?p4 nfo:fileLastModified(nie:isStoredAs(?x)) as ?p5 nfo:fileName(nie:isStoredAs(?x)) as ?p6 nfo:fileSize(nie:isStoredAs(?x)) as ?p7 nie:mimeType(?x) as ?p8 nfo:width(?x) as ?p9 nfo:height(?x) as ?p10 nfo:orientation(?x) as ?p11 WHERE {?x a nmm:Photo . ?x nie:isStoredAs ?file . ?file nie:dataSource/tracker:available true .  FILTER(!fn:starts-with(nie:isStoredAs(?x),'file:///home/defaultuser/Music'))} GROUP BY ?x ORDER BY DESC(nfo:fileLastModified(nie:isStoredAs(?x)))}}"
   file descriptor
         inode: 6406032
         type: fifo
   array [
   ]
1 Like
#4

Ah that’s too bad, because otherwise the fix would have been easy.

Well, the tracker query returns almost immediately and then the waiting begins without any output:

method call time=1729606853.599827 sender=:1.160 -> destination=org.freedesktop.Tracker3.Miner.Files serial=147 path=/org/freedesktop/Tracker3/Endpoint; interface=org.freedesktop.Tracker3.Endpoint; member=Query
   string "SELECT ?p0 ?p1 ?p2 ?p3 ?p4 ?p5 ?p6 ?p7 ?p8 ?p9 ?p10 ?p11  WHERE { GRAPH tracker:Pictures { SELECT ?x as ?p0 nie:isStoredAs(?x) as ?p1 rdf:type(?x) as ?p2 nie:isStoredAs(?x) as ?p3 nie:title(?x) as ?p4 nfo:fileLastModified(nie:isStoredAs(?x)) as ?p5 nfo:fileName(nie:isStoredAs(?x)) as ?p6 nfo:fileSize(nie:isStoredAs(?x)) as ?p7 nie:mimeType(?x) as ?p8 nfo:width(?x) as ?p9 nfo:height(?x) as ?p10 nfo:orientation(?x) as ?p11 WHERE {?x a nmm:Photo . ?x nie:isStoredAs ?file . ?file nie:dataSource/tracker:available true .  FILTER(!fn:starts-with(nie:isStoredAs(?x),'file:///home/nemo/Music'))} GROUP BY ?x ORDER BY DESC(nfo:fileLastModified(nie:isStoredAs(?x)))}}"
   file descriptor
         inode: 200056
         type: fifo
   array [
   ]
method return time=1729606853.608377 sender=:1.65 -> destination=:1.160 serial=499 reply_serial=147
   array [
      string "p0"
      string "p1"
      string "p2"
      string "p3"
      string "p4"
      string "p5"
      string "p6"
      string "p7"
      string "p8"
      string "p9"
      string "p10"
      string "p11"
   ]

I also spotted this ORDER BY DESC(nfo:fileLastModified(nie:isStoredAs(?x))) in the query. I had a quick look at GitHub - sailfishos/qtdocgallery and it looks like it just builds the query and does no sorting on its own, so I suppose sorting isn’t the culprit, right?

The gallery app feels less slow because it doesn’t freeze (nor crash) while waiting - it just shows an endless busy spinner. But I don’t mean faster, I just mean less slow.

#5

I’m curious since I notice ImagePicker pages performance on opening Imageworks. I have about 1800 images on the phone in several directories under Pictures and that opens smoothly. Imageworks opens an imagepicker directly after initializing. I’m curious if it opens for you in something like normal time?

Beside number of images itself, I’m wondering about context.

#6

There seems to be property sorting under the hood. Just a quick perusal: qtdocgallery/src/gallery/qgalleryqueryrequest.cpp at mer-master · sailfishos/qtdocgallery · GitHub

Not a sparql expert or anything, but these are operations on cursor in tight loops. qtdocgallery/src/gallery/tracker/qgallerytrackerresultset.cpp at mer-master · sailfishos/qtdocgallery · GitHub made me look twice.

#7

Sadly no, Imageworks takes just as long.

Interesting - I missed that! These sure look like candidates for profiling.

#8

Ok, thanks. I’m one of those, pull images off phone every year people since I make a book a year. So I guess I’m just lucky to be under the magic number.

That is what I thought :slight_smile:

1 Like
#9

Tried:

  • copied a small test image to device
  • created 10000 copies of if with a shell for loop
  • started composing a new email, add attachment → pictures

Result: loads content in roughly 3 seconds, UI remains fluid.

1 Like
#10

Hmm. It certainly didn’t take 3 seconds to load all 10000 images in the Imagepicker directly after a copy of 10000 600KB HD sized images. I’m still waiting for that to take place after some minutes.

But that’s not a real test since the tracker would usually index all those files which it hasn’t yet if I open up an Imagepicker directly after making a copy on the cli.

#11

Although, the ui DID remain responsive. It’s just that scrolling showed that it was taking a long time to work through the images.

EDIT. Still hasn’t finished indexing all the images and they still don’t all show in the Gallery app. I repeated the test on the 10iii and it’s even slower than my GS5.

#12

Interesting… 3 seconds would be a dream :sweat_smile:. However, this isn’t a very natural environment, of course. I have many different file types (mostly jpg, png, gif, mp4, webp, bmp) and file sizes (ranging between a few KiBs and a few MiBs for images), and each file is usually unique.

Also, another chokepoint could be the thumbnailer, and if all files are identical then it may hide issues there…?

That’s also interesting. Do you mean it’s mainly slow to load the thumbnails?

I don’t want to risk this experiment on my old phone because I’m afraid of wearing out the memory chip too much.

#13

Yes. VERY slow to load thumbnails. In fact, I ran out of time for the experiment (about 20 minutes waiting) and broke it off. I didn’t go further to test if it would load faster after some amount of time for thumbnailer indexing. So, not a very useful test. However, it is interesting to not that with 12,000 images the pickers showed up quickly and didn’t die.

#14

20 minutes is much more patience than I usually muster, but in my case it usually crashes before that anyway. Plus, in my case it’s not primarily that the thumbnails are too slow - they are, but that’s negligible in comparison. The main problem is that everything hangs.

Definitely interesting! I also wonder what’s the difference between the picker and the gallery app, because the gallery app doesn’t hang in my case. (Imageworks does, though…)

#15

I wonder if it might have to do with filters for file types? I noticed on looking at Imageworks that I did’nt have any filters options set. On the ‘filePicker’ view I had:
nameFilters: [ '*.jpg', '*.jpeg', '*.png', '*.tif', '*.tiff', '*.bmp', '*.gif' ]

And yes, I do have a bmp file or two :slight_smile:

1 Like
#16

True, my testing here is a bit artificial so there could be chance for avoiding slowdowns. I’m thinking something like extra sorting which would be avoided. Wouldn’t think the file types or sizes should matter as the model itself is not that interested about them.

What comes to thumbnailer, it should be doing the preview images based on what is needed by the view position. Happens in the background so shouldn’t affect the loading of the model content, slow down other UI, or anything.

If the Gallery app is ok, that’s hinting towards the qtdocgallery not being that bad. It’s also using qtdocgallery for the content.

Did you test the email attachment case I was using? That to rule out any differences on how the picker is set up.

There is one extra complication now on the pickers that it copies the qtdocgallery model content to a separate ListModel for adding name filtering and selected states. That’s not ideal and if a separate model is needed e.g. something based on QSortFilterProxyModel could provide better performance. Having that said, I’m getting ContentModel.qml _update() function called only at start (0 items) and finished (lots of items) and the content copying wouldn’t seem to be that bad. Tried adding console.time(“update”) to start, console.timeEnd(“update”) to end and it’s taking some 945ms on Xperia 10 II. Room for improvement indeed, but not bad enough to explain your experiences.

Maybe that timer is also something you could try.

1 Like
#17

This was always snaily slow in SFOS as long I can think back.
Attaching this

made it a lot faster.

#18

Don’t think that’s related. The changes there are about speeding up transitions, this is something more fundamental on the model handling or similar level.

#19

Yes, main thing I did was speed up (imho annoying) transitions. But I observed a huge speed gain when starting Gallery, LLs Video Player, DeadBeef Silica, loading the previews of the images/clip thumbnails/track lists, and all operating these apps and others is more fluent in general, as loading content goes more fluent. Some apps have their own delays that have to be removed in their qml pages, others are OK without any tweak.
I’m also surprised and unexpectedly delighted and did never await that the UI tweaks would affect such things. Therefore I posted it and hope that it’s useful for someone.

edit: Doing so, I had to ‘harmonize’ the new durations by trial and error, to avoid driving some processes into crash situations and stucking UI. Generally i set values to 1/3 of the original.

1 Like
#20

Yes, it’s just as bad as the other cases.

Well, the difference is that the gallery app doesn’t open the gallery page immediately. It starts, then shows a busy spinner over “pictures” and “videos” for a very long time. Only once the spinner has finished, you can click on the category to open the gallery page. So it’s not that the gallery app is “ok”, it’s just that the gallery app doesn’t hang while loading.

I’ll give it a try

1 Like
#21

By any chance, does tracker3 status complain about errors? Several years ago I had some enormous performance issues on my computer and it turned out to be because tracker choked on a couple of files that were apparently broken in some unexpected way. I deleted those files and the problem was fixed. These days, tracker supposedly doesn’t have that problem anymore, but I believe Sailfish is still using the older version of tracker that doesn’t know what to do with the broken files.

On my device, with about 12,000 pictures according to Gallery, the file picker for pictures takes about 10 seconds to load the list, and then the image previews still need to load in, but at least you can already start scrolling. No issues according to tracker, just a couple of files that don’t have any metadata.

2 Likes
#22

tracker3 status reports:

[nemo@Sailfish ~]$ LC_ALL=en tracker3 status
Currently indexed: 44705 files, 1668 folders
Remaining space on database partition: 6.0?GB (31.31%)
All data miners are idle, indexing complete
40 recorded failures

Problems are reported for jpg, png, pdf, ogg, and mid files. Most issues are Could not get any metadata for uri:... but two stand out (for two pdf files):

Couldn't open PopplerDocument:PDF document is damaged
Crash/hang handling file

The file with the “crash” message has <U+FEFF> in its filename, says tracker’s output. Tracker also seems to have issues with ut8 characters. It shows í (i+´) as ?.

The gallery app shows 1647 videos and about 32’000 pictures. It takes ~10-30s to start (sometimes it fails to start) and then another ~60-90s until the “pictures” spinner stops and the icon becomes clickable.