By any chance, does tracker3 status complain about errors? Several years ago I had some enormous performance issues on my computer and it turned out to be because tracker choked on a couple of files that were apparently broken in some unexpected way. I deleted those files and the problem was fixed. These days, tracker supposedly doesn’t have that problem anymore, but I believe Sailfish is still using the older version of tracker that doesn’t know what to do with the broken files.
On my device, with about 12,000 pictures according to Gallery, the file picker for pictures takes about 10 seconds to load the list, and then the image previews still need to load in, but at least you can already start scrolling. No issues according to tracker, just a couple of files that don’t have any metadata.