I suspect it might be related to the issue with the OBEX service. I experience very similar behaviour (contacts almost never show) with my Xperia 10 III and SFOS 5.1 in my Citroën C5. Restarting the OBEX service sometimes helps, but unfortunately I can’t say that it always works.
This is what I run as the default user with regular privileges. Can you try if it improves anything with your situation?
systemctl restart --user obex.service