I’d love to know what you’re doing “wrong”.
I can have my XA2 sitting outside in full view of a totally clear sky and have it take 15 minutes or much longer to get a lock, yet my Amazfit GTR 2 takes less than a couple of minutes.
My X without a sim card (therefore it cannot use MLS Manager) will get a lock within a minute or two, my Jolla C without a sim card gets a lock almost immediately.
The XA2 can get a lock reasonably quickly occasionally but it’s not reliable enough to be able to use it as a navigation/sports tracking device.