Unfortunately this did not have as happy of an ending as I initially thought.
After having had the battery out, while the phone worked I noticed that entering my security code had a “lag” to it before the phone continued, as if verifying the code suddenly needed more time than before. That in itself wasn’t an issue - but the next thing I noticed was that the phone immediately shut down/reset if I inserted a charger. This was the same regardless of which charger, or whether it was a dedicated charger or a USB-data cable. It was not possible to both charge and have the phone active at the same time.
I also noticed that after having charged, the security code never verified the next startup - it just sat there spinning forever. A forced reset worked to get back to the now-laggier verification and I could get into the phone.
These issues are unfortunately serious enough that I cannot continue using the Jolla as my daily driver and I’ve had to revert to my previous phone. If anyone from Jolla is interested in debugging feel free to reach out - you can be as technical as you like in what my next steps here could be.