Developer of audio output here: I had the same problem, which is why I wrote the application in the first place. It just changes temporarily the sink to which the system outputs audio to. If it is misbehaving, a reboot (or restart of the pulseaudio service) at most should solve it. The app is not modifying any configuration files, it’s talking to the pulseaudio daemon the same way the volume indicators on most Linux desktop are doing it.
Anyway, as a response to @yajo10, in my case with the Xperia X the underlying cause was that there was too much dust inside the 3.5mm jack, causing the part that detects the presence of a plug to malfunction. I ended up removing the dust with a wooden toothpick, although I’m not sure if that could damage it. After that, it has worked just fine for me, so I would suggest you to try cleaning the 3.5mm jack as well
but you can miss an alert, because alerts will not set the audio output to speaker.
I’ve made a note on the Github issue tracker and depending on if time allows it, I’ll try to investigate.