On my XA2, when I take a call, the phone volume is to high ant I can’t lower it.
Solution: I rise it and then lower it. Then it accepts to go lower.
But I guess you tried that.
Anyway, this feature request seems to be close:
The thread also indicates @nephros patch which allows a finer volume control (Patchmanager).
It also talks about a pactl command which should allow to manually lower the volume, but I can’t find the earpiece when listing the outputs.