[C2] Mobile network unavailable, ofono crashes badly, continuously, dumps core

Wellā€¦ :slight_smile:

How I did that is roughly documented here. So itā€™s not a flash-in-ancient-times-and then-update
scenario, itā€™s all manual mangling.
Obviously, with the C2 thatā€™s not possible otherwise.

I guess I could reverse it too if it really makes a difference, but like to be quite subborn in my abhorrence of the default user name.

Thanks for looking - I think so too, but something about it shows up in the logs so I wanted to be transparent about it.


Anyway, my C2 device is in the process of downloading something secret, so I wonā€™t reboot ATM. But Iā€™ll report back after that is done.

Can the SIM card itself play a role in this?

the one Iā€™m using is a bit weird.

Itā€™s new, (bought two or three days ago), and itā€™s from Austrian Budget provider/MVNO ā€œS-Budget Mobileā€.

Now whatā€™s weird about it is that they claim to use T-Mobileā€™s network (called Magenta here), so MCC 232 (Austria) and MNC 03 (T-Mobile).

BUT.

The APN/MMC/SMS configuration that was applied to the Settings app is that of tele.ring (MNC 07).

Actually precisely the settings given in mobile-broadband-provider-info/mobile-broadband-provider-info/serviceproviders.xml at 4a1617295360c10331e748e3a9f42311fbe3325c Ā· sailfishos/mobile-broadband-provider-info Ā· GitHub

Also, if you use their website https://www.s-budget-mobile.at/, while the text makes it clear in many places that the network is T-Mobile/Magenta, the website itself makes a lot of connections to telet.ring named services.

I have added something to the bug report I forgot about: I have a masked aml.service via

nemo@PGJollaC2$ systemctl --user mask aml.service 

Could that cause failure?

Affected phone was left over night, ofono service seemed fine, but related options in the UI (statusbar icon, Topmenu icon, Settings section about SIM/mobile network) were not available.

Restarted ofono a couple of times again.
Now at the moment itā€™s running, but only recognizes one SIM slot (acts as if it were a single-SIM device).

Aha! Solved it!

Tl;DR: it seems it was an unsent SMS sticking around in /var/lib/ofono


Background: That SIM is a prepaid one.

After buying it, you have to

  1. Register it/verify yourself on some website (legal requirement)
  2. Top up the prepaid amount (starts with 0)
  3. Select a data/voice plan (switching away from the default plan). Which can among other options can be done by sending SMS to some internal short number.

While 1. and/or 2. has not been done, one can receive SMS from the provider, and do some things like dial *#101# to check the prepaid balance.

Just for kicks I tried sending the ā€œSelect planā€ SMS twice, and some to another phone number of mine.
These messages were either in ā€œFailedā€, or in ā€œSendingā€ state when I rebooted, and this is the pointthe Ofono crashes began.

After I did the below and rebooted, ofono crashes are gone,
Not sure which one of the files was the reason, but it seems fixed now.
Probably itā€™s the tx_queue thing, and the other ones wouldnā€™t have been necessary, too late to check now.

root@PGJollaC2:/var/lib/ofono/2320xxxxxxxxxxxxxxx/tx_queue # ls
0-14-7107xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
1-14-3F38xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2-14-1B0Dxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
root@PGJollaC2:/var/lib/ofono/2320xxxxxxxxxxxxxxx/tx_queue # rm -rf *

root@PGJollaC2:/var/lib/ofono/2320xxxxxxxxxxxxxxx # ls -la
total 40
drwx------    4 radio    radio         4096 Nov 12 21:21 .
drwx------    4 radio    radio         4096 Nov 12 21:18 ..
-rw-------    1 radio    radio           32 Nov 12 21:21 cbs
-rw-------    1 radio    radio          497 Nov 12 21:16 gprs
-rw-------    1 radio    radio            0 Nov 12 15:28 ims
-rw-------    1 radio    radio           21 Nov 12 15:28 netreg
-rw-------    1 radio    radio           55 Nov 12 21:16 radiosetting
-rw-------    1 radio    radio           72 Nov 12 21:16 sms
drwx------    6 radio    radio         4096 Nov 11 18:16 sms_assembly
drwx------    2 radio    radio         4096 Nov 12 21:29 tx_queue
-rw-------    1 radio    radio           24 Nov 12 21:16 voicecall

root@PGJollaC2:/var/lib/ofono/2320xxxxxxxxxxxxxxx # find sms_assembly/
sms_assembly/
sms_assembly/14D0xxxxxxxxxxxxxxxxxxxxxxxxxx-225-2
sms_assembly/14D0xxxxxxxxxxxxxxxxxxxxxxxxxx-225-2/002
sms_assembly/0ED0xxxxxxxxxxxxxxxxxxxx-197-2
sms_assembly/0ED0xxxxxxxxxxxxxxxxxxxx-197-2/002
sms_assembly/0ED0xxxxxxxxxxxxxxxxxxxx-62-2
sms_assembly/0ED0xxxxxxxxxxxxxxxxxxxx-62-2/002
sms_assembly/14D0xxxxxxxxxxxxxxxxxxxxxxxxxx-29-2
sms_assembly/14D0xxxxxxxxxxxxxxxxxxxxxxxxxx-29-2/002
root@PGJollaC2:/var/lib/ofono/2320xxxxxxxxxxxxxxx # rm -r sms_assembly/* -f

root@PGJollaC2:/var/lib/ofono # cd 2320xxxxxxxxxxxxxxx-3/
root@PGJollaC2:/var/lib/ofono/2320xxxxxxxxxxxxxxx-3 # ls
version
root@PGJollaC2:/var/lib/ofono/2320xxxxxxxxxxxxxxx-3 # cat version
root@PGJollaC2:/var/lib/ofono/2320xxxxxxxxxxxxxxx-3 # cd ..



root@PGJollaC2:/home/.system # cd var/lib/connman
root@PGJollaC2:/home/.system/var/lib/connman # ls
cellular_2320xxxxxxxxxxxxxxx_context1  iptables
global_proxy                       settings
root@PGJollaC2:/home/.system/var/lib/connman # ls -la
total 24
drwx------    5 root     root          4096 Nov 11 15:29 .
drwxr-xr-x    6 root     root          4096 Oct 19 19:32 ..
drwx------    2 root     root          4096 Nov 11 15:31 cellular_2320xxxxxxxxxxxxxxx_context1
drwx------    2 root     root          4096 Nov 12 21:23 global_proxy
drwx------    2 root     root          4096 Nov 12 21:23 iptables
-rw-------    1 root     root           165 Oct 30 21:35 settings
root@PGJollaC2:/home/.system/var/lib/connman # cd cellular_2320xxxxxxxxxxxxxxx_context1/
root@PGJollaC2:/home/.system/var/lib/connman/cellular_2320xxxxxxxxxxxxxxx_context1 # ls -la
total 12
drwx------    2 root     root          4096 Nov 11 15:31 .
drwx------    5 root     root          4096 Nov 11 15:29 ..
-rw-------    1 root     root           176 Nov 11 15:31 settings
root@PGJollaC2:/home/.system/var/lib/connman/cellular_2320xxxxxxxxxxxxxxx_context1 # cat settings
[cellular_2320xxxxxxxxxxxxxxx_context1]
Name=S-BUDGET MOBILE
Favorite=false
AutoConnect=false
Modified=1970-01-01T01:00:00Z
IPv4.method=dhcp
IPv6.method=auto
IPv6.privacy=disabled


root@PGJollaC2:/home/.system/var/lib/connman/cellular_2320xxxxxxxxxxxxxxx_context1 # rm settings
rm: remove 'settings'? y
root@PGJollaC2:/home/.system/var/lib/connman/cellular_2320xxxxxxxxxxxxxxx_context1 # cd ..
root@PGJollaC2:/home/.system/var/lib/connman # rmdir cellular_2320xxxxxxxxxxxxxxx_context1/
``
3 Likes

Unticked the solution checkbox because it appeared again.

It can be triggerered by having ā€œundeliverableā€ or ā€œsendingā€ SMS

3 Likes

Thanks a lot! I can confirm your workaround!

EDIT: Simplification!

My symptoms. Basically after failed SMS ofono lose sanity :wink: . So no mobile network at all. Some did report success after many reboots.

I did try to erase that SMS in the app itself but that didnā€™t help!

So command line it is.

  1. First check that this is the proper issue: ofono in failed state.
[defaultuser@JollaC2 ~]$ systemctl --failed

Note that for now lxc@multi-user.service and pulseaudio.service can be in this list for reason unknown to me.

  1. If ofono is listed this seems to be the same bug and you should stop ofono during this process as root:
[defaultuser@JollaC2 ~]$ devel-su
[root@JollaC2 defaultuser]# systemctl stop ofono.service
  1. Then find the newest stuck SMS in nonempty tx queue:
[root@JollaC2 ofono]# ls -lR /var/lib/ofono/*/tx_queue/* | grep -B 2 radio
/var/lib/ofono/293400130011175/tx_queue/0-15-B43D6FFB5F8E9DBA53425680E83862CCA4DA7982:
total 4
-rw-------    1 radio    radio           86 Nov 25 15:42 000

Which in this case is obviously file 000 (name is after time) in 0-15-B43D6FFB5F8E9DBA53425680E83862CCA4DA7982.

Optionally double check itā€™s content (note that due to different codepage in CLI it might seem corrupted):

[root@JollaC2 defaultuser]# cat /var/lib/ofono/293400130011175/tx_queue/0-15-B43D6FFB5F8E9DBA53425680E83862CCA4DA7982/000 
  ļæ½ļæ½61ļæ½eļæ½& pozdrav,
  1. And then remove that file (in this case 000):
[root@JollaC2 defaultuser]# rm /var/lib/ofono/293400130011175/tx_queue/0-15-B43D6FFB5F8E9DBA53425680E83862CCA4DA7982/000 
  1. And finally reboot or even faster:
    [root@JollaC2 defaultuser]# systemctl restart ofono.service
    C2 works as before!
6 Likes

Thanks. Works for sometimes, then happens again, so we have to repeat the same process.
Hopefully, Jolla can fix this and provide the updated telephony service soon, because right now it is not reliable.

2 Likes

I am trying to follow up on your steps. I cannot identify anything noteworthy in any tx_queue ! All directories under /var/lib/ofono have empty tx_queue directories, though I know I have some failing SMSes.

Is ofono in fail state?

[defaultuser@JollaC2 ~]$ systemctl --failed

Otherwise itā€™s not the same bug.

No, only the ā€œnormalā€ failures (as I read somewhere else ! Why are they normal to fail at this point in time ?) :

  UNIT                   LOAD   ACTIVE SUB    DESCRIPTION
ā— lxc@multi-user.service loaded failed failed LXC Container: multi-user
ā— pulseaudio.service     loaded failed failed PulseAudio (system-wide mode)

No idea. Question for separate thread.

Same problem here. Hereā€™s the trace of ofono crashing

Backtrace of ofono -n :

ofonod[27230]: oFono version 1.29
ofonod[27230]: Initializing AML plugin.
ofonod[27230]: Reading of /etc/ofono/phonesim.conf failed: No such file or directory
ofonod[27230]: [gbinder-radio] Connected to android.hardware.radio.config.IRadioConfig/default
ofonod[27230]: [gbinder-radio] Connected to android.hardware.radio.modem.IRadioModem/slot2
ofonod[27230]: [gbinder-radio] Connected to android.hardware.radio.data.IRadioData/slot2
ofonod[27230]: [gbinder-radio] Connected to android.hardware.radio.messaging.IRadioMessaging/slot2
ofonod[27230]: [gbinder-radio] Connected to android.hardware.radio.network.IRadioNetwork/slot2
ofonod[27230]: [gbinder-radio] Connected to android.hardware.radio.sim.IRadioSim/slot2
ofonod[27230]: [gbinder-radio] Connected to android.hardware.radio.voice.IRadioVoice/slot2
ofonod[27230]: [gbinder-radio] Connected to android.hardware.radio.modem.IRadioModem/slot1
ofonod[27230]: [gbinder-radio] Connected to android.hardware.radio.data.IRadioData/slot1
ofonod[27230]: [gbinder-radio] Connected to android.hardware.radio.messaging.IRadioMessaging/slot1
ofonod[27230]: [gbinder-radio] Connected to android.hardware.radio.network.IRadioNetwork/slot1
ofonod[27230]: [gbinder-radio] Connected to android.hardware.radio.sim.IRadioSim/slot1
ofonod[27230]: [gbinder-radio] Connected to android.hardware.radio.voice.IRadioVoice/slot1
ofonod[27230]: getPreferredNetworkTypeBitmap error 1
ofonod[27230]: getPreferredNetworkTypeBitmap error 1
ofonod[27230]: SIM card OK
ofonod[27230]: Requested file structure differs from SIM: 6fb7
ofonod[27230]: SMS History Probe for modem: 0x27993bc0
ofonod[27230]: Registered interface org.ofono.NetworkTime, path /ril_0
ofonod[27230]: nw selection is already auto
ofonod[27230]: smsc query error GENERIC_FAILURE
ofonod[27230]: Requested file structure differs from SIM: 6fb7
ofonod[27230]: data reg changed 2 ā†’ 0 (unregistered), attached 0
ofonod[27230]: export_entries_one_storage_cb with ME failed
ofonod[27230]: data reg changed 0 ā†’ 2 (searching), attached 0
free(): invalid next size (fast)
ofonod[27230]: Aborting (signal 6) [ofonod]
ofonod[27230]: ++++++++ backtrace ++++++++
ofonod[27230]: +++++++++++++++++++++++++++

Deleting the content in the folder was not enough (the problem still appeared with the folder empty, but messages pending for sending in the messages app).

Finally got it back to a working state after deleting everything that is pending, and restarting ofono, then restarting the whole phone.

Ouch! Iā€™ve been bitten by this bug on my C2 today.

Can somebody please take a look into this race condition in ofono and fix it? Or at least provide detailed instructions on what information to collect if this happens again (core file, logs etc) to provide enough information to developers to fix this.

5 Likes

Me also today, hopefully fixed in next Update.

Thanks for the easy and fast fix @filip.k

1 Like

Sometimes ofono restart is enough (step 5).

Didnā€™t worked for me, because it happend when a sending SMS stalled. Removing the unsend SMS in Messages-App and reboot were not enough, i had to remove it with the shell-commands like you described above.

It seemed has helped me for pretty stable SMS tx, by have the most current config. In my case for Telekom .de , and brands using it, e.g. Pennymobil .de ā€¦, etc. I have recognized with SIM by Pennymobil .de I can use the Telekom .de config: Data-AP as internet.v6.telekom /dual while MMS-AP only work with internet.telekom, (IPv4 only) on proxy 109.237.176.193:8008 maybe that is in use for SMS too?

To use this I need developer mode activation?

Yes. Or ssh access at least.