Exchange Email Synchronisation Reliability

[Steve_Everett]

REPRODUCIBILITY (% or how often): too frequently
BUILD ID = OS VERSION (Settings > About product): 4.0.1.8
HARDWARE (XA2, Xperia 10…): XA2
UI LANGUAGE: English
REGRESSION: (compared to previous public release: Yes, No, ?): No improvement across Sailfish 2.0, 3.0 and 4.0

DESCRIPTION:

Yep, it’s this old, old chestnut again - but since there appears to be no recent summary on SFOS (only years old stuff on Jolla Together), I thought I’d try again…

I have five email accounts configured on my XA2; two of them are imap (which are fine) and three of them are exchange. This bug report only concerns Exchange accounts.

Of the three Exchange accounts, one is large and two are tiny; the bug affects all three identically, so size of mailbox/folder store isn’t an issue.

All three have inbox, sub-folders, contacts and calendar synchronised, two way, with priority to server. No provisioning.

All three have been variously set to ‘always up to date’, timed synchronisation (e.g. every five minutes), custom sync (always up to date at peak, every hour off peak). It makes no difference whatever setting you use in any of the three accounts. Two are are company Office 365 accounts, one is not (my personal email) - one uses an app password as Sailfish can’t handle 2FA, the two others do not.

They are the same accounts I have had for years and have used on SF 2.0, 3.0 and now 4.0.

For years they have failed to synchronise reliably. They:

1. Just simply stop synchronising after a period of time with no error message, and/or
2. Get stuck on ‘Updating’ with a continuous turning circle, and/or
3. They fail (on 4.0), now with a warning triangle and ‘Sync Error’

Emails are therefore almost never up to date which severely affects usability of the app (I now dual carry an iphone for reliable email access).

I have tried:

1. Deleting and re-creating the accounts; They work for a while and then stop

2. Settings/Utilities restarting the network subsystem; Sometimes this fixes things for a while, sometimes it doesn’t - resulting in either a never-ending ‘Updating’ or a ‘sync error’

3. restarting EAS and messageserver from the command line; again sometimes this works, sometimes it doesn’t and you still get a sync error

4. Totally rebooting the phone; this always works immediately, until it fails again a short time later.

I have read an old bug about sometimes the sync stopping when you change networks from wireless to mobile (or vice-versa) so I have had my wife send me an email at home (wireless - it arrives automatically), then I have left home and she has tried again (mobile - it arrives automatically) and then gone back home again (wireless - it arrives automatically); so all good. But then do it a day later and it will fail - so no consistency there.

These issues have (for me and others at least) been outstanding for years, and given that Exchange support is a paid component of Sailfish X, really ought to have been fixed by now.

As an aside (and I suppose a bit off topic) unless Jolla puts some serious effort into fixing the core apps - email, browser and so on - so that they are reliable and usable (how about focusing on just those for a point release?) then people will simply start using reliable Android replacements and Sailfish will just become a fancy UI wrapper. I’m sure nobody wants this.

PRECONDITIONS:

STEPS TO REPRODUCE:

1. create an exchange email account

EXPECTED RESULT:

The account synchronises in line with its configured settings

ACTUAL RESULT:

Synchronisation fails completely after n undetermined period of time. Repeated attempts and inconvenient workarounds required to manually sync.

ADDITIONAL INFORMATION:

(Please ALWAYS attach relevant data such as logs, screenshots, etc…)

[Steve_Everett]

Should point out that on SF 2.0 I would have used imap for these accounts, so not really relevant.

[flypig]

It would certainly be good to get this fixed. Unfortunately the problem is sporadic and so hard to debug unless it’s actually happening on the phone. If you’re experiencing this often, would you be able to collect logs and pick out any errors from them?

To activate logs, you need to create a file ~/.config/eas-sailfish.conf with the following contents (see the Cheat Sheet):

[logging]
Sailfish.eas=d
Sailfish.easnetwork=d
Sailfish.easwbxml=d
Sailfish.logfile=/var/tmp/eas.log

This will store the logs at /var/tmp/eas.log for later checking. The log will contain a lot of data, including potentially the contents of messages, so please don’t post the file here. You should probably also delete the log periodically to avoid it getting too large.

However, if you experience the problem this will allow you to look back through the logs to potentially identify anything unusual (e.g. error messages) that might hint at what the issue is. If you could post that here, removing any personal info, that would be useful.

Don’t worry if you’re not in a position to do this, but if you can, it would be great.

[Steve_Everett]

I can try and do this but my tech skills are a bit rusty! I was a linux dev around 30 years ago so some of this will still make sense to me, but probably not a lot. I assume the file “.config/blah/blah” is a hidden file in the user home directory so I don’t need root permission to create it? If I just create it with a text editor I also assume it will end up with the right permissions, ownership, etc?

[flypig]

Yes, what you write sounds correct on all fronts. I’ve only just started looking in to this properly, so it may be that I can make headway without any need for more info. If you prefer to wait therefore and see how that pans out, and I’ll get back to you if I really do need to know more, then that’s fine as well.

[Steve_Everett]

OK, so done some more testing here. I’m based at home pretty much all of the time at the moment because of the UK lockdown. Therefore my phone is almost always on wireless and not mobile data. For the last few days I have had wireless switched off and exclusively used the phone on mobile data. During this period I have not had a single email synchronisation failure whereas I would usually have three or four per day requiring a phone re-boot to fix. So, some progress here. There’s nothing wrong with my wireless network - it up 24 x 7, my iphone, my wife’s iphone, ipad, deaktop, etc all use it without problem. My Sailfish phone appears to use it without a problem on web sites, etc and email (when it works!). Next step is to disable mobile data and leave the phone purely on wireless for a few days and see if the issue returns …

[Steve_Everett]

Now I’ve had the phone on wireless for two days with mobile data disabled … and again I’ve had no synchronisation problems during this period. I did have one ‘Connection timeout’ error message but using the ‘Update’ on the pulley menu removed this message and the account synchronised fine - no reboot required. So … it seems that the EAS sync process is fine when kept on one network, even when there are dropouts, but it can’t seem to cope with switching from Wireless to mobile data or vice versa reliably every time. My phone was reflashed to 4.0.1.48 this morning, so I’m now going to enable both wireless and mobile data on the reflashed phone to see what happens and if it goes back to the repeated ‘Sync Error’ problems…

[Steve_Everett]

And it does … less that an hour after enabling both wireless and mobile data together the ‘Sync Error’ is back on all of my Exchange accounts - around about once every couple of hours or so (sync is set at ‘Always Up To Date’ and I’ve had quite a few emails in today successfully, so its clearly not every time).
This does tend to suggest that the EAS sync process has a problem when there are two data networks available (wireless and mobile) and switching from one to another after, say a momentary dropout. If there is only one network available (i.e. mobile or wireless - it doesn’t matter which) then it does seem to be able to copy with a dropout unless it is too long - and perhaps that is why I got the ‘Connection Timeout’ message. @flypig does that give you enough to work on, or do you still need logs?

[flypig]

@Steve_Everett

does that give you enough to work on, or do you still need logs?

Thanks for going to such lengths to reproduce this @Steve_Everett. That does seem like a convincing set of factors, and I think it also means there probably won’t be much benefit to having more detailed logs. This will hopefully be enough to get to the bottom of things and I’ll let you know as things progress.

[peterleinchen]

Could this
https://together.jolla.com/question/225150/bug-or-feature-cellular-data-activatedpreferred-while-wlanwifi-is-enabledin-use/
be related?
Sorry, I cannot say whether it might have been fixed, or not meanwhile.

[flypig]

@peterleinchen

Could this
https://together.jolla.com/question/225150/bug-or-feature-cellular-data-activatedpreferred-while-wlanwifi-is-enabledin-use/
be related?

It’s not clear to me whether this is exactly the same issue.

From what we’ve been able to establish, the underlying issue causing the bug reported by @Steve_Everett is that occasionally the server places the client access into a “quarantined” state.

It’s not clear why this happens, since it’s triggered on the server side. Our intention is makes changes so that the user knows this is the problem (i.e. better error reporting to the user).

[Steve_Everett]

@flypig

From what we’ve been able to establish, the underlying issue causing the bug reported by @Steve_Everett is that occasionally the server places the client access into a “quarantined” state.

It’s not clear why this happens, since it’s triggered on the server side. Our intention is makes changes so that the user knows this is the problem (i.e. better error reporting to the user).

My problem was, it seemed, mainly around the switch in networks - from wireless to mobile and vice-versa. When this happened my Exchange accounts failed to sync for whatever reason. As I posted above (and on releases prior to 4.4) I could reliably make the Exchange sync fail by having both wireless and mobile networks enabled, and I could reliably make the Exchange sync succeed if I only had one network type enabled.

One of the community suggested workarounds was to only have one network type enabled at any one time - but its hardly practical to keep manually switching them every time you leave the house, or an office, or …

What was (is?) certain is that it was only Exchange accounts affected by this issue. I got so fed up with it all failing that I moved to IMAP sync (Still Office 365 account, but using IMAP not ActiveSync) several months ago - and I’ve not had a single sync failure since, and that is with both mobile and wireless enabled all the time. This ‘situation of IMAP sync reliability’ started for me on 4.3 and has not changed in 4.4.

I have not gone back to Exchange Sync since, mainly because I know and trust the IMAP sync to be reliable, and secondly the issue with losing connectivity when changing network types still persists in 4.4 - at least with Android apps. I’m still regularly seeing this with WhatsApp and FB Messenger reporting that they have no network whilst native apps remained fully connected to the network.

I guess for ActiveSync it will be important to understand why the server regularly puts the Sailfish client in a quarantined state where it does not do the same for IOS, Android, etc. Could it be because of connectivity issues between client/server because of the change in network type?

[flypig]

Thanks for the clarifications. Exchange sync uses a separate backend service for connecting to the server, with a completely different implementation. Given this, it’s not so surprising that the issue which affects Exchange doesn’t affect IMAP (not that it isn’t useful to have this confirmed).

@Steve_Everett

I guess for ActiveSync it will be important to understand why the server regularly puts the Sailfish client in a quarantined state where it does not do the same for IOS, Android, etc. Could it be because of connectivity issues between client/server because of the change in network type?

Yes, it’s possible, but as yet unclear why this might be. We had hoped that adding in feedback to the user (about why exactly the request was failing) would help better understand whether the two are related.

[Steve_Everett]

A bit of an update on this bug. Over the last year or so I’ve been trying different combinations of settings on my test SFOS phones (XA2 standard and, now, Xperia 10 iii) to see if I could narrow down the problem and I think I am getting somewhere.

When all of the above symptoms occur (i.e. the Exchange or Office365 account stops synchronising, and nothing will fix it short of deleting the account and setting it up again from scratch) I have the account set to synchronise email, contacts and calendar.

However, I have found that if you set the account to synchronise email ONLY (i.e. contacts and calendar are unchecked in the sync profile) then the account will continue to synchronise email indefinitely, even when the network changes from wifi to mobile - so, for me, this is major progress. I’ve had email synchronising reliably on my XA2 Plus for two months now without a problem, and on my X10 iii for as long as it takes until the phone needs to be re-booted for other reasons (loss of sound, loss of mobile data in Android, etc).

This synchronisation reliability still (since 4.4) doesn’t survive a re-boot though. If you re-boot the phone then the Exchange/Office365 accounts just stop working completely and you have to delete/re-create them to get things working again - but it is progress.

Furthermore, if you then enable calendar synchronisation then things still appear to work (more, longer term testing required) - but if you enable one-way/two-way contacts synchronisation then this is when the problems all come back again. So maybe the issue could be with the contacts synchronisation, and it is this that is causing the account to fail?

[flypig]

This is a really useful set of observations @Steve_Everett. I’m no longer in a position to make changes to the code, but I really hope this can help lead to finding the underlying cause. Thanks for taking the time to monitor things over an extended period and for sharing your results.

[Steve_Everett]

Perhaps these observations might also explain why some people have never had problems with Exchange synchronisation -they’ve only ever synchronized just email. It would be interesting to know from them.

[Ras72]

Sorry I haven’t contributed data points earlier. My 10III, fully updated, has two email accounts configured. One is 2FA O365 with calendar and email set to sync, and the other is an Exchange 2019 on-prem server set with calendar, contacts and email to sync.

I used to get the sync errors quite frequently, but some updates back in the last year or so they did something to fix Exchange synching. That solved my problems.

When I was having errors, I could also look in Outlook>email home view>view all folders>Sync Issues and then see what was causing problems and when.

I use my email/calendar/contacts daily and absolutely depend upon the Exchange synchronization to work. It does. I primarily use cellular (T-Mobile USA) but do switch to WiFi when I’m going to download updates or control something on the home LAN from my phone. The email continues to synchronize properly.

I am sorry I cannot think of anything else to help resolve your situation. Just trying to add data to form a clearer picture for those with the ability to solve this for you.

Cheers

[rob_kouw]

Besides the calendar multiplication I reported elsewhere, Exchange syncing is fine, generally speaking. Sometimes Exchange won’t sync for a couple of hours, then it starts going again. When it takes a long time, I will go into the Account and Every three months I have to renew 2FA for syncing again.
I’m on X10iii, using 4 IMAP accounts and one Exchange account. Two way, always up to date during business hours, provisioning is enabled.

Sync detail: I have a recurring event since a long time. Sync settings for Old events show a 1 month period. Changes to only one event in this series, will not be shown correctly. I just changed the period to All events, and I’ll see what happens.

Reading the results from others, I am curious about adding more than one Exchange account. I tried it on my previous X10ii, but it seemed 2FA could only handle one account at a time. Maybe I’ll try during the holidays.

[yomark]

Nerver worked reliably for me. Not with exchange on-prem, not with exchange online. Not with Kopano(z-push opensource activesync).
Mails are not being send, not all calendar entries show up. Appointments created are sometimes lost. I use Android Nine app for reliable operations, but now(since 4.6) it’s opening a tiny screen when creating a new mail.