While I quite happily had my Calendars and Contacts synced (via WebDAV) to my F(x)tec Pro¹ with SFOS 3.3 and 3.2, I fail to get SFOS 3.4 to authenticate successfully against https://apps.kolabnow.com
Now before I start to pull old but officially supported devices from the shelf, upgrade them and try there, is anyone successfully syncing their data from Kolab Now with their SFOS 3.4 device?
On this device, I tried restoring a pre-3.4 backup (after creating a symlink so a /home/nemo pointed at /home/defaultuser) or set up DAV account from scratch (from complete scratch, device was reflashed with 20201125-run-build-lvm-testing for both test), result is the same.
So I pulled my Xperia X off the shelf and did a few tests.
Looks like CalDAV and CardDAV accounts working < 3.4 continue working on 3.4 but flashing 3.4 directly (no OTA) has trouble both setting up CalDAV and CardDAV (for Kolab Now in my case) and trouble restoring a backup (backup of account works, it just never syncs as I can not get it to accept the password)
if I flash Sailfish_OS-Jolla-3.4.0.24-f5121-0.2.1.58 on my Xperia X device, the failure is the same 503[1].
Both for a restored backup and a fresh setup of the DAV account
Adding a CalDAV and CardDAV URL on fresh setup (as opposed to just server URL, username and password) made no difference, still get a 503
if I flash SailfishOS-2.1.3.7-f5121-0.0.1.18 (that version simply because it was the only one < 3.4 for F5121 I had lying around) and then take that with the GUI updater on the phone itself through the stop releases[2] to 3.0.0.8, then restore my backup (to get my Kolab Now sync config back), it works
if I OTA update from that working 3.0.0.8 to 3.2.0.12 the previously restored config continues working
if I OTA update from that working 3.2.0.12 to 3.4.0.24 the previously restored config continues working
Questions to Jolla;
What component and version should I choose at bugs.merproject.org if I want this regression tracked?
Can you reproduce this behaviour with other DAV servers?
Where would I download 3.0.0.8 and 3.2.0.12 for F5121? Your shop page (behind the paywall) only offers me the latest version, is there some place I can authenticate and pull a previous version?
syncing Kolab Now via DAV is generally painless on all my OSes, and it does work on SFOS 3.4 as long as I OTA upgrade, so I do suspect some kind of regression on the SFOS side.
Last login: Sat Jan 2 17:10:32 2021 from 192.168.50.59
,---
| Sailfish OS 3.4.0.24 (Pallas-Yllästunturi)
'---
[defaultuser@XperiaX ~]$ rpm -q buteo-sync-plugin-caldav
buteo-sync-plugin-caldav-0.1.70-1.11.1.jolla.armv7hl
[defaultuser@XperiaX ~]$ devel-su -p
Password:
Enjoy your privileges.
bash: /proc/2722/environ: Permission denied
bash: /proc/2722/environ: Permission denied
[defaultuser@XperiaX ~]$ jolla-settings
[D] unknown:0 - Using Wayland-EGL
[W] unknown:64 - file:///usr/share/jolla-settings/settings.qml:64:5: QML DBusAdaptor: Failed to register service com.jolla.settings
SailfishKeyProvider_storedKey(): error: no scheme or key found for provider/service
SailfishKeyProvider_storedKey(): error: no scheme or key found for provider/service
SailfishKeyProvider_storedKey(): error: no scheme or key found for provider/service
SailfishKeyProvider_storedKey(): error: no scheme or key found for provider/service
[W] unknown:0 - sendAuthenticatedRequest(): request error: QNetworkReply::NetworkError(ServiceUnavailableError) response code: 503 url: "https://apps.kolabnow.com"
Very good reporting and bug tracking work, I would say!
Probably you should add all of that up and create a case at https://jolla.zendesk.com/hc/en-us, so it’ll be tracked there.
And to check out which component to focus on you could also discuss this topic in the next community meeting i think: Community meeting on IRC 14th Jan 2021
If I understand properly, you cannot create a new generic CalDAV / CardDAV account from the UI, but if one already exists it is syncing properly. Restoring a backup is creating a new account so same problem.
Looking at the code, when the account is created, there is a check that is done that the user can authenticate on the server using a request at server address + webdav path. In your case, you didn’t provide a webdav path because the paths differ for CardDAV and CalDAV. Your server is not allowing to authenticate on the root address of the server, so the authentication is failing and the account creation fails. But it’s just the code logic for account creation that is faulty, if an account already exists it is addressing the complete paths directly for each service.
If all my blabla is correct, you should be able to create an account for CalDAV alone, puting the CalDAV path in the webDAV path in the UI.
If so, I’ll discuss with @chris.adams and @blam to see if we can adjust a bit the account creation for such cases (where servers reject authentication at root address).
Not quite. Allow me to list the works and does not work points
works I had working CalDAV and CardDAV syncing on all my SFOS 3.0 and 3.2 devices (not 100% sure about 2.0, while I have a few SOFO devices, I generally only have one on active user as daily driver and the others sit on a shelf)
I can flash my phones and restore my user backup of my CalDAV and CardDAV settings(from µSD card) seemingly fine until SFOS 3.2.0.12 included.
Both restoring a working backup and setting up a …DAV account from scratch on a freshly flashed SFOS 3.4.0.24 fails
OTA updating a 3.2.0.12 with working CalDAV and CardDAV sync to 3.4.0.24 does work.
I meant when filing a bug on bugs.merproject.org I should select a component, I think the buteo…sync one is the correct one but wanted to be sure.
Not selecting one is annoying to the triage people on a bugzilla, so it helps to set the correct one from the start.
As to community meeting, was going to say that I’ll try to attend, but 08:00 UTC falls right into the hour where I get ready for work, so no promises.
Thank you very much! That indeed did work.
So now I have 2 DAV accounts set up for Kolab Now. One for CalDAV and a second one for CardDAV.
Not perfect, but for me this is an acceptable workaround for this regression. Unsure how other Jolla customers feel about the work around, so not marking this topic as resolved/answered.
Basically I would even have been happy with instructions along the lines of edit this file with vi in a shell since without my address book any phone is totally useless to me and not having calendars severely reduces the phone’s viability as a daily driver.
Hopefully the zendesk ticket goes towards a fix for the regression.
I’m afraid that this bug tracker is no longer actively monitored. Jolla advertised to report bug on this forum instead.
I think, I understand the root of the issue, sadly it is located in closed source parts. You can look at /usr/share/accounts/ui/OnlineSyncAccountCreator.qml line 233. You can see that the authentication is done on server address / webdav path, which is not working for your server. The authentication check should be done on one of the service path instead I think. I’ll discuss on IRC with Chris Adams about the best way to actually solve this and be able to use only one account for both services.
Thanks @dcaliste, you saved me from wasting time on the wrong bugtracker.
For me, one of the hardest things in using SFOS is the perceived constant state of flux the servers I need to visit are in.
Crud, closed.
But hey, there’s already an if nextcloud, so maybe adding an if kolabnow is not too much hassle for Jolla.
After poking around it a bit more, it’s also my understanding now that the issue is that authentication check should be done on one of the service path instead. I still fail to understand why I was fine to create a new CalDAV and CardDAV account on SFOS until and including 3.2, but since the component is closed source I cant easily git log OnlineSyncAccountCreator.qml on a local checkout.
Before I dig deeply into RFC 6764 , does the closed source component aim to support RFC 6764 and does Jolla prefer TXT[1] or .well-known[2]?
Being also on FreeNode, I wonder if there is a channel I should join? You should see me lurking on #mer, #meego and #maemo already
(inspired by the DNS section of https://kb.kolabnow.com/documentation/dns-settings-for-group-managers )
but I saw no TXT record giving me a path for _caldavs._tcp (I might just have looked wrong, I do not query SRV and TXT that often)
Plus at Kolab Now the path is different per user and I am unclear how one would deal with that in a TXT record (guess this needs more in depth reading of 6764). I think authenticated .well-known might be better.
[2] quick test, but I might be doing that wrong (obviously I used the correct user@domain)
curl -L --anyauth --user user@domain https://apps.kolabnow.com/.well-known/caldav
Enter host password for user 'user@domain':
[...]
<p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>
[...]
If I used curl correctly, then I can try to ask the Kolab Now people if something is broken on their end, but if the closed component of SFOS does not even try authenticated .well-known and has no plans to do so in the future, then there is no point in me wasting time on that.
As mentioned above, I’m usually discussing with @chris.adams on #sailfishos. We have a periodic meeting every Tuesday at 8 a.m. UTC. Next one will be on January, 12th. You’re welcomed to join if the time slot is convenient for you. You can find logs of the channel if you are interested.
The closed source parts are using the .well-known capability to discover the user CalDAV path. But as far as I can see, this is done later on in the code path, after validation of the authentication on the server path. Which is failing in your case.
After further thinking on the matter, I’m not convinced personally that authenticating on one of the service path is a good idea neither, since this path may not be provided and only guessed later with the .well-known method. Which leaves the question on how to validate the provided authentication information (login and password) opened. That’s where it’s interesting to discuss with Jolla people before running to coding a solution. So let’s keep this topic opened for suggestions and brainstorming up to the 12th. Thanks by the way for the tests on the Kolabnow instance for the .well-known method.
That was sadly a waste of my time. Jolla support unilaterally closed the ticket with a “you seem to have a work around as per forum” even though I did ask for this to not be closed.
I’ll finish dealing with this regression in the forums but I’ll keep support’s reaction in mind next time I run into a bug or regression.
I’ve briefly discussed the issue with @chris.adams this morning as planned on IRC . Here is an extract from the logs:
dcaliste Since 3.4.0 changes in account creation, I think the login / password are checked on the root of the server before going further and retrieving calendar list... 09:03
dcaliste On some servers, the authentication on root is forbidden. 09:03
dcaliste So account creation is failing. 09:03
dcaliste Restoration from storage is failing also for the same reason I guess. 09:04
chriadam eek. 09:04
chriadam so we need to fallback to trying on ... either caldav calendars path, or carddav addressbooks path, or webdav home set path? 09:05
dcaliste Yes, that was my first guess, but there is an issue doing so : the caldav path may be guessed. 09:06
dcaliste So not available at that point. 09:06
chriadam what do you mean by "guessed"? do you mean, looked up via the .well-known endpoint, or something else? 09:06
dcaliste See the kolab issue, there is no webdav path, and caldav and carddav paths are different. 09:07
dcaliste Yes, looked up via .well-known. 09:07
dcaliste (well in the kolab report the .well-known seems to fail but that's another story, more related to server side I think). 09:08
dcaliste So the situation is : 09:09
dcaliste - server root / webdavpath may be forbidden and refused for authentication. 09:09
dcaliste - server root/ webdav path/.well-known may be the solution to try to authenticate 09:10
dcaliste - using the caldav path or the carddav one may not be possible, because not provided by the user at that point. 09:10
chriadam thanks. I've created JB#52704 for this, and will poke Bea to investigate further 09:11
dcaliste Thank you, the solution is not easy in my opinion and relies a lot on server implementation, difficult to find a universal solution. But thanks. I'll keep an eye on the repo and will comment if I can give constructive opinion. 09:12
chriadam yeah, trying to support all different possible servers is ... hard ;-) but we'll do the best we can. thanks very much for bringing it to my attention 09:13
dcaliste No problem, the workaround to create separated accounts puting the *DAV path in the webDAV path to cheat the code highlight the issue but is a bit ugly... 09:14
chriadam yeah definitely shouldn't be required.
He agreed that the workaround is not a proper solution and opened an internal bug on the topic. The bug tracker and the repo it is dealing with are both closed in Jolla. So we cannot have direct access at how they will deal with the bug, but at least developpers know about it. We may ask for advances in a later community meeting for instance.
@dcaliste thanks for the IRC log extract, appreciated.
@chris.adams
If there is an alpha or beta package that needs testing with an actuive …DAV account, feel free to poke me via DM or email, I have an XA2 Ultra for testing now and since that is not my daily driver I’m happy to reflash it or upgrade packages as needed for a test (and then just flash it back to latest official image later)
I can test against Kolab Now and if need be fastmail too. I’m a customer at both, although currently I use fastmail only for email, but my account does include DAV access AFAIR.
FWIW, I tested with curl against server root/ webdav path/.well-known. there’s no file served but at least the output does differentiate between wrong PW and correct PW
with correct user ID and PW (obviously I tested with my ‘user@domain’);
$ curl -L --anyauth --user user@domain https://apps.kolabnow.com/.well-known
Enter host password for user '…':
<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
<s:sabredav-version>2.1.11</s:sabredav-version>
<s:exception>Sabre\DAV\Exception\NotFound</s:exception>
<s:message>File not found: .well-known in 'root'</s:message>
</d:error>
with correct user ID but wrong password (obviously I tested with my ‘user@domain’);
$ curl -L --anyauth --user user@domain https://apps.kolabnow.com/.well-known
Enter host password for user '…':
<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
<s:sabredav-version>2.1.11</s:sabredav-version>
<s:exception>Sabre\DAV\Exception\NotAuthenticated</s:exception>
<s:message>Username or password does not match</s:message>
</d:error>
Thank you @pcfe for the tests. Can you briefly run the three authentication cases you tried already but with -I option ? It’s to get the response headers, I’m wondering if the HTTP status value are coherent.
