Very good reporting and bug tracking work, I would say!
Probably you should add all of that up and create a case at https://jolla.zendesk.com/hc/en-us, so it’ll be tracked there.
And to check out which component to focus on you could also discuss this topic in the next community meeting i think: Community meeting on IRC 14th Jan 2021
Thank you for the detailed reporting.
If I understand properly, you cannot create a new generic CalDAV / CardDAV account from the UI, but if one already exists it is syncing properly. Restoring a backup is creating a new account so same problem.
Looking at the code, when the account is created, there is a check that is done that the user can authenticate on the server using a request at server address + webdav path. In your case, you didn’t provide a webdav path because the paths differ for CardDAV and CalDAV. Your server is not allowing to authenticate on the root address of the server, so the authentication is failing and the account creation fails. But it’s just the code logic for account creation that is faulty, if an account already exists it is addressing the complete paths directly for each service.
If all my blabla is correct, you should be able to create an account for CalDAV alone, puting the CalDAV path in the webDAV path in the UI.
If so, I’ll discuss with @chris.adams and @blam to see if we can adjust a bit the account creation for such cases (where servers reject authentication at root address).
1 Like
Not quite. Allow me to list the works and does not work points
- works I had working CalDAV and CardDAV syncing on all my SFOS 3.0 and 3.2 devices (not 100% sure about 2.0, while I have a few SOFO devices, I generally only have one on active user as daily driver and the others sit on a shelf)
- I can flash my phones and restore my user backup of my CalDAV and CardDAV settings(from µSD card) seemingly fine until SFOS 3.2.0.12 included.
- Both restoring a working backup and setting up a …DAV account from scratch on a freshly flashed SFOS 3.4.0.24 fails
- OTA updating a 3.2.0.12 with working CalDAV and CardDAV sync to 3.4.0.24 does work.
done.
Request #37446
I meant when filing a bug on bugs.merproject.org I should select a component, I think the buteo…sync one is the correct one but wanted to be sure.
Not selecting one is annoying to the triage people on a bugzilla, so it helps to set the correct one from the start.
As to community meeting, was going to say that I’ll try to attend, but 08:00 UTC falls right into the hour where I get ready for work, so no promises.
Thank you very much! That indeed did work.
So now I have 2 DAV accounts set up for Kolab Now. One for CalDAV and a second one for CardDAV.
Not perfect, but for me this is an acceptable workaround for this regression. Unsure how other Jolla customers feel about the work around, so not marking this topic as resolved/answered.
Basically I would even have been happy with instructions along the lines of edit this file with vi in a shell since without my address book any phone is totally useless to me and not having calendars severely reduces the phone’s viability as a daily driver.
Hopefully the zendesk ticket goes towards a fix for the regression.
I’m afraid that this bug tracker is no longer actively monitored. Jolla advertised to report bug on this forum instead.
I think, I understand the root of the issue, sadly it is located in closed source parts. You can look at /usr/share/accounts/ui/OnlineSyncAccountCreator.qml line 233. You can see that the authentication is done on server address / webdav path, which is not working for your server. The authentication check should be done on one of the service path instead I think. I’ll discuss on IRC with Chris Adams about the best way to actually solve this and be able to use only one account for both services.
Thanks @dcaliste, you saved me from wasting time on the wrong bugtracker.
For me, one of the hardest things in using SFOS is the perceived constant state of flux the servers I need to visit are in.
Crud, closed. 
But hey, there’s already an if nextcloud, so maybe adding an if kolabnow is not too much hassle for Jolla.
After poking around it a bit more, it’s also my understanding now that the issue is that authentication check should be done on one of the service path instead. I still fail to understand why I was fine to create a new CalDAV and CardDAV account on SFOS until and including 3.2, but since the component is closed source I cant easily git log OnlineSyncAccountCreator.qml on a local checkout.
Before I dig deeply into RFC 6764 , does the closed source component aim to support RFC 6764 and does Jolla prefer TXT[1] or .well-known[2]?
Being also on FreeNode, I wonder if there is a channel I should join? You should see me lurking on #mer, #meego and #maemo already 
pcfe
[1] this works
$ dig +short _autodiscover._tcp.kolabnow.com SRV
0 0 443 apps.kolabnow.com.
$ dig +short _caldavs._tcp.kolabnow.com SRV
0 1 443 apps.kolabnow.com.
(inspired by the DNS section of https://kb.kolabnow.com/documentation/dns-settings-for-group-managers )
but I saw no TXT record giving me a path for _caldavs._tcp (I might just have looked wrong, I do not query SRV and TXT that often)
Plus at Kolab Now the path is different per user and I am unclear how one would deal with that in a TXT record (guess this needs more in depth reading of 6764). I think authenticated .well-known might be better.
[2] quick test, but I might be doing that wrong (obviously I used the correct user@domain)
curl -L --anyauth --user user@domain https://apps.kolabnow.com/.well-known/caldav
Enter host password for user 'user@domain':
[...]
<p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>
[...]
If I used curl correctly, then I can try to ask the Kolab Now people if something is broken on their end, but if the closed component of SFOS does not even try authenticated .well-known and has no plans to do so in the future, then there is no point in me wasting time on that.
2 Likes
#sailfishos on freenode is the place to hang around as well 
1 Like
As mentioned above, I’m usually discussing with @chris.adams on #sailfishos. We have a periodic meeting every Tuesday at 8 a.m. UTC. Next one will be on January, 12th. You’re welcomed to join if the time slot is convenient for you. You can find logs of the channel if you are interested.
The closed source parts are using the .well-known capability to discover the user CalDAV path. But as far as I can see, this is done later on in the code path, after validation of the authentication on the server path. Which is failing in your case.
After further thinking on the matter, I’m not convinced personally that authenticating on one of the service path is a good idea neither, since this path may not be provided and only guessed later with the .well-known method. Which leaves the question on how to validate the provided authentication information (login and password) opened. That’s where it’s interesting to discuss with Jolla people before running to coding a solution. So let’s keep this topic opened for suggestions and brainstorming up to the 12th. Thanks by the way for the tests on the Kolabnow instance for the .well-known method.
That was sadly a waste of my time. Jolla support unilaterally closed the ticket with a “you seem to have a work around as per forum” even though I did ask for this to not be closed. 
I’ll finish dealing with this regression in the forums but I’ll keep support’s reaction in mind next time I run into a bug or regression.
Me too.
I’ve briefly discussed the issue with @chris.adams this morning as planned on IRC . Here is an extract from the logs:
dcaliste Since 3.4.0 changes in account creation, I think the login / password are checked on the root of the server before going further and retrieving calendar list... 09:03
dcaliste On some servers, the authentication on root is forbidden. 09:03
dcaliste So account creation is failing. 09:03
dcaliste Restoration from storage is failing also for the same reason I guess. 09:04
chriadam eek. 09:04
chriadam so we need to fallback to trying on ... either caldav calendars path, or carddav addressbooks path, or webdav home set path? 09:05
dcaliste Yes, that was my first guess, but there is an issue doing so : the caldav path may be guessed. 09:06
dcaliste So not available at that point. 09:06
chriadam what do you mean by "guessed"? do you mean, looked up via the .well-known endpoint, or something else? 09:06
dcaliste See the kolab issue, there is no webdav path, and caldav and carddav paths are different. 09:07
dcaliste Yes, looked up via .well-known. 09:07
dcaliste (well in the kolab report the .well-known seems to fail but that's another story, more related to server side I think). 09:08
dcaliste So the situation is : 09:09
dcaliste - server root / webdavpath may be forbidden and refused for authentication. 09:09
dcaliste - server root/ webdav path/.well-known may be the solution to try to authenticate 09:10
dcaliste - using the caldav path or the carddav one may not be possible, because not provided by the user at that point. 09:10
chriadam thanks. I've created JB#52704 for this, and will poke Bea to investigate further 09:11
dcaliste Thank you, the solution is not easy in my opinion and relies a lot on server implementation, difficult to find a universal solution. But thanks. I'll keep an eye on the repo and will comment if I can give constructive opinion. 09:12
chriadam yeah, trying to support all different possible servers is ... hard ;-) but we'll do the best we can. thanks very much for bringing it to my attention 09:13
dcaliste No problem, the workaround to create separated accounts puting the *DAV path in the webDAV path to cheat the code highlight the issue but is a bit ugly... 09:14
chriadam yeah definitely shouldn't be required.
He agreed that the workaround is not a proper solution and opened an internal bug on the topic. The bug tracker and the repo it is dealing with are both closed in Jolla. So we cannot have direct access at how they will deal with the bug, but at least developpers know about it. We may ask for advances in a later community meeting for instance.
@dcaliste thanks for the IRC log extract, appreciated.
@chris.adams
If there is an alpha or beta package that needs testing with an actuive …DAV account, feel free to poke me via DM or email, I have an XA2 Ultra for testing now and since that is not my daily driver I’m happy to reflash it or upgrade packages as needed for a test (and then just flash it back to latest official image later)
I can test against Kolab Now and if need be fastmail too. I’m a customer at both, although currently I use fastmail only for email, but my account does include DAV access AFAIR.
FWIW, I tested with curl against server root/ webdav path/.well-known. there’s no file served but at least the output does differentiate between wrong PW and correct PW
with correct user ID and PW (obviously I tested with my ‘user@domain’);
$ curl -L --anyauth --user user@domain https://apps.kolabnow.com/.well-known
Enter host password for user '…':
<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
<s:sabredav-version>2.1.11</s:sabredav-version>
<s:exception>Sabre\DAV\Exception\NotFound</s:exception>
<s:message>File not found: .well-known in 'root'</s:message>
</d:error>
with correct user ID but wrong password (obviously I tested with my ‘user@domain’);
$ curl -L --anyauth --user user@domain https://apps.kolabnow.com/.well-known
Enter host password for user '…':
<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
<s:sabredav-version>2.1.11</s:sabredav-version>
<s:exception>Sabre\DAV\Exception\NotAuthenticated</s:exception>
<s:message>Username or password does not match</s:message>
</d:error>
and for completeness, without auth;
$ curl -L https://apps.kolabnow.com/.well-known
<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
<s:sabredav-version>2.1.11</s:sabredav-version>
<s:exception>Sabre\DAV\Exception\NotAuthenticated</s:exception>
<s:message>No basic authentication headers were found</s:message>
</d:error>
Thank you @pcfe for the tests. Can you briefly run the three authentication cases you tried already but with -I option ? It’s to get the response headers, I’m wondering if the HTTP status value are coherent.
of course.
Like previously, all test were run with my ‘user@domain’.
correct password;
$ curl -I -L --anyauth --user user@domain https://apps.kolabnow.com/.well-known
Enter host password for user '…':
HTTP/1.1 401 Unauthorized
Date: Wed, 13 Jan 2021 20:53:43 GMT
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Security-Policy: connect-src 'self'; child-src 'self'; font-src 'self'; form-action 'self' data:; frame-ancestors 'self'; frame-src 'self'; img-src data: 'self'; media-src 'self'; object-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-eval' 'unsafe-inline'; default-src 'none'; reflected-xss block; referrer no-referrer;
WWW-Authenticate: Basic realm="KolabDAV"
Strict-Transport-Security: max-age=16070400; includeSubDomains
Content-Type: application/xml; charset=utf-8
HTTP/1.1 404 Not Found
Date: Wed, 13 Jan 2021 20:53:43 GMT
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Security-Policy: connect-src 'self'; child-src 'self'; font-src 'self'; form-action 'self' data:; frame-ancestors 'self'; frame-src 'self'; img-src data: 'self'; media-src 'self'; object-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-eval' 'unsafe-inline'; default-src 'none'; reflected-xss block; referrer no-referrer;
X-Sabre-Version: 2.1.11
Strict-Transport-Security: max-age=16070400; includeSubDomains
Content-Type: application/xml; charset=utf-8
wrong password;
$ curl -I -L --anyauth --user user@domain https://apps.kolabnow.com/.well-known
Enter host password for user '…':
HTTP/1.1 401 Unauthorized
Date: Wed, 13 Jan 2021 20:53:50 GMT
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Security-Policy: connect-src 'self'; child-src 'self'; font-src 'self'; form-action 'self' data:; frame-ancestors 'self'; frame-src 'self'; img-src data: 'self'; media-src 'self'; object-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-eval' 'unsafe-inline'; default-src 'none'; reflected-xss block; referrer no-referrer;
WWW-Authenticate: Basic realm="KolabDAV"
Strict-Transport-Security: max-age=16070400; includeSubDomains
Content-Type: application/xml; charset=utf-8
HTTP/1.1 401 Unauthorized
Date: Wed, 13 Jan 2021 20:53:50 GMT
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Security-Policy: connect-src 'self'; child-src 'self'; font-src 'self'; form-action 'self' data:; frame-ancestors 'self'; frame-src 'self'; img-src data: 'self'; media-src 'self'; object-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-eval' 'unsafe-inline'; default-src 'none'; reflected-xss block; referrer no-referrer;
WWW-Authenticate: Basic realm="KolabDAV"
Strict-Transport-Security: max-age=16070400; includeSubDomains
Content-Type: application/xml; charset=utf-8
not using any auth;
$ curl -I -L https://apps.kolabnow.com/.well-known
HTTP/1.1 401 Unauthorized
Date: Wed, 13 Jan 2021 20:54:28 GMT
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Security-Policy: connect-src 'self'; child-src 'self'; font-src 'self'; form-action 'self' data:; frame-ancestors 'self'; frame-src 'self'; img-src data: 'self'; media-src 'self'; object-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-eval' 'unsafe-inline'; default-src 'none'; reflected-xss block; referrer no-referrer;
WWW-Authenticate: Basic realm="KolabDAV"
Strict-Transport-Security: max-age=16070400; includeSubDomains
Content-Type: application/xml; charset=utf-8
Can you send me an email (to chris dot adams at jolla dot com) so I can send you a package for testing? I have implemented a workaround where the user can select a toggle to skip the initial authentication step if required, to avoid the case where we “guess” the auth path wrongly. This may or may not resolve the issue, as we still perform authenticated requests to determine e.g. the calendar paths for caldav a little bit later - but testing should tell us whether that’s the case or not.
2 Likes
Thanks to those who were able to help with testing the workaround - much appreciated.
I have also created a fix to automatically try an authenticated request to the .well-known endpoint if the request to the server root path fails - however I think some changes were made in that library which means that the patch may not apply cleanly to the 3.4.0 codebase. As such, I don’t have 3.4.0+patch packages for testing, for this fix, unfortunately.
Both the “manual workaround” (toggle switch to allow the user to disable authentication at that stage of the account creation flow) change, and the “try .well-known endpoint” change have missed the deadline for 4.0.1 content, so will be part of 4.1.0.
2 Likes
Hi, is it possible to get our hands on the 4.1 package, so we could use it with 4.0?
1 Like
Sorry, somehow I missed this reply. Short answer: you can email me for an experimental build of the packages, but no guarantees that it won’t break your device etc etc.
1 Like
Kvarken 4.1.0’s relnotes mark this as fixed. And it is working as expected on my SFOS 4.1.0 device (an Xperia 10 II)
To have it all in one place (all kudos to the original TJC poster, babba22, for the original instructions)
at Settings / Account / Add account / CalDAV and CardDAV
- Username → user@domain (e.g. george@kolabnow.com)
- Password → …
- Server address → https://apps.kolabnow.com
- Authenticate during account creation → OFF
- Address book path (optional) → /addressbooks/user@domain
- Calendar book path (optional) → /calendars/user@domain
side note: I was kinda expecting a bugbot to update all forum posts listed under Sailfish OS Forum and T.J.C: Issues reported by the community and fixed in this update with a FIXED CURRENTRELEASE 4.1.0 or similar, guess I’m spoiled