[Guide] Fix certificate issues on SailfishOS

General
#1

This guide was primarily written for fixing root certificate issues on SailfishOS ≤ 4.0.1, specifically with the “DST Root CA X3” certificate distributed by Let’s Encrypt.

Note that this guide can be easily adapted, if similar issues with other root certificates occur.

BTW, chapter B of this guide also contains a reminder how to download RPMs from other SailfishOS releases than the installed one.

Documentation:

  1. Basic issue description: DST Root CA X3 Expiration (September 2021) - Let's Encrypt
  2. Detailed issue description: Certificate Compatibility - Let's Encrypt
    States “Jolla Sailfish OS > v1.1.2.16 (version that added ISRG Root X1 unknown)”, i.e. SailfishOS 1.1.2 which deployed ca-certificates-2014.2.1-1 containing CKBI 1.97 from NSS 3.16 does include the ISRG Root X1 certificate, but see 6.
  3. Technical details: Extending Android Device Compatibility for Let's Encrypt Certificates - Let's Encrypt
  4. Nice diagram depicting all this: Chain of Trust - Let's Encrypt
  5. Letsencrypt’s production-chain change-log: Production Chain Changes - API Announcements - Let's Encrypt Community Support
  6. Change-log of SailfishOS’ ca-certificates RPM: https://github.com/sailfishos/ca-certificates/blob/master/ca-certificates.changes
    Likely version 2020.2.41-1 fixes this, rolled out in SailfishOS 4.1.0, 4.2.0 and 4.3.0 (search for “ca-certificates” in change-log from [changelog] Kvarken 4.1.0 )
    or version 2021.2.50-3 is going to fix this, which supposedly is going to be deployed by SailfishOS 4.4.0.
    Side note: As SailfishOS 4.3.0 dropped OpenSSL 1.0.x and only provides OpenSSL 1.1.x (from 4.0.1 to 4.2.0 they were both deployed), it and any newer SailfishOS releases are not affected, regardless of this specific updated certificate bundle RPM.
  7. Just to remind where this is originally coming from: https://github.com/sailfishos/ca-certificates/blob/master/ca-certificates.spec#L42
    I.e., “upstream” for SailfishOS is RedHat’s Fedora distribution, which in turn retrieves and slightly adapts the bundle from Mozilla.

Two approaches to handle this on old SailfishOS releases:

A. Specific to this case: Blacklist the “DST Root CA X3” certificate

  1. Become root:
    devel-su
  2. Change into the blacklist directory:
    cd /etc/pki/ca-trust/source/blacklist
    2a. [Optional] Backup the target bundle:
    cp /etc/pki/tls/certs/ca-bundle.crt /tmp/ca-bundle.crt-backup
  3. Download the offending certificate:
    curl -kLo DST-RootCA-X3.pem https://crt.sh/?d=8395
    3b. [Alternative to 3 / Optional, when 3 worked well] Locally extract the offending certificate and compare it with the downloaded one.
    pkcon install openssl
    trust dump --filter "pkcs11:id=%c4%a7%b1%a4%7b%2c%71%fa%db%e1%4b%90%75%ff%c4%15%60%85%89%10" | openssl x509 > DST-Root-CA-X3.pem
    diff DST-RootCA-X3.pem DST-Root-CA-X3.pem
    If diff did output nothing (i.e., the two files have identical content), all is fine, thus delete one of them:
    rm DST-Root-CA-X3.pem
  4. Create a new target bundle:
    update-ca-trust extract
    4a. [Optional, only feasible when 2a was performed] Verify that the offending certificate is no longer in the target bundle:
    diff /tmp/ca-bundle.crt-backup /etc/pki/tls/certs/ca-bundle.crt
    cat DST-RootCA-X3.pem
    The content between the “Begin” and “End” lines of the diff and the cat output must be the same, except for the < -prefix of the diff output; a visual check is sufficient.

B. Download a recent CA-certificate bundle RPM for SailfishOS,

either per zypper (most elegant)
or on a device with a newer SailfishOS release installed (see Documentation section 5)
or by traversing Jolla’s RPM repositories.

  1. Become root:
    devel-su
  2. Install zypper:
    pkcon install zypper
  3. Add a newer SailfishOS release repository and name it arbitrarily:
    zypper ar -c https://releases.jolla.com/releases/3.4.0.24/jolla/armv7hl foo
    Syntax: zypper addrepo -c https://releases.jolla.com/releases/$releasever/jolla/$arch $name
  4. Test first! For up(date) & in(stall) always use -Dd (--dry-run --download) together (see man-page why).
    Simply let the resolver pick the highest version (because all repos have the same priority) or use --repo to select a specific one; note that the recommended option --from (over --repo) is not available in older zypper versions.
    zypper up -Dd ca-certificates
    Rather do only temporarily accept the repository signing key (because of step 6).
    Aforementioned alternatives, not advantageous here: #zypper up -Dd --repo foo ca-certificates # Better, if available: zypper up -Dd --from foo ca-certificates
  5. When you are sure what you want to do, do the real thing:
    zypper up ca-certificates
  6. Finally remove the temporarily added repository:
    zypper rr -v foo

or

  1. On the newer SailfishOS installation:
    devel-su pkcon download /tmp ca-certificates
    Choose “1”, the “noarch” RPM (not the “source” one).
  2. Copy the downloaded RPM to target machine:
    scp /tmp/ca-certificates-*.jolla.noarch.rpm nemo@<target-machine>:/tmp/
  3. On the target machine (use file name completion per key):
    devel-su pkcon install-local /tmp/ca-certificates-XXXX.Y.ZZ-ABC.jolla.noarch.rpm

or

  1. Prerequisite (optional): Determine the correct URL for the ca-certificates RPM (example for armv7hl, work the same way for aarch64 or i486, but this is a noarch RPM anyway):
    cd /tmp
    curl -kL https://releases.jolla.com/releases/4.3.0.12/jolla/armv7hl/repodata/291c3ba847217d4db39b8bd73baee748409d77f7f06a63518cd6e3527b002ad7-primary.xml.gz | gunzip > repodata_sailfishos-4.3.0.12-armv7hl.xml
    less repodata_sailfishos-4.3.0.12-armv7hl.xml
    Search in less per: /<name>ca-certificates
    Copy the URL part, which is specific for this RPM from the line: <location href="oss/noarch/ca-certificates-2020.2.41-1.4.1.jolla.noarch.rpm"/>
    Prepend the URL with https://releases.jolla.com/releases/4.3.0.12/jolla/armv7hl/ (rsp. aarch64 or i486) to determine the full download URL:
    https://releases.jolla.com/releases/4.3.0.12/jolla/armv7hl/oss/noarch/ca-certificates-2020.2.41-1.4.1.jolla.noarch.rpm
    Note that this is an academic example, because this ca-certificates RPM specifies dependencies (for the first time, i.e. no former version did) on p11-kit, p11-kit-nss-ckbi and p11-kit-trust, but all four packages together seem to be installable under e.g, SailfishOS 3.2.1, for example via the zypper method above. See also the edito below: *
  2. Download the ca-certificates RPM per curl:
    curl -kLO https://releases.jolla.com/releases/4.0.1.48/jolla/armv7hl/oss/noarch/ca-certificates-2018.2.26-1.2.1.jolla.noarch.rpm
    This is the most recent release of the ca-certificates RPM, which still installs flawlessly on SailfishOS 2.x & 3.x. Tested on SailfishOS 2.2.1 and 3.2.1, but the latter has exactly this RPM already installed by default (see pkcon search name ca-certificates).
  3. Install the downloaded RPM (use file name completion per <Tab> key):
    devel-su pkcon install-local ca-certificates-XXXX.Y.ZZ-ABC.jolla.noarch.rpm

Related threads at FSO


P.S.: Backup, text version and license


* Edit: p11-kit, p11-kit-nss-ckbi and p11-kit-trust, all at version 0.23.12, were already provided by SailfishOS 2.2.1 (and have not been updated at least up to SailfishOS 3.4.0), hence one may simply try installing the newest ca-certificates package per method B1.
If that fails due to some unmet dependency, on may use Show home:nephros:j1 / ca-certificates - SailfishOS Open Build Service

9 Likes
XA2 stuck on old 4.30.12 OS version: "SSL handshake failed"
[solved] How to update Jolla from 2.0.0.10 to 3.4?
SSL handshake failed
Storeman and side loading problems (also: Root CA X3 certificate expiration)
Q: Storeman complains about a not existing source - what can i do?
Adding my own root certificate to sailfish os
Which DOD releases should the SailfishOS-OBS support?
Email Update Error: Check Certificate
Let's Encrypt root certificate on SailfishOS 4.x
Jolla 1 with SailfishOS 2 questions
Instagram web don't work anymore in my jolla1
Jolla 1 with SailfishOS 2 questions
#2

Wow, thank you!

Another really well-written, detailed and knowledged guide by @olf.

I am not that deep into that cert stuff, normally do not need it and always forced to dig into it when something happens.
So I did not dare to look for the cert to put it into the blacklist (as it was not in pem format in the bundle).

One noob question I do have: why is my approach only almost? :wink:
I used the source bundle and ran update-ca-trust.
What is “wrong” with that?

1 Like
#3

I am not that deep into that cert stuff, normally do not need it and always forced to dig into it when something happens.

Neither was I WRT how certificates, their updates and manual intervention are practically handled on modern Linux distributions.
But a co-worker firmly stated, “do read the update-ca-trust man-page thoroughly and do not alter the source bundle”, so I went down that road.

So I did not dare to look for the cert to put it into the blacklist (as it was not in pem format in the bundle).

Well, openssl makes these conversions easy (see in the guide).

One noob question I do have: why is my approach only almost? :wink:
I used the source bundle and ran update-ca-trust.
What is “wrong” with that?

One shall not alter the source bundle!
It belongs to an RPM and any update of that RPM (ca-certificates) will overwrite your changes in the source bundle, plus its %post script will trigger update-ca-trust, which overwrites the changes in the target bundle: then your device is back at the start (WRT the certificate issues).
For example, all this will happen during the next SailfishOS upgrade; I would not be surprised, if it fails to finish, due to failing https connections.

Do read the update-ca-trust(8) man page, which additionally explains why manual adaptions by a sysadmin usually should be performed in the /etc/pki hierarchy (priority over /usr/share/pki etc.).

4 Likes
Storeman and side loading problems (also: Root CA X3 certificate expiration)
#4

Thanks again for explanation.

Then, from my perspective, I did it well enough :wink:
knowing my changes will be overwritten…

it was meant for a Jolla1, not getting any more updates
and for newer devices the removal of X3 was said (and with shipping of ssl1.1 the right cert is chosen anyway)

But again, your guide is appreciated to do it right the next time!

#5

Then, from my perspective, I did it well enough :wink:

You sure did, without this hint I would not have known what to ask and look for.

knowing my changes will be overwritten…
it was meant for a Jolla1, not getting any more updates

I needed it for an XperiaX@SFOS3.2.1, which I may update, when ultimately a newer SailfishOS release is published which does not contain flaws seriously impeding my usage.

and for newer devices the removal of X3 was said (and with shipping of ssl1.1 the right cert is chosen anyway)

???
As documented above, only SailfishOS 4.1 to 4.3 provide an updated ca-certificates RPM.
And all this is regardless of the device used.

2 Likes
#6

I have Verla 4.2 since today and the email cannot work, complaining about certificates… Which one is the correct way to fix the isue on Saifish 4.2 ?

#7

Cannot tell, because “the email cannot work, complaining about certificates…” does not include any relevant information.

  • Was it working fine before?
    If so, on which SailfishOS release?
  • Which program do you use for your email?
  • What is the exact error message (Copy & Paste or Screenshot)?

But walking through section A is fully and easily reversible (if you understand it), so you might as well just try.

#8

Email means standard native email. Version is 4.2.0.21. IIRC it stopped working with the previous version 4.1.x.x (I don’t remember which) some days ago. Mail app shows “check certificates” upon sync.

Which is the best approach (A or B) for 4.2.x.x OS ?

I tried approach A but it doesn’t fix the issues with my email provider; the account is an Exchange one. What makes the email accessible is to reset the password and allow untrusted certificates (I really don’t like that).

#9

IMHO none.

B. Download a recent CA-certificate bundle RPM for SailfishOS,

The most recent ca-certificate RPM Jolla currently provides is the one deployed on SailfishOS 4.1.0, 4.2.0 and 4.3.0.

And A fixes something which is already fixed in the recent ca-certificate RPM.

#10

Thus no fix available for now. Nice.

#11

That sounds as if “your email provider” has certificate issues, do not blame SailfishOS / Jolla, rsp. Fedora / RedHat rsp. Mozilla for this: Their certificate bundle is fine.

#12

What about installing openssl
and try to dig what goes wrong in cert chain with
openssl s_client -connect servername:portnumber

1 Like
#13

Very possible.
Thanks a lot

#14

Please mind, that you nowadays also need OpenSSL 1.1 for “the WWW to work”, hence on SailfishOS < 4.0.1 one must:

Install a newer OpenSSL package from here: OpenSSL 1.1.1 + 1.0.2 | OpenRepos.net — Community Repository System

If you are doing this manually (without Storeman), you also have to download the OpenSSL -libs RPM (omit the -devel RPM): openssl-libs 1.1.1 + 1.0.2 | OpenRepos.net — Community Repository System

When using the command line, you have to install these in one go (using the ARMv7 RPMs for e.g., a Jolla1 in this example):
pkcon install-local openssl-1.1.1kgit1-1.7.4.jolla_.armv7hl.rpm openssl-libs-1.1.1kgit1-1.7.4.jolla_.armv7hl.rpm

1 Like
#15

Minor updates to the original guide are at olf (Olf0) (Olf0) / Guide - Fix certificate issues on SailfishOS · GitLab
Because the idiotic default configuration of Discoure (which Jolla will not alter) makes all messages uneditable after a while, except for “wiki”-pages.

2 Likes
#16

Hey there, sailfish 3.2.1.9 running on an aquafish.

I tried the blacklist method and i even installed newer openssl (I couldn’t use zypper as zypper can’t download packages because of the certificate validity).

[root@Sailfish anchors]# rpm -qa | grep openssl
openssl-libs-1.1.1k+git1-1.7.4.jolla.armv7hl
openssl-1.1.1k+git1-1.7.4.jolla.armv7hl

I have verified and I don’t even have the DST Root within /etc/pki/tls/certs/ca-bundle.crt
Nothing works unfortunately! Any suggestions?

* TCP_NODELAY set
* Expire in 149994 ms for 3 (transfer 0xae9d30)
* Expire in 200 ms for 4 (transfer 0xae9d30)
* Connected to www.google.com (142.251.209.4) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, bad certificate (554):
* SSL certificate problem: certificate is not yet valid
* Closing connection 0
curl: (60) SSL certificate problem: certificate is not yet valid
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
#17

Try installing a newer certificate bundle as described under B; choosing the third method may be easiest.

#18

Can’t install anything newer than what I have because of the pk11 packages version constraint, do you know if I can pick them from 3.4.0 and install them on 3.2.19? As if they are supposed to work

#19

Sorry, I do not understand: Please provide the commands you used and their output for clarity.

#20

Hello i just found back my old jolla phone and try to give it a new life :wink:
It was factory reset to 1.0.0.5…i tried to install recent certificate 2021 but then stuck with p11-kit dependencies any suggestion ? thanks in advance

1 Like