I don’t know. The first time I came across it was when the release notes of MLS Manager mentioned it. (@abranson might have some more information?)
I think it’s main purpose is to move “confidential” but system-wide data (like e.g. wifi passwords) from plain old unencrypted /var into an encrypted location.
Hence the existence of
/home/.system/var/lib/connman-vpn
/home/.system/var/lib/sailjail/settings
/home/.system/var/lib/connman
and similar.
Thanks! It’s actually a reasonable solution, you are correct.
It turns out that I have to re-write some of the data loading code in Stellarium anyway. Sigh. Or I make symlinks from /home/.system/usr/share/stellarium → /usr/share/stellarium. Hmmm. I just tested this route (manually moving and symlinking) and it works fine.
If if weren’t such a lot of data, I’d already have implemented loading on first run, but it’s one of those finding a balance between what is sufficient for the first run is, well, tricky. It’s the stars, after all!
Yes that’s about right. /home/.system is a place where system-wide data is kept on the home partition, without becoming inaccessible when another user logs in.
Btw, the addition of the new compression format for rpms is a reproducible builds thing. I think the old one wasn’t always producing the same result, which meant that builds were being triggered when they shouldn’t be, and deltarpm didn’t work very well.
To make it clear, rpms created with the 4.5.0.16 toolchain can no longer be installed on e.g. a Jolla 1 phone, because the rpm version in SFOS 4.3 cannot handle zstd compression it seems.
@nephros, do I understand correctly that you are suggesting to put
%define _binary_payload w2.xzdio
in spec files to set the compression to xz -2 (as it might have been before)?
Notes:
xz -2 is slow on decompression, but better than bzip2 -9 (w9.bzdio).gzip by %define _binary_payload w9.gzdio (= gzip -9): Slightly worse compression ratio, similar compression speed, but much faster decompression. OTOH, SailfishOS 2.2.1 provides xz / liblzma 5.0.4 (released 22 June 2012), so I assume it has been in SailfishOS since its inception (first releases 0.99.x.y in November 2013).xz's default compression level 6 is very slow, memory- and CPU-intensive for both compression and decompression, and its levels {7,8,9} extremely so (even more than zstd's “ultra”-levels {20,21,22}). See xz's man-page, sections “Memory usage” and “-0 … -9” for details (!).%define _source_payload w2.xzdio or something similar (as often suggested), because “SRPM payload compression should stay at gzip (there’s almost no benefit in changing the compression, because SRPM’s contents is compressed already)”. This also implies that any %define _<*>_payload w<?>.gzdio continues to work fine (although only 9 makes sense for substituting <?>).References:
xz -2 to zstd -19: Changes/Switch RPMs to zstd compression - Fedora Project Wiki
rpm -q --qf "%{PAYLOADCOMPRESSOR} %{PAYLOADFLAGS}\n" <package>
zstd -15 would have been slightly more desirable: Much faster compression speed, faster decompression, only slightly worse compression ratio; this can be achieved by setting %define _binary_payload w15.zstdio in a spec file for SailfishOS ≥ 4.4.0 (requires the corresponding SDK release for building).zstd / libzstd is not available for Fedora < 28, RHEL < 8.2, *buntu < 18.10 etc.; I assume it has been introduced into SailfishOS only recently (v4.4.0?). BTW, Linux integrates zstd for kernel space users (no, this does not address persons 
) since 4.14 (November 2017).xz -2 or bzip2 -9 compresses only slightly better than gz -9 at roughly the same compression speed but much slower decompression; but gzip's default 6 is compressing much quicker and usually yields the same compression ratio: Now I remember why I stubbornly kept using tar -cvzf <tarball> and never switched to anything else, also for compatibility’s sake. This is also the reason why Debian 9 / Ubuntu 18.04 started using xz / lzma compression for packages and used gzip prior to that. OpenWRT and other embedded Linux distributions continue to use gzip compression for their packages due to its lightness (WRT CPU, memory and decompression speed).Can someone please try to install an RPM package built with the Sailfish-SDK for 4.5.0 (or the sailfishos-4.5.0 DoD repository at the SailfishOS-OBS) on SailfishOS ≤ 4.3.0, to ensure that nothing else changed. It must emit at least rpmlib(PayloadIsZst) is needed by <package name>-<version>-<release>.<arch>, but any other rpmlib(…) is needed by <…> would be something to also take into account.
Background: With the switch from w9.gzdio to w2.xzdio in RHEL 5→6, also the filedigest was altered from md5 (to sha1, IIRC), see here (it is historic as for RHEL 5→6, the write-up is erratic, the suggestions flawed, it is just to provide a reference and evidence what happened).
Did a few tests with both, “noarch” and true binary RPM files and SRPM files (….src.rpm) build by rpmbuild and checked by rpm -q --qf "%{PAYLOADCOMPRESSOR} %{PAYLOADFLAGS}\n" <package>
| Built on / with |
binary RPM file | noarch RPM file | SRPM file |
|---|---|---|---|
SFOS 2.2.1rpmbuild
|
xz 6 | gzip 9 | |
| SFOS 3.1.0 SFOS-OBS |
xz 6 | xz 6 | |
SFOS 3.2.1rpmbuild
|
gzip 9 | gzip 9 | |
| SFOS 3.2.1 SFOS-OBS |
xz 6T | xz 6T | |
| SFOS 3.4.0 SFOS-OBS |
xz 6T | xz 6T | |
| SFOS 3.4.0 Sailfish‑SDK |
xz 6T | xz 6T | |
| SFOS 4.0.1 Sailfish‑SDK |
xz 6T | xz 6T | |
| SFOS 4.4.0 SFOS-OBS |
xz 6T | xz 6T | |
| SFOS 4.4.0 Sailfish‑SDK |
xz 6T | xz 6T | |
| SFOS 4.5.0 SFOS-OBS |
zstd 19 | zstd 19 |
Contribution are welcome, specifically information about binary RPMs built locally (i.e., on a SailfishOS device) by rpmbuild.
No I’m not suggesting anything, but the information from @adekker above is important for packagers who provide packges for older SFOS versions.
I guess setting the compression scheme back to the previous one is a feasible approach for such packages.
Well, after researching the topic RPM compression much more in-depth than originally intended, I can come up a conclusive suggestion.
but the information from @adekker above is important for packagers who provide packges for older SFOS versions.
I do not see that anybody questioned that, rather the opposite: Jolla’s and @dekker’s information, plus your “non-suggestion” made me research and document this.
I guess setting the compression scheme back to the previous one is a feasible approach for such packages.
Well, “the previous one” is not really conclusive (see my prior post), and I do think that RedHat’s / Fedora’s former choice of xz -2 -T (= %define _binary_payload w2T.xzdio in the spec file) is a better fit for large RPM files (> 100 kbytes), especially for mobile devices. Only for multi-megabyte RPMs I would choose (at most) xz -6 -T (= %define _binary_payload w6T.xzdio), but do understand that Jolla had chosen this as a “one size fits all” setting, because it covers the worst cases well. Note that multithreading for the option -T was introduced in xz 5.2.0, hence it cannot be used for RPMs which may be installed on SailfishOS < 3.2.0!
For small RPMs (< 100 kbytes) and all SRPMs, gzip -9 (= %define _binary_payload w9.gzdio and %define _source_payload w9.gzdio; the latter seems to be the default on most Linux distributions including SailfishOS) is absolutely fine, even its default compression level 6 is almost equivalent (and compresses much faster) and in some cases even 1 (but not 0, because it means “no compression”).
P.S.: On SailfishOS ≥ 4.5.0 Jolla exactly followed RedHat’s / Fedora’s choice of zstd's compression level 19, which is a reasonable as a general default setting, because it covers the worst cases well. I would add (in line with Jolla’s former choice for xz) the parameter -T (= %define _binary_payload w19T.zstdio), which defaults to 0 if -T is set, and to 1 if not (see man-page). For RPMs < 1 Mbyte zstd -10 -T (= %define _binary_payload w10T.zstdio; at most 15) appears to be a better choice. Note that zstd's default at the command line is a compression level parameter of merely 3, but that was likely chosen because of its much faster compression speed, which is much nicer for interactive use (though the compression ratio is lower and decompression slightly slower).
But as the switch to zstd with the backward compatibility issues it causes was the principal point here, xz and / or gzip are the proper choices to obtain backward compatibility.
I just wanted to boost this thread again, since for those of us keeping track of installations < 4.4 it’s necessary. Unless we want to stop supporting everything but latest greatest.
You may find your packages being available in harbour but failing to install if you do not specify. It seems that 4.5 + 3.10 SDK yields rpms that won’t install on older SFOS.
Yes.
Ultimately I settled on either adding …
%define _binary_payload w2.xzdio- _binary_payload;w2.xzdioMacros: section of its yaml fileHello! I am trying to “zypper up” my sfos running on scorpion-windy within 3.4.0.24, but cannot proceed because many packages are affected by seemingly zstd conpression,
( 1/31) Installing: droid-hal-version-scorpion_windy-0.0.1+master.20180915104950.5c60ac8-1.11.10.jolla.armv7hl ..........................................................................................................................................................[error]
Installation of droid-hal-version-scorpion_windy-0.0.1+master.20180915104950.5c60ac8-1.11.10.jolla.armv7hl failed:
Error: Subprocess failed. Error: RPM failed: error: unpacking of archive failed: cpio: Bad magic
error: droid-hal-version-scorpion_windy-0.0.1+master.20180915104950.5c60ac8-1.11.10.jolla.armv7hl: install failed
error: droid-hal-version-scorpion_windy-0.0.1+master.20180915104950.5c60ac8-1.11.9.jolla.armv7hl: erase skipped
Abort, retry, ignore? [a/r/i] (a):
[nemo@sfos-scorpion-windy ~]$ rpm -q --qf "%{PAYLOADCOMPRESSOR} %{PAYLOADFLAGS}\n" /var/cache/pk-zypp-cache/packages/adaptation-community/armv7hl/droid-hal-version-scorpion_windy-0.0.1+master.20180915104950.5c60ac8-1.11.10.jolla.armv7hl.rpm
zstd 19
and even if I update rpm to 4.16.1.3 (which should have supported zstd) could not make these packages installable:
[nemo@sfos-scorpion-windy ~]$ LANG=en_US.UTF-8 zypper lu
Loading repository data...
Reading installed packages...
S | Repository | Name | Current Version | Available Version | Arch
--+-----------------------------+------------------------------------------------------+---------------------------------------------------------+---------------------------------------------------------+--------
v | adaptation-community-common | bluetooth-rfkill-event-hciattach | 1.0.9-1.2.8.jolla | 1.0.9-1.2.9.jolla | armv7hl
v | adaptation-community-common | community-adaptation-devel | 0.0.3-1.9.6.jolla | 0.0.3-1.9.7.jolla | armv7hl
v | adaptation-community | droid-hal-version-scorpion_windy | 0.0.1+master.20180915104950.5c60ac8-1.11.9.jolla | 0.0.1+master.20180915104950.5c60ac8-1.11.10.jolla | armv7hl
v | adaptation-community | geoclue-provider-hybris-hal | 0.2.22-1.5.17.jolla | 0.2.22-1.5.18.jolla | armv7hl
v | native-common | gpsd | 3.23+git2-1.8.5.jolla | 3.23+git2-1.8.6.jolla | armv7hl
v | native-common | gstreamer1.0-vaapi | 1.18.5-1.6.1.jolla | 1.18.5-1.6.3.jolla | armv7hl
v | adaptation-community | hybris-libsensorfw-qt5-hal | 0.11.4-1.6.11.jolla | 0.11.4-1.6.12.jolla | armv7hl
v | native-common | kernel-headers | 5.4.23-1.2.11.jolla | 5.4.23-1.2.12.jolla | armv7hl
v | native-common | libdrm | 2.4.110+git1-1.7.4.jolla | 2.4.110+git1-1.7.5.jolla | armv7hl
v | native-common | libgps | 3.23+git2-1.8.5.jolla | 3.23+git2-1.8.6.jolla | armv7hl
v | adaptation-community | libhybris | 0.0.5.33.tama.1-1.6.8.jolla | 0.0.5.33.tama.1-1.6.9.jolla | armv7hl
v | adaptation-community | libhybris-libEGL | 0.0.5.33.tama.1-1.6.8.jolla | 0.0.5.33.tama.1-1.6.9.jolla | armv7hl
v | adaptation-community | libhybris-libGLESv2 | 0.0.5.33.tama.1-1.6.8.jolla | 0.0.5.33.tama.1-1.6.9.jolla | armv7hl
v | adaptation-community | libhybris-libOpenCL | 0.0.5.33.tama.1-1.6.8.jolla | 0.0.5.33.tama.1-1.6.9.jolla | armv7hl
v | adaptation-community | libhybris-libhardware | 0.0.5.33.tama.1-1.6.8.jolla | 0.0.5.33.tama.1-1.6.9.jolla | armv7hl
v | adaptation-community | libhybris-libnfc | 0.0.5.33.tama.1-1.6.8.jolla | 0.0.5.33.tama.1-1.6.9.jolla | armv7hl
v | adaptation-community | libhybris-libsync | 0.0.5.33.tama.1-1.6.8.jolla | 0.0.5.33.tama.1-1.6.9.jolla | armv7hl
v | adaptation-community | libhybris-libvibrator | 0.0.5.33.tama.1-1.6.8.jolla | 0.0.5.33.tama.1-1.6.9.jolla | armv7hl
v | adaptation-community | libhybris-libwayland-egl | 0.0.5.33.tama.1-1.6.8.jolla | 0.0.5.33.tama.1-1.6.9.jolla | armv7hl
v | adaptation-community | libhybris-tests | 0.0.5.33.tama.1-1.6.8.jolla | 0.0.5.33.tama.1-1.6.9.jolla | armv7hl
v | native-common | libva | 2.6.1+master.20200301232537.48b1b74-1.5.35.jolla | 2.6.1+master.20200301232537.48b1b74-1.5.37.jolla | armv7hl
v | adaptation-community-common | mce-plugin-libhybris | 1.14.1-1.5.10.jolla | 1.14.5-1.3.1.jolla | armv7hl
v | native-common | qt5-qtdeclarative-import-multimedia | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common | qt5-qtmultimedia | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common | qt5-qtmultimedia-gsttools | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common | qt5-qtmultimedia-plugin-audio-pulseaudio | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common | qt5-qtmultimedia-plugin-mediaservice-gstcamerabin | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common | qt5-qtmultimedia-plugin-mediaservice-gstmediacapture | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common | qt5-qtmultimedia-plugin-mediaservice-gstmediaplayer | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common | qt5-qtmultimedia-plugin-resourcepolicy-resourceqt | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common | qtscenegraph-adaptation | 0.7.6-1.3.18.jolla | 0.7.6-1.3.20.jolla | armv7hl
You can see that many shared repositories are affected. Could you have packages in these repositories rebuilt to a compatible format, or is there a way to update rpm and related tools to make zstd-compressed packages installable on old systems?
Which port ist that? Sounds like a question for the maintainer, but I’m really uncertain about this since I’ve only had the issue appear with 4.5 and I only deal with the apps I maintain as advised by @olf
@olf, are you aware of an version rpm (pkcon, too?) which might help resolve this?
Thanks. Upgrade complete by upgrading rpm and rpm-libs to 4.16.1.3+git8-1.10.1 first, as well as related packages. In detail:
$ sudo ssu re 4.4.0.58
$ sudo zypper ref
$ sudo zypper up rpm rpm-libs store-client patchmanager deltarpm zypper sailfish-installationhandler
$ sudo ssu re 3.4.0.24
$ sudo zypper ref
$ sudo zypper up
You are skipping about a gazillion stop releases… no wonder it didn’t work for you.
But sadly the system became broken after upgrading. The unlock screen cannot kick in after reboot.
I have to restore the system from a backup made with twrp recovery.
But with packages within common repositories built with incompatible conpression, it is even impossible to update packages under the same release.
I don’t follow what you are saying… Do you understand what i mean by “stop release”?
For the rest of us we got the prerequisite package in 4.X and only in 4.X+1 was the new compression actually used.
The Issue I met is not related to “stop release”, but what poetaster assumes occurs on packages within common repositories.
But it is, you’re trying to skip a release that introduces new compression method, skipping a stop release doesn’t work