Migrating configuration and data files for sandboxed apps

vige · 3 November 2021 12:00

Hello fellow developers!

I’m sure many of you have read the blog post I wrote about sandboxing. Now that the Sailfish OS 4.3.0 has reached the Early Access stage, you can also move to the next stage with your app: Defining the profile for your app.

Like I mentioned in the blog, we used to have application configuration files in directories named in the format .config/harbour-appname. In the future we no longer want that - the configuration files should instead be located in directories with the format .config/OrganizationName/ApplicationName. So that’s where you should write your configurations from now on. But what about the old configuration files which were already written in .config/harbour-appname? Do we lose access to them? Fear not, that is not the case!

Starting with Sailfish OS 4.3.0, if the .config/harbour-appname directory exists, it will also be visible in the sandbox. So basically, this is what you should do in order to migrate your configuration:

If your configuration files exist in .config/OrganizationName/ApplicationName, use them
If not, check if they exist in .config/harbour-appname, and if they do, read them from there
Write configuration files to .config/OrganizationName/ApplicationName

You can use something like the following code to migrate your configuration from the old location to the new:

void MySettings::migrateSettings()
{
    // The new location of config file
    QSettings settings(QStandardPaths::writableLocation(QStandardPaths::ConfigLocation) + "/my.domain/MyApp/MyApp.conf", QSettings::NativeFormat);

    if (settings.contains("migrated"))
        return;

    QSettings oldSettings(QStandardPaths::writableLocation(QStandardPaths::ConfigLocation) + "/harbour-myapp/harbour-myapp.conf", QSettings::NativeFormat);

    for (const QString& key : oldSettings.childKeys())
        settings.setValue(key, oldSettings.value(key));

    settings.setValue("migrated", "true");
}

Similar migration strategy could be used also for .local/share and .cache directories - you may also want to consider just copying the files over.

olf · 3 November 2021 16:09

Dear @vige, you might clarify if a fresh installation of extant apps, which are not explicitly updated for the sandboxing yet, will still be able to create their directories ~/{.local/share,.config,.cache}/packagename and files in it on SailfishOS ≥ 4.3.0?

Currently you only address "configuration files which were already written in .config/harbour-appname", which supposedly means “before upgrading to SailfishOS 4.3.0”.

Rereading our conversation at the former thread about the topic “configuration and data files of apps with Jolla’s sandboxing” you did not say that all these directories will be creatable and writeable (just “mounted”):

Mind that many apps in the Jolla Store and at Openrepos are not well maintained or unmaintained, although still well working, and if they are not installable any more on recent SailfishOS releases, this will significantly reduce the already slim set of available apps for SailfishOS.
Hence backward compatibility is crucial.

vige · 4 November 2021 05:11

My understanding, which is based on empirical testing on current code (which hopefully becomes release 4.4.0), is that the sailjail implementation creates the harbour-packagename directories for the applications which have not specified their own profiles. For the applications which have specified their profiles, the directories are not created, but are still mounted if they exist.

olf · 4 November 2021 05:33

Thank you very much, that is exactly the clarification / unambiguous wording I wanted to see documented.
Plus WRT its content it is exactly what I hoped for (but became seriously wary).

And as you are the chief designer of SailJail AFAIU, I assume that this also is the intended behaviour, even if the implementation may deviate, right?

P.S.: I am really glad that this is fine on this abstract level, for the more specific / detailed questions and concerns, see the other, older thread.

vige · 4 November 2021 05:45

I’m not the chief designer of sailjail. Your assumption about the intended behaviour is still valid though.

nephros · 5 November 2021 18:29

Thank you for the code snippet for migration.

Couple of follow-up questions:

Is there a similar “Best Practice” solution for pure QML apps (or QML+Python or other non-Qt/QSettings applications), especially those using JS LocalStorage?

So, if my app currently does something like:

invoker --type=silica-qt5 sailfish-qml my-appname

to launch, which changes are needed apart from making it

sailjail -p my-appname.desktop /usr/bin/sailfish-qml my-appname

E.g. is it required/necessary/recommended/forbidden to do something like

ApplicationWindow {
  Component.onCompleted: {
    Qt.application.name         = "appname";
    Qt.application.organization = "my-orga";
  }
[...]
}

… and if yes, how do Qt.application.XXX values map to the ones in the .desktop files.

In my tests (4.2) sailjail frequently did not accept the .desktop appname as valid. What are the rules for names for application and organisation?

specifically it seems for

[X-Sailjail]
ApplicationName=foo

foo is not allowed to contain a hyphen?

vige · 8 November 2021 10:41

No there isn’t. For the time being, my recommendation for the old pure QML apps is to not define their own profiles, i.e. keep using the old locations. For new apps I’d recommend defining the profiles and using the new locations.

I think the answer to all of these is “no”.

As long as you are using libsailfishapp, the values are set automatically, using the values from the .desktop files.

For ApplicationName: Allowed characters are A-Z, a-z, 0-9, underscore (_) and hyphen(-). The name may not start with a number.
For OrganizationName: Same characters as ApplicationName, and a dot (.), which is used for separating components. Any component may not start with a number. There are some names which are not allowed. Currently the list of disallowed OrganizationNames contains only “com.jolla” and “org.sailfishos”.

nephros · 8 November 2021 11:10

Excellent. Thank you.

JulienBlanc · 10 November 2021 08:08

A funny thing is that i made the exact opposite change to comply to harbour policy when i first published the app to jolla store ( https://github.com/Julien-Blanc-tgcm/kontroller/commit/d9007befef4aff4948820e1e9917ede0b3c621d9 )

My understanding is that the appname no longer needs to be harbour- prefixed. Is that correct ? Or is it just for the settings part ?

vige · 10 November 2021 09:44

The ApplicationName in the X-Sailjail section of the .desktop file does not have to be harbour-prefixed. This in turn means that the directory names in ~/{.local/share,.config,.cache} also no longer need to have the harbour- prefix. For the RPM package name and the binary name the old requirements still apply.

nephros · 10 November 2021 11:24

Yea, and I distinctly remember there was a similar migration a long time ago when all apps were moved out of their “orga name” subdirectories. Was that early SailfishOS? Or even Meego Harmattan times? I forget.

attah · 1 December 2021 19:05

If you, like me, have been using QML LocalStorage instead of Settings, @vige’s nice example isn’t applicable right off the bat, so here is my poor attempt at doing the same, but for LocalStorage:

void migrateLocalStorage()
{
    // The new location of the LocalStorage database
    QDir newDbDir(QStandardPaths::writableLocation(QStandardPaths::AppLocalDataLocation) + "/my.domain/MyApp/QML/OfflineStorage/Databases/");

    if(newDbDir.exists())
        return;

    newDbDir.mkpath(newDbDir.path());

    QString dbname = QString(QCryptographicHash::hash(("MyAppDB"), QCryptographicHash::Md5).toHex());

    // The old LocalStorage database
    QFile oldDb(QStandardPaths::writableLocation(QStandardPaths::AppLocalDataLocation) + "/harbour-myapp/harbour-MyApp/QML/OfflineStorage/Databases/" + dbname + ".sqlite");
    QFile oldIni(QStandardPaths::writableLocation(QStandardPaths::AppLocalDataLocation) + "/harbour-myapp/harbour-MyApp/QML/OfflineStorage/Databases/" + dbname + ".ini");

    oldDb.copy(QStandardPaths::writableLocation(QStandardPaths::AppLocalDataLocation) + "/my.domain/MyApp/QML/OfflineStorage/Databases/" + dbname + ".sqlite");
    oldIni.copy(QStandardPaths::writableLocation(QStandardPaths::AppLocalDataLocation) + "/my.domain/MyApp/QML/OfflineStorage/Databases/" + dbname + ".ini");
}

Edit: simplified with QFile.copy()

vige · 2 December 2021 05:09

Why don’t you just use QFile::copy()?

attah · 2 December 2021 06:45

Good point. Qt inexperience plain and simple. I’ll look at amending the example.

poetaster · 10 December 2021 13:14

What’s the proper way to set the properties for QML?

QScopedPointer<QQuickView> view(SailfishApp::createView());
view->rootContext()->setContextProperty("appName", appName);

Seems wrong. But I haven’t found it in the documentation yet.

EDIT: @karry has:

app->setOrganizationDomain(“osmscout.karry.cz”);
app->setOrganizationName(“cz.karry.osmscout”); // needed for Sailjail
app->setApplicationName(“OSMScout”);

poetaster · 10 December 2021 13:25

I’ve managed using attah’s approach to migrate the data and it seems the naming of takes. But while debugging with the QML LiveBench I don’t appear to have access to the new data. I kept the app name for the time being, but:

.pragma library
.import QtQuick.LocalStorage 2.0 as LS
…
var db = LS.LocalStorage.openDatabaseSync(“harbour-dwd”, “1.0”, “DWD Location Cache”, 10000);

Does not work. Do I need to adapt opening the DB in js? No, this is not completely true. I do have read accesss.

EDIT: I’m wrong. All is fine. Sorry for the white noise. For a fully functional version of Attah’s suggestion:

github.com

poetaster/harbour-dwd/blob/main/src/harbour-dwd.cpp

/*
  Copyright (C) 2021 Mark Washeim
  Contact: Mark Washeim <blueprint@poetaster.de>

  You may use this file under the terms of GPLv3
*/

#ifdef QT_QML_DEBUG
#include <QtQuick>
#endif

#include <QDir>
#include <QFile>
#include <QString>
#include <QStandardPaths>
#include <QCryptographicHash>
#include <sailfishapp.h>


void migrateLocalStorage()

This file has been truncated. show original

JulienBlanc · 20 December 2021 12:35

I’ve started playing a bit with sandboxing, and ran into the following issue.

copying from the old configuration directory works fine
deleting the content inside the old configuration directory works fine
deleting the old configuration directory itself does not work if the app is sandboxed (works fine if it is not)

So, to offer a smooth migration path (that correctly cleans up any remnants of the old scheme), i must first release a version without sandboxing, that will do the migration, and then a new version that enables sandboxing. Is this correct or am i missing something ? (or is it just expected that the old directory stays there, after all it is not deleted when we uninstall apps…).

poetaster · 20 December 2021 15:20

Yup. that’s my experience. I hadn’t got there, but I was thinking of trying to move rather than copy and keep a tmp directory with a backup in tld.orgname/tmp. then the worst case is old dangling directories, but no excessive data use.
EDIT: If I’m not mistaken, bool QFile::rename(const QString &newName) instead of copy should do?

EDIT 2: So, just did a test. rename is in fact mv. But I only tested it as above, doing rename instead of copy. That leaves the old folders in place.

EDIT 3: So, for really elegant migration handling @karry :

github.com

Karry/osmscout-sailfish/blob/master/src/Migration.cpp

/*
  OSMScout for SFOS
  Copyright (C) 2021  Lukáš Karas

  This program is free software; you can redistribute it and/or modify
  it under the terms of the GNU General Public License as published by
  the Free Software Foundation; either version 2 of the License, or
  (at your option) any later version.

  This program is distributed in the hope that it will be useful,
  but WITHOUT ANY WARRANTY; without even the implied warranty of
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  GNU General Public License for more details.

  You should have received a copy of the GNU General Public License
  along with this program; if not, write to the Free Software
  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
*/

#include "Migration.h"

This file has been truncated. show original

JulienBlanc · 21 December 2021 05:49

Another option to migrate: https://github.com/Julien-Blanc-tgcm/kontroller/commit/8d1238bd7e48cfb466690eccc201dca61ae57769

Something to be careful of is that, from my experience, the settings directory is automatically created when running through sailjail (my guess is that, to be bound to the jail container, it needs to exists). So, checking against its existence is not something to do if the app is run under sailjail, it will always exists.

poetaster · 21 December 2021 17:19

On the otherhand localstorage and .local/share/appname are not automatically created. But this is also a nice approach! Thanks!

EDIT: It just occured to me, if it’s necessary to bind to the user config dirs, shouldn’t that logic apply to local storage. Sounds like a security issue. Sigh.

EDIT2: Julien’s code not from the commit but in a block:

		QDir dir(QStandardPaths::writableLocation(QStandardPaths::ConfigLocation));
		if (dir.exists("harbour-kontroller"))
		{
			dir.mkpath("tgcm.eu/Kontroller");
			if (dir.exists("harbour-kontroller/kontroller.conf"))
			{
				dir.rename("harbour-kontroller/kontroller.conf", "tgcm.eu/Kontroller/Kontroller.conf");
			}
			dir.rename("harbour-kontroller", "tgcm.eu/Kontroller/old");
		}