Harbour-defender blacklisted in SFOS 3.4.0.24

[peterleinchen]

Definitely not a systemd specialist here.
But as far as I understand:

Often, it is a better choice to use Wants= instead of Requires= in order to achieve a system that is more robust when dealing with failing services.

from
https://www.freedesktop.org/software/systemd/man/systemd.unit.html
or

Systemd: Requires vs wants

Wants/WantedBy is the soft dependency,
which may fail.
And Requires/RequiredBy is the hard deoendency,
which let the higher service fail if not all dependencies could get started.

So again does this sound like a failing service should block booting?
But I do not have any better idea.

[peterleinchen]

So. I invested some time and solved this issue. See here:

My very own 'Sailfish Devember' - Defender

For some reason I post it here. Defender was (and is) a very nice and handy tool. And it eases ad blocking and adds some privacy measures. All kudos to nodevel! Unfortunately it is not maintained anymore and even worse it caused encrypted devices to stop booting (it could be resolved by using recovery image and deinstalling ‘rpm -e harbour-defender’ (or editing path unit manually). I could not (and still cannot) understand that systemd can be so bold to ‘support’ such situation causing full b…

I would have liked to get some pointers earlier from Jolla instead of a blacklisting (which is not needed on unencrypted devices including all Jolla 1, C, Tablets). As I think someone more experienced in systemd would have seen this immediately.

Many thanks to DrYak for this bug report here and confirming my thoughts and needed code changes.

[DrYak]

First: BIG THANK YOU for investigating and fixing the bug!

I had indeed confused the hard/soft requirement, but the final result is the same.

Just for the tiny details:

Wants/WantedBy is the soft dependency,
which may fail.

• Yes, it tolerates a fail (if the units fails, it will not break, but continue starting).
• But it will always initiate a unit start before, no matter what.
• And is will always wait for unit completion
  • One-shot type service units will wait for script’s exit
  • (modern, sysd-aware) daemons service units will wait for successful backgrounding (there’s a systemd dbus signal for that)
  • old-school (SysVInit) daemons service units will wait for the typical double-fork
  • things which are not technically daemon but scripts service units (a.k.a the I wrote a Perl 1-liner and don’t bother with all the daemony stuff) will wait a timeout (i.e.: the script needs to survive without crashing for some time for it be considered a successful start)
  • and path-triggers units require the path to be available (seems logical).

So even if it is a “WantedBy” (so if an actual script did crash booting would have continued), we still end up in a waiting loop, where multi-user waits for the (path trigger) unit to be ready, but the path trigger depends on a mount path (/home) that isn’t mounted yet and thus waits, but the that mount path will never become available before the encryption has been unlocked which is set to happen much later in the boot process order, and that depends on the multi-user target having been already reached (Whereas it’s currently still stuck waiting).

circular dependency hell.

The way you solved it (making sure that the path trigger unit is loaded ONLY AFTER the point at which the GUI is ready and the encryption has been unlocked and mounted) is nice.

There’s a tiny catch in your solution, but not dramatic:

• if the main (admin) user is called “default user” but there’s a secondary (non admin) user created which happens to be called “nemo”, that user gets to do the updates too, even if apparently, according to Jolla’s design only UID 100000 should be able to.

  (but that’s a problem that is shared with any other app which predates the username switch from nemo to default)

• if the users has renamed the admin user to something which is neither defaultuser nor nemo (i.e.: non standard), this line will fail. Best would be to use whatever is the pythonic equivalent of the bash getent passwd 100000 and get the correct path from column 6)

  (but if a user goes to play around with usernames and paths, they should are advanced power users enough to fix this too).

[peterleinchen]

Thanks to you, DrYak, again!

Maybe this is what I could not read out of any information I read about systemd:

@DrYak

And it will always wait for unit completion

• One-shot type service units will wait for script’s exit
• (modern, sysd-aware) daemons service units will wait for successful backgrounding (there’s a systemd dbus signal for that)
• old-school (SysVInit) daemons service units will wait for the typical double-fork
• things which are not technically daemon but scripts service units (a.k.a the I wrote a Perl 1-liner and don’t bother with all the daemony stuff ) will wait a timeout (i.e.: the script needs to survive without crashing for some time for it be considered a successful start)
• and path-triggers units require the path to be available (seems logical).

This makes it more or less clear. And I would have started earlier…
Any pointer to that? (please, even it is now obsolete ) As I would have expected that unit to dangle around but not stop multi-user.target to finish.

About the tiny catches:
I guess I did it to my best knowledge:

@DrYak

• if the main (admin) user is called “default user” but there’s a secondary (non admin) user created which happens to be called “nemo”, that user gets to do the updates too, even if apparently, according to Jolla’s design only UID 100000 should be able to.

This is handled as the existence of /home/defaultuser is checked first. Only if this does not exist it is falling back to old (beloved ) nemo.

@DrYak

if the users has renamed the admin user to something which is neither defaultuser nor nemo (i.e.: non standard), this line will fail. Best would be to use whatever is the pythonic equivalent of the bash getent passwd 100000 and get the correct path from column 6)

This is something that cannot happen as the user names “defaultuser” resp. “nemo” are hard-wired and cannot be changed, afaik. I read that on zendesk, only the clear (UI) name displayed may be changed like on other OSes.
Pythonic could look something like:

import subprocess
HOME_DIR = subprocess("getent passwd 100000 | cut -f 6 -d ':'", shell=True)

But as I am lazy and thought that this is not needed, I kept my approach

[nephros]

@peterleinchen

This is something that cannot happen as the user names “defaultuser” resp. “nemo” are hard-wired and cannot be changed, afaik.

well someone could do it but I don’t think it’s worth dealing with that use case…

# add title to avoid confusing a proper badass with some pathetic little missing fish

usermod -m -d /home/nautilus -l captain_nemo nemo

[peterleinchen]

Uh oh!

Yes, that would be possible of course. But I am pretty sure then something strange will happen on next boot!

Until someone shows me captain_nemo on his nautilus booting, I do see it as safe.

[DrYak]

Well, I think it’s safe to assume that a user that could manage to change their home directory and username, they’d also be able to handle this type of failure modes.

(On the other hand, note that my initial suggestion of using a path inside /var/lib/harbour-defender would also not be affected by that).

In the end, the most important thing is that you fixed defender. And that’s super cool.

[4carlos]

What about this?

https://together.jolla.com/question/215859/adblock-via-etchosts-on-xa2aliendalvik-v8/?answer=215903#post-id-2159038

[nephros]

What about it?

AFAIK/AFAICS, Defender manages the Android Support hosts file as well.

peterleinchen/harbour-defender/blob/master/qml/python/defender_updater.py#L48-L51

Some answers in the thread you linked say that should work, others say AD doesn’t use these hosts entries at all. What is true for current SFOS, current Defender and modern (Xperia10 et al) AD?

For me it reads like that the Defender scripts would have to do the lxc-mount and override as in the linked answer to be effective, correct?

[DrYak]

This works for AlienDalvik for Android 4.x (devices from original Jolla 1 up to Xperia X), as they share the same filesystem as the main linux (the isolation is just inside the Java-like VM that old Android use).

This code will see the files and update them

This does not work AlienDalvik for Android 8.1 (devices starting from Xperia XA 2), as they:

• use an LXC container (and thus their filesystem is isolated from the main Linux)
• store that filesystem in a compressed read-only squashfs system.img file (as opposed to bind-mount some subdirectory that is writeable by Linux).

The code linked above will not work because, the files don’t exist in the namespace of the main linux.

The solution to modify the configuration (the include /var/lib/lxc/aliendalvik/extra_config is handy for that) works, because it bind mounts the Linux file to inside the Android LXC.

[DrYak]

A completely different way to handle this would be to use DNS forwarder daemon on the linux side
(i.e.: it’s own DNS server, e.g.: dnsmasq, systemd-resolved, etc.) (note: on other laptops/smartphomes NetworkManager can even automate that, I don’t know if connman on Sailfish can automate it).

And then configure Android to always use that as a DNS server in the default connection.

The main advantage:

• defender wouldn’t need to modify actual host files. It can modify some include configuration of dnsmasq and normal DNS name solution is handled by the latter.

The main disadvantage:

• much more complex solution involving multiple components (installing dnsmasq, having connman update dnsmasq configuration instead of /etc/resolv.conf, etc.)

[4carlos]

@nephros

AFAIK/AFAICS, Defender manages the Android Support hosts file as well.

Only legacy devices (+ Xperia X I guess)

@DrYak

This does not work AlienDalvik for Android 8.1 (devices starting from Xperia XA 2),

Thank you for your answer. Then I don’t have to answer @nephros .

[peterleinchen]

@DrYak

This code will see the files and update them.

The code linked above will not work because, the files don’t exist in the namespace of the main linux.

The solution to modify the configuration (the include /var/lib/lxc/aliendalvik/extra_config is handy for that) works, because it bind mounts the Linux file to inside the Android LXC.

Yes.
I changed the lxc mount a bit, but in the end it does not really matter, I guess.

Simply edit the file /var/lib/lxc/aliendalvik/extra_config as root and add following line:

lxc.mount.entry = /system/etc/hosts system/etc/hosts none bind,ro,create=file,optional 0 0

• I am using the Android /system/etc/hosts (which gets updated by Defender as well) for the lxc mount.
• Furthermore I added the option ‘ro’ so Android side cannot mess with SFOS file(s).
• And I put option ‘optional’ as when I tried to add also /system/etc/hosts.editable it caused aliendalvik to fail starting (ro file system, could not add that file, alien service failing). This is not needed for system/etc/hosts as this file is present. Just for the records.

[sausset]

I’m encountering a problem with the list updating: it’s already trying to update when I launch the app and it never ends. If I try to update manually it doesn’t change anything.
As lists are not up-to-date, the filtering is ineffective .
Does anybody else has the same behaviour ?

[DrYak]

Yup, not all list are actually still functional. Some option in defender point to now broken lists.

Currently I am using:

• StevenBlack hosts - Unified hosts (adware + malware)
• Disconnect.me / Tracking
• Disctonnect.me / Advertising

Those three seem to still be correctly maintained on their websites and thus Defender can still successfully usethem to update.

[sausset]

Thanks!
I had to uninstall /install the app and now the list update is working with the sources you suggested.

[peterleinchen]

Yes.

Some lists are not working anymore and causing infinite update cycle.
Something to be solved (later). If someone has an idea, create a PR.
For now I did not like to mess with that, knowing that

• hpHosts, Host file
  does not work. Even there were some mirrors for that which could be used.

For me is working:

• MVPS, Hosts file
• Yoyos, Hosts file

[olf]

@DrYak

Best would be to use whatever is the pythonic equivalent of the bash getent passwd 100000 …

AFAIK Jolla tries to get away from all the “hardcoded” configuration. Becoming independent of the fixed “admin user” name “nemo” was just one step in this direction, AFAIU.
Hence UID 10000 may not necessarily be an “admin user”, at least after some configuring.
Plus theoretically there may be multiple “admin users”.

What one can supposedly rely on, is that “admin users” are members of the group “sailfish-system” (since SFOS 3.4.0, before that: “system”).
Edit: Or use loginctl to determine who is logged in at the “primary seat”; see Jolla’s udisksctl-user script for details.

P.S.: To take this into account would just add an indirection to the pythonic one-liner, AFAICS.

[DrYak]

What one can supposedly rely on, is that “admin users” are members of the group “sailfish-system” (since SFOS 3.4.0, before that: “system”).

I would still vote for a separate /varlib/harbour-defender/ with g+rwx for these groups.

[peterleinchen]

I am afraid the update loop you experienced may be caused by my little to zero systemd knowledge.
So please update harbour-defender from openrepos.
And/or see here for some details.

But still Dr.Yak’s and my answer are true and some lists are not maintained anymore and may also not be reachable.