RFC: adding shell scripting capabilities will bring PM2 to the next level

I have submited a feature request about Patch Manager:

I wish to debate this approach here, if you like to partecipate.


DESCRIPTION

Reading the description of this patch, we clearly realise that adding shell script capabilies will bring PM2 to the next level. In order to achieve this there are some type of specific scripts to keep in consideration:

  • pre-install script: pre-install.sh
  • post-install script: post-install.sh
  • pre-remove script: pre-remove.sh
  • post-remove script: post-remove.sh

Which are the same types that are allowed into any kind of package (rpm, deb, etc.). Obviously, these scripts should be executed with root privileges and possibly in a decently set enviroment. Possibly a $PATCH_PATH or $PATCH_DIR (with or without the underscore) could be set in order to refer to folder in which the patch archive is extracted. In this way, more files can be added to be parsed by the scripts or copied into the system.

ADDITIONAL INFORMATION

This feature can be implemented in two different ways:

  • allow to add four scripts for each of that task with .sh extension
  • allow to add a single text file with .shell extension which contains 4 sections.

In the last case, something like that:

[pre-install]
rm -f /etc/resolv.conf

#[post-install]
#[pre-remove]

[post-remove]
ln -sf /run/connman/resolv.conf /etc/resolv.conf

One or another way is indifferent and It is a matter of taste. In .sh case the files do not need to be executable to run but just being interpretable. However, the .sh approach avoid to have a parser to split the .shell single file and moreover, it clears the possible misunderstanding about the chance that every single line will be executed by a different shell instance:

/bin/ash pre-install.sh

instead of

/bin/ash -c "first line"
...
/bin/ash -c "last line"

It is suggested to use /bin/ash as default interpreter which is enough powerful in terms of scripting even if does not implement the full set of bashism. This because the scripts relying on /bin/ash can run on /bin/bash as well but in some system /bin/bash could be optional.


UPDATED

An example of the current limitations of Patch Manager which is great for patching the UI but completely indadeguate for patching the system.

Due to its way of working the Patch Manager is not the right tool for this kind of patches but the dnsmasq RPM can be fixed with this patch:

Changelog: 0.0.6 - connman starts after patchmanager and dnsmasq before connman (in my whishes)

In practice, because this patch will not be applied unless Patch Manager will complete is job, sometimes the connman and dnsmasq services will start before their .service files have been patched and therefore the system will not be able to resolv the domain names. More over, the network restart from SailFish Utilities cannot solve the issue and also rebooting might not solve but usually does.

Answer: [Suggestion] adding shell scripting capabilities will bring PM2 to the next level · Issue #441 · sailfishos-patches/patchmanager · GitHub

3 Likes

There is an intermediate layer of complexity between a RPM which offer a lot of features (full feature) at the cost of having to deal with its specific syntax (which is quite complex because it should deal with all the dependencies of the all the packages in the repository and with the installed ones) and executing shell scripts during a Patch Manager installation. Web Catalog is not the problem because it has to deal with a compressed archive (e.g. pippo.tar.gz) while the Patch Manager have just to execute 4 simple operations as far as all four scripts files exist. There is a HUGE gap between applying a patch (diff -pruN) and preparing a RPM, in this HUGE gap, Patch Manager shell scripting capability has its own place.

@olf - fulfilling that gap will step down the curve of people that can learn from doing a patch, doing a patch + scripting and the develop the skills to package an RPM. The learning curve should be as smooth as possible in order to enlarge the user-base and in this specific case the user-base means advanced smartphone users that get involved into modding / improving their system. If you expect that people jump from patching and packging, many will be lost. P.S.: BTW, hidden profile is not such a great sign of openess and trust towards the others people… :wink:

Heh. The next level of patchmanager is called vi.

3 Likes

@adekker now¹, I put the @olf answer in the right dimension: PM2 is a tool useful to customise the UI because everything else cannot be correctly addressed with PM2.

While the PM3 never get into SFOS because wasn’t able to deal with fakeroot

¹ now - means - after having realised that the documentation on the github, it is the WHOLE documentation which exist.


DOCUMENTATION SHORT COMINGS

update: [WRONG] It is not necessary -p1, also a patch that requires -p0 works as well. Also diff -pruN generates valid patches. [/WRONG] The -p1 works well also with absolute paths like diff -pruN /usr/bin/ciao.sh.orig /usr/bin/ciao.sh

Most of the patches that I saw does not have such .json file. Moreover, creating and adding it would be a task for the Web Catalog which will receive all data from the patch author.

The syntax an example of .qml is completely missing or at least the link to - but in doc there is nothing about.

Finally, it has been said that Web Catalog is in charge for .json but Web Catalog does not add such a file into the archive. Moreover, if a .json is included into the patch then it should be removed to not conflicts with the one generated by Web Catalog. The the explanation of the .json shoud be posticipated after this section otherwise people need to read two times the documentation to understand the plot (it is a crime novel not a man page :rofl:).

DOCUMENTATION LACKS

It is not explained that Patch Manager is not a system service and it removes the patches at the shutdown and brings them up at the boot time because it uses a temporary not persistent folder to store some changes. This breaks the system services that has been patched but raised up BEFORE the Patch Manager completed his job. Because Patch Manager itself is part of the UI, then it depends by d-bus system service.


For everything below IMHO and AFAIK apply.


Patch Manager analisys

For most of system services wait for d-bus is too late or even completely unfeasible expecially if d-bus waits for them. Thus, PM2 need a system handler to apply patches before systemd will be started (e.g. at run-level 1 aka single user). Or it should work in a way that patches are persistent until disabled. This second option is compatible with giving the PM2 simply but powerful script capability.

And now we are again in a loop: persistence could not unversally achived because JailFish create temporary fake root. Therefore system patches that would benefits of the shell scripting capability cannot be delivered by this tool and this explain why @olf wrote that a “complex” patch needs to be delivered by a RPM.

Patch Manager evolution

It is clear that the current Patch Manager is a tool dedicated to customise the UI and nothing more than that. This SHOULD be clearly reported in its documentation.

As wrote before (here) and confuting the @poetaster 's humor that I appreciated very much and I hope you appreciated the analogy with the crime novel plot:

The next level for Patch Manager is a Web Catalog feature that ask the developer/patcher if s/he wants deliver a UI patch or a system patch. In the first case, everything remain the same as by now and in the second case create a RPM to install into the system¹.

I know, it would be a simple RPM that it will apply a patch (or explode an archive) and execute 4 scripts if they exists and it will. After all, doing pkcon search patch we discover that the repository is plenty of patches that the common user is not aware about by UI and Patch Manager.

Conclusion

First of all, I can had overlook or miss something (most probably) because it is my first month anniversary among the sailors.

IMHO, there is a lot of work done (pkcon search patch proves it), poor documentation, break-your-head-against-the-wall attitude and on the other side there is no a product because the current flag ships smartephones Xperia 10 II & III barerly work for the basic functions - they might work in some conditions and for some users - but in general they do not.

Now, it is pretty clear for me because Jolla is going to face a bankruptcy!

My be the last chance to change for the sailors, I suppose. :blush:

Notes

¹ there is no reason because a wo/man should do a work that a robot can do faster and even better unless you are a sadic which likes other people suffer an usless and tireless pain! :rofl:

I think you are subjecting PM to overly intricate design expectations. But, I am my own PM and apply patches by hand, so I have VERY limited design objectives for patchmanager. I do think you should just clone the repo and make PRs though, rather than keep us in suspense.

1 Like

You have two Xes by the judges for this feature now.

Continuing to argue will not further your goal.

Abusing this Forum once more to discuss things belonging elsewhere will not further your goal.

Ad-hominem arguments against a well-respected member of this community will not further your goal.

1 Like

You answered before I had completed my post which was in [saved to continue] state. Therefore, you did not catch my goal at all but completely missed it. Please, read the conclusion of the post you anticipately answered. :blush:

I have to agree with @nephros that you are actually being offensive in tone and ignoring the years of service that some members here have put into PM, as one example only. You have a lot of good impulses to share, but you’re not doing it in a constructive way. Share your ideas in Pull Requests.

3 Likes

I am sad to read that factual reality is offensive but I understand your delusion. I will reply to you, to @nephros and @olf what has been said for centuries: ambassador brings news,
do not shoot the messenger
. Take it with humor or like a game: the nerds against the product¹.

Pulling requests is a community and Jolla employees aim but not mine, also for the future as long as Jolla will manage to avoid pulling the plug to the smartphone market². My aim is to determine the risk of an investment and nothing more than this.

In exchange of your partecipation in this investigation (and “I supposed”, it was a reference to Sherlock Holmes even if the proper definition is SWOT analisys), I gave you my frank opinion, some suggestions and I will leave for new sailors a quick start guide (here) even if it diverged a lot from the initial plannig.

I will just release a couple of updates about my patches, just in case.

notes

¹ spoiler: the nerds won over the product and the market sunk the ship (a few miles along this way of doing).
² without a strong reference market even the linux kernel would never have been spread around and colonised more than 90% of the servers running Internet without Mandriva and the Ubuntu, GNU/Linux would never had managed to be useful for common people with a laptop or a desktop - and it is not about bells and whistles because SFOS is plenty of them but about functionalities, realiability and curve of adoption for end-users and developers, as well the first about the product and the others about the develop/modding tools.

vi? You disappoint me.
ed is the Standard Unix text editor.

2 Likes

Ah here we agree: documentation of PM is in need of improvement, both for end-users and patch developers.

1 Like

sed, everything else is for boys/girls… :rofl:

About changing the sailing route, it would not so hard as you might think:

Check the update #2, please.

sed ‘s/ed/sed/’ ? (insert evil laugh here).

1 Like

It’s not hidden. It’s a well known forum bug/problem.

1 Like

SYSTEM PM2 PATCHES, IMPLEMENTATION EXAMPLE

In applying this Patch Manager patch, two cases arises

case #1: cacerts_gps folder exists

[root@sfos ~]# ls -al /system/etc/security/cacerts_gps
isrgrootx1.pem -> /tmp/patchmanager/.../isrgrootx1.pem
roots.pem -> /tmp/patchmanager/.../roots.pem

case #2: cacerts_gps folder does not exist

[root@sfos ~]# ls -al /system/etc/security/cacerts_gps
cacerts_gps -> /tmp/patchmanager/system/etc/security/cacerts_gps

Both case are wrong because Patch Manager creates symlink where it was supposed to create dirs and files. In fact, symlinks are not the same story because some tools that operate on the filesystem required a specific option to follow symlinks. Like old versions of the tar and tar is one of the widely used tool for doing backups. The approach can be easily changed following these examples:

mkdir /tmp/test
cd /tmp/test
repourl="coderus.openrepos.net/media/documents"
tarball="robang74-x10ii-iii-agps-config-emea-0.2.0.tar.gz"
curl https://$repourl/$tarball | tar xvzf -

The following shell code will works with -p0 and with -p1 because the sed regex deal with /var, new/var and ./new/var indifferently and every relative filepath is converted into an absolute /filepath.

files=$(sed -ne "s,^+++ *\.*[^/]*\([^ ]*\).*,/\\1,p" unified_diff.patch)
grep -qe "^+++ */" unified_diff.patch || false
plvl=$?

Now it is time to do a backup, for future restore but remember that overlay tricks the old tar, therefore a recent tar or busybox tar is needed

busybox tar czf /$store_dirpath/patch-$project_name.tar.gz $files
echo "$files" > /$store_dirpath/patch-$project_name.list

This determines the uninstall procedure by defining a global non-parametric function

patch_uninstall() { 
	rm -f $(cat /$store_dirpath/patch-$project_name.list)
	busybox tar xzf /$store_dirpath/patch-$project_name.tar.gz -C /
}

This is useful before apply the diff patch in order to creates dir and files not symlinks, therefore the symlinks engine could be ignored

patch_apply() {
	ret=0
	for i in $files; do mkdir -p $(basename $i); touch $i; done
	if ! patch -d / $pagrs -p$plvl unified_diff.patch; then 
		patch_uninstall
		ret=$((1+$?))
	fi
	return $ret
}

Is a system patch? For example system_diff.patch instead of unified_diff.patch? No, then uninstall at shutdown time. But why uninstall a patch at shutdown time with the risk that shutdown procedure and the related uninstall procedure can be interupped by a user physical keys controlled switch off? Because the filesystem for UI patches is volatile by default? Also the root filesystem about the reboot? Anyway:

num=$(printf "%05d" $(ls -1 /$store_dirpath/[0-9]*.tgz | wc -l))
busybox tar czf /$store_dirpath/$num-$project_name_applied.tar.gz $files

At boot time, in something functionally equivalent to /etc/rc.local but after all system device has been mounted not just those in /etc/fstab but before every systemd service will start, will be inserted the restoring procedure which will applies all the patch in their correct sequence

files=$(ls -1 /$store_dirpath/[0-9]*_applied.tar.gz)
test "$files" = "" && exit 0
for i in $files; do busybox tar xzf $i -C /; done

That’s all, unless I forgot or overlooked something essential or important.


UPDATE #1

This seems promising to install system updates also for those that do not require a reboot:

   system-update.target, system-update-pre.target,
   system-update-cleanup.service
       A special target unit that is used for offline system
       updates.  systemd-system-update-generator(8) will redirect
       the boot process to this target if /system-update or
       /etc/system-update exists. For more information see
       systemd.offline-updates(7).

It seems a general solution that requires - by documentation - a reboot but the reboot is managed by the configuration and not automatically enforced. However, due to its specific nature and delicacy, it can be a better option to add a service ordering related to system-update.target or even better system-update-pre.target in such a way that the patches which might conflicts with package updates will be applied before making them fail, as expected to be, instead of being overwritten.

UPDATE #2

The Patch Manager patches can be applied and unapplied many times during a user session and this is great feature :heart_eyes:. However, changing the way in which the PM2 works some of them might not fall into this category.

For example DNS Alternative is delivered like a RPM and it proobably it is the best way to have it. Before, it was a RPM package a developer might develop a patch. In general the way of providing a change could be exemplified in three main passages:

  1. system patch → 2. optional RPM package → 3. default RPM package

Which also implies three different level of integration with SFOS: unsopported (community only), supported in terms of the repository consistency (community + Jolla) and Jolla supported (commercial support). Which three different level of SLA and QoS in terms of supporting the end-users.

IMHO, the main difference between a system patch and a RPM package is bringing into the system new binaries rather than modify the system configuration. In this second case, having a system patch seems more reasonable expecially for the end-users that can choose to reconfigure the system as they wish - in the same manner they are doing with UI.

While UI patches can requires an application retart and a fingerprinter reader patch is functionally identical to a patch for e.g. the Settings, those have impact to the network and rely on the installation of 3rd party package e.g. dnsmasq need a little more attention. In fact, restarting network by SailFish Utilities do not consider the case in which dnsmasq is installed and configured (to fix). For all the others, a reboot is almost necessary.

This brings us to the conclusion that there are three patches classes:

  1. those changes UI, app, stand-alone services level: easy to restart
  2. those changes complex services like network/d-bus: might or might not be restarted
  3. those changes the system at such level that a reboot is needed: restart useless

The #1 and the #3 are almost straightforward cases to deal with. The second is a matter of policy: dnsmasq is optional but supported by network restart because it is an important feature that users usually requires to enable. Or on the other side is a 3rd party unsupported services an then the end-user needs to reboot his/her smartphone.

Maybe you misunderstand; pull requests are how you constructively ask for adding new features to existing software on git projects, if not all VCS.

2 Likes

In the very best case, you’ve fixed a bug (or more), diligently documented that, and / or and added a well thought out feature that you’ve clearly explained. Well, one could go on, but, I just made a merge request which disguises a support request. It’s complicated enough, and the person on the other side knows me, so I can probably get away with it, but, it is borderline. I’m really lucky to get PRs which 99.99% of the time simply improve stuff I’m responsible for. I’m very thankful for that.

2 Likes

It is a matter of role separations. Not about feasibility or lack of will to help.

If you don’t want to create pull requests, just say so, nobody is forcing you. But we do not have fixed roles in this users community, it is all voluntarily cherry picking. If you want to help with documentation, you are welcome. Same applies for pull requests or creating apps. And your “role” (whatever you envision as your role) is already ambiguous btw, posting howto’s for beginners and asking for features, but also submitting patches for patchmanager.

2 Likes