RFC: adding shell scripting capabilities will bring PM2 to the next level

Applications Others
#5

@adekker now¹, I put the @olf answer in the right dimension: PM2 is a tool useful to customise the UI because everything else cannot be correctly addressed with PM2.

While the PM3 never get into SFOS because wasn’t able to deal with fakeroot

¹ now - means - after having realised that the documentation on the github, it is the WHOLE documentation which exist.

DOCUMENTATION SHORT COMINGS

update: [WRONG] It is not necessary -p1, also a patch that requires -p0 works as well. Also diff -pruN generates valid patches. [/WRONG] The -p1 works well also with absolute paths like diff -pruN /usr/bin/ciao.sh.orig /usr/bin/ciao.sh

Most of the patches that I saw does not have such .json file. Moreover, creating and adding it would be a task for the Web Catalog which will receive all data from the patch author.

The syntax an example of .qml is completely missing or at least the link to - but in doc there is nothing about.

Finally, it has been said that Web Catalog is in charge for .json but Web Catalog does not add such a file into the archive. Moreover, if a .json is included into the patch then it should be removed to not conflicts with the one generated by Web Catalog. The the explanation of the .json shoud be posticipated after this section otherwise people need to read two times the documentation to understand the plot (it is a crime novel not a man page :rofl:).

DOCUMENTATION LACKS

It is not explained that Patch Manager is not a system service and it removes the patches at the shutdown and brings them up at the boot time because it uses a temporary not persistent folder to store some changes. This breaks the system services that has been patched but raised up BEFORE the Patch Manager completed his job. Because Patch Manager itself is part of the UI, then it depends by d-bus system service.

For everything below IMHO and AFAIK apply.

Patch Manager analisys

For most of system services wait for d-bus is too late or even completely unfeasible expecially if d-bus waits for them. Thus, PM2 need a system handler to apply patches before systemd will be started (e.g. at run-level 1 aka single user). Or it should work in a way that patches are persistent until disabled. This second option is compatible with giving the PM2 simply but powerful script capability.

And now we are again in a loop: persistence could not unversally achived because JailFish create temporary fake root. Therefore system patches that would benefits of the shell scripting capability cannot be delivered by this tool and this explain why @olf wrote that a “complex” patch needs to be delivered by a RPM.

Patch Manager evolution

It is clear that the current Patch Manager is a tool dedicated to customise the UI and nothing more than that. This SHOULD be clearly reported in its documentation.

As wrote before (here) and confuting the @poetaster 's humor that I appreciated very much and I hope you appreciated the analogy with the crime novel plot:

The next level for Patch Manager is a Web Catalog feature that ask the developer/patcher if s/he wants deliver a UI patch or a system patch. In the first case, everything remain the same as by now and in the second case create a RPM to install into the system¹.

I know, it would be a simple RPM that it will apply a patch (or explode an archive) and execute 4 scripts if they exists and it will. After all, doing pkcon search patch we discover that the repository is plenty of patches that the common user is not aware about by UI and Patch Manager.

Conclusion

First of all, I can had overlook or miss something (most probably) because it is my first month anniversary among the sailors.

IMHO, there is a lot of work done (pkcon search patch proves it), poor documentation, break-your-head-against-the-wall attitude and on the other side there is no a product because the current flag ships smartephones Xperia 10 II & III barerly work for the basic functions - they might work in some conditions and for some users - but in general they do not.

Now, it is pretty clear for me because Jolla is going to face a bankruptcy!

My be the last chance to change for the sailors, I suppose. :blush:

Notes

¹ there is no reason because a wo/man should do a work that a robot can do faster and even better unless you are a sadic which likes other people suffer an usless and tireless pain! :rofl:

A collection of tasks and a request for changes
#6

I think you are subjecting PM to overly intricate design expectations. But, I am my own PM and apply patches by hand, so I have VERY limited design objectives for patchmanager. I do think you should just clone the repo and make PRs though, rather than keep us in suspense.

1 Like
#7

You have two Xes by the judges for this feature now.

Continuing to argue will not further your goal.

Abusing this Forum once more to discuss things belonging elsewhere will not further your goal.

Ad-hominem arguments against a well-respected member of this community will not further your goal.

1 Like
#8

You answered before I had completed my post which was in [saved to continue] state. Therefore, you did not catch my goal at all but completely missed it. Please, read the conclusion of the post you anticipately answered. :blush:

#9

I have to agree with @nephros that you are actually being offensive in tone and ignoring the years of service that some members here have put into PM, as one example only. You have a lot of good impulses to share, but you’re not doing it in a constructive way. Share your ideas in Pull Requests.

3 Likes
#10

I am sad to read that factual reality is offensive but I understand your delusion. I will reply to you, to @nephros and @olf what has been said for centuries: ambassador brings news,
do not shoot the messenger. Take it with humor or like a game: the nerds against the product¹.

Pulling requests is a community and Jolla employees aim but not mine, also for the future as long as Jolla will manage to avoid pulling the plug to the smartphone market². My aim is to determine the risk of an investment and nothing more than this.

In exchange of your partecipation in this investigation (and “I supposed”, it was a reference to Sherlock Holmes even if the proper definition is SWOT analisys), I gave you my frank opinion, some suggestions and I will leave for new sailors a quick start guide (here) even if it diverged a lot from the initial plannig.

I will just release a couple of updates about my patches, just in case.

notes

¹ spoiler: the nerds won over the product and the market sunk the ship (a few miles along this way of doing).
² without a strong reference market even the linux kernel would never have been spread around and colonised more than 90% of the servers running Internet without Mandriva and the Ubuntu, GNU/Linux would never had managed to be useful for common people with a laptop or a desktop - and it is not about bells and whistles because SFOS is plenty of them but about functionalities, realiability and curve of adoption for end-users and developers, as well the first about the product and the others about the develop/modding tools.

#11

vi? You disappoint me.
ed is the Standard Unix text editor.

2 Likes
#12

Ah here we agree: documentation of PM is in need of improvement, both for end-users and patch developers.

1 Like
#13

sed, everything else is for boys/girls… :rofl:

About changing the sailing route, it would not so hard as you might think:

Check the update #2, please.

#14

sed ‘s/ed/sed/’ ? (insert evil laugh here).

1 Like
#15

It’s not hidden. It’s a well known forum bug/problem.

1 Like
#16

SYSTEM PM2 PATCHES, IMPLEMENTATION EXAMPLE

In applying this Patch Manager patch, two cases arises

case #1: cacerts_gps folder exists

[root@sfos ~]# ls -al /system/etc/security/cacerts_gps
isrgrootx1.pem -> /tmp/patchmanager/.../isrgrootx1.pem
roots.pem -> /tmp/patchmanager/.../roots.pem

case #2: cacerts_gps folder does not exist

[root@sfos ~]# ls -al /system/etc/security/cacerts_gps
cacerts_gps -> /tmp/patchmanager/system/etc/security/cacerts_gps

Both case are wrong because Patch Manager creates symlink where it was supposed to create dirs and files. In fact, symlinks are not the same story because some tools that operate on the filesystem required a specific option to follow symlinks. Like old versions of the tar and tar is one of the widely used tool for doing backups. The approach can be easily changed following these examples:

mkdir /tmp/test
cd /tmp/test
repourl="coderus.openrepos.net/media/documents"
tarball="robang74-x10ii-iii-agps-config-emea-0.2.0.tar.gz"
curl https://$repourl/$tarball | tar xvzf -

The following shell code will works with -p0 and with -p1 because the sed regex deal with /var, new/var and ./new/var indifferently and every relative filepath is converted into an absolute /filepath.

files=$(sed -ne "s,^+++ *\.*[^/]*\([^ ]*\).*,/\\1,p" unified_diff.patch)
grep -qe "^+++ */" unified_diff.patch || false
plvl=$?

Now it is time to do a backup, for future restore but remember that overlay tricks the old tar, therefore a recent tar or busybox tar is needed

busybox tar czf /$store_dirpath/patch-$project_name.tar.gz $files
echo "$files" > /$store_dirpath/patch-$project_name.list

This determines the uninstall procedure by defining a global non-parametric function

patch_uninstall() { 
	rm -f $(cat /$store_dirpath/patch-$project_name.list)
	busybox tar xzf /$store_dirpath/patch-$project_name.tar.gz -C /
}

This is useful before apply the diff patch in order to creates dir and files not symlinks, therefore the symlinks engine could be ignored

patch_apply() {
	ret=0
	for i in $files; do mkdir -p $(basename $i); touch $i; done
	if ! patch -d / $pagrs -p$plvl unified_diff.patch; then 
		patch_uninstall
		ret=$((1+$?))
	fi
	return $ret
}

Is a system patch? For example system_diff.patch instead of unified_diff.patch? No, then uninstall at shutdown time. But why uninstall a patch at shutdown time with the risk that shutdown procedure and the related uninstall procedure can be interupped by a user physical keys controlled switch off? Because the filesystem for UI patches is volatile by default? Also the root filesystem about the reboot? Anyway:

num=$(printf "%05d" $(ls -1 /$store_dirpath/[0-9]*.tgz | wc -l))
busybox tar czf /$store_dirpath/$num-$project_name_applied.tar.gz $files

At boot time, in something functionally equivalent to /etc/rc.local but after all system device has been mounted not just those in /etc/fstab but before every systemd service will start, will be inserted the restoring procedure which will applies all the patch in their correct sequence

files=$(ls -1 /$store_dirpath/[0-9]*_applied.tar.gz)
test "$files" = "" && exit 0
for i in $files; do busybox tar xzf $i -C /; done

That’s all, unless I forgot or overlooked something essential or important.

UPDATE #1

This seems promising to install system updates also for those that do not require a reboot:

   system-update.target, system-update-pre.target,
   system-update-cleanup.service
       A special target unit that is used for offline system
       updates.  systemd-system-update-generator(8) will redirect
       the boot process to this target if /system-update or
       /etc/system-update exists. For more information see
       systemd.offline-updates(7).

It seems a general solution that requires - by documentation - a reboot but the reboot is managed by the configuration and not automatically enforced. However, due to its specific nature and delicacy, it can be a better option to add a service ordering related to system-update.target or even better system-update-pre.target in such a way that the patches which might conflicts with package updates will be applied before making them fail, as expected to be, instead of being overwritten.

UPDATE #2

The Patch Manager patches can be applied and unapplied many times during a user session and this is great feature :heart_eyes:. However, changing the way in which the PM2 works some of them might not fall into this category.

For example DNS Alternative is delivered like a RPM and it proobably it is the best way to have it. Before, it was a RPM package a developer might develop a patch. In general the way of providing a change could be exemplified in three main passages:

  1. system patch → 2. optional RPM package → 3. default RPM package

Which also implies three different level of integration with SFOS: unsopported (community only), supported in terms of the repository consistency (community + Jolla) and Jolla supported (commercial support). Which three different level of SLA and QoS in terms of supporting the end-users.

IMHO, the main difference between a system patch and a RPM package is bringing into the system new binaries rather than modify the system configuration. In this second case, having a system patch seems more reasonable expecially for the end-users that can choose to reconfigure the system as they wish - in the same manner they are doing with UI.

While UI patches can requires an application retart and a fingerprinter reader patch is functionally identical to a patch for e.g. the Settings, those have impact to the network and rely on the installation of 3rd party package e.g. dnsmasq need a little more attention. In fact, restarting network by SailFish Utilities do not consider the case in which dnsmasq is installed and configured (to fix). For all the others, a reboot is almost necessary.

This brings us to the conclusion that there are three patches classes:

  1. those changes UI, app, stand-alone services level: easy to restart
  2. those changes complex services like network/d-bus: might or might not be restarted
  3. those changes the system at such level that a reboot is needed: restart useless

The #1 and the #3 are almost straightforward cases to deal with. The second is a matter of policy: dnsmasq is optional but supported by network restart because it is an important feature that users usually requires to enable. Or on the other side is a 3rd party unsupported services an then the end-user needs to reboot his/her smartphone.

A collection of tasks and a request for changes
#17

Maybe you misunderstand; pull requests are how you constructively ask for adding new features to existing software on git projects, if not all VCS.

2 Likes
#18

In the very best case, you’ve fixed a bug (or more), diligently documented that, and / or and added a well thought out feature that you’ve clearly explained. Well, one could go on, but, I just made a merge request which disguises a support request. It’s complicated enough, and the person on the other side knows me, so I can probably get away with it, but, it is borderline. I’m really lucky to get PRs which 99.99% of the time simply improve stuff I’m responsible for. I’m very thankful for that.

2 Likes
#19

It is a matter of role separations. Not about feasibility or lack of will to help.

#20

If you don’t want to create pull requests, just say so, nobody is forcing you. But we do not have fixed roles in this users community, it is all voluntarily cherry picking. If you want to help with documentation, you are welcome. Same applies for pull requests or creating apps. And your “role” (whatever you envision as your role) is already ambiguous btw, posting howto’s for beginners and asking for features, but also submitting patches for patchmanager.

2 Likes
#21

Who did the software should take care of documenting and maintaining it because the same time that I spend to understand, documenting and fixing others people code, I can deliver a better solution or simply switch to a better product. Which most probably is what many did before me.

UPDATE #1

First of all, I wish to engage this

We always play roles: we are the father/mother, the friend, the boy/girlfriend, the son/daughter, the manager or the employee. Our lives are about the roles but this roles are something like a jail - a box in which society force to stay in. It is wonderful that a community of free people do not feel the need to fit in a box-schema.

When we approach a situation or a product or a software, we can play many roles: nerd hacker, cyber-security expert, marketing & sales, business owner, product manager, project manager, end-user, advanced user, social engineer, social manager, troll, etc.

This two way of playing roles are completely different.

The first way is about constrains and expectations that OTHERS put over us and we need to conform to them in a way or another. The second way is to put ourselves in the shoes of someone else, someone different from us without being or became someone else. Like playing Dungeon and Dragons, never made anyone being or became an elf or a wizard. However to play that roles, we need to know in deep what’s about that role (set of values) and it is not.

A bit of philosophy - Aristotle wrote, “It is the mark of an educated mind to be able to entertain a thought without accepting it.” Being able to look at & evaluate different values without necessarily adopting them is perhaps the central skill required in changing one’s own life in a meaningful way. (Source: Aristotele cited and commented on Twitter)

Few people can play multiples roles at the same time because few people are acknowledged about that roles enough to play that roles. In fact, companies usually tend to create multi-cultural heterogeneous teams because the complexity of the modern-world products cannot be faced just by 1-single PoV approach or by segmenting the design-to-delivery among separated in-line departments.

This is the reason because community-based products are usually the winner on the long terms because in the Bazaar there is many PoV. Unfortunately, this is not always true. Some company teams are better than some community-bazaars and viceversa.

end of philosophy

Now, it is my time to explain the quick answer I wrote here before going to take care of my real life and then returning back.

Settings:System → Patchmanger:Settings → Activate enabled Patches when booting

Why activating at boot time instead of made them permanent with that option?

This question is extremely relevant - not only for the consequences that brings in terms of constrains - but also because it is a design choice. This design choice should be explained in details into the documentation otherwise this software have to be re-design from scratch.

No, I cannot make a pull request to fix this because this is an information that WHO designed the software should explain. It could be a very good reason/decision or it was a good reason/decision at that time, e.g. SFOS 2.x but not anymore.

This is the reason because this design choices should be documented, they aging and they are still impacting despite aging. The temporary workaroud became the product and the product became the legacy. We need to stop this before even it begins, documenting it.

What this has to do with roles? Unless someone plays the product and project manager role in the community those PoVs are missing and in fact - AFAIK - the part of the documentation which refers to this design choice is missing.

end of theory

We evaluate a change betwee “activate at boot time” and “keep persistent”.

We can do a confrontation:

  • persistence is easier because we “patch & forget”
  • forgetting is not a good practice therefore checking
  • checking but when?
  • every time Patch Manager page is shown
  • how we can implement this check?
  • pach -Rp0 --dry-run can fail or not
  • is it quick enough?
  • yes → done

Here we are, we can have a Persistent Patch Manager with a little of changes. Now, it is your time to play roles or simply express yourself. What good can provide persistence and what problems can cause?

end of doing better

UPDATE #2

Why I am pedanticly collecting this suggestions in the Quick Start Guide. Because one day someone will arrive and with social engineering that will please you, will convince you to do these things or equivalent ones. In the lucky scenario otherwise not. However, the main way is the hard way: community-bazaar should learn how to play roles.

A collection of tasks and a request for changes
#22

Vague philosophy lessons we can do without : check ✓
Very pedantic tone towards others : check ✓
Walls of text changed over and over again : check ✓
Redirect your “answer” to another subject in the same post : check ✓

Despite similar remarks by others I fail to see any change, and giving up hope on any self-reflection. All I can do is give you no more ammunition, so this will the last reaction you will get from me.

6 Likes
#23

Yes, please go ahead with either of these two options; pick the one which suits you best.

This design choice should be explained in details into the documentation otherwise this software have to be re-design from scratch. […] this is an information that WHO designed the software should explain.

Neither you or anyone else is in the position to tell somebody here what should or has to be done. You may make suggestions and try to convince people (though a staccato of half-baked ideas is no very convincing), but basically it is “DIY or leave it”.

We need to stop this before even it begins, …

I wonder who comprises the “we” you keep recurring to: Do you have multiple personalities? If so, this is fine, because then each “we” means “I”; otherwise, I can only reiterate “Neither you or anyone else is in the position to tell somebody here what should or has to be done.”.

5 Likes
#24

Just I did, moving forward for both of them. About the better solution:

  1. I show that it is easy and feasible to patch the filesystem (files and directory) without creating links to a temporary directory
  2. the Patch Manager can move easily from “apply at boot time” in “persistent mode” with check by --dry-run option which probably is just implemented because currently the Patch Manager is able to detect when a patched file is changed
  3. avoid that Patch Manager removes patches when the system is asked to go down for shutdown or reboot

In particular about the point #3, I have tested with success and satisfaction a killall -9 patchmanager. Obviously this would not provide persistence because /tmp/patchmanager is volatile. Now, I have to make another test based on information collected with find /tmp/patchmanager -type f.

The test will be similar to the shell script code I presented here:

  1. collect the list of files using find
  2. backup all the system files when all patches are disabled (original versions) which probably is not necessary because it is reasonable that they are stored somewhere
  3. kill the patchmanger
  4. use the list of files to remove the links and replace with real files
  5. start again the patchmanager to check how is going to behave
  6. do a system reboot instead of point #4

Some tests, just before going to edit the two scripts that apply patches and one in perl and another in shell script.

After that, I will probably discover the SFOS ill-design choice that constrain the Patch Manager to act volatile instead of providing persistence. Or in a lucky scenario, I will simple discover that volatile for Patch Manager is not a constrain (or not anymore).

In both cases the result will be a lot of fun. :blush:

UPDATE

About the point #2, checking the /tmp/patchmanager3/patchmanager.log I found that the check with patch -Rp0 --dry-run is exactly what Patch Manager does to check that each enabled patch is applied correctly.