SFOS refactoring project (news thread)

Platform Development
#1

Premise

Please, on this thread, avoid writing off-topic but technical posts. Instead, to express your opinion, use the dedicated debate thread:

If you find it closed, it is because every 4 hours some people flag it as much as necessary in order to put it in a state of unavailability, and this happens after someone puts the 9th heart on the 2nd time moderated 1st post. Therefore, if you cannot express your opinion there, then complain with the forum staff or, even simpler, send me a private message, but do not do it here.

Introduction

This thread is reserved for the news about the SFOS refactoring project, code name RedFish OS, and here below are the most meaningful full links to start learning about this project:

  • SFOS refactoring project - this page contains the refactoring plan in 4 stages with a lot of links, and most of the notes that I took while I was hanging on in this forum have been reported there, reorganized, and reviewed for better English.

  • SFOS refactoring progress - this page contains the list of tasks about the 1st stage and their progression state, and it will be updated when significant accomplishments are done.

  • SFOS refactoring announce - a thread with the same title as this one but dedicated to the debate.

If you want to improve, update, or ask me about a change to the project or its content, please send me a private message or write to me by email. My email address is clearly presented on the first page of the GitHub project.

Forum

RedFish OS will probably (never say never) never have a dedicated forum, and the reason is that the “forum” is a Latin word that, in its etymological roots, is equivalent to “agorà” in ancient Greek meaning: a public physical place where people living in the same geographical area are used to meet each other for a variety of social activities, and not all of those activities are reputable as good, also for the moral and ethical of that time.

Therefore, if you like the “forum” concept, embrace it in its original meaning and go out of your home or office and make friends in the real world. The only reasonable way to use a platform like this (a virtual forum) is to use it as a wiki or bulletin board, and also for these purposes, it is not the best tool because it easily drives people out of their tracks. In fact, in a virtual environment, the social dynamics are different.

News

From time to time, I will write an update about work in progress as briefly as possible. If you try the software or the procedure that I am going to present here, you are welcome to write a technical post in response. Instead, opinions must go into the dedicated debate thread.

To be honest, the outlook is about completing a few tasks among those started in a way to have a minimum set of working tools and stopping the project for an indefinite time until proper conditions will arise to make sense working on it again. Therefore, I do not think that I will write a lot of updates but just a few during August 2023.

4 Likes
#2

UPDATE #1

This script provides a standard system configuration immediately after the first boot, after the flashing procedure:

The standard term here means about a starting point from which it is supposed everyone wishes to debug and fix the SFOS is put on working for quick results comparison and their reproducibility.

Plus, it adds those command-line tools like pigz and rsync that are useful in doing advanced root filesystem and users home folders backups.

:warning: WARNING NOTE

These adaptation0, aliendalvik, sailfish-eas, xt9 repositories are enabled by default, but it is wrong because they can be refreshed only when the device has a Jolla account. Otherwise, a relative long timeout for each of them keeps the refresh procedure stuck. Therefore, the Jolla account creation and deletion actions must take care of enabling and disabling them, which must be set as disabled by default. Until this fix will be implemented and provided this script deal with this aspect enabling or disabling those repositories.

1 Like
#3

UPDATE #2

  • 01.08.2023, 1st-boot setup after flashing

    description: shell scripts suite for the first boot setup after flashing

    status: ready for advanced users testing

  • 01.08.2023, sysdebug and recovery tarballs

    description: shell scripts that produce tarballs for debugging and recovering

    status: ready for advanced users testing

The first scripts suite has been presented in the update #1 while the second scripts suite are about to provide via SSH tarballs which can be installed even when the system is off-line.

Most - but not all - the command-line binaries included into these tarballs can be replaced by those available into the full features busybox statically linked available at the link here below:

Before installing any of the RPM package contained in that .zip archive read this page.

2 Likes
#4

UPDATE #3

This part has been further developed and now is easy to install

Tested, updated for setting the CPU governors policy and the auto-brightness settings, a quick on-the-fly installation script added.

  • 03.08.2023, first boot setup after flashing

    description: shell scripts suite for the first boot setup after flashing

    status: ready for advanced users testing, quick installation.

The settings should be fine for Xperia 10 II and III. The rationale about the CPU governors policy is here:

Feedback about this analysis are appreciated.

1 Like
#5

UPDATE #4

I rebased the mainbranch with the devel and these are the released updates:

  • 05.08.2023, RedFish OS suite installation

    description: the scripts suite has two distinct instances workstation and mobile device.

    status: ready for advanced users testing, on-the-fly installation script.

  • 06.08.2023, patch downloader and installer

    description: patch downloader and installer alpha-development completed

    status: ready for advanced users testing, patch remover yet to do.

1 Like
#6

UPDATE #5

A lot of changes in the direction of configuring the SFOS after the flashing, at the first boot has been committed into *devel" branch:

Please note that “ready for advanced users adoption” on the devel branch, it is a projection for the future moment in which the devel branch will be merged with the main branch.

The CentOS 8 repositories BaseOS and AppStream have been added but not enabled because enabling them conflicts with RPM update and installation procedures when the same package is provided by RFOS repositories and CentOS.

REQUEST TO HELP

Those like to help, there are three simple questions / tasks to answer / solve:

  1. how to advantage a repository (SFOS) over another (CentOS), if it is possible
  2. how to activate the WiFi tethering by command line
  3. how to activate the USB tethering by command line

This would relief me to investigate about these questions.

1 Like
#7

UPDATE #6

Regarding request #1, I have found three workarounds:

  a. ssu disablerepo ${repo_name}
  b. zypper install -r ${repo_name} ${rpm_name}
  c. zypper install ${rpm_name}-${rpm_version}

The approach a) requires a zypper refresh which is time consuming every time a disable/enable takes place, but protects the system from zypper update and other cases that raise a collision.

The approach b) is probably the cleanest and fastest way because it can also be used with zypper update and force the packages manager to use a set of repositories because multiple -r can be used.

The approach c) can be an option if b) fails to find all the dependencies.

At this point, I managed to add two Centos 8 stream repositories and four Fedora 31 repositories, which is the last version compatible with the glibc installed in the SFOS.

Probably the best way to go is to create a script pkgman or rpmman that fundamentally wraps zypper & co. and provides the magic of a peaceful life with three distribution repositories sets.

When the recovery image will be good enough to administrate the system, I will try the system transition, replacing as many SFOS packages as possible to drive forward Fedora 31, probably.

Why not Centos 8 stream?

  • The last update of Centos 8 stream happened on 2019-12-19, soon 4 year ago.
  • The last update of Fedora 31 happened on 2019-10-23, soon 4 year ago.

More or less both reached their end-of-life (support, updates) at the same time. The main difference is that their distro upgrades are respectively Centos 9 upstream and Fedora 32, 
, 38. Which means that with Centos there is a huge jump forward, while with Fedora seven steps.

Probably, I will try the huge jump, and if it doesn’t work, then the seven-step pathway.

Once most of the packages are replaced, it will not be SFOS anymore but a distro based on Centos or Fedora with some customized RPM mostly related to the applications and platform/hardware adaptation. The HUGE advantage is that a lot of packages will not require any kind of maintenance or update anymore, and we will be free to concentrate ONLY on those packages that make a difference.

This is the reason why I chose RedFish OS, because moving to Centos or Fedora, we are going to have another operative system, a hybrid for a while, but not anymore SFOS.

about tethering persistence

About the tethering persistence respect the SailFish Utils network restart or systemctl restart connman:

About these two, I think that there is a persistence flag somewhere in the UI files or the files the UI use as configuration.

[root@redfishos defaultuser]# grep -rni persistence .

but I did not find it in the defaultuser home. Therefore, it should be a global flag.

1 Like
#8

UPDATE #7

Today, I merged the devel branch with the main branch, and I released the system patch manager shell script suite (basic features but enough to be functional:

  • 12.08.2023, RedFish OS system patch manager suite

    description: the system patch manager scripts suite in its basic features has been completed.

    status: ready for advanced users testing, in particular about the Patch Manager role overlapping.

On the related page, you will find more information about this tool and its role.

personal addition

This release was planned for the 10th, but I took two days of relaxation to carry on one of my favorite activities: exploring new restaurants. You will find the new entry described on my LinkedIn profile—finding which is my LinkedIn profile is part of the quest—and it is not so difficult, IMHO.

1 Like
#9

UPDATE #8

On the devel branch, the suite for backup and restore of the users home folders and the whole system (or different levels of it) is available.:

  • 15.08.2023, user and system backup suite

    description: this shell scripts running on the laptop/PC provide backup and restore capabilities by SSH via USB.

    status: ready for advanced users adoption, full root filesystem backup in 1 minute.

At this point, it starts to become feasible to play with the root filesystem because it can be restored via a previous backup.

Note that these capabilities are available on the devel branch only, and some cases cannot be solved by a simple backup (e.g.: into a service.d folder a faulty.conf, which is new and therefore will not be overwritten nor deleted).

This limitation highlights the need for a recovery image that can provide an integral root filesystem restore from a full previous backup archive.

1 Like
#10

UPDATE #9

Today, I merged the devel branch with the main tagged with v0.3.4 and the most interesting news is about the boot image refactoring that now can boot in recovery or normal boot mode depending if the smartphone is connected to an USB data port, not a charger.

  • 18.08.2023, RedFish OS recovery image

    description: this boot image check for a USB data connection with a Laptop/PC for deciding about going for the recovery or normal boot mode

    status: ready for advanced users adoption, supports restore and backup shell scripts suite.

With the files in the folder, you might not being able to provide yourself a functional image because I did some more changes on it which are still missing in the repository. Those files are intended to show a how the initrd process can be changed to obtain this goal.

Those are interested in a near-future adoption of that boot image, can write me by e-mail.

1 Like
#11

UPDATE #10

I have reworked the telnet recovery menu and all the related scripts plus I made an original logo for RedFish OS with the related animations:

  • 19.08.2023, RedFish OS logo and animation

    description: the RedFish OS logo has been created and added to the recovery image with animations.

    status: ready for advanced users adoption, telnet recovery menu has been reworked.

About the menu:

  • the filesystem check now works as expected
  • added a voice to drop cache and sync filesystem
  • added a voice to power off the smartphone

The reboot and shutdown drop the cache and sync the filesystem leveraging the kernel emergency sysreq features. Keeping pressed the power button, with the RFOS the smartphone shutdown instead of rebooting. To reboot it, it is enough to connect to a USB data port or to a charger.

A bit of marketing has been anticipated also here:

The re-packaging of the recovery image and flashing it on both boot_a and boot_b partitions takes now 4 seconds and not anymore 2 seconds because after having bricked the boot due to a to quick reboot after the write, I have realized that despite the dd oflag options there was not a complete warranty that all the data was transferred. Therefore, I took the decision to attack the problem with multiple approaches 1. drop cache; 2. sysreq emergency kernel filesystem sync; 3. avoid oflag=direct but use conv=sync,fsync. Now, it seems robust enough.

Have a nice Saturday night and lovely Sunday. :blush:

#12

You have a weird definition of what refactoring means.

1 Like
#13

No one is perfect, after all


Screenshot from 2023-08-20 08-08-08

Option F has not been tested yet, might work or not. Depending by Jolla.

Options b and B are going to be developed today because I am tired to wait for the disk decryption which does not matter for those tasks.

image
image830×512 79.9 KB

As you can see, also SFOS 4.5.0.21 has problem with its LVM image and this creates some trouble also at running time. Probably, rather than a refactoring is just a HUGE bugfix plus upgrading the system to a standard distribution like Fedora or CentOS and a cosmetic restyling. Well, in fact it is a refactoring.

IMHO, it is a little hard to fix LVM partitions and volumes on the same running system and it seems not viable by the workstation acting on the image which in its full size is quite huge, like 100GB. Therefore the only viable way is to do it with the recovery image. Unfortunately, the recovery image is almost untouched since 2015 when an Italian developer made it.

I will soon add another option:

  • t) Do chroot LVM rootfs

In such a way, I will be able to start a SSHd and deal with the root filesystem without relying on a complete running system and having to deal with its services and UI but th just the bare minimum of it: the kernel and a static busybox.

#14

UPDATE #11

Finally, RedFish OS received a decent recovery menu:

image
image405×638 58.2 KB

As you can see, it could also have a best-fit to another mobile device screen. You know, Xperia 10 II and III support the USB host-mode therefore one can pilot another one.

Shrinking images like there is not tomorrow!

One more thing, that is worth a bit of attention (about details):

SFOS
RFOS
![
PNG gray 256 x 256 px (29 KB, 58 blocks)
PNG rgb 512 x 512 px (5.3 KB, 11 blocks)

These two images seem similar because of the circle with a logo in the centre. However, the size matters, especially because they are PNG images. Therefore, once included in a cpio.gz they will consume the same space as the originals.

  • 58 x 8 images = 464 blocks

Passing from grey to RGB encoding, the size would increase by a 3x factor, and because the image is 2x larger and 2x higher, the size would increase by another 4x factor:

  • 58 x 8 images x 3 x 4 = 5568 blocks

Instead, the images in RFOS have a total size of 88 blocks, and the ratio of this reworking is

  • 5568 / 88 = 63 times

Because 29 KB is one of the smaller images of the 8, the real ratio is 2⁶ = 64. The trick?

Jolla used a vector template for the animation, like SVG in the original, but trying to bring such a graphic to 16 indexed colors in total creates a lot of artifacts. The artifacts make the animation appear hugly. The solution, is to work pixel by pixel and make the 16 indexed colors look like it were in RGB. Naturally, there is automation to do this work, but the final stage is checking pixel by pixel. This is the main difference between an artwork and an image generated by a vectorial template

Just for those who think that it was just a logo replacement, and nothing else matters


#15

Have you tried piloting?

#16

Do you mean piloting an airplane with engines? Never. Just simulators, those were not just “games” but with a real physic model - In fact, I did more crashes than safe landings - yes, landing is the critical part because the most difficult. However, arriving on the airport strip without auto-pilot nor assisted guidance but just manual controls and visual guide, it is not so easy like in the film. Probably, it was so hard because nobody teach me but just me and the simulator.

#17

Maybe easier with a joystick. But i meant connecting two sailfish os phones and using recovery

#18

Although i also like flying. In theory you have to somehow strictly hold a certain airspeed and be in the glide slope, which you can control with the papi lights visually, there should be two red and two white. Too much red means you’re too low and white means you’re too high. Now the counterintuitive thing is to control airspeed with pitch and vertical speed or glide angle with throttle input.

#19

I meant connecting two RedFish OS smartphones because, with SFOS it is useless.

Connecting two smartphones one that pilots the other one. The best combination is a laptop and a smartphone. Two smartphones are just that scenario, but brought to the extreme. Instead of a laptop, you can use a Rasperri PI or anything else that can deal with a network connection via USB.

#20

UPDATE #12

Most of the news about this merge between devel and main branch has been anticipated here. Therefore, I report here just the update in the news section:

  • 21.08.2023, RedFish recovery telnet menu

    description: the RedFish OS recovery menu has been re-organised, some items need to be worked deeply.

    status: ready for advanced users adoption, telnet recovery menu has a new style.