Reliable crash SfOS through grep command

Bug Reports
,
#1

REPRODUCIBILITY (% or how often):100%
BUILD ID = OS VERSION (Settings > About product):4.3
HARDWARE (XA2, X10, X10 II, …):XA10 II
UI LANGUAGE:German
REGRESSION: (compared to previous public release: Yes, No, ?):?

DESCRIPTION:

Using the “grep” command in the command prompt leads to system reboot

PRECONDITIONS:

Sony XA10ii with current SfOS 4.3

STEPS TO REPRODUCE:

  1. Open terminal
  2. enter (no root privilege required) sth like

grep -ri anysearchtermhere /*

  1. phone will reboot after a short time

EXPECTED RESULT:

search term is found/not found

ACTUAL RESULT:

Phone reboots.

ADDITIONAL INFORMATION:

(Please ALWAYS attach relevant data such as logs, screenshots, etc…)

Not required.

Community meeting on IRC 23rd June 2022
#2

This is “old news”: It always has been like that (retested on an XperiaX@SFOS3.2.1).

  • Note that the trigger is not the garbled output (which may confuse the terminal app):
    fgrep -r foo /* > /dev/null # triggers a reboot
  • It appears that reading some very special file triggers this, because I fail to do so with a simple find:
    find / > /dev/null # does not trigger a reboot
  • [Edit v2] The preferred (IMO correct) syntax (without the *) behaves a bit differently:
    fgrep -r foo / > /dev/null # fgrep equals grep -F
    • It has a much longer run-time (tested three times).
    • Ultimately it seems to be able to also trigger a reboot (I am not 100% sure, yet: have only seen that once) or (have seen that twice as a test result) grep dies without finishing (which also constitutes a bug, because it is not behaving as specified).

By activating the persistent log in /etc/systemd/journald.conf and triggering the bug then you should be able to pinpoint the cause.
I expect that reading some very special file to trigger this (because find does not read the file contents per default), though this also may a bug not specific to SailfishOS.

P.S.:

Further speed optimisation by using the grep options -s and -c (note that -q cannot be used, because it exits right after the first hit):
fgrep -rsc foo / > /dev/null # One might test thoroughly, if " /* " (instead of " / ") makes a difference.

And as a counterpart (not reading file contents, but evaluating every extant filename; i.e., doing the same as the fgrep -r above except for reading file contents):
find / -printf \c

1 Like
#3

Thanks for your comment. I came across this bug by chance, and probably it’s not a priority 1 thing since it does normaly not appear in a common usage scenario. However, being able to reproducably trigger a reboot as a normal user might become a security issue?
I thought it’s at least worth mentioning:-)
Cheers

#4

I cannot and do not concur, because as long as nobody understands what triggers this “grep causes OS crash / reboot”-bug, one must assume that any grep statement grepping in the filesystem may do, which does not run in confined, known-good space (e.g., ~/Documents).

#5

Can’t a -print be added to find [..] -exec grep to determine the last filename read?

#6

Yes, this might be helpful to progress with the analysis of this in conjunction with Systemd’s Journal in persistent mode.
Actually I would consider dropping grep first and see, if a
systemd-cat find / -type f,c,b -print -execdir head -qc 0 {} \;
does trigger it. Note that I could not determine how to redirect STDout to /dev/null within the -exec, otherwise one may also use cat or head -qc 1.
But likely the main issue is that logging the name of the offending file will be presumably performed after the crash, i.e., it will not be performed. Well, one may try and play with this.

P.S. / Edit: A viable route likely is the -fprint option, because no journal is needed and the output can be easily discarded:
find / -type f,c,b -fprint /filelist -execdir fgrep -sc foo {} + > /dev/null
or find / -type f,c,b -fprint /filelist -execdir head -qc 1 {} > /dev/null +
etc.
Using \; instead of + will be a lot slower, but may be necessary to exactly pinpoint the offending file.

#7

What about first building a list, then grep each file contained in this list after having echoed the filename to a second list.
Wouldn’t be the last file in the second list be the culprit one?

Create the second list

touch /foo/list_of_grepped_files.txt

Build the list of files to be greped

find . -type f,c,b -print >| /foo/list_of_files_to_grep.txt

Log and Grep the files

for line in $(cat list_of_files.txt); do; echo $line >> /foo/list_of_grepped_files.txt && grep -i anysearchtermhere $line ;done

#8

That is an extremely slow and multi- line & -step implementation of what I described under “P.S. / Edit:”.

Either way, go ahead and try, if you want. I am curious to know accessing which file triggers this.

1 Like
#9

Some additional testing revealed:

  • fgrep -rsc foo / > /dev/null # Always ends in grep hanging
  • fgrep -rsc foo /* > /dev/null # Always ends in rebooting the SailfishOS device

I currently have no idea what this difference indicates.

#10

Wouldn’t / include /.config while /* doesn’t?

#11

Yes, but all non-dotfiles should be equally grepped.
So a theory might be that grep hangs before reaching the offending file when taking the first route.

#12

Ok, didn’t crash my volla. didn’t find my search term either. sigh. EDIT: strange should have crashed. There are a number of binaries that have lines long enough to lead to grep running out of memory. hmmmm.

But, this is not a bug. it’s a constraint of grep. just open the each file in vi and /

#13
  • Which variant? Searching in / or /*?
  • Did it successfully finish or hang?

EDIT: strange should have crashed. There are a number of binaries that have lines long enough to lead to grep running out of memory. hmmmm.
But, this is not a bug. it’s a constraint of grep. just open the each file in vi and /

Nope, this is not how grep works: One can easily grep for any string in multi-megabyte binaries on much more memory constrained machines.

#14

Ah. / (recursive shouldn’t require a wildcard glob?) and it finished without match … Retrying with /* …

and that is interesting but probably a complete waste of time. I see a blinking screen:
SNIP…
apping found
grep: /boot: Permission denied
grep: /config/sdcardfs: Permission denied
grep: /d/mmc1/mmc1:0001/status: Permission denied
grep: /d/mmc1/mmc1:0001/state: Permission denied
grep: /d/mmc1/clock: Permission denied
grep: /d/mmc1/ios: Permission denied
grep: /d/sys_timer/debug: Permission denied
grep: /d/mmc0/mmc0:0001/ext_csd: Permission denied
grep: /d/mmc0/mmc0:0001/status: Permission denied
grep: /d/mmc0/mmc0:0001/state: Permission denied
grep: /d/mmc0/clock: Permission denied
grep: /d/mmc0/ios: Permission denied
grep: /d/keypad/debug: Permission denied

CTRL-C
reset
[defaultuser@VollaPhone ~]$

Hmmm. Why are we doing this?

As for memory constraints on grep, I’ve encountered it while searching through very large, encrypted blobs (proprietary wrapers around XFS filesystems) … but those were 100GB+ range. And it depends on the length/size of a line, if I remember correctly. This is, however more than 10 years ago, so I don’t have any more details.

EDIT. Just looked it up. On Linux, there should be no distinction between text and binary. By default GNU grep is in binary mode, but not binary AS text mode. That suggests that a binary with a very long no null bytes sequence could run out of memory. unless mmap is used which, I believe, is not the case?

#15

I’m actually not sure but I believe it was ‘finished’. Since it appeared that output was being interpreted (hence the flashing), it’s not so easy to discern. But, I probably didn’t need to hit CTRL-C and could just have recovered my screen with reset.

EDIT: just ran it again with a re that won’t match as often (bluescreenofdeath) …
… last line grep: /d/keypad/debug: Permission denied …

Then it sits there. I lose my patience. CTRL-C … during the entire procedure I could continue surfing in the browser.

#16

To determine the cause for the “Reliable crash [of] SfOS through grep command”.

#17

Ok. More informed speculation from #sailfishos-porters:

<T42_> for me reading from /sys/module/wlan/parameters/fwpath crash kernel, i guess XA10 II mentioned by OP ( whatever that device is) also have similar pararmeter with broken read

#18

And, does it (on you your Volla phone)?

On my XperiaX@SFOS3.2.1 a fgrep -rsc foo /sys/ > /dev/null does not crash, but the grep hangs after a while (stops using CPU as seen in top).

#19

I don’t have that path, but after repeats it appears that consistently ‘hangs’ at:

grep: /d/keypad/debug: Permission denied

#20

grep hanging there doing nothing indicates the file does not “return” any data.
Such as you might simulate with

mkfifo myfifo; grep foo myfifo ; rm myfifo # <-- grep hangs waiting for data coming through the pipe.     

grep foo /dev/zero # <-- grep hangs because /dev/zero dev-zeroes

(DO remove that myfifo file after testing, many programs do not like named unconnected pipes hanging around.)

Quite possible some of the weird files under /sys do something like this.

Also I suspect it is possible that there are infinite symlink loops under /sys which would also look like a hang. strace -e file will tell, and grep being in a loop should show quite some CPU/IO usage.

1 Like