[4.0.1.48] Cyrillic symbols in Terminal (or ssh) are not shown [busybox replaced with bash]

Bug Reports
,
#1
REPRODUCIBILITY (% or how often) : 100%
BUILD ID = OS VERSION (Settings > About product) : 4.0.1.48
HARDWARE (XA2, Xperia 10…) : XA2+
UI LANGUAGE : Russian
REGRESSION : (compared to previous public release: Yes, No, ?) : Yes

DESCRIPTION:

After update, i was remove busybox-symlinks-bash, than i install bash ( because i use some bash specific scripts, which not work in busybox)

PRECONDITIONS:

remove *busybox-symlinks-bash, install bash

devel-su
pkcon remove busybox-symlinks-bash
pkcon install bash

STEPS TO REPRODUCE:

mkdir ~/test
cd ~/test
for each in 1 2 3 4 5 6 7 8 ; do touch Файл_${each} ; done
ls

EXPECTED RESULT:

should show files list like this:

Файл_1  Файл_2  Файл_3  Файл_4
Файл_5  Файл_6  Файл_7  Файл_8

ACTUAL RESULT:

????_1  ????_3  ????_5  ????_7
????_2  ????_4  ????_6  ????_8

ADDITIONAL INFORMATION:

Снимок_Экрана_20210216_002
Снимок_Экрана_20210216_0021080×2160 273 KB

P.S.
forgot to tell:
i already tried :

export LC_ALL=`locale -a|grep ru` 
# or just export LC_ALL=ru_RU.utf8
export LANG=`locale -a|grep ru`
# or jusr export LANG=ru_RU.utf8
1 Like
#2

don’t know if this is a terminal issue, but it is happening because the terminal has no idea that there is UTF-8 filename. It is using (obviously) ASCII.
try setting the terminal to appropriate locale if possible.

#3

forgot to say:
i was already try :

export LC_ALL=`locale -a|grep ru` 
# or just export LC_ALL=ru_RU.utf8
export LANG=`locale -a|grep ru`
# or jusr export LANG=ru_RU.utf8
1 Like
#4

Confirmed. Anything beyond finnish non-ASCII characters gets replaced by question marks in the directory entries, regardless of obvious shell i18n settings. This is on 3.4.0.24, so it’s not a regression from this version.

1 Like
#5

strange, on 3.4 i missing this bug )

#6

Maybe it’s filesystem specific? I could test on btrfs, my test above was on ext4.

#7

I wonder if locales were generated for the language in question

#8

I can confirm this problem.

It’s not dependent on terminal emulator: I can observe the same effect using ssh connection. (Actually, if you look carefully, you can see Cyrillic letters in the screenshot of the terminal emulator in a message above, just not in ls output.)

Running ls | od -c shows that ls indeed outputs ? signs instead of Cyrillic letters. However, bash (from gnu-bash package) shows Cyrillic letters without problems, in command line and in output of echo *.

On the other hand, Busybox shell replaces Cyrillic letters with ? signs when trying to input them in command line editor, but still preserves them: entering echo привет results in command line looking like echo ??????, but the output is still привет. And echo * in Busybox shell shows Cyrillic letters in filenames correctly.

As /bin/ls is a symlink to /usr/bin/busybox (through busybox-symlinks-coreutils), this looks to me like an issue in Busybox (or libraries it uses).

But I also can confirm that ls was replacing Cyrillic letters with ? signs in filenames in Sailfish 3 as well (up to 3.4), so it can be also a similar issue in coreutils package (or, again, in libraries it uses).

1 Like
#9

Should presence of locales matter? As long as a character is a valid UTF-8 sequence, it should be printable, no matter what locale is used.

For exampe, in Fedora I can type LANG=C.UTF-8 ls, and it shows me all filenames with Cyrillic letters correctly. On the other hand, LANG=C.UTF-8 busybox ls in Fedora replaces Cyrillic letters in filenames with ? signs. Coincidence? I don’t think so.

1 Like
#10

I think so (man locale) and what is the output of locale -m.
When I setup LC_ALL to i.e. bg_BG.UTF-8 and do locale --help I see it in the appropriate language. Same with ru_RU.UTF-8.
In my opinion it is a problem in “ls” - bug?!
I saved a perl script and it displays the dir listing also properly

opendir(FH, '.') || die "Can't opendir: $!";
while (readdir FH) { print "$_\n"; }
closedir FH;

regards

#11

I’ve installed GNU Coreutils instead of Busybox ones: zypper in gnu-coreutils, then select option 1 (remove conflicting busybox-symlinks-coreutils package). And now ls shows filenames with Cyrillic letters correctly! So, yes, this is obviously a bug (or missing functionality) in Busybox.

1 Like
#12

Another observation that might be useful.

I’ve turned on my old Jolla-1 phone that still has Sailfish 3. It has the same problem with ls not showing Cyrillic letters in filenames. But what I did not know is that /bin/ls on Sailfish 3 is also a symlink to /usr/bin/busybox through busybox-symlinks-coreutils package. (For some reason I had a false impression that Sailfish has regular GNU Coreutils, not Busybox ones.)

But if you’re stuck with Sailfish 3 on your Jolla-1 phone, you can still install GNU Coreutils: zypper in coreutils (please note that the package in Sailfish 3 is called coreutils, not gnu-coreutils and has much older version than the one from Sailfish 4, presumably to remain under GPLv2 license).

As a sidenote, let me express a great pleasure about Jolla finally allowing up-to-date GNU packages under GPLv3 license into Sailfish 4 (although not pre-unstalled, but still available for installation using zypper).

1 Like
#13

thanks for your answers, i’ll try install gnu-coreutils , if it help - it would be enough for me now