SDK 3.2: unable to deploy package to emulator with Docker based build engine

Bug Reports
#1

REPRODUCIBILITY: always
BUILD ID: 3.2.10
HARDWARE: VirtualBox emulator
UI LANGUAGE: Czech :slight_smile:
REGRESSION: No

DESCRIPTION:

I recently unistall my SDK and install it again and configure Docker-based build engine. I do it twice, before second attempt, I was deleting all (I hope) SDK configuration: rm -rf ~/.config/QtProject ~/.config/Sailfish*, but the result is the same. I cannot deploy my project to emulator VM machine using sfdk command. It works fine with physical devices.

PRECONDITIONS:

Running on Kubuntu 20.04 with docker 19.03.8, virtualbox 6.1.12

STEPS TO REPRODUCE:

  • I have these configured devices:
$ sfdk device list
#0 "Sailfish OS Emulator 3.3.0.16"
    emulator         autodetected  nemo@127.0.0.1:2223
    private-key: ~/SailfishOS/vmshare/ssh/private_keys/Sailfish_OS-Emulator-latest/nemo
#1 "Jolla"
    hardware-device  user-defined  nemo@192.168.1.101:22
    private-key: ~/.ssh/id_rsa_jolla
#2 "Intex"
    hardware-device  user-defined  nemo@192.168.1.112:22
    private-key: ~/.ssh/id_rsa_intex
  • I am able to connect to Emulator with ssh and even with sfdk device exec:
$ sfdk emulator start "Sailfish OS Emulator 3.3.0.16"
$ sfdk config "device=Sailfish OS Emulator 3.3.0.16"
$ sfdk device exec -- /bin/bash
[nemo@SailfishEmul ~]$ whoami
nemo
[nemo@SailfishEmul ~]$ sudo ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,DYNAMIC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:7e:50:7f brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,DYNAMIC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:5a:11:00:01 brd ff:ff:ff:ff:ff:ff
    inet 10.220.220.1/24 brd 10.220.220.255 scope global eth1
       valid_lft forever preferred_lft forever
  • but when I build my project, I am unable to deploy it with sfdk:
$ cd ~/SailfishOS/projects/osmscout-sailfish
$ sfdk config "device=Sailfish OS Emulator 3.3.0.16"
$ sfdk config "target=SailfishOS-3.3.0.16-armv7hl"
$ sfdk --quiet build

$ sfdk --debug deploy --sdk
sfdk: [D] About to run on build engine: "mb2" arguments: ("--device", "Sailfish OS Emulator 3.3.0.16", "--target", "SailfishOS-3.3.0.16-armv7hl", "deploy", "--sdk")
sfdk: [D] Mapping "/home/karry/SailfishOS/mersdk/targets/SailfishOS-3.3.0.16-armv7hl/" as "/"
sfdk: [D] Mapping "/home/karry/SailfishOS/mersdk/targets/SailfishOS-3.3.0.16-armv7hl" as "/"
sfdk: [D] Mapping "/home/karry" as "/home/mersdk/share"
sfdk: [D] Mapping "/home/karry" as "/home/src1"
sfdk: [D] Command after mapping engine paths: "mb2" arguments: ("--device", "Sailfish OS Emulator 3.3.0.16", "--target", "SailfishOS-3.3.0.16-armv7hl", "deploy", "--sdk") CWD: "/home/mersdk/share/SailfishOS/projects/osmscout-sailfish"
sfdk: [D] Task Sfdk::RemoteProcess(0x7ffe46c20890) transition: NotStarted --Started--> Running
sfdk: [D] Remote process ID: 39836 all data: "39836\n"
ssh: connect to host 10.220.220.1 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.3]
sfdk: [D] Task Sfdk::RemoteProcess(0x7ffe46c20890) posting Exited
sfdk: [D] Task Sfdk::RemoteProcess(0x7ffe46c20890) transition: Running --Exited--> NotStarted

EXPECTED RESULT:

package is deployed on emulator

ACTUAL RESULT:

it fails

ADDITIONAL INFORMATION:

  • my docker daemon configuration:
$ cat /etc/docker/daemon.json 
{
    "dns": [
        "10.7.3.3",
        "10.7.2.2",
        "1.1.1.1", 
        "8.8.8.8"
    ],
    "default-address-pools": [
      {
        "base" : "100.64.0.0/10",
        "size" : 24
      }  
    ]
}

May be problem in non-default address space for docker? I tried to remove this configuration, restart docker daemon, but without efect. It is necessary to have VirtualBox and Docker in same network?

My docker networks:

$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
f0edef2eb79e        bridge              bridge              local
d9198c269aa9        host                host                local
382503fd7e21        none                null                local

$ docker network inspect f0edef2eb79e d9198c269aa9 382503fd7e21
[
    {
        "Name": "bridge",
        "Id": "f0edef2eb79ec7da9e5ecb01ce2bb8fc1bc8efbe78d5e5207737d1549eab1313",
        "Created": "2020-08-29T15:09:02.982828619+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "100.64.0.0/24",
                    "Gateway": "100.64.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "bcb601c834d32c84e7925e8706224e68e5e40c88758be1b6d02f6ff3c0503ba9": {
                "Name": "sailfish-os-build-engine",
                "EndpointID": "2d5a786a90aa1fa4f3bbd28ac7f1f75baf8644311636227eec18be263f043143",
                "MacAddress": "02:42:64:40:00:02",
                "IPv4Address": "100.64.0.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    },
    {
        "Name": "host",
        "Id": "d9198c269aa99dde21bcbce25318f9dba8aa8febebeca1a91033954724f66ae0",
        "Created": "2019-12-13T14:54:48.742810928+01:00",
        "Scope": "local",
        "Driver": "host",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": []
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {},
        "Labels": {}
    },
    {
        "Name": "none",
        "Id": "382503fd7e21ef6b8696ac3d81bda22707eae8724e8c13294c6deef13bd12132",
        "Created": "2019-12-13T14:54:48.699517625+01:00",
        "Scope": "local",
        "Driver": "null",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": []
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {},
        "Labels": {}
    }
]
  • network configuration on host:
$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s31f6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 10:65:30:24:e6:89 brd ff:ff:ff:ff:ff:ff
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 34:41:5d:6b:16:1f brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.113/24 brd 192.168.1.255 scope global dynamic noprefixroute wlan0
       valid_lft 316577sec preferred_lft 316577sec
    inet6 fe80::7cff:ba67:d657:c983/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 10:65:30:97:6c:04 brd ff:ff:ff:ff:ff:ff
5: vpn0: <POINTOPOINT,MULTICAST,NOARP> mtu 1406 qdisc fq_codel state DOWN group default qlen 500
    link/none 
6: vboxnet0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 0a:00:27:00:00:00 brd ff:ff:ff:ff:ff:ff
7: vboxnet1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 0a:00:27:00:00:01 brd ff:ff:ff:ff:ff:ff
9: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:57:9d:df:15 brd ff:ff:ff:ff:ff:ff
    inet 100.64.0.1/24 brd 100.64.0.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:57ff:fe9d:df15/64 scope link 
       valid_lft forever preferred_lft forever
35: vetha13cf69@if34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default 
    link/ether 5e:e2:50:d7:a6:9f brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::5ce2:50ff:fed7:a69f/64 scope link 
       valid_lft forever preferred_lft forever
#2

Have you tried executing sfdk engine exec ssh nemo@10.220.220.1? If you get " ssh_dispatch_run_fatal: Connection to 10.220.220.1 port 22: message authentication code incorrect" back, you may have the same problem as me:


`

#3

I tried it and got network timeout.

$ sfdk engine exec ssh nemo@10.220.220.1
ssh: connect to host 10.220.220.1 port 22: Connection timed out
Error running command on the build engine: Remote process crashed.

My problem is that this network is not reachable even from host system, so it is not routed even from docker container… I don’t understand how it should work :slight_smile: Some vboxnet* interface should be configured statically on host, I guess. But what component should be responsible for that?

#4

Ah, I read too quickly and overlooked some parts. For me Docker networks just works, without any configuration (on OpenSUSE tumbleweed at least). Are you sure there isn’t a firewall intervening somewhere?

Edit: once again I’m reading too fast. Haven’t tried deploying from Docker to the Emulator yet, since VirtualBox is broken on my distro.

#5

Indeed the issue is caused by your non-default network configuration. The connection to emulator is handled by DNAT rules, which are set up in /usr/libexec/sdk-setup/dnat-emulators (inside the build engine). Inside that file you will find a line starting with “DEF_EMULATOR_IP”. Replace the IP address on that line with 100.64.0.1 and restart the build engine. That should help.

#6

So, what I did:

  • create backup tag of current build docker:
docker tag sailfish-os-build-engine sailfish-os-build-engine:backup
  • create Dockerfile with IP replacement:
FROM sailfish-os-build-engine
RUN sed  's/172.17.0.1/100.64.0.1/' -i /usr/libexec/sdk-setup/dnat-emulators
  • build new image:
docker build --tag sailfish-os-build-engine .
  • kill running container, try sfdk deploy --sdk again… and there is no change (emulator is running).
$ sfdk deploy --sdk
ssh: connect to host 10.220.220.1 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.3]

Firewall in container seems to be updated:

karry@LatitudeMachine:~/SailfishOS/projects/osmscout-sailfish$ docker ps
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS                                          NAMES
b812d9f5ec53        sailfish-os-build-engine   "/usr/bin/setarch i3…"   9 seconds ago       Up 3 seconds        0.0.0.0:2222->22/tcp, 0.0.0.0:8080->9292/tcp   sailfish-os-build-engine
karry@LatitudeMachine:~/SailfishOS/projects/osmscout-sailfish$ docker exec -it b812d9f5ec53 /bin/bash
[root@b812d9f5ec53 /]# iptables-save 
# Generated by iptables-save v1.8.2 on Mon Aug 31 07:28:14 2020
*nat
:PREROUTING ACCEPT [17:4478]
:INPUT ACCEPT [5:300]
:OUTPUT ACCEPT [19:1192]
:POSTROUTING ACCEPT [21:1312]
-A OUTPUT -d 10.220.220.1/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 100.64.0.1:2223
-A OUTPUT -d 10.220.220.2/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 100.64.0.1:2224
-A OUTPUT -d 10.220.220.3/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 100.64.0.1:2225
-A OUTPUT -d 10.220.220.4/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 100.64.0.1:2226
-A OUTPUT -d 10.220.220.5/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 100.64.0.1:2227
-A OUTPUT -d 10.220.220.6/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 100.64.0.1:2228
-A OUTPUT -d 10.220.220.7/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 100.64.0.1:2229
-A OUTPUT -d 10.220.220.8/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 100.64.0.1:2230
-A OUTPUT -d 10.220.220.9/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 100.64.0.1:2231
-A OUTPUT -d 10.220.220.10/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 100.64.0.1:2232
COMMIT
# Completed on Mon Aug 31 07:28:14 2020

Should not be emulator address 10.220.220.1 reachable from the host? It is not :frowning:

#7

It should not. But your question made me realize, that there is still one piece of the puzzle missing: The NAT configuration of the emulator virtual machine. If you open the settings of the virtual machine (In Oracle VM VirtualBox Manager), you will find “Port Forwarding” in “Network” settings. There is a rule “guestssh4docker”, which quite likely has “172.17.0.1” as the host IP. You should change that to 100.64.0.1 as well.

#8

Thank you @vige! It works now and I finally know how :slight_smile: