[4.1.0.23] Android 10 has broken DNS

[benaryorg]

REPRODUCIBILITY (% or how often): 100%
BUILD ID = OS VERSION (Settings > About product): 4.1.0.23
HARDWARE (XA2, Xperia 10…): XA2 (H3113)
UI LANGUAGE: English (UK)
REGRESSION: (compared to previous public release: Yes, No, ?): Yes

DESCRIPTION:

Disclaimer: Having freshly flashed the device I was still in the process of downloading the latest update while starting to configure all the bits and pieces, which means I cannot 100% rule out the possibility until the factory reset is done, but I’m pretty sure.

I’m running NAT64&DNS64 locally, with no IPv4, only IPv6 with public IPs assigned through SLAAC.
Everything does seem to handle that pretty well, however as soon as I had upgraded to 4.1, most notably with the Android 10 support, my Android apps were not able to resolve DNS anymore.
Example from Aptoide:

05-19 21:05:38.539  1246  1408 W System.err: cm.aptoide.pt.dataprovider.exception.NoNetworkConnectionException: java.net.UnknownHostException: Unable to resolve host "ws75.aptoide.com": No address associated with hostname

Poking more deeply into the inner workings (by the way, nice job with that LXC, looks very neat) it seems that Android at some point after 8.0 may have removed the known cogs for tweaking DNS settings (i.e. net.dns1 and friends are all gone apparently).
The native Android settings yield the DNS settings to be set to “automatic” by default, and setting them to “off” or e.g. “dns64.dns.google” or “dns64.cloudflare-dns.com” does not seem to fix anything.
There isn’t much to go by other than anything and everything within that LXC container reporting resolution failures, despite having a public IPv6 address with no firewall whatsoever (i.e. that thing is public in the internet and can reach everything it wants, as long as it’s either IPv6, or it resolves via DNS64 to a 64:ff9b::/96 IP) and pings work just fine.
More probing sadly didn’t yield anything more, and I can’t seem to get anything useful out of tcpdump either, it almost looks like the container bluntly decided to not even try resolving any DNS record at all.

PRECONDITIONS:

A working NAT64/DNS64 setup I guess?
Seeing as IPv6-only networks are getting more traction in mobile environments this might be more common, although in my case it’s WiFi.
Switching over to a regular NAT4 WiFi fixes everything.

STEPS TO REPRODUCE:

0. factory reset your device to 4.0 & setup NAT64&DNS64 WiFi
1. connect to WiFi
2. use Android apps (notice: they work)
3. update to 4.1
4. use Android apps (notice: they can’t resolve DNS names)

EXPECTED RESULT:

Android Apps can resolve hostnames and do so via DNS64 (i.e. the way SailfishOS itself does).

ACTUAL RESULT:

Android Apps cannot resolve any hostnames.

ADDITIONAL INFORMATION:

Things I tried:

• setting the net.dns properties at runtime
• adding the net.dns properties to /usr/sbin/alien-generate-properties.sh
• rebooting, restarting Android Support, reinstalling Android support, etc.
• setting the “Private DNS” option to Cloudflare’s DNS64, or Google’s DNS64, or turning off the Private DNS entirely (i.e. not “automatic”)
• running tcpdump, but it seems the container doesn’t do anything at all while the resolution errors are being logged (sudo lxc-attach -n aliendalvik -- tcpdump -Ani any 'not port 22')

Factory reset yielded that it works at 4.0, but not at 4.1, with no changes other than the upgrade.

[wickedsp1d3r]

It seems like Android apps are no longer using /etc/resolv.conf for DNS resolving under Kvarken (4.1.0.24).
Does anyone know from where they take their configuration now?

[benaryorg]

To me it seems (though I can’t say for sure) that Android only respects the interface specific configuration (i.e. the extended WiFi dialogue) and the global DNS settings.
The latter doesn’t seem to do anything for me, while the former isn’t applicable with aliendalvik (since the interface is passed through to the container directly so there is no network setup running on Android side).

Those are a lot of guesses and assumptions of course.
However if this holds true then I suspect that the Android “way to go” of using the global DNS settings (which do DoT rather than regular DNS) does not work because the initial resolution of those DNS servers fail due to the absence of an interface-specific DNS setting.

Regardless of any suspicions of mine it very likely needs an upstream (aliendalvik) fix which reimplements/overrides the Android internal DNS resolver with one that specifically queries the resolver running on localhost, which might work without too much trouble if the network namespace is not fully separated.

[iDont]

Just want to add that this also affects VPN configurations that push a DNS server address to the client (Sailfish).

All of my Android apps can’t resolve hostnames behind my VPN server anymore starting from Sailfish 4.1, since my own DNS server isn’t queried by them. Native apps resolve them fine though.

[iDont]

Just a quick note that this regression still exists in the Sailfish 4.3 update, for those interested.

[iDont]

aliendalvik’s DNS resolver still doesn’t seem to respect the DNS server address pushed by the VPN server to Sailfish with the latest 4.5 update.

Unfortunately, my workaround of editing and then bind-mounting Sailfish’ /etc/hosts over the one from aliendalvik also does not seem to work anymore with 4.5:

[root@Xperia10III defaultuser]# cat /var/lib/lxc/aliendalvik/extra_config
lxc.mount.entry = /etc/hosts system/etc/hosts none bind,create=file 0 0

If anyone has another workaround, please let me know!

[miau]

The config file for the hosts entry has changed. Take a look here:

SFOS 4.5 feedback thread

Ok, found out how it works (@peterleinchen interested??). Start app-support, ps auxw|grep lxc shows you –rcfile=/tmp/appsupport/aliendalvik/config, there in you have lxc.include = /tmp/appsupport/aliendalvik/10-bsp_config. Hunting the source of this, since this is volatile. systemctl status aliendalvik gives you the script /usr/lib/systemd/system/aliendalvik.service, there in Requires=aliendalvik-prepare.service and there in …"$CONTAINER_CONFIG_PATH/10-bsp_config". Bingo! Added thereafter: e…

[kan_ibal]

@iDont

aliendalvik’s DNS resolver still doesn’t seem to respect the DNS server address pushed by the VPN server to Sailfish

As far as I understand /etc/hosts doesn’t solve anything in this regard. You need force aliendalvik to use /etc/resolv.conf instead.

[iDont]

Actually it does for my use case. I hardcode my internal DNS names to a local ip in the hosts file. The ip addresses I set get routed correctly when I’m using both WiFI or Mobile Data + VPN. I just want to access my own services from Android apps when I’m on the road.

Ideally, aliendalvik would just query the DNS server that is pushed by the VPN server, so the same DNS names would resolv to the same ips without hardcoding them. This used to work in earlier versions than Sailfish 4.1.
As far as I know, there is no way to make aliendalvik respect the resolv.conf from Sailfish, which does respect the DNS server.
If you know any way, I suppose this bug report can be closed with that fix

[benaryorg]

Small update, there seems to be more to it than that on Android 10.
Opening something like Fennec I can clearly see DNS requests, opening a range of other apps including Conversations I see nothing at all.
Those apps that cause no internet traffic whatsoever seem to be the exact apps that rely on the internet connectivity reported by Android itself (i.e. app asks “can I internet?”, Android says “yes you can”).
In this case however the Android API does not report connectivity, only partial connectivity at best.
So the issue is not that apps can’t access the internet, it seems they simply won’t because to them it looks like they don’t have internet.

On a network engineering level I could work around virtually every issue here, but AppSupport reporting no connectivity causes the apps to not do anything to start with, which is pretty much the only thing I can’t work around.

Edit: by the way this also happens if the SailfishOS WiFi connection reports full connectivity, which is something you can achieve by turning off the IPv4 in the connman configuration files, otherwise it will report limited connectivity it seems.