cli: failure to run tests inside termux

[tomac4t]

(Moved from the Slack thread for better tracking this issus.)

Describe the bug

Running ooniprobe-cli on Termux will quilckly failed on DNS lookup... Then I tried curl -v https://dns.quad9.net/dns-query command, the output is normal. The one of DoH services which ooniprobe use doesn't blocked in my region. https://gist.github.com/tomac4t/18f6b47fab366d22e1a9727956853738#file-termux_ooniprobe_output-txt (Slack message)

To Reproduce

Welcome to Termux!

Community forum: https://termux.com/community
Gitter chat:     https://gitter.im/termux/termux
IRC channel:     #termux on libera.chat

Working with packages:

 * Search packages:   pkg search <query>
 * Install a package: pkg install <package>
 * Upgrade packages:  pkg upgrade

Subscribing to additional repositories:

 * Root:     pkg install root-repo
 * X11:      pkg install x11-repo

Report issues at https://termux.com/issues

~ $ curl https://github.com/ooni/probe-cli/releases/download/v3.15.0/ooniprobe-linux-arm64 -fsSLO
~ $ chmod +x ooniprobe-linux-arm64
~ $ ./ooniprobe-linux-arm64 run im
   • Running im tests
[engine] iplookup: using ubuntu
[engine] sessionresolver: http3://cloudflare-dns.com/dns-query... Post "https://cloudflare-dns.com/dns-query": unknown_failure: lookup cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused
[engine] sessionresolver: system:///... lookup geoip.ubuntu.com on [::1]:53: read udp [::1]:46897->[::1]:53: read: connection refused
[engine] sessionresolver: https://dns.google/dns-query... Post "https://dns.google/dns-query": unknown_failure: lookup dns.google on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused
[engine] sessionresolver: https://dns.quad9.net/dns-query... Post "https://dns.quad9.net/dns-query": unknown_failure: lookup dns.quad9.net on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused
[engine] sessionresolver: https://cloudflare-dns.com/dns-query... Post "https://cloudflare-dns.com/dns-query": unknown_failure: lookup cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused
[engine] sessionresolver: http3://mozilla.cloudflare-dns.com/dns-query... Post "https://mozilla.cloudflare-dns.com/dns-query": unknown_failure: lookup mozilla.cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused
[engine] sessionresolver: https://mozilla.cloudflare-dns.com/dns-query... Post "https://mozilla.cloudflare-dns.com/dns-query": unknown_failure: lookup mozilla.cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused
[engine] sessionresolver: http3://dns.google/dns-query... Post "https://dns.google/dns-query": unknown_failure: lookup dns.google on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused
[engine] sessionresolver: https://doh.powerdns.org/... Post "https://doh.powerdns.org/": unknown_failure: lookup doh.powerdns.org on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused
[engine] iplookup: using cloudflare
[engine] sessionresolver: http3://cloudflare-dns.com/dns-query... Post "https://cloudflare-dns.com/dns-query": unknown_failure: lookup cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused
[engine] sessionresolver: system:///... lookup www.cloudflare.com on [::1]:53: read udp [::1]:37938->[::1]:53: read: connection refused
[engine] sessionresolver: https://doh.powerdns.org/... Post "https://doh.powerdns.org/": unknown_failure: lookup doh.powerdns.org on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused
[engine] sessionresolver: http3://dns.google/dns-query... Post "https://dns.google/dns-query": unknown_failure: lookup dns.google on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused
[engine] sessionresolver: https://mozilla.cloudflare-dns.com/dns-query... Post "https://mozilla.cloudflare-dns.com/dns-query": unknown_failure: lookup mozilla.cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused
[engine] sessionresolver: http3://mozilla.cloudflare-dns.com/dns-query... Post "https://mozilla.cloudflare-dns.com/dns-query": unknown_failure: lookup mozilla.cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused
[engine] sessionresolver: https://cloudflare-dns.com/dns-query... Post "https://cloudflare-dns.com/dns-query": unknown_failure: lookup cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused
[engine] sessionresolver: https://dns.quad9.net/dns-query... Post "https://dns.quad9.net/dns-query": unknown_failure: lookup dns.quad9.net on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused
[engine] sessionresolver: https://dns.google/dns-query... Post "https://dns.google/dns-query": unknown_failure: lookup dns.google on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused
[engine] iplookup: using stun_ekiga
[engine] iplookup: using stun_google
   ⨯ Failed to lookup the location of the probe error=lookupProbeIP failed: all IP lookuppers failed: [ Get "https://geoip.ubuntu.com/lookup": sessionresolver: LookupHost failed: [ <http3://cloudflare-dns.com/dns-query> Post "https://cloudflare-dns.com/dns-query": unknown_failure: lookup cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <system:///> lookup geoip.ubuntu.com on [::1]:53: read udp [::1]:46897->[::1]:53: read: connection refused; <https://dns.google/dns-query> Post "https://dns.google/dns-query": unknown_failure: lookup dns.google on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <https://dns.quad9.net/dns-query> Post "https://dns.quad9.net/dns-query": unknown_failure: lookup dns.quad9.net on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <https://cloudflare-dns.com/dns-query> Post "https://cloudflare-dns.com/dns-query": unknown_failure: lookup cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <http3://mozilla.cloudflare-dns.com/dns-query> Post "https://mozilla.cloudflare-dns.com/dns-query": unknown_failure: lookup mozilla.cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <https://mozilla.cloudflare-dns.com/dns-query> Post "https://mozilla.cloudflare-dns.com/dns-query": unknown_failure: lookup mozilla.cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <http3://dns.google/dns-query> Post "https://dns.google/dns-query": unknown_failure: lookup dns.google on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <https://doh.powerdns.org/> Post "https://doh.powerdns.org/": unknown_failure: lookup doh.powerdns.org on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused;]; Get "https://www.cloudflare.com/cdn-cgi/trace": sessionresolver: LookupHost failed: [ <http3://cloudflare-dns.com/dns-query> Post "https://cloudflare-dns.com/dns-query": unknown_failure: lookup cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <system:///> lookup www.cloudflare.com on [::1]:53: read udp [::1]:37938->[::1]:53: read: connection refused; <https://doh.powerdns.org/> Post "https://doh.powerdns.org/": unknown_failure: lookup doh.powerdns.org on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <http3://dns.google/dns-query> Post "https://dns.google/dns-query": unknown_failure: lookup dns.google on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <https://mozilla.cloudflare-dns.com/dns-query> Post "https://mozilla.cloudflare-dns.com/dns-query": unknown_failure: lookup mozilla.cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <http3://mozilla.cloudflare-dns.com/dns-query> Post "https://mozilla.cloudflare-dns.com/dns-query": unknown_failure: lookup mozilla.cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <https://cloudflare-dns.com/dns-query> Post "https://cloudflare-dns.com/dns-query": unknown_failure: lookup cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <https://dns.quad9.net/dns-query> Post "https://dns.quad9.net/dns-query": unknown_failure: lookup dns.quad9.net on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <https://dns.google/dns-query> Post "https://dns.google/dns-query": unknown_failure: lookup dns.google on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused;]; dial udp: lookup stun.ekiga.net on [::1]:53: read udp [::1]:37270->[::1]:53: read: connection refused; dial udp: lookup stun.l.google.com on [::1]:53: read udp [::1]:44659->[::1]:53: read: connection refused;]
[engine] sessionresolver: [{"URL":"http3://cloudflare-dns.com/dns-query","Score":9.090099090900012e-8},{"URL":"system:///","Score":9.099099999999995e-9},{"URL":"https://dns.google/dns-query","Score":0},{"URL":"https://dns.quad9.net/dns-query","Score":0},{"URL":"https://cloudflare-dns.com/dns-query","Score":0},{"URL":"http3://mozilla.cloudflare-dns.com/dns-query","Score":0},{"URL":"https://mozilla.cloudflare-dns.com/dns-query","Score":0},{"URL":"http3://dns.google/dns-query","Score":0},{"URL":"https://doh.powerdns.org/","Score":0}]
   ⨯ failed to run im          error=lookupProbeIP failed: all IP lookuppers failed: [ Get "https://geoip.ubuntu.com/lookup": sessionresolver: LookupHost failed: [ <http3://cloudflare-dns.com/dns-query> Post "https://cloudflare-dns.com/dns-query": unknown_failure: lookup cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <system:///> lookup geoip.ubuntu.com on [::1]:53: read udp [::1]:46897->[::1]:53: read: connection refused; <https://dns.google/dns-query> Post "https://dns.google/dns-query": unknown_failure: lookup dns.google on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <https://dns.quad9.net/dns-query> Post "https://dns.quad9.net/dns-query": unknown_failure: lookup dns.quad9.net on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <https://cloudflare-dns.com/dns-query> Post "https://cloudflare-dns.com/dns-query": unknown_failure: lookup cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <http3://mozilla.cloudflare-dns.com/dns-query> Post "https://mozilla.cloudflare-dns.com/dns-query": unknown_failure: lookup mozilla.cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <https://mozilla.cloudflare-dns.com/dns-query> Post "https://mozilla.cloudflare-dns.com/dns-query": unknown_failure: lookup mozilla.cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <http3://dns.google/dns-query> Post "https://dns.google/dns-query": unknown_failure: lookup dns.google on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <https://doh.powerdns.org/> Post "https://doh.powerdns.org/": unknown_failure: lookup doh.powerdns.org on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused;]; Get "https://www.cloudflare.com/cdn-cgi/trace": sessionresolver: LookupHost failed: [ <http3://cloudflare-dns.com/dns-query> Post "https://cloudflare-dns.com/dns-query": unknown_failure: lookup cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <system:///> lookup www.cloudflare.com on [::1]:53: read udp [::1]:37938->[::1]:53: read: connection refused; <https://doh.powerdns.org/> Post "https://doh.powerdns.org/": unknown_failure: lookup doh.powerdns.org on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <http3://dns.google/dns-query> Post "https://dns.google/dns-query": unknown_failure: lookup dns.google on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <https://mozilla.cloudflare-dns.com/dns-query> Post "https://mozilla.cloudflare-dns.com/dns-query": unknown_failure: lookup mozilla.cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <http3://mozilla.cloudflare-dns.com/dns-query> Post "https://mozilla.cloudflare-dns.com/dns-query": unknown_failure: lookup mozilla.cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <https://cloudflare-dns.com/dns-query> Post "https://cloudflare-dns.com/dns-query": unknown_failure: lookup cloudflare-dns.com on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <https://dns.quad9.net/dns-query> Post "https://dns.quad9.net/dns-query": unknown_failure: lookup dns.quad9.net on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused; <https://dns.google/dns-query> Post "https://dns.google/dns-query": unknown_failure: lookup dns.google on [scrubbed]: read udp [scrubbed]->[scrubbed]: read: connection refused;]; dial udp: lookup stun.ekiga.net on [::1]:53: read udp [::1]:37270->[::1]:53: read: connection refused; dial udp: lookup stun.l.google.com on [::1]:53: read udp [::1]:44659->[::1]:53: read: connection refused;]

System information (please complete the following information):

• Device: [redacted]
• OS: Android 11
• OONI Probe version: OONI Probe CLI v3.15.0

Additional context

Termux 0.118.0 from F-droid: https://f-droid.org/en/packages/com.termux/

[tomac4t]

@xhdix provide a workaround on Slack is work for me:

pkg install proot
termux-chroot
./miniooni example

An example output:

~ $ termux-chroot
~ $ ./ooniprobe-linux-arm64 geoip
┏━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ GeoIP lookup             ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━┛
[engine] iplookup: using ubuntu
[engine] sessionresolver: http3://cloudflare-dns.com/dns-query... Post "https://cloudflare-dns.com/dns-query": generic_timeout_error
[engine] sessionresolver: system:///... ok
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ asn: AS56046                                          ┃
┃ country_code: CN                                      ┃
┃ ip: [Redacted]                                        ┃
┃ network_name: China Mobile Communications Corporation ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
[engine] sessionresolver: [{"URL":"system:///","Score":0.9000000009099101},{"URL":"http3://cloudflare-dns.com/dns-query","Score":9.090099090900013e-9},{"URL":"https://doh.powerdns.org/","Score":0},{"URL":"http3://dns.google/dns-query","Score":0},{"URL":"https://mozilla.cloudflare-dns.com/dns-query","Score":0},{"URL":"http3://mozilla.cloudflare-dns.com/dns-query","Score":0},{"URL":"https://cloudflare-dns.com/dns-query","Score":0},{"URL":"https://dns.quad9.net/dns-query","Score":0},{"URL":"https://dns.google/dns-query","Score":0}]

@bassosimone pointed it out that the reason is probe-cli "does not use getaddrinfo, which means the Go code is trying to search for /etc/resolv.conf which is missing on the platform".

@xhdix 's screenshot:

~ $ echo $PREFIX
/data/data/com.termux/files/usr
~ $ cat /etc/resolv.conf
cat: /etc/resolv.conf: No such file or directory
~ $ cat $PREFIX/etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4
~ $ termux-chroot
~ $ cat /etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4

[bassosimone]

Thanks a lot! So, I think the core problem here is that we're cross compiling and cross compiling prevents us from linking with libc therefore the "system" resolver provided by Go does not use getaddrinfo but rather attempts to parse /etc/resolv.conf, fails, and then we see the failure you documented. I see two solutions here:

1. do not cross compile miniooni but rather compile using the Android C compiler so we can link with getaddrinfo build: do not cross compile miniooni #2119

2. write a custom DNS resolver that bypasses the one that Go would use when we're not linking with libc and such a resolver will have pre-configured UDP resolvers such as 8.8.8.8, 8.8.4.4, etc etc netxlite: consider bypassing net.Resolver completely #2118

I will certainly do 1, because it seems to me the most logical fix. I am unsure about 2, but I felt like documenting the idea anyway and put it into the icebox for future consideration.

Also, I think I should modify my normal workflow to include Termux on Android into the equation.

Thanks a lot for letting me know about this issue!