Cross-compiling go-sendxmpp.

01 March 2021

I used to joke that the day setting up a cross-compilation environment was easy we'd be one short step away from having true artificial general intelligence. For the most part neither has happened yet. However, I must admit that Go has come pretty close to making it easy, but it's also kind of opaque unless you go all-in on Go to the exclusion of all other languages. It's not really a language that you can just toy around with, kind of like FORTH.

Long-time readers know that I'm all about XMPP as a command and control channel for my exocortex. However, when it comes to embedded environments (like my networking hardware) I've had to resort to some pretty nasty hacks to get any kind of monitoring on those machines. I periodically hunt for any kind of command-line XMPP client to play with, in the somewhat futile hope that I'll find one that'll run on OpenWRT without needing to fill up what little storage space there is on the unit with lots of dependencies. I did find one implemented as a couple of shell scripts but even after lots of hacking I was never able to get it to work.

Then, one day, I got lucky and found go-sendxmpp, a command line tool that logs into an XMPP server and can send and receive messages. You can even have it follow a source of data and send whatever comes down the pike. It was even pretty easy to compile on Windbringer to play around with, but then the question of how to cross-compile it for another hardware platform came up. My home wireless routers have ARMv7 processor cores, and the wireless bridge on the batphone has a MIPS core. Windbringer is a 64-bit Intel machine. So... how do you compile Go code on one hardware platform for a different one?

As it turns out this is surprisingly easy to do. Rather than setting up an entirely different toolchain you can set a couple of environment variables and the Go compiler will produce a fully operational executable for that hardware platform. First, though, you have to figure out what hardware platform you're building for. I SSH'd into the closest wireless router and queried the Linux kernel directly:

root@wednesday:~# head /proc/cpuinfo 
processor   : 0
model name  : ARMv7 Processor rev 1 (v7l)
BogoMIPS    : 1332.00
Features    : half thumb fastmult vfp edsp neon vfpv3 tls vfpd32 
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x4
CPU part    : 0xc09
CPU revision    : 1

ARMv7. Now we know what we're dealing with. Now to check out go-sendxmpp, following the instructions to do so:

{15:05:59 @ Sun Feb 21}
[drwho @ windbringer ~] () $ go get salsa.debian.org/mdosch/go-sendxmpp

Here's the tricky part: Actually compiling. This is the command I used:

{15:06:01 @ Sun Feb 21}
[drwho @ windbringer ~] () $ GOOS=linux GOARCH=arm GOARM=7 go build
    salsa.debian.org/mdosch/go-sendxmpp
  • GOOS=linux - Compiling for a Linux.
  • GOARCH=arm - Compiling for an ARM processor core.
  • GOARM=7 - It's running a version 7 core.

Because you are cross-compiling the Go compiler won't put its output in the place one might expect (i.e., $GOPATH/bin), it'll put it in the directory you happen to be sitting in:

{15:15:45 @ Sun Feb 21}
[drwho @ windbringer ~] () $ ls -ltr | tail -5
drwxr-xr-x drwho users   464 KB Sat Feb 20 21:12:51 2021 graphics
drwxr-xr-x drwho users    68 KB Sat Feb 20 21:22:20 2021 mp3
drwxr-xr-x drwho users    16 KB Sat Feb 20 21:23:54 2021 video
drwxr-xr-x drwho drwho     4 KB Sun Feb 21 15:05:52 2021 go
.rwxr-xr-x drwho drwho   6.3 MB Sun Feb 21 15:07:24 2021 go-sendxmpp

You can be sure that it's a cross-compiled binary by either trying to run it, or by asking the OS what it thinks the file is:

{15:16:37 @ Sun Feb 21}
[drwho @ windbringer ~] () $ ./go-sendxmpp 
bash: ./go-sendxmpp: cannot execute binary file: Exec format error

{15:16:39 @ Sun Feb 21}
[drwho @ windbringer ~] () $ file go-sendxmpp 
go-sendxmpp: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked,
    Go BuildID=m-ECc4RTyAZ9b4c3szLA/bwh366SB79bzn4dTX_7c/hDEoh_ptPwsELUe-BPO5
    jS1D_A3In21_1O8hny4b, not stripped

"ARM, EABI5 version 1." That's what we want to see. Let's copy that file over to my router and create a config file with login credentials for my XMPP server:

root@wednesday:~# ./go-sendxmpp 
2021/02/21 23:17:36 No recipient specified.

# Yay!  It ran!

root@wednesday:~# ls -alF
drwxr-xr-x    1 root     root           520 Jan 29 20:34 ./
drwxr-xr-x    1 root     root           608 Dec 31  1969 ../
-rw-r--r--    1 root     root           187 Sep  2  2019 .iftoprc
-rw-------    1 root     root           295 Dec 13 19:51 .lesshst
-rw-------    1 root     root            84 Jan 29 20:35 .sendxmpprc
-rwxr-xr-x    1 root     root       6587517 Jan 29 19:38 go-sendxmpp*
-rwxr-xr-x    1 root     root          2525 Jul  5  2019 monitoring.sh*

root@wednesday:~# cat .sendxmpprc 
username: wednesday@xmpp.example.com
password: AreTheyMadeFromRealGirlScouts?

Let's test by sending a message:

root@wednesday:~# ./go-sendxmpp drwho@exocortex.virtadpt.net
Nobody gets out of the Bermuda Triangle, not even for a vacation.
^D

Yeah!

As a proof of concept, let's cross-compile for MIPS. Again, we'll politely ask the Linux kernel running on my wireless bridge what architecture its processor core is running:

root@cyclopsis:~# head /proc/cpuinfo 
system type     : MediaTek MT7620N ver:2 eco:6
machine         : Nexx WT3020 (8M)
processor       : 0
cpu model       : MIPS 24KEc V5.0
BogoMIPS        : 385.84
wait instruction    : yes
microsecond timers  : yes
tlb_entries     : 32
extra interrupt vector  : yes
hardware watchpoint : yes, count: 4, address/irw mask: [0x0ffc, 0x0ffc, 0x0ffb, 0x0ffb]

Good, we know what we're dealing with. A little bit of googling shows that, to build for this particular hardware platform you have to refer to it as mipsle. Let's use that knowledge to cross-compile:

{15:27:05 @ Sun Feb 21}
[drwho @ windbringer ~] () $ rm go-sendxmpp 

{15:27:32 @ Sun Feb 21}
[drwho @ windbringer ~] () $ GOOS=linux GOARCH=mipsle go build
    salsa.debian.org/mdosch/go-sendxmpp

{15:28:01 @ Sun Feb 21}
[drwho @ windbringer ~] () $ ls -alF go-sendxmpp 
.rwxr-xr-x drwho drwho 6.9 MB Sun Feb 21 15:28:01 2021   go-sendxmpp*

Give it a quick test:

{15:30:29 @ Sun Feb 21}
[drwho @ windbringer ~] () $ ./go-sendxmpp 
bash: ./go-sendxmpp: cannot execute binary file: Exec format error

{15:30:57 @ Sun Feb 21}
[drwho @ windbringer ~] () $ file go-sendxmpp 
go-sendxmpp: ELF 32-bit LSB executable, MIPS, MIPS32 version 1 (SYSV),
    statically linked, Go BuildID=87xk9hqtqJFJiL0sjm3R/j6N70-Ma-o38Ga7sYKPW
    DQHhGko2R4KuTJOVapg5/jdEGRR3j9gq7Az6-T3VA, not stripped

Fantastic. However, in my case there's a problem - the wireless bridge doesn't have enough storage capacity to hold the executable.

root@cyclopsis:~# df -h /
Filesystem                Size      Used Available Use% Mounted on
overlayfs:/overlay        3.9M    900.0K      3.0M  23% /

Crap.

There are some funky tricks you can use to get the file smaller, but in this particular case it's not really worthwhile because the wireless bridge in question has four megabytes of storage on board (for the record, my wristwatch has an order of magnitude more). That isn't to say that your environment will be quite so constrained, I just happened to have a tiny mobile router on hand when I built the wireless bridge.

As far as I can tell, this particular trick should be generalizable to cross-compiling pretty much anything written in Go. I can't say for sure, I'm not a Go expert by any means, but the research I've done points to this being an officially supported technique for building for other platforms. I don't know which versions of Go it'll work for; I've tried it on v1.15.5, v1.15.0, and v1.14.0 and it seems to work as expected.

Good luck.