Antarctica Starts Here.

I've mentioned in the past that my exocortex incorporates a number of different kinds of bots that do a number of different things in a slightly different way than Huginn does.  Which is to say, rather than running on their own and pinging me when something interesting happens, I can communicate with them directly and they parse what I say to figure out what I want them to do.  Every bot is function-specific so this winds up being a somewhat simpler task than it might otherwise appear.  One bot runs web searches, another downloads files, videos, and audio, another wakes up and look sat system stats every minute... but where does this all start?  How does it all fit together?

It starts with Jabber, the humble XMPP protocol.

XMPP is not new and fresh, it's not the sexy new hotness.  It's been around for a bit over twenty years so it's well understood, it's battle-tested, and there are enough implementations out there of both the client and server that you can pretty much pick whatever you want.  I started off using a couple of accounts on a public Jabber server and eventually migrated everything over to my own server running Prosody; I pretty much used these instructions and haven't looked back.

To communicate with my bots I use a number of different Jabber clients.  On my mobile devices I use Conversations.  At work I use Swift on my laptop and converse.js in my web browser.  On Windbringer I use Pidgin because I still haven't found any desktop clients for Linux that suck any less.  When I'm SSH'd into one of my shell boxes I use Profanity, which is a text-mode XMPP client that looks a lot like IRC.  I won't walk you through installing and setting up a client, but you will want to add your account (you@jabber) per the instructions for your software.  This is what you'll talk to your bots with.

Before I go any farther, I should probably describe the example I'm about to give.  Let's say that you have a Prosody server called jabber which you want to run parts of Halo on to do stuff for you.  Specifically, you want to do basic system monitoring with Systembot.  We're not going to talk about what else that server is actually doing because at this point it doesn't matter.  This is a bare-bones overview of how everything hooks together.  We're not going to assume any advanced knowledge - no automation with Ansible or Puppet, no creative programming tricks.  Where possible, I'll link to tutorials which do a better job of explaining some specific stuff so I don't overspecialize in this post.  It doesn't matter what kind of machine this is, either: jabber could be a virtual machine running in AWS, it could be a server in a rack, or it could even be an embedded Linux machine in a set-top box or smart television.

Right.

So, jabber has a Prosody server running.  We're going to set up two Jabber accounts, one for you and one for each server involved (which is to say, the box called jabber).  I strongly recommend that you make the Jabber username the same as the server's hostname so it's intuitively obvious what server maps to what account.  Each account will have its own unique password; I use my Diceware utility for this but you can use whatever you're comfortable with.  Log into jabber and run the following command to create an account for yourself: sudo prosodyctl adduser you@jabber

When prompted, enter a password for your XMPP account.  Now create one more account for the server and set a unique password on it: sudo prosody ctl adduser jabber@jabber

Now.. why?

Part of the Halo is a bot called the XMPP bridge, which logs into a Jabber server with a set of credentials you give it (one of the four accounts you just finished creating) by pretending to be an XMPP client like the one on your desktop.  The XMPP bridge also has a small embedded web server that implements a REST API.  Every bot that's supposed to talk to that bridge has a message queue named after it which holds commands sent from its owner.  The message bridge also has a message queue called /replies which every bot can send messages to that are then be relayed to the bot's owner.  As long as a bot can speak HTTP it can interact with the message bridge regardless of what you programmed it in.  If you wanted to, you could write a bot in Go or even a shell script that took orders from the XMPP bridge.

Now we need to set up the XMPP bridge on jabber.  Assuming that you're still logged in, check out the latest version of exocortex-halo/, set up an environment to hold the dependencies for the XMPP bridge, and make a copy of the sample config file:

• git clone https://github.com/virtadpt/exocortex-halo
• cd exocortex-halo/exocortex_xmpp_bridge
• virtualenv env
• source env/bin/activate
• pip install -r requirements.txt
• cp exocortex_xmpp_bridge.conf.example exocortex_xmpp_bridge.conf

When you edit the new exocortex_xmpp_bridge.conf file you just copied you're going to set a couple of options: the bot's owner, the bot's Jabber username and password (to log into Prosody), and the names of the other bots that'll communicate with it (for this example, there will be just one).  Here are the settings you're going to use:

owner = you@jabber
username = jabber@jabber
password = <jabber's password here>
loglevel = INFO
agents = Jabber

Now, start up the XMPP bridge:

(env) [you@jabber exocortex_xmpp_bridge]$ ./exocortex_xmpp_bridge.py 

If your configuration is correct, you should see something like this:

INFO: Logged into XMPP server.
INFO: Negotiating TLS
INFO: Using SSL version: TLS 1.0
INFO: JID set to: jabber@jabber/a0082f86-176b-4312-a897-10e70fbb27e0
INFO: CERT: Time until certificate expiration: 21 days, 0:24:12.159715

If your XMPP client is connected, you should get a message from the XMPP bridge as it starts up.  In another shell (if you're using Screen or tmux) or SSH session into the server called jabber, you can use cURL to talk to the XMPP bridge:

[you@jabber ~]$ curl http://localhost:8003/
{"active agents": ["Jabber", "replies"]}

What this means is that the bot you're going to set up next called Jabber is going to take its orders by contacting the URL http://localhost:8003/Jabber, and it's going to send its output to http://localhost:8003/replies.  You don't have to worry about this because it's handled for you, but it's good to keep in the back of your mind for later.  Because the XMPP bridge listens to localhost only, only things running on the same box can contact it so I didn't add HTTPS support.  Please, do yourself a favor and do not reconfigure an XMPP bridge to listen on anything else unless you absolutely have to.  I added support for that for the purpose of debugging and maybe running in some very odd enviroments but that's about it.

That said, it's time to test your first XMPP bridge.  Fire up your Jabber client, log into you@jabber and send a message to jabber@jabber:

(03:53:15 PM) You: help
(03:53:16 PM) Jabber: 
Supported commands:
- help - This online help.
- Robots, report. - List all constructs this bot is configured to communicate with.

To send a command to one of the constructs, use your XMPP client to send a message
that looks something like this:

"[bot name], do this thing for me."

Individual constructs may have their own online help, so try sending the command
"[bot name], help."

If you get output that looks like the above, it's working.  Good job!  Now let's try one of the commands:

(03:53:07 PM) You: Robots, report.
(03:53:08 PM) You: Contents of message queues are as follows:

Agent Jabber: []

You've just asked the XMPP bridge what bots it's configured to work with (which at the same time asks it what message queues it has) and what's in those messag queues (by default an empty list, denoted by []).  In case you're wondering what "Robots, report" actually means, I cribbed it from the game Suspended by Infocom.  I couldn't think of a better way to get status reports from bots on a server and haven't found anything better yet.  Far from perfect, but I don't think terrible, either.

Now we're going to set up your first bot called, unimaginatively, Jabber, which will be an instance of Systembot.  Log into the server called jabber again, or spawn a new shell, and create another virtualenv to hold Systembot's dependencies:

• cd ~/exocortex-halo/system_bot
• virtualenv env
• source env/bin/activate
• pip install -r requirements.txt
• cp system_bot.conf.example system_bot.conf

Make a copy of the file system_bot.conf.example called system_bot.conf.  Open system_bot.conf in your favorite text editor and make the following changes:

bot_name = Jabber
polling_time = 5
loglevel = info

Now start up Systembot:

(env) [you@jabber system_bot]$ ./system_bot.py 
INFO: Everything is configured.

In your Jabber client you should get the following message from Systembot, which means that your configuration was correct:

(04:31:05 PM) Jabber: Got a message from Jabber:

Jabber now online.

Now let's interact with Systembot:

(04:39:37 PM) You: Jabber, help
(04:39:41 PM) Jabber: Got a message from Jabber:

My name is Jabber and I am an instance of ./system_bot.py.

    I continually monitor the state of the system I'm running on, and will send alerts
any time an aspect deviates too far from normal if I am capable of doing so via the
XMPP bridge.  I currently monitor system load, CPU idle time, disk utilization, and
memory utilization.  The interactive commands I currently support are:

    help - Display this online help.
    load/sysload/system load - Get current system load.
...

Systembot can understand quite a few commands, so try a few out to get a feel for it:

(04:43:00 PM) You: Jabber, system load.
(04:43:01 PM) Jabber: Got a message from Jabber:

The current system load is 0.87 on the one minute average and 1.38 on the
five minute average.

(04:43:06 PM) You: Jabber, disk usage.
(04:43:11 PM) Jabber: Got a message from Jabber:

The system has the following amounts of disk space free:
    /boot - 88.76%
    / - 28.81%

(04:44:09 PM) You: Jabber, uptime.
(04:44:16 PM) Jabber: Got a message from Jabber:

The system has been online for 21 days, 21:54:56.320000.

Congratulations.  You've just added a message bridge and a bot to your exocortex.  The reason you gave a copy of Systembot the name "Jabber" is because, conceptually speaking, you're asking the server called jabber questions about its state.  Think of it as a cognitive abstraction layer.  Now let's try something a little different: One of the tools I wrote was a command line tool which lets you send arbitrary messages to the XMPP bridge on your server.  Like my other bots it uses the Requests module; for the purposes of this experiment let's install the native package for your server.  Assuming for the moment that the server called jabber is running Ubuntu or Debian (which are pretty common at cloud hosting providers), this is how you'd do it:

• cd ~/exocortex-halo/command_line_messenger
• sudo apt-get install python-requests

Pretty straightforward stuff.  Now let's do something silly: ls -alF | ./send_message.py -

In a couple of seconds, your XMPP client should pop up a window with a new message in it:

(05:05:37 PM) Jabber: Got a message from replies:

total 24
 drwxr-xr-x  2 you you 4096 May 30  2017 ./
 drwxr-xr-x 20 you you 4096 May 20 17:14 ../
 -rw-r--r--  1 you you 1970 May 29  2017 README.md
 -rw-r--r--  1 you you    9 May 26  2017 requirements.txt
 -rwxr-xr-x  1 you you 4904 May 26  2017 send_message.py*

You just listed the files in the current working directory and piped the output through send_message.py, which sent the text to the XMPP bridge, which sent it to your XMPP client.  Pretty neat, huh?  You can use this utility to send whatever text you want, and you can redirect the output of commands to it, also.  How about this?

[you@jabber command_line_messager]$ ./send_message.py --message
    "I just logged into your box and I'm running commands."

Again, in a couple of seconds you should get a message in your XMPP client:

(05:14:05 PM) Jabber: Got a message from replies:

I just logged into your box and I'm running commands.

Well, it's uncomfortably hot and I'm just about out of things to write.  I'll have another couple of posts coming in the near future about some of the more specialized and unusual bots.