Parsing simple commands in Python.

Feb 04 2017

A couple of weeks ago I ran into some of the functional limits of my web search bot, a bot that I wrote for my exocortex which accepts English-like commands ("Send me top 15 hits for HAL 9000 quotes.") and runs web searches in response using the Searx meta-search engine on the back end.  This is to say that I gave my bot a broken command ("Send hits for HAL 9000 quotes.") and the parser got into a state where it couldn't cope, threw an exception, and crashed.  To be fair, my command parser was very brittle and it was only a matter of time before I did something dumb and wrecked it.  At the time I patched it with a bunch of if..then checks for truncated and incorrect commands, but if you look at all of the conditionals and ad-hoc error handling I probably made the situation worse, as well as much more difficult to maintain in the long run.  Time for a rewrite.

Back to my long-term memory field.  What to do?

I knew from comp.sci classes long ago that compilers use things called parsers and grammars to interpret code so that it can be converted into an executable.  I also knew that the parser Infocom used in its interactive fiction was widely considered to be the best anyone had come up with in a long time, and it was efficient enough to run on humble microcomputers like the C-64 and the Apple II.  For quite a few years I also ran and hacked on a MOO, which for the purposes of this post you can think of as a massive interactive fiction environment that the players can modify as well as play in; a MOO's command parser does pretty much the same thing as Infocom's IF parser but is responsive to the changes the user's make to their environments.  I also recalled something called a parse tree, which I sort-of-kind-of remembered from comp.sci but because I'd never actually done anything with them, I only recalled a low-res sketch.  At least I had someplace to start from so I turned my rebooted web search bot loose with a couple of search terms and went through the results after work.  I also spent some time chatting with a colleague whose knowledge of the linguistics of programming languages is significantly greater than mine and bouncing some ideas off of him (thanks, TQ!)

But how do I do something with all this random stuff?

Huginn: Writing a simple agent network.

Jan 15 2017

EDIT: 20170123 - My reviewers have suggested some edits to the article, many of which I've applied.

It's been a while since I wrote a Huginn tutorial, so let's start with a basic one to get you comfortable with the idea of building an agent network.  This agent network will run every half hour, poll a REST API endpoint, and e-mail you what it gets.  You'll have to have access to an already running Huginn instance that can send outbound e-mail.  This post is going to be kind of lengthy, but that's because I'm laying out some fundamentals.  Once you understand those you can skip past the explanations and move on to the good stuff.

First, a little background - what's a REST API?  If you already know just skip down past the cut and move on, but if you don't know what I'm talking about I'll try to explain.  I'm going to assume that you've been able to install Huginn using my instructions or someone else's, or you've got access to a running instance.  I'm also going to assume that you're not a hardcore coder, you're someone who's trying to apply a useful tool to your everyday life.

At its simplest, an API (Application Program Interface) is a way to interact with a system or part of a system.  It's (hopefully) designed to be regular, which means that once you understand the basics you can apply that knowledge to figure out the more complex parts with a little messing around because the basics continue to apply.  Let's say that I've written a library called myLib, which implements a bunch of really annoying stuff (like opening and closing files and sorting elements of data) so you don't have to.  My library has a bunch of functions that carry out those tasks (openStupidFile(), readAllOfFilesContents(), sortIntegers(), sortFloatingPointValues(), searchThisCrapForAString()) when you call them in your own code.  Those functions are part of my library's API.  In the documentation are instructions for calling each function, which includes the arguments you need to pass to each function (e.g., openStupidFile() takes two arguments, a full path to a file and 'r' for read-only or 'rw' for read-write, and it returns a handle to the file that you can pass to another function or NULL if it failed).  The data type each function returns (the file handle or NULL value) is part of the API, as are the arguments each function takes (path to the file and 'r' or 'rw').

The same principle has been applied to the Web in several different ways.  What concerns us right now is something called the RESTful API (REpresentational State Transfer), which basically means interacting with a web service using HTTP verbs (GET, PUT, POST, and so forth) and referencing  URLs instead of functions in a library.  Like HTTP, requests are stateless, which means that you make a request, the server responds, and there's no further context beyond that.  You can think of RESTful APIs as fire-and-forget.  The general idea is that there is a web server of some kind, which could be a traditional one like Apache or a specialized one running inside a web app built around a server like web.py which responds to those URLs in some way.  If you make a GET request to a URL, it'll send you some data.  If you make a PUT request you replace something on the server at that URL with something you send it.  If you make a POST request you create a new something on the server.  If you make a DELETE request that something on the server gets erased.  All of this depends on the HTTP verbs the server supports (not all REST APIs need to support all of them), your access permissions (not every account can do everything), whether or not you've authenticated to the server (it is sometimes the case that read-only access doesn't require an account but read-write access does require an account or an API token or something else along those lines), or who owns a particular resource (Alice's resources are read-only for every other account on the server, but read-write for her alone), of course.  REST makes life easier but it's not carte blanche to run hog wild.  Additionally, many REST API services enforce access limits - you get so many requests per minute, hour, or day and after that it returns errors.  For example, Twitter's API will return an Error 420 (enhance your calm) if you trip their rate limiter.

The 2016 election and weird patterns on Twitter.

Dec 18 2016

You've already read my opinion of the 2016 election's outcome so I'll not subject you to it again. However, I would like to talk about some weird stuff I (we, really) kept noticing on Twitter in the days and weeks leading up to Election Day.

As I've often spoken of in the past, a nontrivial portion of my Exocortex is tasked with monitoring global activity on Twitter by hooking into the back-end API service and pulling raw data out to analyze. Those agents fire on a stagged schedule, anywhere from every 30 minutes to every two hours; a couple of dozen follow specific accounts while others use the public streaming API and grab large samples of every tweet that hits Twitter around the world.

If you want to look at a simplified version of that agent network to see how it works I've made it available on Github. As you can see, the output of that particular agent network is batched into e-mails of arbitrary size using the Email Digest Agent and is sent to one of my e-mail addresses as a single batch. The reason for this is twofold; it's easier to scan through a large e-mail and look for patterns visually than it is to scan through several dozen to several hundred separate messages in sequence, and it uses fewer system resources on my e-mail provider to store and present to me that output.

Six or seven weeks before Election Day, Lifeline (the recognition code for the agent network which carries out these sorts of tasks for me) started sending me gigantic e-mail digests every hour or so, containing something like several hundred tweets at a time (the biggest was nearly a thousand, as I recall). Scanning through those e-mails showed that most of the tweets were largely identical, save for the @username that sent them. Tweets about CNN and the Washington Post being GRU and SVR disinformation projects or on-the-ground reporting tagged with #fakenews. Links pointing to Infowars articles (the tweets consisted of the titles of posts, links, and the same sets of hashtags; if you ran the Twitter-compressed URLs through a URL unshortener they all pointed to the same posts). Anti-Bernie and anti-Hillary tweets that all had the same content and the same hashtags. Trump as the second coming messages and calls to action. Rivers of bile directed at political comentators and reporters. Links to fake Wikileaks Podesta e-mails that went to Pastebin or other post-and-forget sites (there wasn't even enough data in the fakes to attempt to validate them (by the bye, the method linked to is really easy to automate)). I saw the same phenomenon with #pizzagate tweets, only the posts came in shorter bursts more irregularly. It went on and on, day and night for weeks, hundreds upon hundreds of unique copies of the same text from hundreds of different accounts. I had to throw more CPUs at Exocortex to keep up with the flood.

All of these posts, when taken together as groups or families consisted of exactly the same text each and every time, though the t.co URLs were different (a brief digression: Twitter's URL shortening service seems to generate different outputs for the same input URL to implement statistics gathering and user tracking as part of its business strategy). Additionally, all of those posts went up more or less within the same minute. The Twitter API doesn't let you pull the IP addresses tweets were sent from but the timestamps are available to the second. If you looked at the source field of each tweet (you'll need to scroll down a bit), they were all largely the same, usually empty (""), with a few minor exceptions here and there. The activity pattern strongly suggests that bots were used to strafe circles of human-controlled accounts on Twitter that roughly correspond to memetic communities. Figuring that somebody had already done some kind of visualization analysis (which I suck at), I had Argus (one of my web search bots) do some digging and he found a bunch of pages like this study, which seem to back up my observations.

The sort of horsepower needed to create such an army of bots would be very easy to assemble: Buy a bunch of virtual machines on Amazon's EC2. Write a couple of bots using Ruby or Python. Sign up for a bunch of Twitter accounts or just buy them in bulk. Make a Docker image that'll effectively turn one EC2 instance into as many as you can reasonably run without crashing the VM. Deploy lots and lots of copies of your bots into those Docker containers. Use an orchestration mechanism like Ansible to configure the bots with API keys and command them en masse; if you're in a time crunch you could even use something like pssh to fire them all up with a single command. Turn them loose. If you've been in IT for a year, this is a Saturday afternoon project that won't cost you a whole lot, but could make you a lot of money.

"Well, yeah, there was an army of bots advertising on Twitter. What else is new?" you're probably saying.

What I am saying is simply this: This post describes a little bit about how this sort of media strategy works, what the patterns look like at the 50000 foot view, and my/our observations. I don't think I did anything really ground-breaking here, only in the sense that I used a bunch of AI systems that stumbled across what was going on by accident. It was the hardcore data scientists who did the real academic work on it (though that work is a bit inaccessible unless you're a computer geek).

Memetic warfare is here, and our social networks at the battlegrounds. Armor up.

Upgrading Ubuntu Server 14.04 to 16.04.

Oct 29 2016

A couple of days ago I got it into my head to upgrade one of my Exocortex servers from Ubuntu Server 14.04 LTS to 16.04 LTS, the latest stable release. While Ubuntu long-term support releases are good for a couple of years (14.04 LTS would be supported until at least 2020) I had some concerns about the packages themselves being too stale to run the later releases of much of my software. To be more specific, I could continue to hope that the Ruby and Python interpreters I have installed could be upgraded as necessary but at some point the core system libraries would be too old and they'd no longer compile. Not good for long-term planning.

First off, whenver you're about to do a major upgrade of anything, read the release notes so you know what you're getting yourself into. You'll also usually find some notes about all the new goodies you'll be able to play with.

In the past I've had nothing but trouble using the documented Ubuntu release upgrade process, so much so that I've had clients sign "I told you so," documents when they pressured me to do so because the procedure could reliably be expected to leave the system completely trashed, and a full rebuild was the only recourse. This time I set up a testbed in Virtualbox which consisted of a fully patched Ubuntu Server 14.04.5 LTS install. I ran through the documented upgrade process, and much to my surprise it went smoothly, leaving me with a functional virtual machine at the end of a 45 minute procedure (most of which was automatic, I only had to answer a few questions along the way). The process consisted of logging in as the root user (sudo -s) and running the updater (do-release-upgrade).

So, if it's so easy, why am I writing a blog post about it? Why worry?

Why worry, indeed. Read on.

Exocortex: Setting up Huginn

Sep 11 2016

In my last post I said that I'd describe in greater detail how to set up the software that I use as the core of my exocortex, called Huginn.

First, you need someplace for the software to live. I'll say up front that you can happily run Huginn on your laptop, desktop workstation, or server so long as it's not running Windows. Huginn is developed under Linux; it might run under one of the BSDs but I've never tried. I don't know if it'll run as expected in MacOSX because I don't have a Mac. If you want to give Huginn a try but you run Windows, I suggest installing VirtualBox and build a quick virtual machine. I recommend sticking with the officially supported distributions and use the latest stable version of Ubuntu Server. At the risk of sounding self-serving, I also suggest using one of my open source Ubuntu hardening sets to lock down the security on your new VM all in one go. If you're feeling adventurous you can get a VPS from a hosting provider like Amazon's AWS or Linode. I run some of my stuff at Digital Ocean and I'm very pleased with their service. If you'd like to give Digital Ocean a try here's my referral link which will give you $10us of credit, and you are not obligated to continue using their service after it's used up. If I didn't like their service (both commercial and customer) that much I wouldn't bother passing it around.

As serious web apps go, Huginn's system requirements aren't very high so you can build a very functional instance without putting a lot of effort or money toward it. You can run Huginn in about one gigabyte of RAM and one CPU, with a relatively small amount of disk space (twenty gigabytes or so, a fairly small amount for servers these days). Digital Ocean's $10us/month droplet (one CPU, one gigabyte of RAM, and 30 gigabytes of storage) is sufficient for experimentation and light use. To really get serious usage out of Huginn you'll need about two gigabytes of RAM to fit multiple worker daemons into memory. I personally use the following specs for all of my Huginn virtual machines: At least two CPUs, 60 gigabytes of disk space, and at least four gigabytes of RAM. Chances are, any physical machine you have on your desk exceeds these requirements so don't worry too much about it (but see these special instructions if you plan on using an ultra-mini machine like the Raspberry Pi). If you build your own virtual machine, take into account these requirements.