Antarctica Starts Here.

EDIT: 20170123 - My reviewers have suggested some edits to the article, many of which I've applied.

It's been a while since I wrote a Huginn tutorial, so let's start with a basic one to get you comfortable with the idea of building an agent network.  This agent network will run every half hour, poll a REST API endpoint, and e-mail you what it gets.  You'll have to have access to an already running Huginn instance that can send outbound e-mail.  This post is going to be kind of lengthy, but that's because I'm laying out some fundamentals.  Once you understand those you can skip past the explanations and move on to the good stuff.

First, a little background - what's a REST API?  If you already know just skip down past the cut and move on, but if you don't know what I'm talking about I'll try to explain.  I'm going to assume that you've been able to install Huginn using my instructions or someone else's, or you've got access to a running instance.  I'm also going to assume that you're not a hardcore coder, you're someone who's trying to apply a useful tool to your everyday life.

At its simplest, an API (Application Program Interface) is a way to interact with a system or part of a system.  It's (hopefully) designed to be regular, which means that once you understand the basics you can apply that knowledge to figure out the more complex parts with a little messing around because the basics continue to apply.  Let's say that I've written a library called myLib, which implements a bunch of really annoying stuff (like opening and closing files and sorting elements of data) so you don't have to.  My library has a bunch of functions that carry out those tasks (openStupidFile(), readAllOfFilesContents(), sortIntegers(), sortFloatingPointValues(), searchThisCrapForAString()) when you call them in your own code.  Those functions are part of my library's API.  In the documentation are instructions for calling each function, which includes the arguments you need to pass to each function (e.g., openStupidFile() takes two arguments, a full path to a file and 'r' for read-only or 'rw' for read-write, and it returns a handle to the file that you can pass to another function or NULL if it failed).  The data type each function returns (the file handle or NULL value) is part of the API, as are the arguments each function takes (path to the file and 'r' or 'rw').

The same principle has been applied to the Web in several different ways.  What concerns us right now is something called the RESTful API (REpresentational State Transfer), which basically means interacting with a web service using HTTP verbs (GET, PUT, POST, and so forth) and referencing  URLs instead of functions in a library.  Like HTTP, requests are stateless, which means that you make a request, the server responds, and there's no further context beyond that.  You can think of RESTful APIs as fire-and-forget.  The general idea is that there is a web server of some kind, which could be a traditional one like Apache or a specialized one running inside a web app built around a server like web.py which responds to those URLs in some way.  If you make a GET request to a URL, it'll send you some data.  If you make a PUT request you replace something on the server at that URL with something you send it.  If you make a POST request you create a new something on the server.  If you make a DELETE request that something on the server gets erased.  All of this depends on the HTTP verbs the server supports (not all REST APIs need to support all of them), your access permissions (not every account can do everything), whether or not you've authenticated to the server (it is sometimes the case that read-only access doesn't require an account but read-write access does require an account or an API token or something else along those lines), or who owns a particular resource (Alice's resources are read-only for every other account on the server, but read-write for her alone), of course.  REST makes life easier but it's not carte blanche to run hog wild.  Additionally, many REST API services enforce access limits - you get so many requests per minute, hour, or day and after that it returns errors.  For example, Twitter's API will return an Error 420 (enhance your calm) if you trip their rate limiter.

Enough theory.  Let's play around a little.

I thought about throwing together a little REST API service to play around with using the BaseHTTPServer module, but I figured that somebody had probably already done that.  A little searching revealed the JSONPlaceholder, which has a tiny REST API service at https://jsonplaceholder.typicode.com/ that everybody can play around with.  So, we're going to use that.  Open this URL in your browser: https://jsonplaceholder.typicode.com/posts/1

What you see in your browser window is a good example of what a REST API returns when you make a GET request - you see some data in key/value format.  When you load that document and reference a key (let's take $.title as an example) you'll get the value "sunt aut facere repellat provident occaecati excepturi optio reprehenderit."  Ask for a key, get a value.  Pretty simple.

Let's set up a scenario to hold our example agent network in Huginn.  Log into your Huginn instance and click on Scenarios at the top of the screen.  If you haven't been playing around with Huginn you won't see anything here.  Let's create a scenario to keep things nice and neat: Scroll to the botton and click the New Scenario button.  Give it the name "Test Scenario" (no quotes, let's keep life easy) and click the Save Scenario button.  Huginn will display a screen called "Test Scenario Scenario" with a bunch of empty columns (Name, Age, Schedule, Last Check, Last Event Out, dot dot dot).  If you don't see this you can navigate to it manually (Scenarios -> Test Scenario).  It's empty but we'll fix that.

Step one: Get data into Huginn.  We do that with Website Agent, which is designed to interact with websites as its name implies.  Put the mouse cursor on Agents at the top of the screen, then click New Agent.  Click the "Select an Agent Type" drop-down and search for Website Agent by typing the first few letters; click on it to select it.  I like to give my agents names that reflect the scenario they're part of because it makes them easier to keep track of in the long run, so give this agent the name "Test Scenario - Hit JSON Placeholder".  Click on the Schedule drop-down and set it to run every 30 minutes (you can change this if you need to, 30 minutes is good to start with).  The Keep Events field tells Huginn how long to keep the data this agent collects in the database; "Forever" is just that, which means that eventually you'll run out of disk space.  I keep events for a week, usually, two weeks on the outside.  For our purposes, set it to an hour.  The Sources field is a way of setting the name of an agent whose events it'll ingest.  This is handy when adding stuff to fairly large agent networks but it doesn't hurt to leave it blank.  The Recievers field is much the same, only it tells the new agent which existing agent to send events to.  Same use case, same "it won't hurt anything to leave this blank."  The Scenarios field is the name of the scenario or scenarios (an agent can be a member of more than one scenario but that makes life difficult in the long run - I recommend against it) the agent is part of.  Type the first few letters of "Test Scenario" and click on it when it comes up.

A brief digression: If you don't hook stuff up right the first time, you won't break anything.  Honest.  Events will back up in the database untl there is a way to move them into another agent, and the moment it can the pipeline will clear.  You can also delete events from an agent to clear stuff out and try again if you need to.  We're playing around here, so treat it like play: You're not going to hurt anything, and every new thing you do will teach you something new.  If you need permission, you have my permission to stop treating it like serious business and start having fun.

Now we've got a body for this agent, so let's give it a brain.  Scroll down a bit and you'll see the agent's configuration editor, which is basically a souped-up JSON editor.  The first option we're interested in is expected_update_period_in_days, which is the number of days which should be able to pass before the agent decides that something is wrong with it and disables itself.  A good value for this is 365.  The second (and most critical) is url, which is the URL of the REST API that the agent is going to contact.  Copy and paste the URL https://jsonplaceholder.typicode.com/posts/1 into the field.  You'll also want to load that URL in a separate browser tab because you might need to refer back to its output later.  The type field is the type of data that the service will return; your options are XML, JSON, text (raw text - nothing special about it at all) and HTML (in case you find yourself needing to scrape and parse raw HTML).  Set it to "json".  The value of mode specifies when the agent should emit events ("all" seems to mean "always", "on_change" means "whenever the result from the server changes from last time", and "merge" means "take the output from the service and merge it with the event that triggered this one, then emit resulting bundle of data").  Set it to "all" for the moment.

Now we need to clear out the bits we don't need because the agent is set up by default for HTML parsing (which is icky, let me tell you, but that's a tale for another time) and we want JSON parsing.  If you scroll down, in the online help on the right-hand side you'll see a paragraph called "Scraping JSON."  That's the part we care about.  So, we're going to use the JSON editor's shortcuts: If you look on the line that says "hovertext:", at the very end there's a minus sign inside the red icon.  Click that.  It deletes an entire block of stuff we don't need.  Do the same thing on the "title:" line and one more time on the "url:" line.  The agent editor should look like this.  Perfect - a tabula rasa for you to work with.  Time to pick apart the JSON document.  I like to use jsonpath.com for that, because I can interactively pick out the stuff I want.  It's a bit of black art which takes some trial and error to learn, but once you get the pattern down the process becomes second nature.  Select everything in that page you opened and paste it into the "JSON" window.  The format of that JSON document you grabbed is this (follow along, it's easy): You have an { object } containing a userId, an id, a title, and a body.

  {
    "userId": 1,
    "id": 1,
    "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
    "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit..."
  }

(Note: The \n parts are line breaks.)

For the purposes of our experiment, we want the "id" and "title" fields.  So, put your cursor in the "JSONPath Syntax" line in your jsonpath.com tab and delete everything.  The top level of the JSON document is represented with a dollar sign, so type one in there (but DO NOT hit enter - that'll wipe everything out!) Periods are used to separate the parts of the document (think of them like steps).  Next, you want the ID code of that post (just pretend it's a blog post), so you'll type a period, which means "we're going one level down" and the word "id", which is the key in the document that you want the value for.  If you look at the Evaluation Results window, you'll see that its contents changed - it now has the contents of the JSON document, sort of.  As you change the contents of the JSONPath Syntax line, the contents of the Evaluation Results window will change in response to show you what you're getting so you know if you're on the right track.  Your JSONpath will look like this: $.id

Go back to your Huginn tab and just below the "Extract" part of the agent editor you should see a green button with a white plus sign inside of it.  If you click on that you'll add a line to the extract bit, which will let you specify a name for what you're picking out of the JSON.  You could get cute and call it "fred" or something, but do yourself a favor and be descriptive.  I recommend calling it "id" and leaving it at that.  Next to it you'll see two question marks, which means that the key's waiting for a value.  In this case, the key's going to be a JSON path into the document.  If you look closely you'll see that at the far end of that line is a {, a [, and a red circle with a minus sign (which you've used previously to delete stuff).  If you scroll down a bit, in the online help you'll see that it's expecting another { object }, with a key called "path" and a value containing the JSON path.  So, click the { and you'll be presented with a new pair of curly braces and another blank line starting with "??".  In place of the "??" type the word "path".  The cursor will jump to the other part of that line, which means it's asking for a value.  Type in your JSON path: $.id

Now let's run a test to see what we get.  Click the button that says "Dry Run", which will trigger your Website Agent and show you what happens.  A window will pop up asking you for an event to send.  While this is useful under some circumstances, don't worry about it, just hit the Dry Run button.  You'll be shown exactly what Huginn sees internally: A JSON document consisting of the key "id" and the value 1.  Perfect.  Now we want the title of our sample blog post.  We're going to use the same general JSON path, only we're going to change "id" out for "title", because if you scroll back a bit you'll see that "title" is one of the keys in the JSON that represents our hypothetical blog post.  Click the little 'x' button in the top-right corner of the pop-up window and then click the little green plus icon beneath "extract" again.  You'll get a brand-new blank line, which you'll fill out using the exact same pattern if not text: You'll be prompted for the name, so give it "title".  Click the curly brace at the end to open a new object and give it the name "path".  The value for "path" will be the JSON path, which you'll be freehanding: $.title

Roughly speaking, that JSON path means "Relative to the top level of the document ($), give me the value of the key called title (.title)."  Your agent editor should look something like this.  As before, click the Dry Run button, and then the Dry Run button in the pop-up window.  You should see a little block of JSON that looks like this:

[
  {
    "id": 1,
    "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit"
  }
]

Perfect.  You did it.  This is what an event emitted by your test Website Agent will look like.  Get rid of the Dry Run Results popup window as before and click the Save button to commit your agent to the database.  You'll get dropped back to the Your Agents window.  Click on the little "Test Scenario" lozenge at the end of your agent's title to go back to the screen which shows only the agents in the current scenario.  If you wanted to, you could go back and reference a few more JSON keys in the REST API's output, or edit the output text to display things a little differently.  There's no reason that an agent has to remain static once you've got it working, you can make changes if you need to.  My advice to you on this point is to make one change at a time, test it, and then make the next.

A bit complex?  Maybe.  I could just have given you what to put in there, but then you'd have less of an idea of how to build your own stuff later, and ultimately that's the point.  You wouldn't have any examples for anything else you wanted to do and nothing to copy from, plus if you downloaded somebody else's you'd have a hard time figuring out what all the components did.  Additionally, once you have a single agent up and running you can clone it over and over and just tweak the particulars (like the URL it contacts) to save time.  If you spend the time getting it right once you can reap the benefits later.

You've got an input, now you need an output.  Remember how I said that you needed working outbound e-mail from your Huginn instance?  We're going to create an agent that sends e-mail every time it receives an event from your Website Agent.  Click on Agents at the top, then New Agent.  This time you want to pick an Email Agent (start typing and it'll sort through them until it's got some hits, then click on the appropriate title.  Give it the name "Test Scenario - Send Email".  Again, you only want it to store events for an hour, to prevent cluttering up your database.  This agent you want to connect to your Website Agent, so click in the Sources box, and when it pops up click the "Hit JSON Placeholder" entry when it appears; if this is a brand-new instance it'll be the only agent, but occasionally you'll have to hunt around a little.  The "Propagate Immediately" tickybox means that the agent will immediately send an event to the next in the chain when it fires; ordinarily agents wait for the next minimum schedule cycle (one minute) before it does so.  My advice is to not worry about it, it's not terribly useful.  Put the agent into the Test Scenario in the appropriate box.

Now you have a choice to make: You can save the Email Agent as-is and be done with it, in which case it'll e-mail every key in the event it's given without any sort of formatting or context, or you can copy the contents of the event into a message in such a way that it fills in blanks in the text.  This example is going to show you the latter, because making use of the templating system (a very simple language called Liquid) is what lets you chain lots of agents together by referencing pieces of events in other places.  In the agent editor, change the value of the expected_receive_period_in_days field to 365, just like before.  Now, if you scan through the docs on the right-hand side, you'll see that the body field isn't required, but if you add it that'll create a formal e-mail body.  Hit the green plus bubble and call the new field body.  In the text field, type in the following: "Blog post {{ id }} has gone up with the title {{ title }}." (no double quotes)

The {{ double curly brackets }} tell the agent that the words within are a template variable that should be swapped out with the value of the matching variable on the right hand side... scroll all the way down and you'll see a smaller box with the heading Agents::WebsiteAgent and a truncated view of the events it expects to receive.  So, by putting the names of those fields into the appropriate locations in the text, you'll get the values of those fields.  Hit the Save button.  You're back to the Your Agents screen.

Let's test things out.  Click on your "Test Scenario - Hit JSON Placeholder" agent to bring up the agent overview.  Now click on the Actions drop-down on the left-hand side and select Run to intentionally trigger the agent.  If it ran as expected, you should see "Last event created: <1m ago", which means that it ran, it got data (the JSON document from JSONplaceholder), processed the data, built an outbound e-mail, and transmitted it.  Now, if you know you have e-mail set up correctly, sit back and wait for a few minutes.  You'll eventually receive an e-mail from your Huginn instance with the title "You have a notification!" and a message body of "Blog post 1 has gone up with the title sunt aut facere repellat provident occaecati excepturi optio reprehenderit."  Good work, you just built your first agent network.

Now, one more trick.  Looking at a list of agents isn't terribly interesting.  Assuming that you have Graphviz set up correctly (and you should, it's pretty easy), click on the View Diagram button.  Huginn will display a handy visual map that shows agents as bubbles and the data flows between them as arrows.  As your scenarios get bigger and bigger, you'll be able to pan around and see how things are connected.  You can also click on an agent in the visual map and go right to its overview page.

Really the only other advice I have for you is to read the online help for each agent on the right-hand side.  It'll tell you what you need to know about each kind of agent as well as fill you in on some less obvious stuff and non-standard configuration options, which as time goes by you'll be making a lot of use of.  If you want to see what this agent network looks like as a whole, I've put an exported copy of Test Scenario in my Github repository of agent networks here.  Please, feel free to mess around with it, tweak it, and build on it.