Integrating Huginn with a Matrix server.

Jan 19 2020

Throughout this series I've shown you how to set up a Matrix server and client using Synapse and Riot, and make it much more robust as a service by integrating a database server and a mechanism for making VoIP more reliable.  Now we'll wrap it up by doing something neat, building a simple agent network in Huginn to post what I'm listening to into a Matrix Room.  I have an account on libre.fm that my media players log to which we'll be using as our data source.  Of course, this is only a demonstration of the basic technique, you can, in theory plug whatever you want into a Matrix server because the API was designed to be extensible.

We're going to assume that you've already set up a Matrix server and have an account on it, and that you have access to a working Huginn install.

Making a Matrix server STUN-enabled.

Jan 18 2020

Previously in this series I showed you how to migrate a Matrix server to use Postgres, a database server designed for busy workloads, such as those of a busy chat server.  This time around I'll demonstrate how to integrate Synapse with a STUN/TURN server to make the voice and video conferencing features of the Matrix network more reliable.  It's remarkably easy to do but it does take a little planning.  Here's why I recommend doing this:

If you are reading this, chances are you're behind a NATting firewall, which means that your device doesn't have a publically routable IP adresss.  In addition to rewriting all of your network traffic so that it doesn't look like it's coming from a private network, the firewall is also doing port forwarding to pass inbound traffic to your device (least of all replies from web servers), again so it doesn't look like you're behind a firewall.  This works just ducky with TCP traffic because TCP sets up bidirectional connections; TCP packets are acknowledged every time which has the additional effect of letting the firewall keep the connection together.  VoIP traffic, on the other hand, tends to use UDP, which is not connection-oriented.  One way to look at UDP is as a fire-and-forget protocol: The packet gets launched toward its destination, and it may or may not arrive depending upon network core weather patterns, luck, the phase of the moon... packets may also not necessarily arrive in the correct order.  It's an inherently unreliable protocol.  This is what makes it useful for streaming data traffic like audio or video, because it's inherently low latency.  If you've ever been on a call and heard it break up or go into robot mode (or for that matter, seen a television program glitch out) this is probably what happened.  The occasional glitchout is the price you pay for a relatively snappy data stream.

Converting a Matrix server to use Postgres.

Jan 12 2020

In my last post about the Matrix network I covered how to set up a public Synapse server as well as a web-based client called Riot.  In so doing I left out a part of the process for the sake of clarity (because it's a hefty procedure and there's no reason not to break it down into logical modules), which was using a database back-end that's designed for workloads above and beyond what SQLite was meant for.  I'll be the first to tell you, I'm not a database professional, I don't know a whole lot about how to use or admin them aside from installing stuff that uses databases, so there's probably a lot of stuff I'm leaving out.  I'd love to talk to some folks who are more knowledgable than I am.

That said, the only other database server that Synapse supports is Postgres, which seems to have a user interface very different from that of what I use most often (MySQL) so the same procedures don't apply.  I'm going to do my best to break things down.  To make it more clear, I will not be referring back or comparing to MySQL because that will needlessly muddy already unclear waters.

Setting up a private Matrix server.

Jan 11 2020

EDIT - 20200804 - Updated the Nginx stanzas because the newer versions of Certbot do all the work of setting up SSL/TLS support for you, including the most basic Nginx settings.  If you have them there you'll run into trouble unless you delete them or comment them out.  Also, Certbot centralizes all of the appropriate SSL configuration and hardening settings into a single includable file (/etc/letsencrypt/options-ssl-nginx.conf) for ease of maintenance.

A couple of years ago I spent some time trying to set up Matrix, a self-hosted instant messaging and chat system that works a little like Jabber, a little like IRC, a little like Discord and a little like Slack.  The idea is that anyone can set up their own server which can federate with other servers (in effect making a much larger network), and it can be used for group chat or one-on-one instant messaging.  Matrix also has voice and video conferencing capabilities so you could hold conference calls over the network if you wanted.  For example, one possible use case I have in mind is running games over the Matrix network.  You could even build more exotic forms of conferencing on top of Matrix if you wanted to.  Even more handy is that the Matrix protocol supports end-to-end encryption of message traffic between everyone in a channel as well as between private chats between pairs of people.  If you turn encryption on in a channel it can't be turned off; you'd have delete the channel entirely (which would then cause the chat history to be purged).

Chat history is something that was a stumbling block in my threat model the last time I ran a Matrix server, somewhen in 2016.  Things have changed quite a bit since then.  For usability Matrix servers store chat history in their database, in part as a synchronization mechanism (channels can exist across multiple servers at the same time) and in part to provide a history that users can search through to find stuff, especially if they've just joined a channel.  For some applications, like collaboration inside a company this can be a good thing (and in fact, may be legally required).  For other applications (like a bunch of sysadmins venting in a back channel), not so much.  This is why Matrix has three mechanisms for maintaining privacy: End to end encryption of message traffic (of entire channels as well as private chats), peer-to-peer voice and video using WebRTC (meaning that there is no server that can record the traffic, it merely facilitates the initial connection), and deleting the oldest chat logs from the back-end database.  While it is true that there is no guarantee that other servers are also rotating out their message databases, end-to-end encryption helps ensure that only someone who was in the channel would have the keys to decrypt any of it.  It also seems feasible to set up Matrix channels such that all of the users are on a single server (such as an internal chat) which means that the discussion will not be federated to other servers.  Channels can also be made invite-only to limit who can join them.  Additionally, who can see a channel's history and how much of it can be set on a by-channel basis.

For the record, on the server I built for writing this article the minimum lifetime of conversation history is one calendar day, and the maximum lifetime of conversation history is seven calendar days.  If I could I'd set it to Signal's default of "delete everything before the last 300 messages" but Synapse doesn't support that so I tried to split the difference between usability and privacy (maybe I should file a pull request?)  A maintenance mole crawls through the database once every 24 hours and deletes the oldest stuff.  I could probably make it run more frequently than that but I don't yet know what kind of performance impact that would have.

One of the things I'm going to do in this article is gloss over the common fiddly stuff.  I'm not going to explain how to create an account on a server because I'm going to assume that you know how to look up instructions for doing that.  Hell, I google it from time to time because I don't do it often.  I'm also going to break this process up into a couple of articles.  This one will give you a basic, working install of Synapse (a minimum viable server, if you like).  I also won't go over how to install Certbot (the Let's Encrypt client) to get SSL certificates even though it's a crucial part of the process.  I will explain how to migrate Synapse's database off of SQLite and over to Postgres for better performance in a subsequent article.  For what it's worth I have next to no experience with Postgres, so I'm figuring it out as I go along.  Seasoned Postgres admins will no doubt have words for me.  After that I'll talk about how to make Matrix's VoIP functionality work a little more reliably by installing a STUN server on the same machine.  Later, I'll go over a simple integration of Huginn with a Matrix server (because you just know it's not a technical article unless I bring Huginn into it).

A piece of advice: Don't try to go public with a Matrix server all at once.  The instructions are complex and problematic in places, so this article is written from my notes.  Take your time.  If you rush it you will screw it up, just like I did.  Get what you need working, then move on to the next bit in a day or so.  There's no rush.