Setting up a private Matrix server.

Jan 11 2020

A couple of years ago I spent some time trying to set up Matrix, a self-hosted instant messaging and chat system that works a little like Jabber, a little like IRC, a little like Discord and a little like Slack.  The idea is that anyone can set up their own server which can federate with other servers (in effect making a much larger network), and it can be used for group chat or one-on-one instant messaging.  Matrix also has voice and video conferencing capabilities so you could hold conference calls over the network if you wanted.  For example, one possible use case I have in mind is running games over the Matrix network.  You could even build more exotic forms of conferencing on top of Matrix if you wanted to.  Even more handy is that the Matrix protocol supports end-to-end encryption of message traffic between everyone in a channel as well as between private chats between pairs of people.  If you turn encryption on in a channel it can't be turned off; you'd have delete the channel entirely (which would then cause the chat history to be purged).

Chat history is something that was a stumbling block in my threat model the last time I ran a Matrix server, somewhen in 2016.  Things have changed quite a bit since then.  For usability Matrix servers store chat history in their database, in part as a synchronization mechanism (channels can exist across multiple servers at the same time) and in part to provide a history that users can search through to find stuff, especially if they've just joined a channel.  For some applications, like collaboration inside a company this can be a good thing (and in fact, may be legally required).  For other applications (like a bunch of sysadmins venting in a back channel), not so much.  This is why Matrix has three mechanisms for maintaining privacy: End to end encryption of message traffic (of entire channels as well as private chats), peer-to-peer voice and video using WebRTC (meaning that there is no server that can record the traffic, it merely facilitates the initial connection), and deleting the oldest chat logs from the back-end database.  While it is true that there is no guarantee that other servers are also rotating out their message databases, end-to-end encryption helps ensure that only someone who was in the channel would have the keys to decrypt any of it.  It also seems feasible to set up Matrix channels such that all of the users are on a single server (such as an internal chat) which means that the discussion will not be federated to other servers.  Channels can also be made invite-only to limit who can join them.  Additionally, who can see a channel's history and how much of it can be set on a by-channel basis.

For the record, on the server I built for writing this article the minimum lifetime of conversation history is one calendar day, and the maximum lifetime of conversation history is seven calendar days.  If I could I'd set it to Signal's default of "delete everything before the last 300 messages" but Synapse doesn't support that so I tried to split the difference between usability and privacy (maybe I should file a pull request?)  A maintenance mole crawls through the database once every 24 hours and deletes the oldest stuff.  I could probably make it run more frequently than that but I don't yet know what kind of performance impact that would have.

One of the things I'm going to do in this article is gloss over the common fiddly stuff.  I'm not going to explain how to create an account on a server because I'm going to assume that you know how to look up instructions for doing that.  Hell, I google it from time to time because I don't do it often.  I'm also going to break this process up into a couple of articles.  This one will give you a basic, working install of Synapse (a minimum viable server, if you like).  I also won't go over how to install Certbot (the Let's Encrypt client) to get SSL certificates even though it's a crucial part of the process.  I will explain how to migrate Synapse's database off of SQLite and over to Postgres for better performance in a subsequent article.  For what it's worth I have next to no experience with Postgres, so I'm figuring it out as I go along.  Seasoned Postgres admins will no doubt have words for me.  After that I'll talk about how to make Matrix's VoIP functionality work a little more reliably by installing a STUN server on the same machine.  Later, I'll go over a simple integration of Huginn with a Matrix server (because you just know it's not a technical article unless I bring Huginn into it).

A piece of advice: Don't try to go public with a Matrix server all at once.  The instructions are complex and problematic in places, so this article is written from my notes.  Take your time.  If you rush it you will screw it up, just like I did.  Get what you need working, then move on to the next bit in a day or so.  There's no rush.

Okay, down to business.  As of writing this article I installed the latest stable version of Synapse, the Matrix back-end server which happened to be v1.8.0.  The server in question is hosted at Digital Ocean (referral link) running a fully patched install of Ubuntu Linux v18.04 and has 2GB of RAM, 50GB of disk space, and one CPU.

Things to keep in mind: When you name a Matrix server you can't change it later without a full reinstall, so be sure to pick something you can live with.  I went with matrix.jackpoint.virtadpt.net.  To separate privileges I created a separate, unprivileged user account with a locked password that I could then log into as the root user.  The general idea is that nobody should ever have to log in as the Matrix server's account unless they're already logged into the server, and logged in as the root user, and are working on Synapse itself.  Synapse is written in Python so a couple of pre-requisite Ubuntu packages had to be present: build-essential, libffi-dev, libjpeg-dev, libssl-dev, libxslt1-dev, python3-dev, python3-pip, python3-setuptools, python3-virtualenv, and sqlite3.

Installing and configuring Synapse was done as the Matrix pseudo-user:

drwho@jackpoint:~(3) $ sudo -s
Password: 
root@jackpoint:~() # su - matrix
matrix@jackpoint:~$ 

Overall, the process was this: Create a virtualenv, install Synapse, and generate a basic configuration file.  The commands looked like this:

matrix@jackpoint:~$ python3 -mvenv synapse
matrix@jackpoint:~$ . synapse/bin/activate
(synapse) matrix@jackpoint:~$ pip install --upgrade pip setuptools
(synapse) matrix@jackpoint:~$ pip install matrix-synapse
(synapse) matrix@jackpoint:~$ python -m synapse.app.homeserver
    --server-name matrix.jackpoint.virtadpt.net
    --config-path homeserver.yaml --generate-config --report-stats=no

The generated homeserver.yaml config file is extensively commented so I filtered all the comments and blank lines out of mine (grep -v '#' homeserver.yaml | grep -v '^$').  To be honest they're needlessly verbose and somewhat confusing.  The comments I've added below, however should be more informative.  Rather than copy-and-pasting this config file, I recommend that you generate your own and search through the file for the bits I've given below to minimize the possibility of later releases of Synapse changing things out from under you.

# This is the name your server responds to.  It can't be changed later!
server_name: "matrix.jackpoint.virtadpt.net"

# Where the process ID of the Synapse process goes.
pid_file: /home/matrix/homeserver.pid

# The canonical URL that Matrix clients must access.
public_baseurl: https://matrix.jackpoint.virtadpt.net/

# Never federate with any of the listed netblocks.
federation_ip_range_blacklist:
  - '127.0.0.0/8'
  - '10.0.0.0/8'
  - '172.16.0.0/12'
  - '192.168.0.0/16'
  - '100.64.0.0/10'
  - '169.254.0.0/16'
  - '::1/128'
  - 'fe80::/64'
  - 'fc00::/7'

# Configure the part of Synapse that talks to Matrix clients.
listeners:
  - port: 8008

    # We're going to rely upon Nginx to do this for us.
    tls: false
    type: http

    # Needed by Nginx for proxying to work correctly.
    x_forwarded: true

    # We're proxying through Nginx so only listen on localhost.
    bind_addresses: ['::1', '127.0.0.1']

    # Synapse subsystems connected to this port.
    # 'client' is the part that talks to the user's client.
    # 'federation' is the part that talks to other Matrix servers.
    resources:
      - names: [client, federation]
        compress: false

# Email address of the admin.
admin_contact: 'mailto:me@example.com'

# Chat history.
retention:
  # This can probably be disabled by setting it to 'false'.
  enabled: true

  # This is how long to store chat history for.
  default_policy:
    min_lifetime: 1d
    max_lifetime: 7d

database:
  # Database type?
  name: "sqlite3"

  # Database configuration.
  # It's SQLite so there isn't much to speak of.
  args:
    # Where to put the database.
    database: "/home/matrix/homeserver.db"

# Configure Python's logging capability.
log_config: "/home/matrix/matrix.jackpoint.virtadpt.net.log.config"

# Where to put stuff that people paste into chats.
media_store_path: "/home/matrix/media_store"

# Enable account creation, else it won't even work on the command line.
enable_registration: true

# These can be blank.
account_threepid_delegates:
metrics_flags:

# Don't report Matrix usage statistics to the Matrix projects.
report_stats: false

# This is an automatically generated crypto key for signing access tokens.
macaroon_secret_key: "a bunch of garbage"

# An automatically generated key used for protecting HTTP form requests.
form_secret: "more garbage"

# The location of the crypto key that does the actual token signing.
# This is automatically generated.
signing_key_path: "/home/matrix/matrix.jackpoint.virtadpt.net.signing.key"

# Set so that non-admins can create their own channels.
enable_group_creation: true

user_directory:
  # Users can search the local directory of accounts.
  enabled: true

  # Users cannot search the directory of all accounts seen by the server.
  # This takes some infrastructure work, so it's not enabled right now.
  search_all_users: false

The next thing to do is start up Synapse so that it's only accessible from localhost (this is how it's configured above so it's the default) and create an admin account.

(synapse) matrix@jackpoint:~$  synctl start
(synapse) matrix@jackpoint:~$  register_new_matrix_user -c homeserver.yaml
    http://localhost:8008/

# This means the username.
New user localpart: jhardcastle
Password:
Confirm password:
Make admin [no]: yes
Success!

Now for the tricky bit: Making Nginx proxy the Synapse server.  I won't go into how to install Nginx because Digital Ocean has fairly good documentation and I had Nginx installed on Jackpoint already.  I set up a DNS record for matrix.jackpoint.virtadpt.net pointing to the same IP address as jackpoint.virtadpt.net (this is dependent upon whoever handles DNS for your domain) and then, because best practice for Nginx is to create a separate config file for every application, domain, or website a server manages, I created a file /etc/nginx/sites-available/matrix.jackpoint.virtadpt.net with the following contents:

server {
        listen 443;
        server_name matrix.jackpoint.virtadpt.net;

        # Yes, this is commented out for the moment.
        #ssl on;
        ssl_dhparam /etc/ssl/jackpoint.virtadpt.net_dhparams.pem;
        ssl_session_timeout 10m;
        ssl_session_cache shared:SSL:50m;

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
        ssl_ciphers ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA;
        ssl_prefer_server_ciphers on;

        location /_matrix {
                proxy_pass http://localhost:8008;
                proxy_set_header X-Forwarded-For $remote_addr;
        }
        access_log /var/log/nginx/matrix.jackpoint.virtadpt.net.access.log;
        error_log /var/log/nginx/matrix.jackpoint.virtadpt.net.error.log;
}

# Yes, this goes in the file also.  Keep them together.
server {
        listen 8448;
        server_name matrix.jackpoint.virtadpt.net;

        # Yes, this is commented out for the moment.
        #ssl on;
        ssl_dhparam /etc/ssl/jackpoint.virtadpt.net_dhparams.pem;
        ssl_session_timeout 10m;
        ssl_session_cache shared:SSL:50m;

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
        ssl_ciphers ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA;
        ssl_prefer_server_ciphers on;

        location / {
                proxy_pass http://localhost:8008;
                proxy_set_header X-Forwarded-For $remote_addr;
        }

        access_log /var/log/nginx/matrix.jackpoint.virtadpt.net.access.log;
        error_log /var/log/nginx/matrix.jackpoint.virtadpt.net.error.log;
}

I enabled this configuration file (sudo ln -s /etc/nginx/sites-available/matrix.jackpoint.virtadpt.net /etc/nginx/sites-enabled/matrix.jackpoint.virtadpt.net) and kickstarted Nginx (sudo nginx -s reload).  I used Certbot to set up a Let's Encrypt SSL certificate for matrix.jackpoint.virtadpt.net.  Certbot will configure both server{} blocks in that configuration file to use the same SSL certificate for you.  I then edited the config file to uncomment the "ssl on" bit, so it now looks like this:

server {
        listen 443;
        server_name matrix.jackpoint.virtadpt.net;

        ssl on;
        ssl_dhparam /etc/ssl/jackpoint.virtadpt.net_dhparams.pem;
        ssl_session_timeout 10m;
        ssl_session_cache shared:SSL:50m;

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
        ssl_ciphers ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA;
        ssl_prefer_server_ciphers on;

        location /_matrix {
                proxy_pass http://localhost:8008;
                proxy_set_header X-Forwarded-For $remote_addr;
        }
        access_log /var/log/nginx/matrix.jackpoint.virtadpt.net.access.log;
        error_log /var/log/nginx/matrix.jackpoint.virtadpt.net.error.log;

    ssl_certificate /etc/letsencrypt/live/matrix.jackpoint.virtadpt.net/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/matrix.jackpoint.virtadpt.net/privkey.pem; # managed by Certbot
}

# Yes, this goes in the file also.  Keep them together.
server {
        listen 8448;
        server_name matrix.jackpoint.virtadpt.net;

        ssl on;
        ssl_dhparam /etc/ssl/jackpoint.virtadpt.net_dhparams.pem;
        ssl_session_timeout 10m;
        ssl_session_cache shared:SSL:50m;

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
        ssl_ciphers ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA;
        ssl_prefer_server_ciphers on;

        location / {
                proxy_pass http://localhost:8008;
                proxy_set_header X-Forwarded-For $remote_addr;
        }

        access_log /var/log/nginx/matrix.jackpoint.virtadpt.net.access.log;
        error_log /var/log/nginx/matrix.jackpoint.virtadpt.net.error.log;

    ssl_certificate /etc/letsencrypt/live/matrix.jackpoint.virtadpt.net/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/matrix.jackpoint.virtadpt.net/privkey.pem; # managed by Certbot
}

Kickstart Nginx again (sudo nginx -s reload).  You should now have a brand-new Matrix server running.  Now we need to point a client at the server to see if it works.  Matrix clients have come a very long way since 2016.ev, so I went with Riot.  I grabbed the latest stable build from Github, uncompressed it into my home directory (not the Matrix user's), and configured it to operate with my Matrix server using the official instructions.  To make life significantly easier (unlike last time) I set up a separate virtual host to put Riot into.  Riot consists of a bunch of HTML and client-side Javascript, so I'm not too worried about it being accessible.  Incidentally, you can use any installed copy of Riot to log into any Matrix server you have an account on, so you could in theory use mine to log into the official Matrix get-to-know-you channel.  There are also mobile and desktop versions of Riot you can use.

Now, one more thing: Federation.  Making your Matrix server talk to other matrix servers.  This is at once tricky as hell to get right and surprisingly simple.  First off, look back at the second block of the above Nginx config file - see how it says "listen 8448"?  What that tells Nginx is, "Open port 8448/tcp and listen for HTTP requests for the host matrix.jackpoint.virtadpt.net."  If your host is running a firewall like UFW (which I highly recommend you look into, it makes IPtables suck much less) you'll want to make sure that port is exposed to the outside world (sudo ufw allow 8448).  That is only half the process; the other half involves telling the rest of the Internet how to find your server so they can federate.  This is surprisingly tricky but I think I can break it down.

There are two ways that you can set up Matrix for a domain.  The first is to make it so that anyone can ping an account directly (@drwho:matrix.jackpoint.virtadpt.net), which implies that the server can also be contacted directly (matrix.jackpoint.virtadpt.net).  The second is to set it up so that the account looks like it's "just" at the domain (@drwho:virtadpt.net) but the connection is actually made to the server matrix.jackpoint.virtadpt.net.  The former is fairly easy.  The second is significantly less easy (especially if you don't run much of the domain's infrastructure).  I went with the former because, surprise surprise, I always screwed up the latter.

These days in webshit there is a thing called the .well-known/ directory.  This is due to a standard (RFC 5785) that basically says that, for any kind of random crap that someone might have running on their server or somewhere on their domain, that random crap should have a directory in a common place where "here's how you find my random crap" files can be kept.  In our case, the random crap is a Matrix server, and the file in question is /.well-known/matrix/server.  The contents of this file tell other Matrix servers how to contact yours for the purpose of federation.  Mine looks like this:

{
    "m.server": "matrix.jackpoint.virtadpt.net:443"
}

As the Matrix pseudouser I created a file /home/matrix/server.txt that contains the above JSON.  I then added the following to the /etc/nginx/sites-available/matrix.jackpoint.virtadpt.net file, right beneath the "location /_matrix" block:

        location = /.well-known/matrix/server {
            alias /home/matrix/server.txt;
        }

What this means is, "Whenever someone requests the file https://matrix.jackpoint.virtadpt.net/.well-known/matrix/server, send them the contents of the /home/matrix/server.txt file."  That someone is usually another Matrix server trying to connect to mine, and will use the sort-of-but-not-really URL in that JSON document to do so.  Restart Nginx again and try to hit the URL you just created.  If you see a block of JSON that looks like that, it worked.  Now, one more thing: Set it up so that Synapse starts every time the system reboots.  I did this by grabbing a copy of the matrix-synapse.service file for systemd, moving it into the directory /etc/systemd/system and editing it a bit so that it reflected my system configuration:

[Unit]
Description=Synapse Matrix homeserver

[Service]
Type=notify
NotifyAccess=main
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-abort

User=matrix
Group=matrix

WorkingDirectory=/home/matrix/synapse
ExecStart=/home/matrix/synapse/bin/python -m synapse.app.homeserver --config-path=/home/matrix/homeserver.yaml
SyslogIdentifier=matrix-synapse

[Install]
WantedBy=multi-user.target

Now for the moment of truth: Will it blend^Wwork?

(synapse) matrix@jackpoint:~$ synctl stop
(synapse) matrix@jackpoint:~$ exit
root@jackpoint:~() # systemctl start matrix-synapse.service
root@jackpoint:~() # systemctl status matrix-synapse.service

● matrix-synapse.service - Synapse Matrix homeserver
   Loaded: loaded (/etc/systemd/system/matrix-synapse.service; enabled; vendor pres
   Active: active (running) since Sat 2020-01-11 21:17:39 EST; 12min ago
 Main PID: 14506 (python)
    Tasks: 21 (limit: 2339)
   CGroup: /system.slice/matrix-synapse.service
           └─14506 /home/matrix/synapse/bin/python -m synapse.app.homeserver --conf

Jan 11 21:29:58 jackpoint matrix-synapse[14506]: 2020-01-11 21:29:58,569 - synapse.
Jan 11 21:29:58 jackpoint matrix-synapse[14506]: 2020-01-11 21:29:58,576 - synapse.
Jan 11 21:29:58 jackpoint matrix-synapse[14506]: 2020-01-11 21:29:58,579 - synapse.
Jan 11 21:29:58 jackpoint matrix-synapse[14506]: 2020-01-11 21:29:58,734 - synapse.
Jan 11 21:29:58 jackpoint matrix-synapse[14506]: 2020-01-11 21:29:58,735 - synapse.
Jan 11 21:29:58 jackpoint matrix-synapse[14506]: 2020-01-11 21:29:58,855 - synapse.
Jan 11 21:29:58 jackpoint matrix-synapse[14506]: 2020-01-11 21:29:58,860 - synapse.
Jan 11 21:29:59 jackpoint matrix-synapse[14506]: 2020-01-11 21:29:59,765 - synapse.
Jan 11 21:29:59 jackpoint matrix-synapse[14506]: 2020-01-11 21:29:59,770 - synapse.
Jan 11 21:29:59 jackpoint matrix-synapse[14506]: 2020-01-11 21:29:59,803 - synapse.

(Note: systemctl status always runs its output through less, which doesn't play well with Screen when the lines are really long.  It's the "active (running)" part that you're looking for.)

If you go to your copy of Riot (or someone else's) and log into your server you should see the Riot user interface.