Using Ansible to restart a bunch of services running under systemd in --user mode.

05 December 2019

ansible exocortex git howto playbooks systemd updates

Let's say that you have a bunch of servers that you admin en masse using Ansible.  You have all of them listed and organized in your /etc/ansible/hosts file.  Let's say that each server is running a system service (like my Systembot) running under systemd in --user mode.  (Yes, I'm going to use my exocortex-halo/ repository for this, because I just worked out a good way to keep everything up to date and want to share the technique for everyone new to Ansible.  Pay it forward, you know?)  You want to use Ansible to update your copy of Systembot across everything so you don't have to SSH into every box and git pull the repo to get the updates.  A possible Ansible playbook to install the updates might look something like this:

---
- name: Update checkout of exocortex-halo/ from Github.
  hosts: all
  become: no

  tasks:
    - name: Pull from Github.
      git:
        repo: "https://github.com/virtadpt/exocortex-halo"
        dest: "./exocortex-halo"
        update: yes

What this playbook does is:

  • For all nodes listed in your /etc/ansible/hosts file...
  • Don't try to become the root user.
  • If the ./exocortex-halo directory does not exist, run git clone https://github... to create it.
  • If it does exist, change into it and run git pull to update it.

When this playbook is done running it will have updated or created the git repository checkouts on all of your hosts.

Now.. because you're running Systembot as an unprivileged user on each of your machines under systemd in --user mode (which is something I really need to do a writeup of because it's a huge pain in the ass if you don't know the trick), you need to restart the service you just updated so the changes will take effect.  Now, you could do so node by node individually because sometimes systemd is really finicky about such things:

{20:59:16 @ Mon Dec 02}
[drwho @ windbringer ~] () $ ssh alpha.example.com
Host key fingerprint is SHA256:a6acffb1b41e4342ce1b5dc8158b472d150291cf62f7
drwho@alpha:~/ $ systemctl --user restart system_bot.service
...
{21:01:01 @ Mon Dec 02}
[drwho @ windbringer ~] () $ ssh beta.example.com
Host key fingerprint is SHA256:0f51899cd4ee979bde784aac50aff6c62a526cc4fa6a
drwho@beta:~/ $ systemctl --user restart system_bot.service
...

...but there's a significantly easier way of going about it, using an Ansible ad-hoc command instead of a playbook:

ansible all -m systemd -a "scope=user name=system_bot.service state=restarted"

This means:

  • Run the ansible command against every host in the default host inventory file (/etc/ansible/hosts)
  • Call the module systemd
  • The scope of the thing you're manipulating is in --user mode
  • The name of the service you're manipulating is system_bot.service
  • Restart the service (literally, "Make the current state of the service be 'restarted'.")

Sit back and wait.  You'll be all done before you know it.  Of course, this method will work for any --user mode service, and by changing the value of scope to "system" you can restart system-wide services (which, of course, will require root privileges) or "global" to restart a service that runs in --user mode for every user on the system simultaneously.

The reason I'm writing this up is because I've done this particular process this particular way far more times than I care to count.  It sucks.  I had so much trouble getting my bots working with systemd that I didn't want to mess with it in the context of Ansible.  However, much to my surprise with five minutes of reading this afternoon, spitballing a command in a Joplin note, and then testing that command at home because I've been tinkering with Systembot again, it worked without any expectation of doing so.  So, rather than forget I decided to do a quick writeup of how to do it with a real-world example, not only so it's easier for me to find later if I need to but so that I can save other folks a whole lot of time and tears.

back to top