Antarctica Starts Here.

EDIT - 20171011 - Added a bit about getting real login shells inside of this Screen session, which fixes a remarkable number of bugs.  Also cleaned up formatting a bit.

To keep the complexity of parts of my exocortex down I've opted to not separate everything into larger chunks using popular technologies these days, such as Linux containers (though I did Dockerize the XMPP bridge as an experiment) because there are already quite a few moving parts, and increasing complexity does not make for a more secure or stable system.  However, this brings up a valid and important question, which is "How do you restart everything if you have to reboot a server for some reason?"

A valid question indeed.  Servers need to be rebooted periodically to apply patches, upgrade kernels, and generally to blow the cruft out of the memory field.  Traditionally, there are all sorts of hoops and gymnastics one can go through with traditional initscripts but for home-grown and third party stuff it's difficult to run things from initscripts in such a way that they don't have elevated privileges for security reasons.  The hands-on way of doing it is to run a GNU Screen session when you log in and start everything up (or reconnect to one if it's already running).  This process, also, can be automated to run when a system reboots.  Here's how:

When I set up a new box for my exocortex, I create a personal GNU Screen config file in my home directory, ~/.screenrc.  In this file I add some configuration options and commands for Screen to run when a new session starts up.  Here's a snippet of one of these files:

# Make the default login shell bash.
# shell "/usr/bin/bash"
defshell -bash

# Detach the session on disconnect.
autodetach on

# Don't display copyright information.
startup_message off

# Make every new shell a login shell.
shell -$SHELL

# Store 4000 lines of scrollback for every window.
defscrollback 4000

screen -t "System monitor" top

screen -t "Network traffic" slurm -d5 -i eth0

chdir /home/drwho/exocortex-halo/exocortex_xmpp_bridge
screen -t "Exocortex XMPP Bridge" python2 ./exocortex_xmpp_bridge.py

chdir /home/drwho/exocortex-halo/web_search_bot
screen -t "Exocortex Web Search Bot" python2 ./web_search_bot.py

The commented lines should be pretty self-explanatory; I refer you to the "Customization" section of the Screen manpage for more information.  The lines beginning with "screen" are commands to Screen to start new shells.  The "-t foo bar baz" option sets a title with an arbitrary string on each new shell that is really only useful when listing the running shells (with the control-a, w keystroke inside of Screen).  The chdir command sets the current working directory of the new shell to the directory specified; I like full directory paths to minimize the possibility of mistakes.  The subsequent commands (python2 ./exocortex_xmpp_bridge.py ...) are executed by the new shell as-is and, in this case, fire up an XMPP bridge.  So, every time a new copy of Screen starts up it runs these commands, one after the other.  This brings us to the next question, which is "How do you make this happen at boot-time?"

The answer comes from the manpage for crontab.  There is a time specifier for cron which lets you specify the next time a job will run as "the next time the server boots up."  I put the following line into my personal crontab with the command crontab -e:

@reboot /usr/bin/screen -d -l -m

Roughly translated, this means "After the next reboot, run the command /usr/bin/screen with the options '-d -l -m'."  This, of course, means that screen runs the ~/.screenrc file I sketched out above when it starts up and all of the bots I have in there start up automatically.  The "-d -l -m" command line switches for Screen mean:

• -d - Start in detached mode, which is to say run in the background until something grabs it.
• -l - Start up in login mode (run the shells' config files to set up the environments).
• -m - Don't bother trying to grab a TTY, just run in the background as if it were a daemon.  This is important when using the -d switch.

End result: All of my stuff starts up roughly a minute after my server boots up, and I don't have to do anything manually to make that happen.