Random knowledge XVII.

05 February 2007

MD5 hashes are not only good for digital signatures, they're good for making sure that files you've downloaded weren't corrupted. However, a directory full of files can prove problematic because you have to cat the text file of MD5 sums to expect, run md5sum against a file, compare it visually.. it can get annoying afte a short period of time. Thankfully, the GNU implementation of the md5sum utility has the option --check, which lets you pass a text file of lines ("MD5 = ") to the utility. It will automatically compute an MD5 sum of each file in the text file and compare it to the expected value in the file, printing out an message for each file that passes.

Whenever your web browser hangs for a while because it's waiting for a file download to start, you'll inevitably jump over to another window to do something else. Just when you start typing in that other window the dialog box for the filename to save to will pop up with a suggested filename and grab focus, so whatever commands you were typing will replace that filename. This is great for losing files because they're named with snippets of text, such as rm -rf /tmp/sourcecode.

They're also a cast-iron bitch to remove or rename after the fact.

Sometimes it is necessary to download multiple files as required from a system that is well-insulated from the rest of the Net (say, by three or four firewalls). Proxy server software, such as Privoxy, can be used to proxy the connections for you, but the problem remains of connecting the proxies together.

First of all, you need access to at least one bastion host to run the proxy server software on. If you can, compile it on each server, but if you're in such a restricted environment that you've got multiple firewalls to deal with you probably won't have development kits available to you on them, in which case you'll have to precompile the software and transfer the executable and config file to each one.

Now, assuming that your workstation has mostly unrestricted access to the Net, set up a copy of Privoxy on your workstation that listens on the loopback address (, port 8118. On your first bastion host, set up another copy of Privoxy that also listens on the loopback interface but on port 8000. Also configure this copy (as well as all subsequent copies) to accept connections from anywhere. While this is technically a security loophole only the system that Privoxy is running on will be able to access the proxy port. Also set this copy up to forward everything to port

Configure all subsequent copies of privoxy to listen at localhost:8000 but forward everything back to localhost:7000.

Start privoxy on each server: sbin/privoxy etc/config

When you SSH to your bastion host, forward port 7000 back to port 8118 on your workstation, like this: ssh -C -R 7000:localhost:8118 bastion. This opens up port 7000 on your bastion host and forwards it back to the copy of Privoxy on your workstation.

From your bastion host, SSH to the next bastion host in the chain: ssh -C -R 7000:localhost:8000. This opens port 7000 on the next bastion host and forwards it back to the previous server in the chain on port 8000.

On the final SSH link in the chain, forward port 7000 as before, but in your shell (usually you'll have su'd to the root user - this IS so you can run a network application like apt, yum, or CPAN, right?) set an environment variable that sets the HTTP proxy for whatever application you're running: export http_proxy=http://localhost:7000

Now, test your connection: telnet localhost 7000. If you get a valid connection and hitting the enter key a few times results in a 400 error, it's working. If it doesn't work then either you've accidentally turned on access filtering on a copy of Privoxy somewhere, or you've misforwarded a port someplace. Log out through the chain and each step of the way, try telnetting to localhost:7000 - the server it doesn't work on is the one you have to fix.

Always be the last one to hang up from the teleconference.

If you see a web application server like Tomcat or BEA Weblogic that isn't pitching Java exceptions all over the place... chances are it's not running at the moment.

Excedrin on an empty stomach is a bad idea.

An infrastructure of any kind that is advertised as being 'fault tolerant' or 'redundant' sure as hell isn't.