Antarctica Starts Here.

Note: I'm retraining on a new keyboard as I write this, so I apologize for any egregious typos in advance.

Over on birbsite a couple of weeks back a thread was spun up about your worst fuckup on the job and I figured that, because it's been nearly twenty years I'd tell my worst story. However, much to my chagrin and concern I found that I'd bobbled a few of the details. Seeing as how it was one of my career's formative moments this scared me quite a bit. I'd been considering putting some work in on my memoirs anyway (because, as I write this, I'm coming up on the first anniversary of my mom's Greater Feast), so why not start things off this way? And it worries me a bit how the memory of such an important thing could have faded so much. So, time to do a little background research and fix it up a little.

In late 2003.ev, after finishing my degree I finally landed my first full time gig in the security field (as a permatemp, but still) for a county government back home. I started out there on a team of IT temp workers taking inventory of the facility, but a couple of weeks into the gig it turned out that I was working alongside a guy that I ran into at Defcon in 2001.ev (we geeked out on cryptography for a couple of hours) and he pulled a couple of strings to get me detached from the IT inventory team and onto his security team (bringing the number of people on the team up to two). I was duely issued a desk in the cubicle gulag and a fairly old Dell workstation (Pentium-II CPU, 32 megs of RAM, 2GB hard drive, Windows XP), which was one of the more powerful workstations on the floor, and was told to familiarize myself with the network because I was going to be building some IDSes with Snort as my first assignment.

I figured that the easiest way to go about that was to run an nmap scan of the network, record the output, and read through it to see what I was dealing with. I'd been using nmap since college so I was already familiar with it and what it could do. I hasten to add that, in the documentation, the user it warned that port scanning can crash services or entire systems if you're not careful. I'd been using nmap since 1998.ev or therabouts and had yet to crash any machines in my network, so I'd chalked the warning up to an overabundance of caution. I figured out the network addressing configuration from the output of ifconfig, fired off a scan, and went to get lunch because it was going to take a while.

Something I was not aware of was that the network in question had a dinosaur pen with an honest-to-pete mainframe, an IBM S/390 running OS/390. Inside of that monster of a machine were running a large array of virtual machines, each of which was running a crucial bit of business infrastructure - payroll, archival (I found out later that there was a team of greybeards maintaining an entire library of reel to reel magnetic tape storage - they were migrating away from reel to reel to tape cartridges that were newer, held more data, and were smaller. Their VM was running in there, too.

When I got back from lunch, my boss was standing in my cubicle holding an armful of greenbar printout about four inches thick. I hadn't seen greenbar since undergrad, because we used it to print out our source listings on the campus VAX, so this immediately worried me. You don't break out the greenbar hardcopy without good reason. Pete threw the printout on my desk and ordered me to look at it and tell him what it was. I flipped through it and felt my blood run cold. I'm no mainframe expert, but when I see the strings "OS/390," "virtual machine," and "crash dump" over and over and over again, each time referencing a different hostname, I can add 2 and 2 and get 4. The presence of the string "TCP/IP" strongly implied that my nmap'ing the network had caused the TCP/IP stack of each virtual machine to shit itself, causing the mainframe to print a crash dump to the default printer. Eventually, OS/390 had enough of my shenanagains and shot itself in the head.

One mainframe, dead on arrival.

I quietly told my boss what I thought it meant, and he confirmed my worst fears. "Do you know how long it's going to take the admins to IPL that S/390? At least two weeks, maybe more."¹

At that moment my dreams of getting into security as a career seemed to turn turtle and sink. I'd be lucky if I could land an IT position fixing printers after that. Maybe I could go to grad school for a few years, get a job as a TA and a masters' degree, and let the heat die down. So, I turned, walked to the copy room, grabbed a mostly empty cardboard box of copier paper (putting the last two reams on the shelf), and began packing my few office belongings for the frog march out of the building.

To this day, I still don't know why Pete didn't treat me like a Russian oligarch. I'd certainly earned it. I didn't get fired. I definitely got one of the worst dressings down of my life; the greybeards in the dinosaur pen beating the hell out of me would have hurt less. I'm pretty sure that they wanted my head as well, not that I could blame them. This was, as a bit of IBM folklore would have it, a million dollar education.²

What my boss drilled into my head that day (note: if you're easily squicked, don't click on that link - you have been warned) was that passive measures are, in some ways, just as effective as active ones. You can learn a great deal about a network (if you're positioned in the right places, anyway) using, say, tcpdump and analyzing the packet captures, or examining the ARP cache of your workstation or a server you have access to. Firing a metric assload of strangely configured packets at a couple of million IP addresses that may or may not exist³ is just begging to get whacked upside the head by Murphy's law.

The other thing I learned was that warnings are published for a reason. There are theoretical problems, to be sure, but if someone puts a notice in their documentation to the effect of "You can kill a machine by running this tool against it and fuck things up for weeks," chances are they made the same mistake (possibly on a smaller scale) and put the warning in there so that others could learn from their mistake.

So, if you're just getting into the business, please learn from my mistake. I got really, REALLY lucky. You might not.