A few thoughts on what it means to cut a country off.

11 February 2011

The hot topic these days is the January 25th revolution in Egypt: the people rose up and demanded that their president (who is known for, among other things, having bloggers raided, torture, censorship, and general repression of the people of an entire country) step down and do whatever it is that retired dictators do (which is usually not what the people wish he or she would do). For the record, the United States was well aware that this was happening, and in fact aided the government of Egypt to the tune of 1.5 billion US dollars a year because doing so was mutually beneficial to both countries. I won't go into very many details of this because there are people who know a hell of a lot more about the situation than I do (mostly because they've got the time and CPU power to stay on top of it, and have significant numbers of contacts in the country; I have none of those things). What I do want to talk about, however, are some of the measures that were taken by the Egyptian government to minimize the effects of people talking to one another and organizing into the crowds you've probably seen on television who were calling for the Mubarak regeime to pack up and get the hell out of Dodge so that a true democracy could be set up.

For a couple of days the Egyptian government turned off the cellular sectors of its communications infrastructure, leaving the phones of millions of people useless. This was done in an attempt to prevent people from organizing because they were calling one another and relaying new developments with text messages. More importantly, the Egyptian government forced the ISPs operating within its borders to functionally shut down in a bid to prevent its citizens from organizing further. Thanks to the efforts of a group of ham radio operators, ISPs within the European Union offering free dialup modem pools, some very talented hackers, and Internet activists like We Rebuild the people of Egypt were able to pass information around, plan, organize, and coordinate. You can get a sense for their efforts by watching this wiki page if you've a mind to (or if you want to see how you can help out). This was due in part to Noor Networks of Egypt not being able to go completely offline due to the fact that the Egyptian stock exchange relied upon them for connectivity, and Mubarak was unwilling to weather the economic damage that would have occurred if they went completely dark.

I bet you're wondering how they went about cutting an entire country off from the rest of the global Net. It seems unthinkable, doesn't it? After all, there is optical fibre all over the place, satellite links, and even some PSTN trunk lines still alive and kicking, and we can scarcely go a day without interacting with the Net in some fashion (even if it's making a simple phone call).

It was actually pretty simple (for backbone engineering values of 'simple')...

Okay. Network architecture 050, Routing for Mortals.

For the moment, assume that a router is a computer that does one thing very well and blindingly fast: it takes a packet of data traversing a network (say, part of an HTTP request to a web server), looks at the destination IP address, and figures out where to send that packet to bring it one hop closer to its destination. Routers typically have scads of network interfaces; your home router (which is actually a firewall but that's not important at the moment) has at least two, an external and an internal. A router of the sort you'd find in an office's wiring closet or a small company's data center probably has about as many interfaces but they're designed for sorts of cables that you're not likely to see elsewhere, like optical fibre entering the building from a telephone pole or something more exotic. The routers that you'd find in an IXP (Internet Exchange Point, where ISPs cross-connect with each other (a practice called peering)) are, as they say, a whole 'nother smoke. The costs of these routers start at five significant digits and might go up to seven, have dozens of network interfaces, and can route gigabits of traffic every second. If optical fibre is the skeleton of the Net, these routers are the musculature. Without either entire sectors of the Net vanish. Sure, networks themselves might still be running, but there would be no way for traffic to enter or leave them, so they might as well be powered down. Information functions by being in motion, after all.

The Net is diced up into so many netblocks (few are geographically contiguous) that it isn't feasible to assemble a routing table that works for every possibility, they'd be too complex to manage and so couldn't be updated to stay current. So what they do is give the routers general guidelines for handling traffic and let them fight it out amongst themselves with a protocol called BGP (Border Gateway Protocol). What happens is that the ISP configures a router to know what blocks of IP addresses it should handle traffic for. This-and-such netblock is handled by interface-0, that-and-such netblock is handled by interface-1. Traffic destined for other networks is handled by interface-2, interface-3, and interface-4. Router interfaces 2, 3, and 4 (in this example) are plugged into routers operated by three other ISPs, whose routers announce what networks they're authoritative for.

There are actually a couple of routing protocols in use today, such as OSPF, RIP, and IS-IS, as well as BGP. Most of these are considered interior routing protocols, which is to say that they were designed to permit the routers of a single ISP (or autonomous system, to be technical about it) to keep each other abreast of the best pathways between one another. Routers can and do run multiple protocols to exchange routing information; not all methods are ideal for all environments, and sometimes there are a couple of pieces of hardware on your network that just don't have what you need to have a homogeneous environment.

The purpose of BGP is twofold: first, every router tells all of the other ISPs' routers it's connected to that it's operational and thus a candidate for handling traffic. Every minute every router sends a heartbeat packet to its immediate neighbors. If a router ever drops off the Net (say, it has a close call of the fibre-seeking backhoe kind) its neighbors can shift gears in mid-stream and pick a different router to send packets through. Second, BGP is used to announce to its neighbors what networks it can relay traffic to, a path from that particular router to a netblock X, and the net number of hops toward a netblock X. Ordinarily, neighboring routers decide on the shortest path depending on how specific a match is found between a route and a destination. For example, if a packet is destined for the IP address and a router has two possible routes that would work ( and, the closer match ( is the one it will go with. These decisions can be overridden by policies programmed into a router under certain circumstances, such as a gentlebeing's agreement to favor a particular ISP, political reasons, network latency, or the phase of the moon. A static route is preferred over all others because it usually means a direct connection to a destination. If a destination was found using one of the interior routing protocols I mentioned earlier, then that one is used because it means that the traffic won't have to cross into someone else's network. As a last resort, routes acquired using BGP are chosen. Any of the ISPs our hypothetical router is connected to could conceivably handle traffic headed anywhere else on the Net; the point of BGP is to give routers the information needed to make a determination more or less autonomously.

Oh, one more thing: BGP can also be used to rescind candidacy for handling traffic, in effect telling neighboring routers "I'm not going to play with you anymore, go talk to someone else. Ommmmmmmmmm......" This is how Egypt vanished from the Net for a couple of days. Officials high up in the old Egyptian government went to the five tier-one ISPs in the country and told them to configure their routers to stop announcing routes to everyone else (there are about 3,000 routes in those tables, I hear). The sysadmins did as they were told. The routers upstream forgot that the Egyptian netblocks existed, and thus did an entire country go dark. Or nearly so - as I mentioned earlier, the ISP which Egypt's stock market uses (Noor) supported minimal connectivity so that commerce could continue. It was over this network that a trickle of traffic was able to get through. No physical damage had to be done, no lines had to be cut, no fire axes had to be used. All it took was a couple of keystrokes and a little time. It should also be noted, I am informed, that nearly all of Egypt's net.traffic eventually runs through a single IXP just outside of Cairo (CR-IX), and the links leaving that IXP terminate somewhere in Italy. Effectively a chokepoint, this no doubt made the task of cutting things off even easier.

The same thing could possibly be done here but it wouldn't be nearly as easy or effective. There are eleven tier one ISPs in the United States, which is to say that there are eleven networks big and far reaching enough to reach every other network on the planet without having to pay anyone for the privilege. There are also a lot more than just two or three IXPs in the United States; if these stats are correct there are at leasts twenty-four of them around the country (MAE East and MAE West are both listed under MAE on that chart and there might be more). The routing tables relevant to the United States are also large enough that it would likely not be possible to knock out all connectivity (there are on the order of hundreds of thousands of routes per table). Suffice it to say that a given packet would probably find at least one way to go from point A to point B despite some subset of routers going black. On top of this, hypothetically speaking, if the US government tried to pull the plug they'd be effectively cutting themselves off at the knees because they make use of the same networks for communication that we do. While it might be possible to force ISPs to only route traffic from netblocks registered to the military and federal agencies it would probably be a futile effort as the number of valid routes left would still be enough for civilian communication to continue. It's also just not possible to line up sufficient numbers of leased lines on such short notice and these days satellite links don't have the kind of bandwidth that long runs of optical fibre do.

Just the same, a couple of days ago Senators Susan Collins, Joseph Liberman, and Thomas Carper proposed a plan that would allow the President to order the disconnection of any computer networks in the event of an infowar attack. It wasn't a bill, just an idea for one, but it had freedom of speech and electronic liberty advocates up in arms, and rightly so because what was proposed was pretty much what happened in Egypt last week and in Tibet back in 2005. Sure, the proposal said that it would never be used to silence dissent in the United States, but they also said that the NSA would never eavesdrop upon American citizens (lie), and the US was not the sort of country that would torture and wrongfully imprison people on an indefinite basis (another lie). I use the past tense when discussing this proposal because the hue and cry was so great that the proposal was taken off the table yesterday, but it was tried once in August of 2010, and it'll no doubt be tried again, probably as a rider on a bill that "no true American would vote against."

Now, back to domain seizure. Yes, it's back. A few weeks ago the registered domain of a Spanish website called Rojadirecta.org, which was made available indexes of links to streaming and cached professional sports footage was seized by US Immigration and Customs Enforcement. Roja Directa was taken to court twice for publishing links and twice it was ruled that it is, in fact, legal to link to other websites and talk about what's on them. They didn't host anything illegal or pirated, they just had links to videos. The reason that ICE was able to jump international jurisdiction and seize the domain was because it was registered through GoDaddy, which is a company incorporated and operating out of the United States. It's now officially registered to the Department of Homeland Security. The article I linked to above was another shot over the bow because another set of domains were taken over in the same effort since the original version was published online. Right now there are a few rumblings from the Senate but it remains to be seen what'll come of Ron Wyden asking the hard questions. As it stands now, the safest thing to do is to register domains that are not controlled by organizations operating within the United States. To see a list of domain registrars outside of the US in case you want to hedge your bets take a look at this list.

By the way, if anybody knows how I could register a couple of domains under the .va TLD, please drop me a line.

Last but not least, on 25 January 2011 the US Department of Justice announced that they're unhappy that ISPs don't record more information about what their customers do online at a hearing held by the House Subcommittee on Crime, Terrorism, and Homeland Security. Supposedly, criminal investigations are too difficult because they can't get their hands on what IP addresses are issued to customers at a given time, nor can they be given lists of URLs and IP addresses that customers visit through the course of an average day. Interestingly, the hearing held that morning included a direct attack on the EFF, which not only advocates for privacy, fair use, and anonymity online but publishes best practice papers on defense against surveillance, search engine privacy, and legal guidelines for bloggers. The suggested data retention guidelines would require at a minimum that records of IP addresses issued to customers, probably along with internally used information (such as subscriber ID, MAC address of cable or DSL modem, unique line identifier, or what have you) for a certain period of time, probably about two years. There are conflicting reports on the other sorts of information they would like retained - most of them are probably common sense extrapolations of what law enforcement would find useful for tracking people, but there's always some jetwash thrown in.

It should strike you as interesting (but unsurprising) that the DoJ is asking for the storage of data that they refuse to maintain themselves.

Coda: Special thanks to Alexius Pendragon, The Wrong Hands, and Jacob Smith for fact checking me on some things and correcting me on others.