Project Byzantium called to assist relief efforts in New York.

14 November 2012

As a result of the damage done to New York City by Hurricane Sandy the week before last, Project Byzantium was contacted by representatives of several NGOs and non-profit organizations we've been in contact with as a result of our work on community wireless mesh networks. We were asked if Byzantium Linux might be useful in assisting relief efforts in New York City by restoring communications on the local level. As this is one of our primary use cases, we responded in the affirmative, and were told that we might be asked to go to New York City to help out. It wasn't a definite thing at the time, but just in case we immediately began work on a new version of Byzantium Linux which took into account some of the information we'd been given about the situation.

The new version is ultra-stripped down and has almost no user interaction. To save compute cycles we pulled out the hosted applications so that there would be one less thing for users to worry about. We also took the configuration code out of the control panel and wrote a daemon that automatically configured every wireless network interface it found on the system, effectively turning it into wireless infrastructure inside of sixty seconds. We also replaced the IP routing protocol with OLSR because there was a small Commotion Wireless network in Brooklyn, and representatives of the community wanted to extend its coverage rather than replace it. The Commotion project was kind enough to send us their configuration file so that our respective projects would seamlessly and automatically support each other.

This might form the foundation for the next official release of Byzantium but I think I'm getting ahead of myself.

Through our respective networks of contacts, we put out word of what we thought we'd need: Spare laptop and netbook computers, USB keys, blank CDs, and information. Specifically, information about the state of the communications and power grids in New York City, contact info for people whom we could contact, and places to stay. Over the next few days we slowly gathered information about cellular coverage, power, water, and food distribution in areas that were hit the hardest by Hurricane Sandy. Then, last Friday afternoon we recieved word from one of the people we'd been in contact with: Get to New York City.

We dropped everything and did.

From the get-go, logistics were sketchy. We didn't have many specifics on what we had to do until we actually got to New York City, though given our expertise in wireless networks we figured that would be a large part of what we'd be working on. Sitwon, Haxwithaxe and I stopped off at the halfway point of the drive to stretch, check our e-mail, and make a few phone calls. After we caught up and discussed things a bit we decided to get a hotel room for the evening in New Jersey and figure things out rather than drive straight through to New York City without a plan or real options. Haxwithaxe and I installed the Sandy respin of Byzantium Linux on a few handfuls of USB keys and programmed our hand-held amateur radios for NYC-specific frequencies and repeaters. Project Byzantium received word later that evening from one of our contacts - go to the FEMA disaster relief facility in Brooklyn and hook up with relief teams on site. They'd set up on the top floor of the Ikea, a landmark of considerable size to be sure. Somewhat serendipitously we'd misjudged how difficult it would be to get there (post-Sandy traffic to and from New York being somewhat unpredictable) so we were on site a good two hours early. We ran a radio check and an inventory, and then placed a phone call to F-, who then asked us to scout a couple of square blocks to see what places had power and get a lay of the land.

We got back into Sitwon's SUV and did so, and mapped out some likely installation locations for wireless nodes at the same time. At the appointed time we met up with F-, who then conducted us inside to the FEMA relief office upstairs. Along the way he laid out what they wanted to accomplish, we told him what we needed to do the job, and then set to work. We'd been given use of a table in the Ikea food court within ready distance of a power drop, so we taped down an extension cord, unpacked our kit, and set to work. Sitwon, Haxwithaxe, and I got automatic gateway configuration and propagation working and we tested interoperability with an existing Commotion Wireless router. It was seamless. No trouble at all.

On the other hand, the net.connection at Ikea was sketchy at best. It wouldn't stay online for longer than a few minutes at a time so our mesh's uplink kept bouncing. F- informed us that the windows were shielded, which is why signal strength from sources external to the building were nearly non-existent, so using someone else's link wasn't much of an option at the moment. Still, we had demonstrated that Byzantium Linux would work in the field. Unfortunately, everything did not go perfectly in other areas. One of the groups who had contacted us to say that they had equipment we could deploy in Red Hook did not follow through, which left us with lots of copies of Byzantium Linux but no computers to run them on.

We immediately started work on our backup plan. There is a burgeoning community mesh network already in place in Red Hook, which meant that there were routers available. Specifically, Ubiquiti Nano- and Picostations. Looking back on it, we got lucky in that respect. The Red Hook Initiative had been trying to set up a community wireless network for some time but City Hall kept blocking the project until the hurricane convinced them otherwise. Without that stash of Ubiquiti equipment on site, the telecom restoration effort would probably have been dead in the water. Plan B in place, we packed up our kit and hiked to the Red Hook neighborhood of Brooklyn, New York.

Red Hook is a small up and coming neighborhood that was hit hard by Hurricane Sandy. We arrived at the Red Hook Initiative building, and after straightening out some miscommunication we hooked up with the team there. RHI is a non-profit organization working with the community of Red Hook to improve the neighborhood. They've come a long way, and not just because it's now home to college students and artists. The people there are some of the nicest I've ever met. When faced with adversity they pulled together to help one another, and I don't think that they're going to stop after the last broom is hung up and the last bulb is lit.

Once we'd gotten our bearings on the ground, a few of us began flashing Commotion Firmware onto the routers cached in the loft of the building while a small team of us went hiking through Red Hook looking for buildings in which we could set up routers. Unfortunately, not everybody is comfortable with having a wireless router in their window or on their roof. Public housing laws also make it difficult to participate in community projects like this, even with Federal assistance cutting red tape and fending off the notoriously Gollum-like telecommunications providers. Not all buildings have power drops on the roof and we didn't have any solar panels, nor could we source any in time, so our only workable option was power over Ethernet, which meant finding people who'd let us plug in from inside their apartments. We left the logistics of that to our contacts from Red Hook because they know the community better than we did.

That evening we set up a Commotion router on the rooftop of a nearby garage as the first step of our effort to construct a larger network. This involved a certain amount of climbing across rooftops, stringing cable, and wiring up connectors under poor working conditions. In hindsight, I think it's time that I start going back to the gym and lose a little weight because it was not easy playing rooftop ninja. Moving the equipment from point to point involved dismantling it, carrying the pieces, and then rigging up a means of lowering them down a twenty foot drop with an extension cable. At one point we also had to patch the plastic sheath of the network cable because a sharp angle iron had peeled away some of the shielding; water would likely have gotten into the cable and caused the wires to degrade over time, eventually forcing replacement. I don't know how long the teflon tape will hold, but I hope it holds long enough.

One of the folks we were working with was kind enough to let us crash in his apartment that night. We relaxed a bit, plugged our kit in to recharge, and sprawled out on whatever horizontal service happened to be handy. After a hasty breakfast on Sunday morning we were back on site at RHI to continue where we'd left off the night before. To provide a bit of context for that day's activities, the telecom grid in New York City was in bad enough shape that there was little to no data connectivity in the neighborhood. Making phone calls was occasionally problematic as well; at one point, a room full of people placing cellphone calls were connected to wrong numbers on Sunday afternoon, suggesting that the telephony switches were messed up. In short, the community needed to be able to communicate with the rest of the world. We had a lead on an uplink across town but no way to connect the community wireless mesh to it, hence, our call for microwave equipment because it was impractical to set up enough nodes to bridge the gap. We briefly considered some nifty wifi antenna tricks but decided that we needed something more permanent than a couple of Pringles cans and some coathanger yagi antennae. So we helped set up a 15 megabit satellite uplink on the roof of the RHI building. Somebody (I'm not sure whom) got word out to a satellite Internet provider, and they trucked a semi-portable satcomm rig out to Brooklyn. It took the rest of the morning to haul all of the components up to the roof, run cable, assemble the unit, and get it aimed correctly.

After replacing a dead router in the RHI loft we patched the mesh into the satellite link and restored some Internet connectivity to the rest of the neighborhood. Haxwithaxe and I then set about flashing the rest of the Ubiquiti routers and configuring them for rapid deployment while Sitwon met with some of the relief organizers and Federal liasons. By the time we'd left on Sunday night we'd assembled enough network harware to cover most of, if not all of the neighborhood once it's deployed. Everything just needs to be plugged in, functionality has to be confirmed, and then they'll be online and pushing traffic for users. Sitwon was part of the team that set up the router in Coffee Park, which represents significant growth in the community's network.

There is a saying in the military that goes like this: No plan ever survives first contact with the enemy. This holds true with disaster relief efforts as well. Several miscommunications pertaining to logistics with the Project Byzantium team left us spinning our wheels for a while, most notably because nobody on site at RHI was expecting us. For our part, we were on the wrong side of the building. Also, "reply to all" is not the standard response in all mobile Gmail apps, which meant that some messages didn't go through to folks when we were trying to find crash space in New York City. If someone says that they're going to assist a relief effort by providing anything, from clothing to equipment, you do not have their assistance until the kit is actually, physically, in your hand. Don't count on it until you can drop kick it. It is also vitally important that everyone involved exchange information by any means necessary. When cellular phones didn't work, we used ham radios and FRS. All of these measures came in handy when we were blocks apart. In fact, always have at least two ways of getting in touch with anybody you'll be working with, just in case batteries die, a unit breaks, or something else comes up. As I learned from Haxwithaxe, always have a stash of 550 paracord or climbing rope in your field kit. Haxwithaxe rigged up an ingenious rope and bucket system for hauling equipment up into the loft at RHI, and from there onto the roof of the building for deployment. This included a bunch of bricks that weighed about 50 pounds each, which were used to weight down the satlink's base.

Project Byzantium would like to take this time to thank everybody who was involved in the relief and restoration efforts in New York City, and the Red Hook neighborhood in particular. The Red Hook Initiative is putting in a lot of long hours and working hard to make their bit of the world a better place to live. All of us would not have achieved as much as we did without assistance from all of our contacts on the ground from far and sundry and who are too numerous to mention (probably up into the triple digits). Thanks to Occupy Sandy for hauling and distributing relief supplies to the people of Red Hook, including food and clothing. Thanks to the hacktivists of Telecomix for putting us in touch with folks on the ground and in the streets of Red Hook. Thanks to our contacts and colleagues inside the DC Beltway for tearing through red tape and paperwork day and night. Most of all, thank you everyone who boosted the signal for us on Twitter, mailing lists, Facebook, and elsewhere across the Net and around the globe. Your help was invaluable, and we couldn't have done it without you.