Deep learning gone wild, direct neural interface techniques, and hardware acceleration of neural networks.

Jun 05, 2016

There is a graphic novel that is near and dear to my hearts by Warren Ellis called Planetary, the tagline of which is "It's a strange world. Let's keep it that way." This first article immediately made me go back and reread that graphic novel...

The field of deep learning has been around for just a short period of time insofar as computer science is concerned. To put it in a nutshell deep learning systems are software systems which attempt to model highly complex datasets in abstract ways using multiple layers of other machine learning and nonlinear processing algorithms stacked on top of one another, the output of one feeding the input of another. People are using them for all sorts of wild stuff these days, from sifting vast databases of unstructured data for novel patterns to new and creative ways to game the stock, bond, and currency markets. Or, if you're Terence Broad of London, accidentally get DMCA takedown requests.

Broad is working on his master's degree in Creative Computing, and as part of that work developed a deep learning system which he trained on lots of video footage to see if it became a more effective encoder by letting it teach itself how to watch video, in essence. It's not an obvious thing but representing video as data ("encoding") is a wild, hairy, scary field... there are dozens of algorithms for doing so and even more container formats for combining audio, video, and other kinds of data into a single file suitable for storage and playback. Broad built his deep learning construct to figure out more efficient ways of representing the same data in files all by itself, without human signal processing experts intervening. He then ran the movie Bladerunner through his construct, dumped its memory and uploaded it to video sharing site Vimeo. What happened shortly thereafter was that one of Warner Brothers' copyright infringement detection bots mistook the video output by Broad's deep learning construct by dumping its memory for a direct rip of the movie because the output of his deep learning system was so accurate and sent an automatic takedown request to the site because it couldn't tell the difference from the original. One of the videos in the article is a short side-by-side comparison of the original footage to the construct's memory. There are differences, to be sure - some of the sequences are flickering, rippling blotches of color that are recognizable if you look back at the original every few seconds, but other sequences are as a good a replica as I've ever seen. Some of the details are gone, some of the movement's gone, but a surprising amount of detail remains. If you grew up watching nth-generation bootlegs of the Fox edit of Bladerunner where the color's all messed up, you know what I'm talking about.

Fellow children of the 80's may have grown up the way I did dreaming of the creation of DNI, direct neural interface between organic brains and computers. My cyberpunk-flavored Twitter silliness of a few nights ago aside, a surprising development in the field of neural dust is going to be presented in just a couple of days. Professor Michel Maharbiz of the University of California at Berkeley is going to present work he's done using neural dust in vivo to control the actions of large insects. Maharbiz has been implanting various types of neural dust into the brains of very large beetles along with subdural transceivers that communicate with the neural dust using ultrasound and what appears to be a very short range transceiver that communicates with a "backpack" consisting of an off-the-shelf microcontroller that packs a longer-range transceiver among other apparatus. Via this wireless link, professor Maharbiz is able to very accurately steer the beetles as they fly and walk around from a distance because the use of neural dust allows for very precise control over which muscles are triggered inside the beetle, how they contract, and at what times. In a similar line of endeavor neural lace is back in the news thanks to Elon Musk speaking at Codecon a few weeks back. While there he talked about the possibility of employing neural lace implants on a large scale in an attempt to keep organic brains (albeit heavily modified ones) higher up in the food chain when living in the same world as advanced deep learning and AI constructs. It's pretty clear that he was talking about interfacing humans with exocortices to provide a better chance by providing a significantly faster I/O interface for the human brain. He's got a point in that speech recognition will only get you so far, and manual data input has some pretty serious failure modes (RST aside, you can only type so fast, even with next-generation virtual keyboards like those built into Android nowadays).

One last thing I wanted to put on everybody's radar is another product for speeding up machine learning and deep learning applications. This one comes out of Google's labs and is called the Tensor Processing Unit, ostensibly after their open source machine intelligence development kit called TensorFlow. As far as we know the TPU is a hardware platform which offloads many of the machine operations that underly practical AI onto dedicated-purpose silicon which is orders of magnatude faster than even the GPUs normally used for neural processing and AI work these days. The TPUs are also designed to accept less accuracy in the data they process in much the same way that human brains operate with and process incomplete or sometimes misleading data yet somehow still come up with usable (note I didn't say 'correct') answers. The TPUs are designed to use significantly less power than the rest of the systems they're plugged into, as anybody who's ever had to buy a one kilowatt power supply to drive their new graphics card can attest to (not that I've had to do that recently or anything). This seems due, in part, to the fact that the ASICs are designed to deal with incomplete or partially inaccurate data, meaning that more of the transistors on board the chips are dedicated to crunching raw numbers and not dealing with the inaccuracies in same, resulting in a greater number of operations per second. As seems to be common for Google they've been running these dedicated hardware devices in their own data centers for over a year now, as evidenced by some improvements in Gmail, Voice Search and Google Maps. Something I found interesting was the turnaround time: From prototype of the first TPU chips to full implementation in production was just 22 days in-house. No parts of me have found out yet if Google plans to sell Tensor Processing Units on any markets or anything about their software development kits (or even if TensorFlow supports them). Time will tell.