Nov 30 2019
UPDATE: 20191230 - Uploaded a copy to my Peertube account.
From time to time I carp about how generally lousy our bandwidth is out here. Verizon (our CLEC in the Bay Area) has all but given up on maintaining their infrastructure out here, aside from the bare minimum to keep the copper from turning to verdigris. They gave up on deploying fiber some years ago (local mirror) some years ago, and from the poking around I've done on their side of the fence, their general stance in the Bay Area appears to be "Get everyone on celllar so we can ignore the rest of the network." Which sucks and does nobody but Verizon's shareholders any good in the long run.
Anyway, after yet another afternoon wasted on the phone with tech support because our speed fell to pre-dialup speeds for reasons unknown, I decided to take the bull by the horns and put some old skills to work. Out came the fox and hound and my old lineman's test set, and I set about figuring out which lines in the fist-sized morass of ancient wiring outside, if any, were actually hooked up. The way a fox and hound works is, you clip or plug a tone generator (the fox) into the line you want to trace, and you use a matching inductive probe (the hound) to listen for the sound. Telephony cables are almost never insulated so you don't need to touch the copper directly, the faint EM field around the wire is sufficient.
I was able to trace the line successfully, but in so doing I found out why our bandwidth was so terrible. Thankfully, after demonstrating the problem to the contractor that Verizon sent out, we were able to work together to not only rip out the dead cabling outside, but mostly resolve the interference.
Oct 14 2018
Long time readers are probably wondering where I've been lately. The answer is kind of long and is worth a post all on its own. The short version of the story is, work's been eating me alive lately. This is our busiest time of year and it's been all hands on deck for a couple of weeks now. In point of fact, last week was our quarterly all-hands meeting, where everybody on my team was flown into town for a solid week of meetings. All day, every day. Most of my visible activity lately took the form of parts of my exocortex running on automatic with some hit-and-run posting while waiting for the coffee maker at work to top me up in between meetings.
This also means that I haven't had a whole lot of patience for interacting with people. Not in the sense that people can feel frustrated with other people or their actions, but in the sense that interacting with people in a meaningful way - having a real conversation - takes more compute cycles than I have available right now. After fourteen hours in a conference room with 40 other people, not only am I out of social, but I'm mentally exhausted.
Dec 18 2016
You've already read my opinion of the 2016 election's outcome so I'll not subject you to it again. However, I would like to talk about some weird stuff I (we, really) kept noticing on Twitter in the days and weeks leading up to Election Day.
As I've often spoken of in the past, a nontrivial portion of my Exocortex is tasked with monitoring global activity on Twitter by hooking into the back-end API service and pulling raw data out to analyze. Those agents fire on a stagged schedule, anywhere from every 30 minutes to every two hours; a couple of dozen follow specific accounts while others use the public streaming API and grab large samples of every tweet that hits Twitter around the world.
If you want to look at a simplified version of that agent network to see how it works I've made it available on Github. As you can see, the output of that particular agent network is batched into e-mails of arbitrary size using the Email Digest Agent and is sent to one of my e-mail addresses as a single batch. The reason for this is twofold; it's easier to scan through a large e-mail and look for patterns visually than it is to scan through several dozen to several hundred separate messages in sequence, and it uses fewer system resources on my e-mail provider to store and present to me that output.
Six or seven weeks before Election Day, Lifeline (the recognition code for the agent network which carries out these sorts of tasks for me) started sending me gigantic e-mail digests every hour or so, containing something like several hundred tweets at a time (the biggest was nearly a thousand, as I recall). Scanning through those e-mails showed that most of the tweets were largely identical, save for the @username that sent them. Tweets about CNN and the Washington Post being GRU and SVR disinformation projects or on-the-ground reporting tagged with #fakenews. Links pointing to Infowars articles (the tweets consisted of the titles of posts, links, and the same sets of hashtags; if you ran the Twitter-compressed URLs through a URL unshortener they all pointed to the same posts). Anti-Bernie and anti-Hillary tweets that all had the same content and the same hashtags. Trump as the second coming messages and calls to action. Rivers of bile directed at political comentators and reporters. Links to fake Wikileaks Podesta e-mails that went to Pastebin or other post-and-forget sites (there wasn't even enough data in the fakes to attempt to validate them (by the bye, the method linked to is really easy to automate)). I saw the same phenomenon with #pizzagate tweets, only the posts came in shorter bursts more irregularly. It went on and on, day and night for weeks, hundreds upon hundreds of unique copies of the same text from hundreds of different accounts. I had to throw more CPUs at Exocortex to keep up with the flood.
All of these posts, when taken together as groups or families consisted of exactly the same text each and every time, though the t.co URLs were different (a brief digression: Twitter's URL shortening service seems to generate different outputs for the same input URL to implement statistics gathering and user tracking as part of its business strategy). Additionally, all of those posts went up more or less within the same minute. The Twitter API doesn't let you pull the IP addresses tweets were sent from but the timestamps are available to the second. If you looked at the source field of each tweet (you'll need to scroll down a bit), they were all largely the same, usually empty (""), with a few minor exceptions here and there. The activity pattern strongly suggests that bots were used to strafe circles of human-controlled accounts on Twitter that roughly correspond to memetic communities. Figuring that somebody had already done some kind of visualization analysis (which I suck at), I had Argus (one of my web search bots) do some digging and he found a bunch of pages like this study, which seem to back up my observations.
The sort of horsepower needed to create such an army of bots would be very easy to assemble: Buy a bunch of virtual machines on Amazon's EC2. Write a couple of bots using Ruby or Python. Sign up for a bunch of Twitter accounts or just buy them in bulk. Make a Docker image that'll effectively turn one EC2 instance into as many as you can reasonably run without crashing the VM. Deploy lots and lots of copies of your bots into those Docker containers. Use an orchestration mechanism like Ansible to configure the bots with API keys and command them en masse; if you're in a time crunch you could even use something like pssh to fire them all up with a single command. Turn them loose. If you've been in IT for a year, this is a Saturday afternoon project that won't cost you a whole lot, but could make you a lot of money.
"Well, yeah, there was an army of bots advertising on Twitter. What else is new?" you're probably saying.
What I am saying is simply this: This post describes a little bit about how this sort of media strategy works, what the patterns look like at the 50000 foot view, and my/our observations. I don't think I did anything really ground-breaking here, only in the sense that I used a bunch of AI systems that stumbled across what was going on by accident. It was the hardcore data scientists who did the real academic work on it (though that work is a bit inaccessible unless you're a computer geek).
Memetic warfare is here, and our social networks at the battlegrounds. Armor up.
Dec 01 2016
Sometimes, very occasionally, when using the Lastpass plugin with Google Chrome, you may find that Lastpass will start acting wonky. Specifically, if you've had Chrome running for a couple of days, you will notice that Lastpass has logged you out, even if you're in an Incognito Window. When clicking on the browser plugin's icon, you will be able to log into it as usual; multifactor authentication will similiarly work as expected. If you wait a few seconds, the plugin's icon will go dark again. If you're quick and drop into "My Vault," you'll see that screen for a second or two before you get bounced out again. You won't be able to log into anything, and you'll eventually start cursing the day you decided to stop using a password manager like Keepass. You might do this a dozen or two times, scratching your head all the while.
To break out of this frustrating loop, clear your browser cache (Chrome menu icon -> History -> History -> Clear Browsing Data, check Browsing history and Download history, uncheck everything else -> Clear browsing data), fully terminate Chrome (don't just close all of your windows), and start it up again. That should fix the problem.