Chicago Sun-Times
Staff reports on all things politics - from City Hall to Springfield to Washington, D.C.

ORCA Woes: How a tech FAIL stymied the Romney campaign on Election Day

| 3 Comments | No TrackBacks

romney_app.jpg

By Andy Ihnatko

The software development community had never been entertained by its own equivalent of a "Fail" video before. And then, the Romney campaign commissioned ORCA, an ambitious software platform that was supposed to collect onsite voting information from tens of thousands of volunteers nationwide on election day, and send it to strategists at campaign HQ.

You know what I mean by a Fail video? I'm talking about those viral clips that usually begin with someone saying "Here, hold my beer and watch me do this" and ends with the camera rushing over to the spot on the side of the road where this guy is now rocking back and forth, clutching his groin in agony next to broken bits of his skateboard . . . as well as the railing that he apparently believed was made of a soft and spongy kind of iron.

These videos are entertaining because they document an absolutely unambiguous disaster that's being suffered by someone other than you. And they're genuinely fascinating, because . . . well, criminy, man! A higher lifeform wouldn't even consider making a jump from the bed of a moving flatbed truck onto a roadside trampoline. What the hell was this person even thinking?

There were so many fails about ORCA. The webapp was meant to connect tens of thousands of volunteers to a single central webserver This lone server was soon shut down by the campaign's ISP, because the sudden incoming flood of geographically-diverse hits appeared to be a denial-of-service attack. The server appeared to be inadequate for the flood of traffic anyway.

Romney campaign Digital Director Zac Moffatt talked to CNET about the traffic blast and amount of data being served:

"The primary issue was we beta-tested in a different environment than the Garden [Boston Garden, where the 800 campaign staffers were working]. There was so much data coming in -- 1200 records or more per minute -- it shut down the system for a time. Users were frustrated by lag, and some people dropped off and we experienced attrition as a result."

The login IDs and passwords provided to statewide volunteers were incorrect, and barred them from accessing the app. The campaign didn't even release the app until election day. They didn't even release its 60 pages of documentation until the night before...so nearly 40,000 people needed to get up to speed on ORCA at the moment they needed to actually start logging data with it.

And! They were given a URL for ORCA that pointed to a nonexistent http//: address, instead of the correct https:// one.

"Here, hold my beer while I start coding..."

You couldn't say that the success or failure of the Romney campaign hinged on ORCA. In fact Moffatt defends ORCA, telling Ars Technica that his system performed nearly to expectation - and certainly nowhere near the Fail it's being branded as:

"I can tell you that data from 91 percent of counties in the targeted states came in and that we had 14.5 million people who were marked as having voted. And there were 4,397 reports of incidents that we were able to pass to our legal department." 

But it's obvious that the ability to send voters and mobilize volunteers to specific districts that needed shoring up on election day was going to be an immense help. The enormity of the Fail is baffling.

I've barely done any web development at all. It's fair to say that I know only enough PHP and Ruby to order off the menu in a server-side-scripting-themed restaurant. And yet, even I knew that every choice that ORCA's developers had made was almost exactly the wrong way to do that.

So I talked to some folks who do this sort of thing for a living, to see if I was crazy.

My first Whiskey Tango Foxtrot reaction: the idea that all of this traffic would be accepted by just one server, connected to the Internet via a Comcast account. Sure, it was a Business account, but that still made no sense; it's not a service that any rational human would use for a mission-critical project.

Do keep in mind that the overall goal was to make Their Guy the ruler of the free world.

Chris Cieslak is a developer at Stand Alone, a Chicago-based developer of mobile apps. He's also one of the folks behind Buster, the top-rated iPhone app for tracking Chicago public transit. So he knows a thing or two about transacting lots of data from lots of users, though the demands of even a public transit app are likely to be an order of magnitude lower than those of ORCA.

The Buster app uses Amazon Web Services for its backend. Chris immediately suggested that it also would have been perfect for ORCA (at least as he understands the needs of the campaign). AWS is an invisible hero of the Web. I myself use it to host high-volume content on my blog, to ensure that a sudden spike in traffic won't shut down my server, or result in ungodly bandwidth fees.

Amazon offers a service called "Elastic Beanstalk" that hosts and runs your webapp. If there's a spike in traffic or bandwidth, AWS automatically handles it without any intervention whatsoever.

"With Amazon you can upload your app to Beanstalk and it will literally do all the load balancing/spinning up new instances/etc for you," Chris enthused. "You don't need to know ANYTHING. It's all web interface, even."

So campaign volunteers would still connect to a webapp. Unbeknownst to them, it's hosted by Amazon. The campaign pays for metered usage, and campaign headquarters collects data. They never need to know what AWS needed to do in order to handle the load.

Admittedly, Amazon couldn't offer the Romney campaign a bundle deal on the ORCA hosting plus phone service and Cinemax like Comcast. But really, there are more important things than just striking the best bargain.

What about the snafu regarding the https:// URL? It's easy to configure a server to automatically redirect inbound http:// requests to the correct secure server instead of just rejecting the connection.

Chris has a theory, informed by the report that ORCA used 11 database servers but just one application server, as well as by the (kindhearted, and likely accurate) presumption that ORCA's developers weren't total idiots.

"They originally had a load balancer and a bunch of app servers, but for some reason couldn't get it to work properly," Chris hypothesized. "So they tore out the load balancer, which probably handled the http->https redirect, and forgot to add it back to the now-internet-facing app server."

That's pure conjecture, of course. But as with any disaster, from an air crash to a terrorist act to the green-lighting of a network TV show featuring a monkey dressed like a doctor, we all instinctively struggle to find sense and reason. It's a way to reassure ourselves that we don't live in a world of chaos and disorder, nor that the operating system of the human computing platform is fundamentally alpha.

I'm hearing from many developers whose overall take was that the ORCA development team was way, way, way behind schedule on the project, and that they were forced to throw many things together at the last minute. The tone and codswalloping weirdness of ORCA's problems are familiar. It's possible that its developers shipped ORCA in its final form not because the project was ready, but because it was the day before the election.

That would explain the lack of live-fire testing, as well as the almost criminal lack of training given to ORCA's users. Real people trying to access a partially-deployed system would have encountered the http:// redirect problem, as well as all the problems with user login credentials . . . and they would have found those problems when it just didn't matter. Better, it would have identified potential troubles with the user interface, and fixed problems that the Romney volunteers didn't get a chance to encounter thanks to ORCA's larger-scale hopelessness.

How did the Obama campaign avoid this problem? Reportedly, they had their own system up and running months in advance. Their app management team produced a more integrated solution to the data problem. On Tuesday, they simply flipped a switch in the app to change it from "Campaign Mode" to "Election Day" mode. By then, everybody who needed to use the system had already had plenty of experience with it, and all of the showstopping bugs had long been squashed.

I thought ORCA's developers were smart to implement it as a webapp. A web application will run on pretty much any device with an internet connection. Also, setting it up on tens of thousands of devices running various operating systems would have required nothing more than typing a URL into an address bar.

Was that really the right call?

Ben Gottlieb, also of Stand Alone (and developer of the rather lovely "Roger Ebert's Great Movies" app), admits to being prejudiced against that approach. Nonetheless, he makes a strong case for deploying ORCA as a pair of iOS and Android apps. The developer has access to the full bag of tricks provided by the native operating system. ORCA could have been a beautiful, easy-to-use app instead of an interactive webpage. Native apps also have access to all of the resources of the device: the camera, GPS coordinates, the people in the user's address book, etc.

Furthermore, if ORCA's sole functions could be reduced to "collect data from the voting place" and "send data to a central server," then a native app is the only solution that would guarantee at least 50% success. Even after the campaign's central server became inaccessible, the volunteer could continue to collect and log data. If the servers remained offline (or if the volunteer lost his or her Internet connection) they'd then have to read the data to someone over the phone. But the value here is in the actual information about who's voting and who isn't. That's certainly better than nothing, which is what ORCA's webapp delivered.

It was an interesting conversation with Ben. To play devil's advocate: a modern HTML5 webapp can run independently of a server connection. It can access GPS and the device's camera, and it can also store data locally. Amazon, for example, has built a web edition of their Kindle app that's almost indistinguishable from the native edition, feature for feature. Finally, ORCA's users were going to represent people with a widely diverse range of tech savvy and devices. I have an iPhone and I understand how to install an app. Would the Romney campaign have been up to the challenge of walking their volunteers through the process of installing a native app? Would they have made sure that the volunteers were properly equipped to begin with?

(Oh, wait, we already know the answer to that one, don't we? Remember that guy, curled up by the side of the road, still gurgling and groaning?)

The "Obama For America" app (which is still available for download from the App Store) pretty much proves Ben's point, though. Yup, it served up photos and news releases from the campaign trail. But it also was a clever and effective tool for mobilizing volunteers. When a local office needed help staffing phone banks in a certain area, the app put out the word. When a district needed canvassing, the app found volunteers in the user's area, and furnished them with maps and data.

And on Election Day, it pushed "get out the vote" notifications out to its users, targeted by geolocation as needed. It's kind of brilliant. Cold-calling people from a random list risks alienating and annoying people. But anybody who made the active choice to seek out and install the Obama app had clearly indicated a desire to listen to messaging. And at best, they indicated a willingness to throw on a hat and coat and join the street team in their community.

The ORCA Fail continues to inspire discussion and debate. I'm certain that most of the big IT conferences in 2013 will feature in-depth papers analyzing it. Everyone's going to continue to sift through the wreckage for years, looking for wisdom and lessons.

ORCA leaves me with an additional worry. What if it had worked?

Political observers tentatively agree that it wouldn't have changed the results. The electoral math  - Nate Silver's FiveThirtyEight blog had Obama as a 92 percent favorite heading into Election Day - was pointing to a clear win for Obama.

But ORCA, and the Obama campaign's voter apps, point to a new kind of weaponization of data. 2012 was the year in which the databases and models that shaped the 2008 election were wired together. It produced a new understanding of the electorate's weather systems, and allowed strategists to forecast with some accuracy the patterns that were once considered to be an Act of God.

It's just the beginning . . . and not necessarily of something good. 

Remember that every site you visit and every "Like" and "Friend" you create on a social network is being recorded somewhere, as a molecule of data to be sold to someone else, for God knows what purpose. Everything you do that in any way involves a transaction with a computer is being recorded, analyzed, and sold, without your knowledge.

If you're an optimist, your hope is that every election will be won by the candidate who manages to identify a single set of aspirational goals that unites the country. If you're a pessimist, you should worry that it'll be won by the computer that identifies the 218,987 voters in specific districts that can definitely be swung to a party's candidate, by exactly the right message at exactly the right time.

For years, we've been hearing about the power of swing states. In the future, elections might hinge on swing demographics. 

PBS NewsHour's Hari Sreenivasan explored the depths of data mining in Ohio just a week before the election as the Romney and Obama campaigns waged all out war in the all-important state for an ever-more-narrow subset of voters:


At 2:17 PM on Tuesday, November 8, 2016, a computer might convince a campaign strategist that the only voters worth chasing after at this point are married men between the ages of 33 to 41 who visit ESPN.com at least three times a day, have registered a boat within the last seven years, have at least $4000 in credit card debt, and who have never rented a David Lynch movie.

The campaign ads we see at 4 p.m. on that day ought to be hysterically funny.

No TrackBacks

TrackBack URL: http://blogs.suntimes.com/cgi-bin/mt-tb.cgi/51847

3 Comments

This is by far the best written-for-the-non-geek account of the Romney campaign's ORCA #FAIL.

And I've read 'em all.

Weeping with laughter.

-- MrJM

I had publicly expressed that Obama would have been deprived of his rightful election. That was an error or a joke from Destiny. I was again reminded the secret to happiness is never to expect.
No point talking about Zionists. They had tried to fix an election and now they are sad because after having failed with all their money they still are wondering what happened.
I learned the secret to happiness is never to expect.
There is only One Destiny but is not one for us to know. It is the eternity of quantum mechanics that is not constricted within the concepts of Time, Space and Movement as we are instead.
http://www.wavevolution.org/en/index.html

In 1955 Isaac Asimov wrote a short story called "Franchise," about what happens when computers have a better understanding of our politics than we do ourselves. It's a funny, scary little story that everyone should read, because I'm afraid that Asimov nailed it, like he did so many things. The difference here is that we actually still have the power to control our own destiny -- if we're smart enough to start caring. Anybody wanna take odds on the likelihood of that?

Leave a comment