Is AOC right about AI?

Conservative Twitter is up in arms today over Rep. Alexandria Ocasio-Cortez saying at an MLK Day event that algorithms are biased. (Of course “bias” has been translated into “racism.”) The general response from the right has been, “What a dumb socialist! Algorithms are run by math. Math can’t be racist!” And from the tech experts on Twitter: “Well, actually….”

I have to put myself in the latter camp. Though I’m not exactly a tech expert, I’ve been researching the impact of technology like AI and algorithms on human well-being for a couple of years now, and the evidence is pretty clear: people have bias, people make algorithms, so algorithms have bias.

When I was a kid, my dad had this new-fangled job as a “computer programmer”. The most vivid and lasting evidence of this vocation was huge stacks of perforated printer paper and dozens upon dozens of floppy disks. But I also remember him saying this phrase enough times to get it stuck in my head: “garbage in, garbage out.” This phrase became popular in the early computer days because it was an easy way to explain what happened when flawed data was put into a machine – the machine spit flawed data out. This was true when my dad was doing…whatever he was doing… and when I was trying to change the look of my MySpace page with rudimentary HTML code. And it’s true with AI, too. (Which is a big reason we need the tech world to focus more on empathy. But I won’t go on that tangent today.)

When I was just starting work on my book, I read Cathy O’Neil’s Weapons of Math Destruction (read it.), which convinced me beyond any remaining doubt that we had a problem. Relying on algorithms to make decisions for us that have little to no oversight and are entirely susceptible to contamination by human bias – conscious or not – is not a liberal anxiety dream. It’s our current reality. It’s just that a lot of us – and I’ll be clear that here I mean a lot of us white and otherwise nonmarginalized people – don’t really notice.

Maybe you still think this is BS. Numbers are numbers, regardless of the intent/mistake/feeling/belief of the person entering them into a computer, you say. This is often hard to get your head around when you see all bias as intentional, I get that, I’ve been there. So let me give you some examples:

There are several studies showing that people with names that don’t “sound white” are often passed up for jobs in favor of more “white-sounding” names. It reportedly happens to women, too. A couple of years ago, Amazon noticed that the algorithm it had created to sift through resumes was biased against women. It had somehow “taught itself that male candidates were preferable.” Amazon tweaked the algorithm, but eventually gave up on it, claiming it might find other ways to skirt neutrality. The algorithm wasn’t doing that with a mind of its own, of course. Machine-learning algorithms, well, learn, but they have to have teachers, whether those teachers are people or gobs of data arranged by people (or by other bots that were programmed by people…). There’s always a person involved, is my point, and people are fallible. And biased. Even unconsciouslyEven IBM admits it. This is a really difficult problem that even the biggest tech companies haven’t yet figured out how to fix. This isn’t about saying “developers are racist/sexist/evil,” it’s about accounting for the fact that all people have biases, and even if we try to set them aside, they can show up in our work. Especially when those of us doing that work happen to be a pretty homogeneous group. One argument for more diversity in tech is that if the humans making the bots are more diverse, the bots will know how to recognize and value more than one kind of person. (Hey, maybe instead of trying to kill us the bots that take over the world will be super woke!)

Another example: In 2015, Google came under fire after a facial recognition app identified several black people as gorillas. There’s no nice way to say that. That’s what happened. The company apologized and tried to fix it, but the best it could do at the time was to remove “gorilla” as an option for the AI. So what happened? Google hasn’t been totally clear on the answer to this, but facial recognition AI works by learning to categorize lots and lots of photos. Technically someone could have trained it to label black people as gorillas, but perhaps more likely is that the folks training the AI in this case simply didn’t consider this potential unintended consequence of letting an imperfect facial recognition bot out into the world. (And, advocates argue, maybe more black folks on the developer team could have prevented this. Maybe.) Last year a spokesperson told Wired: “Image labeling technology is still early and unfortunately it’s nowhere near perfect.” At least Google Photos lets users to report mistakes, but for those who are still skeptical, note: that means even Google acknowledges mistakes are being – and will continue to be – made in this arena.

One last example, because it’s perhaps the most obvious and also maybe the most ridiculous: Microsoft’s Twitter bot, Tay. In 2016, this AI chatbot was unleashed on Twitter, ready to learn how to talk like a millennial and show off Microsoft’s algorithmic skills. But almost as soon as Tay encountered the actual people of Twitter – all of them, not just cutesy millennials speaking in Internet code but also unrepentant trolls and malignant racists – her limitations were put into stark relief. In less than a day, she became a caricature of violent, anti-semitic racist. Some of the tweets seemed to come out of nowhere, but some were thanks to a nifty feature in which people could say “repeat after me” to Tay and she would do just that. (Who ever would have thought that could backfire on Twitter?) Microsoft deleted Tay’s most offensive tweets and eventually made her account private. It was a wild day on the Internet, even for 2016, but it was quickly forgotten. The story bears repeating today, though, because clearly we are still working out the whole bot-human interaction thing.

To close, I’ll just leave you with AOC’s words at the MLK event. See if they still seem dramatic to you.

“Look at – IBM was creating facial recognition technology to target, to do crime profiling. We see over and over again, whether it’s FaceTime, they always have these racial inequities that get translated because algorithms are still made by human beings, and those algorithms are still pegged to those, to basic human assumptions. They’re just automated, and automated assumptions, it’s like if you don’t fix the bias then you’re automating the bias. And that gets even more dangerous.”

(This is the “crime profiling” thing she references, by the way. I’m not sure where the FaceTime thing comes from but I will update this post if/when I get some context on that.)

Update: Thanks to the PLUG newsletter (which I highly recommend) I just came across this fantastic video that does a wonderful job of explaining the issue of AI bias and diversity. It includes a pretty wild example, too. Check it out.


Driverless empathy

Algorithms and big data affect our lives in so many ways we don’t even see. These things that we tend to believe are there to make our lives easier and more fair also do a lot of damage, from weeding out job applicants based on unfair parameters that ignore context to targeting advertisements based on racial stereotypes. A couple of weeks ago I got to see Cathy O’Neil speak on a panel about her book Weapons of Math Destruction, which is all about this phenomenon. Reading her book, I kept thinking about whether a more explicit focus on empathy on the part of the engineers behind these algorithms might make a difference.

The futurist and game creator Jane McGonigal suggested something similar to me when I spoke to her for this story earlier this year. We talked about Twitter, and how some future-thinking and future-empathizing might have helped avoid some of the nasty problems the platform is facing (and facilitating) right now. But pretty soon Twitter may be the least of our worries. Automation is, by many accounts, the next big, disruptive force, and our problems with algorithms and big data are only going to bet bigger as this force expands. One of the most urgent areas of automation that could use an empathy injection? Self-driving cars.


I’ll be honest – until very recently I didn’t give too much thought to self-driving cars as part of this empathy and tech revolution that’s always on my mind. I thought of them as a gadget that may or may not actually be available at scale over the next decade, and that I may or may not ever come in contact with (especially while I live in New York City and don’t drive). But when I listened to the recent Radiolab episode “Driverless Dilemma,” I realized I’d been forgetting that even though humans might not be driving these cars, humans are deeply involved in the creation and maintenance of the tech that controls them. And the decisions those humans make could have life and death consequences.

The “Driverless Dilemma” conversation is sandwiched around an old Radiolab episode about the “Trolley Problem,” which asks people to consider whether they’d kill one person to save five in several different scenarios. You can probably imagine some version of this while driving: suddenly there are a bunch of pedestrians in front of you that you’re going to hit unless you swerve, but if you swerve you’ll hit one pedestrian, or possibly kill yourself. As driverless technology becomes more common, cars will be making these split-second decisions. Except it’s not really the cars making the decisions, it’s people making them, probably ahead of time, based on a whole bunch of factors that we can only begin to guess at right now. The Radiolab episode is really thought-provoking and I highly recommend listening to it. But one word that didn’t come up that I think could play a major role in answering these questions going forward is, of course, empathy.

When I talked with Jane McGonigal about Twitter, we discussed what the engineers could have done to put themselves in the shoes of people who might either use their platform for harassment or be harassed by trolls. Perhaps they would then have taken measures to prevent some of the abuse that happens there. One reason that may not have happened is that those engineers didn’t fit into either of those categories, so it didn’t occur to them to imagine those scenarios. Some intentional empathy, like what design firms have been doing for decades (“imagine yourself as the user of this product”) could have gone a long way. This may also be the key when it comes to driverless cars. Except the engineers behind cars’ algorithms will have to consider what it’s like to be the “driver” as well as other actual drivers on the road, cyclists, pedestrians, and any number of others. And they’ll have to imagine thousands of different scenarios. An algorithm that tells the car to swerve and kill the driver to avoid killing five pedestrians won’t cut it. What if there’s also a dog somewhere in the equation? What if it’s raining? What if the pedestrians aren’t in a crosswalk? What if all of the pedestrians are children? What if the “driver” is pregnant? Car manufacturers say these are all bits of data that their driverless cars will eventually be able to gather. But what will they do with them? Can you teach a car context? Can you inject its algorithm with empathy?