Weapons of Math DestructionThe hand-wringing continues about robots, and for whose jobs they’re coming next. But the “robots” needn’t be tangible to transform our lives. Actually, they’re already here, in the form of big data algorithms – predictive mathematical models fueled by astounding computing power and endless supplies of data.

This doesn’t have to be ominous.  Well-designed models, properly applied, are a beautiful thing.  But some models are toxic, and such bad modeling has become ubiquitous, with far-reaching impacts on where we go to school; how we get a job and how we’re evaluated; how we get and maintain financial credit and insurance; what information we access online; how we participate in elections and civic life; and how we are treated by law enforcement and the judicial system.  That’s why Cathy O’Neil’s book Weapons of Math Destruction is such an important book for our time. 

O’Neil is a data scientist who writes like Malcom Gladwell, using interesting stories that make a complicated topic easily accessible.  With a Ph.D. in mathematics from Harvard, O’Neil worked as a hedge fund quant, with a front-row seat for the 2008 crash.  She left the hedge fund in 2011 to work for e-commerce start-ups, becoming yet more dissolusioned by the “dark side of big data.”  O’Neil is now an evangelist for understanding both the promise and also the dangers of big data algorithms.

And there’s danger aplenty.  O’Neil’s name for big data gone bad is weapons of math destruction, or WMDs.  O’Neil explores numerous WMDs spreading across our society, driving harmful results in teacher performance evaluations; college admissions and costs; online advertizing; policing, sentencing, and parole; the employment market; on-the-job performance evaluation; consumer credit; insurance; and civic life in general.

Unlike beneficial, effective predictive models (think baseball as Moneyball), WMDs share three dangerous flaws:

  1. Lack of Transparency.  WMDs are opaque.  We have limited access to the statistical data they use, and we cannot see how the model interprets the data.
  2. Lack of Statistical Rigor.  WMDs rely on data that are not highly relevant to the predicted outcomes.  Instead, stand-in, or proxy, statistics are used, which may miss the mark.  Batting averages are highly relevant to a baseball player’s performance, but my zip code, though suggestive, may not be highly relevant to whether I timely pay my debts.
  3. Lack of Continual Updating.  WMDs are commonly untested by reality.  There may be limited or no effort to compare the model’s predictions with what actually happens, so the opportunity to adjust and correct the model is lost. Worse yet are self-fufilling WMDs, such as credit models that trap the poor in poverty, or sentencing models that breed recidivism.

Most importantly, WMDs cloak flawed, unverified assumptions in the unappealable “objectivity” of science:

Models, despite their reputation for impartiality, reflect goals and ideology…. Our own values and desires influence our choices, from the data we choose to collect to the questions we ask.  Models are opinions embedded in mathematics.

Those constructing the models build them upon someone’s opinion of what is “success” for the model.  For example, does a successful law enforcement model focus on targeting limited law enforcement resources on crime in general, or solely on violent crime?  This makes a big difference to white-collar criminals and their victims, doesn’t it?

And when highly relevant input data is scarce (or too “hard” to use), the selection of proxy data can be highly subjective.  For example, if “success” is having better results in secondary education, are student standardized test scores highly predictive of teacher performance, or are the student test results driven by too many other factors outside of the teacher’s reach?  Since those other factors are really hard to measure, the readily quantified test scores are such a tempting proxy for teacher quality, aren’t they?  And do we bother to go back and look independently at whether success is actually improved, or do we simply continue to measure the flawed proxy, using a deceptive feedback loop to declare success?

The solution starts, as always, with awareness of what’s actually going on around us.  Big data predictive analytics is here to stay.  It isn’t inherently bad, and it needn’t be destructive.  But as the object of the analytics, we need to be aware of what works well and what does not. Weapons of Math Destruction is an eye-opener.