by Felix Salmon
"Disruption by data is a good thing — as long as humans stay in control of it."
Jim Lerman's insight:
I found this to a marvelously well-informed piece and informative as well. Among numerous compelling ideas Salmon shares, I really liked his "four stages of the rise of quants":
1. Pre-disruption - the time before big data takes over (example: pre-2002, before the Oakland As were moneyballed)
2. Disruption - big data analytics take over (examples: 2002 Oakland As, Obama's 2012 election campaign)
3. Overshoot - too much dependence on data, to the exclusion of institutional knowledge of the field (examples: statistics-driven policing, the 2008 mortgage crisis)
4. Synthesis - combining computers with human judgment (examples: best computers can beat best humans at chess, but they can be beaten by humans aided by computers, Boston Red Sox winning the World Series for the first time by using both statistics and scouting).
Salmon devotes important space to a discussion of Campbell's Law": "The more any quantitative social indicator is used for social decision making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor."
Campbell's Law is certainly in evidence in the manner in which high stakes testing has produced widespread cheating from top to bottom in schools. This was well documented by David Berliner (past President of AERA) in the excellent book he co-authored with Sharon Nichols: Collateral Damage: How High-Stakes Testing Corrupts America's Schools