The Washington Post has a great article today about the paradox of humans and automation. John D. Lee, a professor at the University of Wisconsin at Madison has a choice quote:
The better you make the automation, the more difficult it is to guard against these catastrophic failures in the future, because the automation becomes more and more powerful, and you rely on it more and more.
While the article talks about a few ways to help with the problem, such as having operators occassionally train with no automation or making more “polite” automation, there’s no silver bullet. There is one great line which really gets to the heart of the matter:
The more reliable the system, the more likely it is that humans in charge will “switch off” and lose their concentration, and the greater the likelihood that a confluence of unexpected factors that stymie the algorithm will produce catastrophe.
This is really the biggest problem, which receives very little attention I think. We’re running a number of studies here at HALab which are looking at this confluence of humans, automation and the conflicts between them.
Metrorail Crash May Exemplify Automation Paradox, Washing Post, 6/29/09