I have voraciously read endless pro and con scenarios about artificial intelligence since first writing about it years ago. At this point, there is no doubt that concerns about the dangers of runaway AI raised by Elon Musk, Stephen Hawking, Bill Gates, Bill Joy and others are genuine.
There also is no doubt whatsoever that the new organizations aimed at mitigating the dangers — OpenAI, The Future of Life Institute, Machine Intelligence Research Institute and others — are extremely important developments.
Clearly, no sane person or organization wants to see, let alone encounter, runaway AI. However, a base problem is that no one knows where the actual crossover point — the edge or tipping point — exists, and thus we mortals are unlikely to be able to prevent it from occurring. Said differently, there is a very high probability that we will misjudge where that crossover point is and will thus go beyond the key threshold. Overshooting is the norm in biology and in most, if not all, evolving systems, but especially man-made ones.
Part of the overshoot problem is related to the fact that we are really talking about the dynamics of nonlinear systems. It is what Nassim Taleb called “black swan” events and saw as a source of the 2008 economic meltdown. That is, where the statistically improbable becomes probable if only because of the highly probable convergence of other statistically improbable trends or events. This leads directly into the other part of the problem, human nature, aka hubris. While we talk a good prophylactic game, and all those working on AI want to believe, “us versus them” constitutes “realpolitik.”