Technology and computational evangelists quickly learn that human psychology is a key component of any project roadmap. The truth in the quip that it took one social genius plus 500,000 scientists and engineers to put a man on the moon can be appreciated when one observes the social issues that surface in technical discussions within even small groups of people. As the name implies, data-intensive computing requires large amounts of data.
My article “Big Money for Big Data” in the June 2012 issue of HPC Source notes that massive parallelism is the only path forward for organizations wishing to cope with the ever-increasing size of big data sets. Just as technology is changing the meaning of “big” data, so is it increasing what is meant by “massive” parallelism. At the moment, GPU programmers work with tens of thousands of threads, while multi-core programmers utilize tens of threads. This technical dichotomy has created a tension between the CTO (chief technical officer) who is responsible for defining the data processing goals to keep an organization competitive over time, and those within the organization vested with the responsibility to evaluate, integrate and manage the technology used to reach those goals within a production environment.
Both parties are making a rational risk/reward calculation, but when and how should the performance gains attributed to massively parallel technology be included in the organization roadmap? While technology ultimately dictates production capability, the choice of the technology roadmap to meet the performance goals for the organization is a very human process. People differ in their risk affinity and perception of a reward. So, they naturally make different judgments. Uncertainty increases the importance of testing, but opinion tends to guide the decision-making process about what to test. Unfortunately, personal biases are a known danger that can inadvertently affect the test process to support a pre-conceived notion.