It is easier than ever to measure and monitor people and machines, but the technology of Big Data is not without its shortcomings.
Intuition helps but there's no substitute for big data technology and analysis. Technology to ingest, store, search and analyze big data will proliferate across government and commerical sectors in 2013 and beyond.
Techniques for segmentation, clustering and modeling data allow organizations to revel meaning hidden inside massive amount of data.
Experienced analysts can recommend independent attributes for inclusion in the analysis, variables that have explanatory power in a model or segmentation scheme. When this experience is coupled with data discovery methods to explore data in an unconstrained way, analysts can pinpoint data elements and relationships that may be correlated with a specific outcome and therefore improve the accuracy of the model. It’s the combination of intuition, proven experience, flexible discovery tools, proven statistical methods and a full set of data that lead to the fastest, most significant insights.
This freedom to explore data has other benefits including identifying missing or poor quality data yielding improved standards and collection processes. Most of this can be discovered through profiles and histograms of each data field.
Data discovery tools, when combined with the approaches referenced above, allow analysts to confirm findings and expand the way analyst’s model data. During this process, the analyst may discover new ways to transform data, group continuous data into categorical data or calculate new data attributes to be used in the analysis.
This class of tools has the added advantage of telling a story about your data using a full complement of visualizations designed to focus the audience on insights and conclusions. Once the data story is presented to the business, they rapidly draw conclusions to shape programs.
Look at the work MarkLogic has done with Tableau Software to better understand the enhanced power of data discovery using the full breadth of both unstructured and structured data. Sure, intuition is important. But proven analytical methods that leverage new forms of data in real time will give you the most bang for your buck.