Big Data & Digital Marketing

In my opinion, Spark should NOT be compared with Hadoop but with MapReduce. However, people usually compare Hadoop and Spark (probably because they are buzzwords).

5 things to keep in mind:

1. They do different things -

Hadoop is a distributed data infrastructure (HDFS),

Spark is a data-processing tool.

2. Hadoop is more complete -

Hadoop also includes a data-processing tool (MapReduce),

Spark does not have its own filesystem and needs to be integrated with some.

3. Spark is (much) faster -

MapReduce operates in step;

Spark operates in one shot (because it is in-memory).

4. Speed is not always what you need -

For batch processing you do not need Spark's high velocity;

Common applications for Spark are those requiring real-time analysis.

5. Failure recovery -

both Hadoop and Spark are resilient to failures.

There is a substantial spending on big data, with more than 75% of companies (from different industries) investing in big data in the next two years.

Each industry vertical has its own challenges and solutions. Here is a great article and infographics by simplilearn that describe for 10 verticals both main challenges and applications.

The 10 industries are:

1. Banking

2. Communication

3. Healthcare

4. Education

5. Manufacturing

6. Government

7. Insurance

8. Retail

9. Transportation

10. Energy

A list of awesome visualisations!

Not only are they clear and beautiful, but also they keep you focused on the subject and succeed in conveying the story behind the data.

If you need to stimulate your creativity for your next dashboard or infographics have a close look at them!

Here is my list of top 5:

1. Are we alone?

Stimulate user’s imagination

http://www.bbc.com/future/story/20120821-how-many-alien-worlds-exist

5. Infographics with Real images

http://marion-luttenberger.squarespace.com/#/infographics/

Predictive analytics, or the ability to make accurate predictions, is based on good data and good algorithms. Every sensible data scientist knows this. It's good to see that this knowledge is spreading. Data alone, no matter how big, cannot give the right answers.

I believe that useful predictive analytics gives *accurate* predictions, not *exact* predictions. Predictive analytics which gives exact predictions is used in Science, not in Business.

1. IBM is improving its cloud-based elastic data warehouse;

2. Cloudera announces Ibis: a project for letting data-scientists use Python directly on Hadoop;

3. Amazon is making Elastic MapReduce almost plug-and-play.

What's especially interesting about all of these announcements is that they are signs of a maturing market. No one's introducing a brand new data processing engine. Instead, IBM is improving, acquiring and competing; Amazon is streamlining; Cloudera is re-platforming; and Apache NiFi is integrating.

Data Lakes as the Big Data alternative to Data Warehouses.

Their main features are:

1. Contain ALL of the data, both structured and unstructured, internal and external;

2. Store all data in RAW format;

3. No SCHEMA is imposed on the data.

I like Data Lakes because they:

1. Are flexible

2. Break silos

This increases the success rate of Big Data projects.

Are you a Data Scientist coding in Python? Then you might find this tutorial helpful.

Virtualenvwrapper is a tool for managing virtual environments under Python.

This guide and video tutorial well explain the basic usage of the package and also give suggestions on how to go beyond that.

My 3 takeaways from 2015 Gartner's research on Big Data:

1. Big Data solutions and technologies are becoming mainstream;

2. Big Data is mainly used for enhancing customer experience;

3. Business Leaders are as active as CIO/CTO in launching Big Data projects.

Organisations are overwhelmingly targeting enhanced customer experience as the primary goal of big data projects (64 percent). Process efficiency and more-targeted marketing are now tied at 47 percent. As data breaches continue to make headlines, enhanced security capabilities saw the largest increase, from 15 percent to 23 percent.

Last year, 37 percent of big data projects were initiated by the CIO, while 25 percent were initiated by business unit heads. In 2015, the roles are nearly tied, at 32 percent and 31 percent, respectively.

How is Big Data changing the banks business? And how this can add value for customers?

Today it is imperative to use machine learning for detecting frauds and mitigating risks. The next step should be solidifying the relationship with clients, by offering what they really need, not what the bank wants to sell.

The 6 uses of Big Data in banking

1. Increase efficiency - shorten the time to get information from db

2. Reduce cost - in maintaining the data infrastructure

3. Prevent fraud - self-evident

4. Accountability - better evaluation of clients' risk

5. New revenue streams - the more you know the more you can offer

6. Client satisfaction - by offering a better customer experience

A key topic when trying to leverage Big Data is data integration.

Data integration can take long time and is crucial to really benefit from big data.

Silo breaking, made possible by data integration, is what can let a company move from applying short-term tactics to creating a long-term strategy.

It goes without saying that without some good questions (i.e. business objectives) even good data integration is of little use.

One good suggestion for defining the goal is to put the customer in the center, for real.

8 basic question to help you get started on the right track:

1. Who is your customer?

2. What do they need?

3. What data should you be looking for to see if you are delivering?

4. Where is the data coming from?

5. How is it stored/organized?

6. Who looks at it and how often?

7. Who is analyzing it?

8. Who is presenting it?

SAP HANA and Hadoop are very different, that's why they could be a good combination in terms of complementing each other.

For example, SAP HANA is in-memory and uses predefined schema, while Hadoop is on disk and has no schema.

A scalable column-oriented database management for real-time analytics (SAP HANA) meets a technology platform that supports any kind of data for analyzing massive amount of data (Hadoop).

After screening the whole Twitter looking for #BigData posts, counting tweets, retweets, favourites and followers, the article reports the top50 accounts that best performed in the month of July 2015.

Here are the top5:

1. KPMG  (Score = 100.0%)

2. World Economic Forum (83.8%)

3. CloudEXPO ® (57.6%)

4. UN Development (45.4%)

5. Popular Science (41.4%)

Following them could be a wise idea for those who want to know more about the field!

"Good intentions don't work, but a repeatable, measurable and improvable process can" by Anurag Gupta, at Amazon.

Nobody wants to hire the wrong person and nobody wants to work for the wrong company. By following processes, it is possible to reduce the probabilities for things like these to happen.

1. Raise the bar, steadily and honestly;

2. Build a hiring funnel. By consistently repeating a process, you improve it. Do yourself a favour: filter candidates at each step;

3. Don't go easy on culture. This is the hardest to standardize, and yet it's one of the main reason why employee will not stay.