Corporate Challenge of Big Data

Last week, President Obama launched the Administration's new Open Data Policy and Executive Order aimed at ensuring that data released by the government will be as accessible and useful as possible.

Project Open Data is an online, public repository intended to foster collaboration and promote the continual improvement of the Open Data Policy.

Pinterest has been riding an exponential growth curve, doubling every month and half. Pinterest’sYashwanth Nelapati and Marty Weiner, tell the dramatic story of Pinterest’s architecture evolution in a talk titled Scaling Pinterest.

 

Two lessons from the talk:

Architecture is doing the right thing when growth can be handled by adding more of the same stuff.Evaluate tool choices with a preference for tools that are: mature; really good and simple; well known and liked; well supported; consistently good performers; failure free as possible; free. Using these criteria they selected: MySQL, Solr, Memcache, and Redis.

Unfortunately, the job description for a data scientist has become quite lofty. Unless your company is Google-level cool, you're going to struggle to hire your big data dream team (well, at least right now), and few firms out there could recruit them for you. Ultimately, most organizations will need to enlist the support of existing staff to achieve their data-driven goals, and train them to become data scientists. To accomplish this, you must determine the basic elements of data scientist "DNA" and strategically splice it into the right people. - By Jim Kaskade, CEO, Infochimps

John Treadway of Cloud Technology Partners (also called CloudTP), discussed the state of the industry in an interview with SiliconAngle founding CEO John Furrier and Wikibon's Stu Miniman.

Furrier opens the discussion by asking Treadway if the growing number of cloud acquisitions is a sign that the market is consolidating.

Treadway concludes the interview by pointing out that EMC's cloud expertise gives it an edge over traditional consultancies that are less familiar with modern architectures.

In the spirit of this International Year of Statistics, the McDonald’s analytical approach speaks to some of the fundamentals of the field. It is a great example of how statistics can dramatically affect a mega-corporation

In industries where good talent can be hard to find, companies are using algorithms to scour the Web for recruits, and in the process, they are challenging some traditional notions of hiring.

AWS Identity and Access Management (IAM) enables you to create multiple Users and manage the permissions for each of these Users within your AWS Account.

Rufus Pollock, Founder and Co-Director of the Open Knowledge Foundation, discusses why current hype around big data misses the point, and why the real revolution is small data, loosely joined

Dr. Data is IN... The U.S. spends more on healthcare than any other nation: around $9,000 per person in 2012. Can data scientists help?

Banks are trying to become more focused on the specific needs of their customers and less on the products that they offer. They need to:

Engage customers ininteractive/personalized conversations (real-time)Provide a consistent, cross-channel experience including real-time touch points like web and mobileAct at critical moments in the customer sales cycle (in the moment)Market and sell based on customer real-time activities

Kathy Giusti founded the Multiple Myeloma Research Foundation and John Quackenbush is the director of the Center for Cancer Computational Biology at the Dana-Farber Cancer Institute form a new collaboration to upend cancer research...
Two or three years down the road, tens of thousands of patients could become part of this large data resource, and help drive innovation...
We needed a way to put all of the data into a system so that researchers, clinicians, and bioinformatics specialists could use it. Late in our search for a partner, we saw what GenoSpace was doing and went, "Oh my gosh, this is amazing, amazing technology."

Aditya Agarwal, Director of Engineering at Facebook, gave an excellent "Scale at Facebook" talk that covers their architecture, but the talk is really more about how to scale an organization by preserving the best parts of its culture. 

 

The Key Points :-

 

1. Scaling takes Iteration 
2. Don't Over Design 
4. Choose the right tool for the job, however your choice comes with overhead. 
5. Get the culture right.

6. Move Fast - break things.

7. Huge Impact - small teams.

8. Be bold - innovate.

Machine data is changing the 'big data' landscape irrevocably

Correlating that machine data – data at rest, dark data in sensors and human unstructured data – could help you to see opportunities and threats earlier than the competition. 

Machine data is now the fastest growing element of big data and is set to represent 90 per cent of all data.

The ability to analyse and derive insight from all this machine data is where the big prize is in big data, but up until now, its value has been largely ignored by businesses.

“We are honored to be recognized by SC Magazine with another award that demonstrates how essential flexibility, speed and scale are for security intelligence platforms,” said Tim Mather, chief information security officer and vice president of security and compliance, Splunk. “Splunk software delivers the future of security intelligence today. Attackers are intimately familiar with the security gaps created by legacy, pre-fabricated solutions that cannot collect and analyze all data from an organization. More than 2,000 organizations around the world use Splunk software to gain the upper hand in security investigations by creating a real-time understanding of their end-to-end security.”

Social Inbox brings together the power of your social and email channels.

Using Social Inbox to Enhance Your Events Participation

Here's an example of how you can use Social Inbox to amplify your company's impact at live events.

Geo Location Targeting:

You walk into the grocery store and your smartphone beeps. “Welcome back,” it says. “There’s a sale on tomato sauce in aisle 6. 

You’re driving down the highway on your way to work. As you approach, a billboard changes to show an advertisement for your favorite coffee shop, two miles away. After you pass, the billboard switches to show a trailer for the new Disney movie, just as a school bus rounds the corner.

Scientists at IBM Research have made the world's smallest movie by moving individual atoms with a scanning tunneling microscope. 

 

Recently, IBM Research announced it can now store single bit of information in just 12 atoms as opposed to roughly one million atoms currently. From 1,000,000 to 12 — that’s a dramatic breakthrough that not only has the potential to make our computers and devices smaller and more powerful, but also holds enormous implications for the way entire industries operate. If used for data storage "you could carry around not just one or two   movies on your IPhone you could carry every movie ever produced" contends Andreas Heinrich, Principal Investigator, IBM Research. 

It was Atwal's long-held belief that football needs better performance statistics that led him and a friend to set up Squawka in June 2012. It gives fans access to stats collected about every game in Europe's top national leagues. The site collects every pass made by every player and whether it was successful or not, the number of tackles, corners, shots, goals and so on...

Squawka is just one of a new generation of companies that are exploiting advances in the way data is collected, stored and handled. It cannot afford the computing infrastructure to host and serve up its data on a global scale. Instead, it buys cheap storage, processing power and virtually unlimited bandwidth in the cloud. This lets it cope with traffic spikes on busy match days.

Data scientists: Modern-day knowledge prospectors, panning for insight. How do we accelerate tomorrow’s business? How can tomorrow’s businesses mine data more effectively? How does relying on data get better results than judgment alone?

While some abandonments are never a good thing, abandoned baskets in the online channel can be a rich source of revenue.

Millions of shoppers are trying out products in store before buying online to save money. Learn more about the trend of showrooming.. Showroomers experiment with products in-store then buy them online later for less money. The trend is hitting big retailers like Best Buy and Walmart, as customers use the brick-and-mortar stores to test-drive devices like smartphones and e-readers before buying from retailers likeAmazon, where products are cheaper on average.

It is time for organisations to more seriously consider enterprise social networking, particularly within collaboration strategies, as the market begins to mature and transition from the introductory to growth phase, according to analyst firm, Ovum.