Big Data Technolo...
Follow
Find
9.9K views | +5 today
Big Data Technology, Semantics and Analytics
Trends, success and applications for big data including the use of semantic technology
Curated by Tony Agresta
Your new post is loading...
Your new post is loading...
Scooped by Tony Agresta
Scoop.it!

Where Do You Collect “Now Data”?

Where Do You Collect “Now Data”? | Big Data Technology, Semantics and Analytics | Scoop.it
If you're new here, you may want to subscribe to our RSS feed. Thanks for visiting! TweetShareShare on TumblrEmailSharebar TweetShareShare on TumblrEmail Related posts:What’s the Big Deal About Big Data?
Tony Agresta's insight:

"Now Data" is another way of saying "event triggered data."  How you detect significant events to identify signals through the noise?  The actions you take once you accomplish this goal are as important.  While this article references marketing applications, it applies to everything from customer on-boarding in financial services to real time market trading information.    Other classic examples of using event triggered data detection come from federal government and local law enforcement applications that combine real time feeds of unstructured data interpreted using semantic processing and then connected to pre-existing data to assess risk and deploy resources.

 

There are several approaches that could be used to solve these types of problems and help answer the question "what set of events triggers should I need to be aware of?"  At the very least, they allow you to prioritize the workload and focus on very specific cases.  Predictive models will isolate a set of independent data with varying weights of explanatory power.   Identifying these attributes is helpful allowing you to score incoming data in real time leading to notification of the right individual when a resulting score exceeds thresholds.  Simpler approaches include building rule libraries (sets of queries) that can be used to classify big data streaming into the enterprise. When a data element matches the pre-defined query, event triggered notifications can be sent.

 

Technology to achieve these goals exists today.  An Enterprise NoSQL Database can ingest a combination of structured, unstructured and semi-structured data, score the data in real time, match the incoming data to pre-defined query libraries and trigger notifications.  To learn more about this approach, you can go here:  http://www.marklogic.com/what-is-marklogic/enterprise-nosql/

 

There's also an upcoming event completely dedicated to Enterprise NoSQL where case studies from some of the largest companies in the world will be presented.  MarkLogic World 2013 is being held in Las Vegas starting April 8th.  You can learn more and register for this event here: http://www.marklogic.com/events/


more...
No comment yet.
Rescooped by Tony Agresta from Real World NoSQL
Scoop.it!

A NoSQL database for the Enterprise...

Want to use a NoSQL database, but also want Enterprise features? Read on... MarkLogic (my employer!) have just announced a new Enterprise Developer license. This is aimed at Developers that want to...

Via Adam Fowler
more...
Adam Fowler's curator insight, February 13, 2013 4:49 PM

Analysts are starting to realise the hype around Big Data and NoSQL is biting customers with Enterprise needs. MarkLogic can help prevent organisations waste a lot of time in trying to crowbar and patch Open Source alternatives to make them do what they need. With MarkLogic, you can already have no pre-defined schema, XML and JSON storage, ACID compliant transations, and comprehensive search functionality - all in a single product.

Rescooped by Tony Agresta from Big Data, IoT and other stuffs
Scoop.it!

How Big Data Revolutionized the Finance Industry: From Wall Street to the iPhone

How Big Data Revolutionized the Finance Industry: From Wall Street to the iPhone | Big Data Technology, Semantics and Analytics | Scoop.it

Wikibon’s Big Data Vendor Revenue and Market Forecast noted several factors that contributed to the growth of Big Data in 2012. One factor was that of “an increase in awareness of the benefits of Big Data as applied to…financial services.” What makes the financial services and Big Data such a good marriage?


Via Ed Stenson, Toni Sánchez
more...
No comment yet.
Scooped by Tony Agresta
Scoop.it!

Nate Silver's big-data insights -- FCW

Nate Silver's big-data insights -- FCW | Big Data Technology, Semantics and Analytics | Scoop.it
In his latest book, statistician and predictive analytics expert Nate Silver describes his approach to forming forecasts out of data.
Tony Agresta's insight:

“Big data is not a cure-all, and it is inherently filled with noise and uncertainty, but it does have tremendous potential if people approach it the right way. ‘The world is not lacking for techniques, it's more about the right goals and right attitudes,’ Silver said.”  Having goals associated with big data analysis is a must.   Applying technology and techniques to achieve those goals is not far behind. 


Different approaches to analysis, some of which are presented in this article, complement one another and allow you to reach those goals faster. Let's take three classic approaches - dashboards, predictive models and data visualization – and the problem of fraud detection.  Let’s say our goals include improved fraud detection for incoming insurance claims and more efficient allocation of resources to investigate those claims.  If analysts can prioritize the workload for investigators, they can find fraud faster and reduce costs.


BI dashboards typically show key metrics which may lead the analyst to spot trends that they want to model using predictive analysis.   They also point analysts to independent data that may have some explanatory power in the model.   For example, a BI dashboard showing recent insurance claims by postal code may show a spike in certain areas which could lead to deeper analysis where geographic indicators (city, zip+4) are selected as attributes to predict fraudulent claims.   While knowing that the insurance claim has a higher likelihood of being fraudulent is important, understanding the ring of people linked to that claim is potentially more important. Are those people linked to other claims that have been investigated and found to be fraudulent?  Do these people share the same address?  Are they using the same doctor or pharmacy?  Have they worked together in the past?  


Data visualization allows you to explore those relationships and picks up where predictive models leave off.  In this case, all of the major types of analysis were used to achieve the goal of identifying suspicious claims and ultimately identifying a fraud ring.


Different approaches to analysis can complement one another.  Business Intelligence and dashboards provide one level of visibility.  They point the analyst to key trends and relationships that may require a model to be built.  Results of those models (scores or yes/no indictors) can be used with data discovery tools to understand relationships, identify patterns of behavior, show connections between seemingly disparate data and rapidly draw conclusions.   Identifying goals up front will allow analysts to formulate questions they want to ask of the data.  Using different types of analysis helps address challenges with big data. 


To learn more about how you achieve your goals using Enterprise NoSQL, you can go here:


http://www.marklogic.com/what-is-marklogic/analytics-business-intelligence/

more...
No comment yet.
Scooped by Tony Agresta
Scoop.it!

The new reality for Business Intelligence and Big Data

The new reality for Business Intelligence and Big Data | Big Data Technology, Semantics and Analytics | Scoop.it
You know about Big Data and its potential, how it creates greater understanding of our world, reduces waste and misuse of resources, and dramatically increases efficiency.
Tony Agresta's insight:

Data discovery tools allow you to reveal hidden insights in your data when you don't know what to look for in advance.  These highly interactive tools allow you to visualize disparate data in various forms - charts, timelines, graphs, geo-spatial and tables – and explore relationships in data to uncover patterns that static dashboards cannot.  

 

With the explosion of big data, organizations are now using these tools with structured, semi-structured and unstructured data.  This approach allows them to consolidate data without having to build complex schemas, search the data instantly, deliver new content products dynamically and analyze all of their data in real time.  A transformational shift in data analysis is underway allowing organizations to do this with documents, e-mails, video and other sources.   Imagine if you could load data into Hadoop, enrich it, ingest the data into an enterprise NoSQL database in real-time, index everything for instant search & discovery and analyze that data using Tableau or Cognos.   As the only Enterprise NoSQL Database on the market, MarkLogic allows you to do just that.

 

You can learn more here:  http://www.marklogic.com/what-is-marklogic/analytics-business-intelligence/

 




more...
No comment yet.
Scooped by Tony Agresta
Scoop.it!

Five Things To Know About XML

Five Things To Know About XML | Big Data Technology, Semantics and Analytics | Scoop.it
XML is at the foundation of what is driving digital content today.   Most eBooks have an XML foundation.  Many websites are driven by XML content.  So it is important if you are considering expandi...
more...
No comment yet.
Scooped by Tony Agresta
Scoop.it!

5 Skills Data Scientists Should Cultivate in 2013 | SiliconANGLE

5 Skills Data Scientists Should Cultivate in 2013 | SiliconANGLE | Big Data Technology, Semantics and Analytics | Scoop.it
more...
No comment yet.
Scooped by Tony Agresta
Scoop.it!

If I haven't heard of it it's probably NoSQL! | Bloor

It's all too easy to think that all noSQL vendors fall into the same camp. MarkLogic is a notable exception.
Tony Agresta's insight:

Relevant insights from Philip Howard at the Bloor Group about the NoSQL market.

more...
No comment yet.
Scooped by Tony Agresta
Scoop.it!

MarkLogic Simplifies Development of Enterprise-Ready Applications: Free Developer License for MarkLogic(R) Enterprise Edition Now Available

SAN CARLOS, CA, Feb 12 (Marketwire) -- MarkLogic Corporation, provider of the only enterprise NoSQL database,today announced the availability of a free Developer License forMarkLogic(R) Enterprise Edition.
Tony Agresta's insight:
From Gary Bloom: "By providing a free Developer License we enable developers to quicklydeliver reliable, scalable and secure information and analyticapplications that are production-ready," said Gary Bloom, CEO andPresident of MarkLogic. "Many of our customers first experimented withother free NoSQL products, but turned to MarkLogic when they recognizedthe need for search, security, support for ACID transactions and otherfeatures necessary for enterprise environments. Our goal is to eliminatethe cost barrier for developers and give them access to the bestenterprise NoSQL platform from the start."

Learn more here:
http://www.marklogic.com/press-releases/marklogic-simplifies-development-of-enterprise-ready-applications-free-developer-license-for-marklogic-enterprise-edition-now-available/


more...
No comment yet.
Scooped by Tony Agresta
Scoop.it!

Personalized Learning Is Sweeping College Campuses, Courtesy of Big Data

Personalized Learning Is Sweeping College Campuses, Courtesy of Big Data | Big Data Technology, Semantics and Analytics | Scoop.it
Data mining and analytics teach educators about their students.
more...
No comment yet.
Scooped by Tony Agresta
Scoop.it!

Professional Sports & Big Data - Gaining the Competitive Edge via Analytics and Understanding

Professional Sports & Big Data - Gaining the Competitive Edge via Analytics and Understanding | Big Data Technology, Semantics and Analytics | Scoop.it
The NFL season has just ended and a new era of Big Data analytics is just beginning.
more...
No comment yet.
Scooped by Tony Agresta
Scoop.it!

DNA - The New Big Data Storage Solution of the 21st Century? | CloudTimes

DNA - The New Big Data Storage Solution of the 21st Century? | CloudTimes | Big Data Technology, Semantics and Analytics | Scoop.it
Scientists are constantly looking for alternative methods of storing information that will ensure long-term preservation of data at low cost and operating cost.
more...
No comment yet.
Rescooped by Tony Agresta from visual data
Scoop.it!

[INFOGRAPHIC] BIG DATA: What Your IT Team Wants You To Know

[INFOGRAPHIC] BIG DATA: What Your IT Team Wants You To Know | Big Data Technology, Semantics and Analytics | Scoop.it

The purpose of Big Data is to supply companies with actionable information on any variety of aspects. But this is proving to be far more difficult than it looks with over half of Big Data projects left uncompleted.


Two of the most often reported reasons for project failures are a lack of expertise in data analysis. Reports show that data processing, management and analysis are all difficult in any phase of the project, with IT teams citing each of those reasons more than 40% of the time.

However, failures in Big Data projects may not solely lie on faulty project management. In a recent survey, a staggering 80% of Big Data’s biggest challenges are from a lack of appropriate talent. The field’s relative infancy is making it hard to find the necessary staff to see projects through, resulting in underutilized data and missed project goals.

IT teams are quickly recognizing a chasm between executives and frontline staffers whose job it is to apply findings from Big Data. In the end,it may not be the anticipated cure-all for 21st century business management. It is only as good as good as the system that runs it.


Via Peter Azzopardi, Berend de Jonge, Lauren Moss
Tony Agresta's insight:

Very interesting infographic.  Why do they fail?  For all of the reasons above and then some...    Over 80% of the data being collected today is unstructured and not readily stored in relational database technology burdened by complex extract, transform and load.  There's also pre-existing data, sometimes referred to as "dark data" that includes documents which need to be included and made discoverable for a host of reasons - compliance and regulatory issues are one.   Log activity and e-mail traffic used to detect cyber threats and mitigate risk through analysis of file transfers is yet another set of data that requires immediate attention.

 

Social and mobile are clearly channels that need to be addressed as organizations continue to mine data from the open web in support of CRM, product alerts, real time advertising options and more.  

 

To accomplish all of this, organizations need a platform with enterprise hardened technology that can ingest all of these forms of data in real time, without having to write complex schemas.   Getting back to the point - What do most projects fail?   If companies attempt to do this with technology that is not reliable, not durable and does not leverage the skills of their existing development organization, the project will fail.  

 

We have seen this time and time again.   MarkLogic to the rescue.   With over 350 customers and 500 big data applications, our Enterprise NoSQL approach mitigates the risk.  Why?  Our technology stack includes connectors to Hadoop, integration with leading analytics tools using SQL, Java and Rest APIs, JSON support, real time data ingestion, the ability to handle any form of data, alerting, in database analytics functions, high availability, replication, security and a lot more.  

 

When you match this technology with a world-class services organization with proven implementation skills, we can guarantee your next Big Data project will work.  We have done it hundreds of times with the largest companies in the world and very, very big data.

 

www.marklogic.com



more...
Olivier Vandelaer's curator insight, January 30, 2013 2:45 AM

Looking at the infographic, it clearly reminds me about the start of "Enterprise Data Warehouse": failures by "Innacurate scope", "Technical Roadblocks" & "Siloed data and no collaboration". It looks so familiar.

Adrian Carr's curator insight, January 30, 2013 10:27 AM

This is a great infographic - it shows that whilst everyone is doing it (it being "Big Data" - whatever that is...), talent is rare, technology is hard to find and the projects never end.  A far cry from the speed with which companies such as the BBC deployed MarkLogic to serve all data for the sport websites through the Olympics.  Now that was big data, delivered by a talented team in a short space of time.

Scooped by Tony Agresta
Scoop.it!

Data Analytics Deep is the New Big

Data  Analytics Deep is the New Big | Big Data Technology, Semantics and Analytics | Scoop.it
Meeting pharmas valuebased reimbursement challenge Marc Iskowitz looks at how companies, driven by payer and provider pressure, are leveraging Big Data across the product lifecycle
Tony Agresta's insight:

Improving patient outcomes using deep data analytics is on the rise.   Pharma companies and payers recognize that results from clinical trials don't address all of the issues required to reduce risk. Claims data and electronic health records are two sources of data at the tip of the data analytics spear but they don't always address patient reported outcomes.

 

Social networks, research from companies like IPSOS and payer-pharma integrated databases like the one in use from Humana-Pfizer offer promising new approaches to dig deeper into outcomes from therapies, continuation-of-care analysis, treatment decision management and early stage performance.

 

"To meet the demand, companies are bolstering their Big Data infrastructure. 'If there is this foundation of information that can be created into an evidence base,'  Resnick says, 'it can allow for greater efficiency and value.'  Some companies, says Narayanan, are moving toward having one central source of integrated real-world data, housed in the company, that marketing and other departments can tap into to understand the current real-world landscape. 'That's where the ROI comes,' he says.

 

Proven infrastructure including an Enterprise NoSQL database can be used to consolidate the data, dig deeper using search applications, discover new insights and deliver analytical results to all members of the ecosystem including patients, providers, payers, big pharma, pharmacies and researchers.

 


more...
Patti Peeples's curator insight, March 8, 2013 8:54 AM

Well, in an ideal world, they are doing this...but pharma product marketers seem clueless about this so far.

Rescooped by Tony Agresta from Real World NoSQL
Scoop.it!

Idiots guide to Hadoop...

Found a great idiots guide video on YouTube for Hadoop... From EMC of all people!... http://www.youtube.com/watch?v=XtLXPLb6EXs Let me know what you think!

Via Adam Fowler
more...
Adam Fowler's curator insight, February 15, 2013 12:16 PM

Good video intro to Hadoop

Scooped by Tony Agresta
Scoop.it!

MarkLogic Enhances Enterprise NoSQL Database Applications & Analytic Solutions with Intel® Distribution for Apache Hadoop Software | MarkLogic

Visit MarkLogic.com to read MarkLogic Enhances Enterprise NoSQL Database Applications & Analytic Solutions with Intel® Distribution for Apache Hadoop Software.
more...
No comment yet.
Scooped by Tony Agresta
Scoop.it!

Big Data Vendor Revenue And Market Forecast 2012-2017 - Wikibon

Big Data Vendor Revenue And Market Forecast 2012-2017 - Wikibon | Big Data Technology, Semantics and Analytics | Scoop.it
Tony Agresta's insight:

"Leading the way in terms of revenue in the Hadoop/NoSQL subsegment of the Big Data market in 2012 was a 10-year-old firm, MarkLogic. The company’s NoSQL document store is in use at Bank Of America, the Defense Intelligence Agency and Warner Brothers, among other household names in the media and financial services industries."

more...
No comment yet.
Scooped by Tony Agresta
Scoop.it!

Use Case: Sophisticated Queries | MarkLogic

Visit MarkLogic.com to read MarkLogic in Defense & Intelligence - Use Case #2: Sophisticated Queries.
more...
No comment yet.
Scooped by Tony Agresta
Scoop.it!

Big Data Analytics and Drug Surveillance

Big Data Analytics and Drug Surveillance | Big Data Technology, Semantics and Analytics | Scoop.it
HIMSS13 presentation on usingBig Data analytics for drug surveillance.
more...
No comment yet.
Scooped by Tony Agresta
Scoop.it!

DbRevolutionReportFinal.pdf

Tony Agresta's insight:

Followers may be interested in this white paper from The Bloor Group which summarizes the differences between database technologies  It's meaty.


Here are a few additional points that Bloor has written about MarkLogic's Enterprise NoSQL approach:


  • MarkLogic is also a true transactional database. Most NoSQL databases have compromised the ACID (Atomicity, Consistency, Isolation and Durability) properties that are important for transaction processing, MarkLogic is fully equipped to be a transactional database, and if you simply wanted to use it for order processing, there would be no problem in doing so.
  • The database has been built to enable rapid search of its content in a similar manner to the way that Google’s search capabilities have been built to enable rapid search of the Internet.
  • As some of MarkLogic’s implementations have grown to above the single petabyte level, fast search of massive amounts of data is one of its most important features. To enable its search capability MarkLogic indexes everything on ingest; not just the data, but also the XML metadata. This provides it with the ability to search both text and structure. For example, you might want to quickly find someone’s phone number from a collection of emails.
  • With MarkLogic you could pose a query such as: “Find all emails sent by Jonathan Jones, sort in reverse order by time and locate the latest email that contains a phone number in its signature block.”
  • You may be able to deduce from this that Mark Logic knows what an email is, knows how to determine who the sender is, knows what a signature block is and knows how to identify a phone number from within the signature block. If you were looking for a mobile phone number then you would simply add the word “mobile” in front of phone number. It should be clear from this that very few databases could handle such a query, because most databases are straight-jacketed by whatever version of SQL they implement and, even if it were possible to bend SQL in such a way as to formulate this kind of query, most databases cannot dig into data structures they hold in the way that MarkLogic.


With the release of MarkLogic 6 last fall, MarkLogic also provided SQL support through integration with Tableau and Cognos, in-database analytic functions, JSON support, JAVA and REST APIs and more.  For more information on this release, you can go here:


http://www.marklogic.com/what-is-marklogic/marklogic-6/



more...
No comment yet.
Scooped by Tony Agresta
Scoop.it!

How Big Data is Being Used to Help Raise the IQ of School Buildings « A Smarter Planet Blog

How Big Data is Being Used to Help Raise the IQ of School Buildings « A Smarter Planet Blog | Big Data Technology, Semantics and Analytics | Scoop.it
By Christopher Luongo More school systems across the U.S.
more...
No comment yet.
Scooped by Tony Agresta
Scoop.it!

Financial Firms Overhaul Data Management Strategies

Financial Firms Overhaul Data Management Strategies | Big Data Technology, Semantics and Analytics | Scoop.it
As capital markets organizations focus on ways to reduce risk, data management is receiving a top-to-bottom makeover.
Tony Agresta's insight:

"In the current environment, smaller is sometimes better. For instance, firms are no longer spending money on gigantic warehouses and agreeing on a single data model that would fit all the data in one place," says Amir Halfon, CTO of financial services at MarkLogic, a provider of database technology. "It's that aspect that introduces the most cost. The traditional approach of a big data warehouse doesn't cut it anymore."

more...
No comment yet.
Scooped by Tony Agresta
Scoop.it!

Big Data “Hype” Coming To An End | SiliconANGLE

Big Data “Hype” Coming To An End | SiliconANGLE | Big Data Technology, Semantics and Analytics | Scoop.it
Tony Agresta's insight:

"Organizations have fascinating ideas, but they are disappointed with a difficulty in figuring out reliable solutions,” writes Sicular from The Gartner Group.

 

"Their disappointment applies to more advanced cases of sentiment analysis, which go beyond traditional vendor offerings.  Difficulties are also abundant when organizations work on new ideas, which depend on factors that have been traditionally outside of their industry competence, e.g. linking a variety of unstructured data sources.”

 

Today, organizations are coming to the realization that free or low cost open source technology to handle big data requires intense development cycles that burn costs and time.  Solving demanding challenges in these four areas has proven difficult:

 

  • Search & Discovery
  • Content Delivery
  • Analytics and Information Products
  • Data Consolidation

 

Organizations need to work with proven technology that's reliable and durable.  They need to work with technology that handles ACID transactions, enterprise security, high availability, replication, real time indexing and alerting - without having to right 10,000+ lines of code. 

 

Major financial institutions, healthcare payors, government agencies, media giants, energy companies, and state & local organizations have standardized on big data technology proven to increase developer productivity, create new revenue streams and address mission critical operations in a post 9-11 era. 

 








more...
Adrian Carr's curator insight, February 11, 2013 11:11 AM

IT does it again.  Build a technology up until we start to believe it will solve all world problems.  It generates huge "science projects" and then everything comes tumbling down.  Finally a voice of reason says...maybe we set expectations unrealistically...One more trough of disillussionment !

Rescooped by Tony Agresta from Big Data and Personalization
Scoop.it!

Predicting the Future By Mining Online News and Other Web Data | MIT Technology Review

Predicting the Future By Mining Online News and Other Web Data | MIT Technology Review | Big Data Technology, Semantics and Analytics | Scoop.it
Prototype software can give early warnings of disease or violence outbreaks by spotting clues in news reports.

Via Fred Thiel, Edward Chenard
more...
No comment yet.
Scooped by Tony Agresta
Scoop.it!

Debunking NoSQL Database Myths | Data Informed

Debunking NoSQL Database Myths | Data Informed | Big Data Technology, Semantics and Analytics | Scoop.it
There are still myths about exactly what a NoSQL database can and can't do.
Tony Agresta's insight:

If you are interested in big data, please listen to the podcast with Adam Fowler.  He gets to the truth about big data technology. The fact is, enterprise NoSQL databses are available and proven to work. 

more...
Adrian Carr's curator insight, February 11, 2013 11:12 AM

Adam Fowler provides some good answers to what is realin the NoSQL space. The fact is, enterprise NoSQL databses are available and proven to work.  You just have to select carefully and be discerning abotu what is REAL and what is "on the Roadmap" i.e. Not real.