FrenchTech SAAS

Pour rappel, il existe une distinction essentielle entre le lac de données et l’entrepôt de données (et le data mart). Le lac de données stocke les données brutes, quelle que soit la forme fournie par la source de données :

• Données structurées issues de bases de données relationnelles (lignes, colonnes)
• Données semi-structurées : fichiers de logs, CSV, JSON, XML, etc.
• Données non structurées : emails, documents, PDFs mais aussi potentiellement des données binaires (images, audio, vidéo)

Il n’y a pas d’hypothèses sur le schéma des données, chaque source de données peut utiliser le schéma de son choix. Il incombe aux utilisateurs des données du lac de leur donner un sens pour leurs propres besoins.

Le lac de données est généralement défini comme un réservoir de données pour des opérations de type ETL (Extract-Transform-Load), ou comme un hub central pour l’analyse de données en libre-service. Le lac de données se réfère également aux outils de gestion de la donnée au sein de la plate-forme (stockage, monitoring, etc.)

In Data-Oriented Architecture (DOA), systems are still organized around small, loosely-coupled components, as in SOA microservices. But DOA departs from microservices in two key ways:

1. Components are always stateless

   Rather than componentizing and federating data stores for each relevant component, DOA mandates describing the data or state-layer in terms of a centrally managed global schema.

2. Component-to-component interaction is minimized, instead favoring interaction through the data layer

   In our trading system, a component receiving prices for different securities is just publishing prices in a canonical form in our data store. A system can consume these prices by querying the data layer for prices, rather than request prices from a specific service (or set of services) through a specific API.

   Here, the integration cost is linearized. A DOA schema change means that up to N components might need to be updated, rather up to N2 connections between them.

Where this really shines is when individual high-level data types are populated by different providers. If we replace one service with one table, then we haven’t simplified things much. The benefit is if there are multiple sources of the same general data type. If a trading system is connecting to multiple marketplaces, who each publish requests from customers to an RFQ table, then downstream systems can query this one table and not worry about where a customer request is coming from.

In past some new tools have emerged, which are especially capable of handling the process of integrating data from outside, often called Data Ingestion. From an outside perspective, they are very similar to a traditional Enterprise Service Bus infrastructures, which in larger organization are often in use to handle message-driven and service-oriented systems. But there are also important differences, they are typically easier to scale in a horizontal fashion, offer a more distributed setup, are capable of handling high-volumes of data/messages, provide a very detailed monitoring on message level and integrate very well with the Hadoop ecosystem. This session will present and compare Apache NiFi, StreamSets and the Kafka Ecosystem and show how they handle the data ingestion in a Big Data solution architecture.

Big Data Architecture & Patterns

The Best Way to a solution is to “Split The Problem”. Big Data Solution can be well understood using Layered Architecture. The Layered Architecture is divided into different layers where each layer performs a particular function.

This Architecture helps in designing the Data Pipeline with the various requirements of either the Batch Processing System or Stream Processing System. This architecture consists of 6 layers which ensure a secure flow of data.

• Data Ingestion Layer

This layer is the first step for the data coming from variable sources to start its journey. Data here is prioritized and categorized which makes data flow smoothly in further layers.

• Data Collector Layer

In this Layer, more focus is on the transportation of data from ingestion layer to rest of data pipeline. It is the Layer, where components are decoupled so that analytic capabilities may begin.

• Data Processing Layer

In this primary layer, the focus is to specialize in the data pipeline processing system, or we can say the data we have collected in the previous layer is to be processed in this layer. Here we do some magic with the data to route them to a different destination, classify the data flow and it’s the first point where the analytic may take place.

• Data Storage Layer

Storage becomes a challenge when the size of the data you are dealing with, becomes large. Several possible solutions can rescue from such problems. Finding a storage solution is very much important when the size of your data becomes large. This layer focuses on “where to store such large data efficiently.”

• Data Query Layer

This is the layer where active analytic processing takes place. Here, the primary focus is to gather the data value so that they are made to be more helpful for the next layer.

• Data Visualization Layer

The visualization, or presentation tier, probably the most prestigious tier, where the data pipeline users may feel the VALUE of DATA. We need something that will grab people’s attention, pull them into, make your findings well-understood.

You May also Love to Read Why We Need Modern Big Data Integration Platform & Data Ingestion Patterns