Tech
2.4K views | +4 today
Follow
 
Your new post is loading...
Your new post is loading...
Scooped by Matthieu Blanc
Scoop.it!

Long-running Spark Streaming Jobs on YARN Cluster - Passionate Developer

Long-running Spark Streaming Jobs on YARN Cluster - Passionate Developer | Tech | Scoop.it
A long-running Spark Streaming job, once submitted to the YARN cluster should run forever until it is intentionally stopped.
Any interruption …
more...
No comment yet.
Scooped by Matthieu Blanc
Scoop.it!

An overview of gradient descent optimization algorithms

This blog post looks at variants of gradient descent and the algorithms that are commonly used to optimize them.
more...
No comment yet.
Scooped by Matthieu Blanc
Scoop.it!

Kafka Reliability - When it absolutely, positively has to be there

Best practices for making sure messages don't get lost - ever.
more...
No comment yet.
Scooped by Matthieu Blanc
Scoop.it!

Neural networks and deep learning

Neural networks and deep learning | Tech | Scoop.it

Neural Networks and Deep Learning What this book is about On the exercises and problems Using neural nets to recognize handwritten digits How the backpropagation algorithm works Improving the way neural [...]...

more...
No comment yet.
Scooped by Matthieu Blanc
Scoop.it!

(BDT309) Data Science & Best Practices for Apache Spark on Amazon EMR

Organizations need to perform increasingly complex analysis on their data — streaming analytics, ad-hoc querying and predictive analytics — in order to get bet…
more...
No comment yet.
Scooped by Matthieu Blanc
Scoop.it!

Demystifying Stream Processing with Apache Kafka

Demystifying Stream Processing with Apache Kafka | Tech | Scoop.it
Neha Narkhede describes the core features of Apache Kafka and Apache Samza, which include scalability and parallelism through data partitioning, fault tolerance and event processing order guarantees, support for stateful stream processing, and handy stream processing primitives such as windowing.
more...
No comment yet.
Scooped by Matthieu Blanc
Scoop.it!

Data processing platforms architectures with SMACK: Spark, Mesos, Akka, Cassandra and Kafka

Data processing platforms architectures with SMACK: Spark, Mesos, Akka, Cassandra and Kafka | Tech | Scoop.it
This post is a follow-up of the talk given at Big Data AW meetup in Stockholm and focused on different use cases and design approaches for building scalable data processing platforms with SMACK(Spark, Mesos, Akka, Cassandra, Kafka) stack. While...
more...
No comment yet.
Scooped by Matthieu Blanc
Scoop.it!

Ingest Tips / From Relational into Kafka - Ingest Tips

Ingest Tips / From Relational into Kafka - Ingest Tips | Tech | Scoop.it
Overview of tools for migrating data from relational databases like MySQL, Postgres and Oracle to Apache Kafka.
more...
No comment yet.
Scooped by Matthieu Blanc
Scoop.it!

Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms…

Talk given by Reynold Xin (@rxin) at Strata New York 2015
more...
No comment yet.
Scooped by Matthieu Blanc
Scoop.it!

Some Important Streaming Algorithms You Should Know About | MapR

Some Important Streaming Algorithms You Should Know About | MapR | Tech | Scoop.it
Ted Dunning, Chief Applications Architect for MapR, presented a session titled: “Some Important Streaming Algorithms You Should Know About” at the Spark Summit 2015 conference held in San Francisco. During the session, he highlighted some newer streaming algorithms such as t-digest and streaming k-means. This article is adapted from his talk.
more...
No comment yet.
Scooped by Matthieu Blanc
Scoop.it!

A collection of links for streaming algorithms and data structures

A collection of links for streaming algorithms and data structures
more...
No comment yet.
Scooped by Matthieu Blanc
Scoop.it!

Airbnb New User Bookings, Winner's Interview: 3rd place: Sandro Vega Pons

Airbnb New User Bookings, Winner's Interview: 3rd place: Sandro Vega Pons | Tech | Scoop.it
AirBnB New User Bookings was a popular recruiting competition that challenged Kagglers to predict the first country where a new user would book travel. This was the first recruiting competition on …
more...
No comment yet.
Scooped by Matthieu Blanc
Scoop.it!

How-to: Do Data Quality Checks using Apache Spark DataFrames - Cloudera Engineering Blog

How-to: Do Data Quality Checks using Apache Spark DataFrames - Cloudera Engineering Blog | Tech | Scoop.it
Apache Spark’s ability to support data quality checks via DataFrames is progressing rapidly. This post explains the state of the art and future possibilities.
Apache Hadoop and Apache Spark make Big Data accessible and usable so we can easily find value, but that data has to be correct, first. This post will focus on this problem and how to solve it with Apache Spark 1.3 and Apache Spark 1.4 using DataFrames. Read More
more...
No comment yet.
Scooped by Matthieu Blanc
Scoop.it!

Extending the Yahoo! Streaming Benchmark | data Artisans

Extending the Yahoo! Streaming Benchmark | data Artisans | Tech | Scoop.it

Until very recently I’ve been working at Twitter and focusing primarily on stream processing systems. While researching the current state-of-the-art in stateful streaming systems I came across Apache Flink.

more...
No comment yet.
Scooped by Matthieu Blanc
Scoop.it!

Elements of Scale: Composing and Scaling Data Platforms - ben stopford

Elements of Scale: Composing and Scaling Data Platforms - ben stopford | Tech | Scoop.it
This transcribed talk explores a range of data platforms through a lens of basic hardware and software tradeoffs.
more...
No comment yet.
Scooped by Matthieu Blanc
Scoop.it!

Distributed real time stream processing- why and how

In this talk you will discover various state-of-the-art open-source distributed streaming frameworks, their similarities and differences, implementation trade-…
more...
No comment yet.
Scooped by Matthieu Blanc
Scoop.it!

Effective testing for spark programs Strata NY 2015

This session explores best practices of creating both unit and integration tests for Spark programs as well as acceptance tests for the data produced by our Sp…
more...
No comment yet.
Scooped by Matthieu Blanc
Scoop.it!

Functors, Applicatives, And Monads In Pictures - adit.io

Aditya Bhargava's personal blog.
more...
No comment yet.