Large-scale Incremental Processing
10.7K views | +0 today
Follow
Large-scale Incremental Processing
Your new post is loading...
Your new post is loading...
Scooped by Jaeboo Jeong
Scoop.it!

The infrastructure behind Twitter: efficiency and optimization | Twitter Blogs

The infrastructure behind Twitter: efficiency and optimization | Twitter Blogs | Large-scale Incremental Processing | Scoop.it
Our first blog as part of the infrastructure series gives an overview of our data centers and the hardware engineering journey at Twitter.
more...
No comment yet.
Scooped by Jaeboo Jeong
Scoop.it!

Applying Kafka Streams for internal message delivery pipeline « LINE Engineers' Blog

Applying Kafka Streams for internal message delivery pipeline «  LINE Engineers' Blog | Large-scale Incremental Processing | Scoop.it
IntroductionHello, my name is Yuto Kawamura. I'm a LINE server engineer in charge of developing and operating LINE's core storage facilities such a
more...
No comment yet.
Scooped by Jaeboo Jeong
Scoop.it!

Making the Elephant Fly in the Cloud - Hortonworks

Making the Elephant Fly in the Cloud - Hortonworks | Large-scale Incremental Processing | Scoop.it
Ram Venkatesh also contributed to this blog series  Why Apache Hadoop in the Cloud? Ten years ago, Hadoop, the elephant started the Big Data journey inside
more...
No comment yet.
Scooped by Jaeboo Jeong
Scoop.it!

Facebook’s Artificial Intelligence Research lab releases open source fastText on GitHub

Facebook’s Artificial Intelligence Research lab releases open source fastText on GitHub | Large-scale Incremental Processing | Scoop.it
Every day, billions of pieces of content are shared on Facebook. To keep up with the data, Facebook has been using a variety of tools to classif
more...
No comment yet.
Scooped by Jaeboo Jeong
Scoop.it!

Data Reprocessing with Kafka Streams: Resetting a Streams Application

Data Reprocessing with Kafka Streams: Resetting a Streams Application | Large-scale Incremental Processing | Scoop.it
In this blog post we describe how to tell a Kafka Streams application to reprocess its input data from scratch.
more...
No comment yet.
Scooped by Jaeboo Jeong
Scoop.it!

Michael Spector's Blog: Why we moved from Mesos to Yarn (and from Chronos to Airflow)

Michael Spector's Blog: Why we moved from Mesos to Yarn (and from Chronos to Airflow) | Large-scale Incremental Processing | Scoop.it
more...
No comment yet.
Scooped by Jaeboo Jeong
Scoop.it!

cURL with HTTP2 Support - A Minimal Alpine-based Docker Image | I care, I share, I'm Nathan LeClaire.

cURL with HTTP2 Support - A Minimal Alpine-based Docker Image | I care, I share, I'm Nathan LeClaire. | Large-scale Incremental Processing | Scoop.it
more...
No comment yet.
Scooped by Jaeboo Jeong
Scoop.it!

3 Node Swarm Cluster in 30 seconds (Docker 1.12)

Docker 1.12 has been released!Last June during the keynote of DockerCon we saw a demo where a 3 node swarm cluster was created in 30 seconds using the new swarmkit integration into Docker Engine 1.12 Impressive...
more...
No comment yet.
Scooped by Jaeboo Jeong
Scoop.it!

The Netflix Tech Blog: Distributed Resource Scheduling with Apache Mesos

The Netflix Tech Blog: Distributed Resource Scheduling with Apache Mesos | Large-scale Incremental Processing | Scoop.it
more...
No comment yet.
Scooped by Jaeboo Jeong
Scoop.it!

HDFS: Big data analytics' weakest link

HDFS: Big data analytics' weakest link | Large-scale Incremental Processing | Scoop.it
Hadoop's distributed file system isn't as fast, efficient, or easy to operate as it should be
more...
No comment yet.
Scooped by Jaeboo Jeong
Scoop.it!

Building Your Own Apache Hadoop Distribution

Building Your Own Apache Hadoop Distribution | Large-scale Incremental Processing | Scoop.it
While BUILDING.txt includes a lot of hints about what the various options are to build Apache Hadoop, it is overwhelming to turn those directions into something that can actually be deployed. In fa…
more...
No comment yet.
Scooped by Jaeboo Jeong
Scoop.it!

How to use SparkSession in Apache Spark 2.0

How to use SparkSession in Apache Spark 2.0 | Large-scale Incremental Processing | Scoop.it
This is the first in the series of how-to use blog posts on new features and functionality in Apache Spark 2.0
more...
No comment yet.
Scooped by Jaeboo Jeong
Scoop.it!

Disaster Recovery Strategies for Big Data - Disaster Recovery Journal - Dedicated to Business Continuity

Organizations are increasingly analyzing operational data in order to reveal a significant competitive advantage, increase revenue, or mitigat
more...
No comment yet.
Scooped by Jaeboo Jeong
Scoop.it!

Securing NoSQL Databases: Use the Force | MapR

Securing NoSQL Databases: Use the Force | MapR | Large-scale Incremental Processing | Scoop.it
With stories of the thefts of millions of credit card records and sensitive employee data at some of the world’s largest companies and government agencies dominating recent headlines, it’s not surprising that organizations are doubling down on...
more...
No comment yet.
Scooped by Jaeboo Jeong
Scoop.it!

Distributed Application Bundles (Tour Around Docker 1.12 Series)

Distributed Application Bundles (Tour Around Docker 1.12 Series) | Large-scale Incremental Processing | Scoop.it
The new Swarm bundled in Docker 1.12+ is a vast improvement compared to the old orchestration and scheduling.
more...
No comment yet.
Scooped by Jaeboo Jeong
Scoop.it!

Securing Apache Spark Shuffle using Apache Commons Crypto - Cloudera Engineering Blog

Securing Apache Spark Shuffle using Apache Commons Crypto - Cloudera Engineering Blog | Large-scale Incremental Processing | Scoop.it
Learn how the performance advantages of the Crypto cryptographic library will provide an upgrade for Spark shuffle encryption over the current approach.
more...
No comment yet.
Scooped by Jaeboo Jeong
Scoop.it!

Uber’s case for incremental processing on Hadoop

Uber’s case for incremental processing on Hadoop | Large-scale Incremental Processing | Scoop.it
Near-real-time processing yields increased efficiency and an opportunity for unified architecture.
more...
No comment yet.
Scooped by Jaeboo Jeong
Scoop.it!

Docker for Mac

Developers love Docker. You can see this from the amount of attention Docker has got the last couple of years. But, one of the biggest issues developer
more...
No comment yet.
Scooped by Jaeboo Jeong
Scoop.it!

Multi-node Clusters with Cloudera QuickStart for Docker - Cloudera Engineering Blog

Multi-node Clusters with Cloudera QuickStart for Docker - Cloudera Engineering Blog | Large-scale Incremental Processing | Scoop.it
Getting hands-on with a multi-node cluster for self-learning or testing is even easier, now.
more...
No comment yet.