CDN Breakthroughs
2.6K views | +0 today
Follow
CDN Breakthroughs
Monitoring innovations in CDN, CDN Federation, Cloud, SDN, Security, Caching, Load Balancing, Video/DRM delivery strategies & tools
Curated by Nicolas Weil
Your new post is loading...
Your new post is loading...
Scooped by Nicolas Weil
Scoop.it!

Netflix open sources its data traffic cop, Suro

Netflix open sources its data traffic cop, Suro | CDN Breakthroughs | Scoop.it

Netflix is open sourcing a tool called Suro that the company uses to direct data from its source to its destination in real time. More than just serving a key role in the Netflix data pipeline, though, it’s also a great example of the impressive — if not sometimes redunant — ecosystem of open source data-analysis tools making their way out of large web properties.

 

Netflix’s various applications generate tens of billions of events per day, and Suro collects them all before sending them on their way. Most head to Hadoop (via Amazon S3) for batch processing, while others head to Druid and ElasticSearch (via Apache Kafka) for real-time analysis. According to the Netflix blog post explaining Suro (which goes into much more depth), the company is also looking at how it might use real-time processing engines such as Storm or Samza to perform machine learning on event data.

more...
No comment yet.
Rescooped by Nicolas Weil from Video Breakthroughs
Scoop.it!

2013 CDN Summit Netflix Keynote

more...
Nicolas Weil's curator insight, June 29, 2013 4:50 AM

Interesting information set on the Open Connect program & appliances

Scooped by Nicolas Weil
Scoop.it!

Netflix builds a tool for jumping between Amazon clouds

Netflix builds a tool for jumping between Amazon clouds | CDN Breakthroughs | Scoop.it

Netflix clearly has learned from its Christmas Eve outage, which involved the failure of Amazon Web Services’ Elastic Load Balancing service, and has created a tool called Isthmus to solve the problem.

 

In a Friday blog post, Netflix’s Ruslan Meshenberg explained that Isthmus manages Elastic Load Balancing services in multiple regions in order to keep latency low for users in the event that ELB goes down in one region. On Christmas Eve, the issue was that state data got deleted, causing issues for the control plane tasked with managing load-balancer configuration and bringing some down some ELB load balancers. Now that sort of error could be less likely to affect Netflix service.

more...
No comment yet.
Scooped by Nicolas Weil
Scoop.it!

Netflix tackles the black art of DNS

Netflix tackles the black art of DNS | CDN Breakthroughs | Scoop.it

Netflix yesterday pushed version 1.1 of its Denominator DNS automation system onto its public GitHub account. This open-source tool allows developers and administrators alike to automate DNS migrations, changes and rules without having to edit DNS records by hand. With this release, the project now includes geographic-based controls for DNS routing, and thus is nearly ready to be deployed internally at Netflix.

 

When a developer is hired at Netflix, they learn a few different ways of doing things off the bat. For starters, there are very few managers, and all developers are senior level. Secondly, developers tend to find their own niches rather than have one assigned to them. So it went for Adrian Cole, who joined Netflix in December and found himself running a brand new open-source project by January.

 

That project, proposed by Adrian Cockcroft (Netflix’s cloud architect), came to be called Denominator. “Some of the outages we had last year could be solved by running multiple [cloud] regions,” he said. “We were looking for ways to direct our customers to more than one region. We do that by managing DNS, but we can't have a hand-managed configuration the way most people do DNS; we needed a RESTful API.”

more...
No comment yet.
Scooped by Nicolas Weil
Scoop.it!

Netflix's Denominator: A Multi-Vendor Interface for DNS

Netflix's Denominator: A Multi-Vendor Interface for DNS | CDN Breakthroughs | Scoop.it

We announced Denominator at our February NetflixOSS meetup, and now we are ready to release the first code as open source.  Denominator is a portable Java library for manipulating DNS clouds.  Denominator has pluggable back-ends, initially including AWS Route53, Neustar Ultra, DynECT, and a mock for testing.  We also ship a command line version so it's easy for anyone to try it out.
The reason we built Denominator is that we are working on multi-region failover and traffic sharing patterns to provide higher availability for the streaming service during regional outages caused by our own bugs and AWS issues. To do this we need to directly control the DNS configuration that routes users to each region and each zone. When we looked at the features and vendors in this space we found that we were already using AWS Route53, which has a nice API but is missing some advanced features; Neustar UltraDNS, which has a SOAP based API; and DynECT, which has a REST API that uses a quite different pseudo-transactional model.  We couldn’t find a Java based API that grouped together common set of capabilities that we are interested in, so we created one. The idea is that any feature that is supported by more than one vendor API is the highest common denominator, and that functionality can be switched between vendors as needed, or in the event of a DNS vendor outage.

more...
No comment yet.
Scooped by Nicolas Weil
Scoop.it!

Why Netflix Is Fixing AWS Instead Of Switching To OpenStack

Why Netflix Is Fixing AWS Instead Of Switching To OpenStack | CDN Breakthroughs | Scoop.it

Last week Netflix released ice, a publicly available tool that offers a granular look at Amazon Web Services usage and the associated costs in more detail than Amazon itself provides. Netflix, which accounts for a third of all North American Internet traffic on any given night, relies on the Amazon's cloud platform because it's the only company with the capacity to deliver all of that data to customers.

 

But when AWS runs into problems, like when its Elastic Load Balancer that routes network traffic went down, causing anembarrassing Christmas Eve outage of Netflix's streaming service, there is no recourse. As a result, Netflix is forced to create its own tools for dealing with problems it encounters in AWS.

more...
No comment yet.
Scooped by Nicolas Weil
Scoop.it!

How Netflix built its OpenConnect cache to speed up your video streams

How Netflix built its OpenConnect cache to speed up your video streams | CDN Breakthroughs | Scoop.it

Netflix’s decision to build its own caching boxes for optimizing its video delivery was in part influenced by the work of online backup provider Backblaze.

 

OpenConnect boxes are being used to cache Netflix content within an ISP’s data center, bringing it closer to the end user and thus improving the quality of the service. Cockroft said that Netflix opted to build its own systems for this kind of caching because it was much cheaper than buying it from a major hardware vendor. Part of the reason for that is that the legacy vendors build multipurpose systems that have to be tested and optimized for many different use cases. “We wanted a box that would do one thing,” Cockroft said.

more...
No comment yet.
Scooped by Nicolas Weil
Scoop.it!

Object Cache for Scaling Video Metadata Management

Object Cache for Scaling Video Metadata Management | CDN Breakthroughs | Scoop.it

As the largest Internet TV network, one of the most interesting challenges we face at Netflix is scaling services to the ever-increasing demands of over 36 million customers from over 40 countries.


Each movie or TV show on Netflix is described by a complex set of metadata. This includes the obvious information such as title, genre, synopsis, cast, maturity rating etc. It also includes links to images, trailers, encoded video files, subtitles and the individual episodes and seasons. Finally there are many tags that are used to create custom genres, such as “upbeat”, “cerebral”, “strong female lead”. These all have to be translated into many languages, so the actual text is tokenized and encoded.


This metadata must be made available for several different services, which each require a different facet of the data. Front-end services for display purposes need links to images, while algorithms that do discovery and recommendations use the tags extensively and search thousands of movies looking for the best few to show to a user. Powering this while utilizing resources extremely efficiently is one of the key goals of our Video Metadata Services (VMS) Platform.

more...
No comment yet.
Scooped by Nicolas Weil
Scoop.it!

Netflix's 5 Secrets For Maximizing Amazon Cloud Value

Netflix's 5 Secrets For Maximizing Amazon Cloud Value | CDN Breakthroughs | Scoop.it
Netflix chief cloud architect Adrian Cockcroft shares five money-saving maneuvers for big Amazon Web Services users.

 

Instead of trying to build out its own data centers for its rapidly expanding film and video distribution business, Netflix finds the better strategy is to use Amazon Web Services' cloud resources. At Cloud Connect 2013, the architect of that strategy disclosed some of his secrets for optimizing use of the Amazon cloud.

more...
No comment yet.
Scooped by Nicolas Weil
Scoop.it!

Netflix Announces EVCache: Distributed in-memory datastore for Cloud

Netflix Announces EVCache: Distributed in-memory datastore for Cloud | CDN Breakthroughs | Scoop.it

EVCache is a distributed in-memory caching solution based on memcached &spymemcached that is well integrated with Netflix OSS andAWS EC2 infrastructure. Today we are announcing the open sourcing of EVCache client library on Github.


What is an EVCache App?
EVCache App is a logical grouping of one or more memcached instances (servers). Each instance can be a

EVCache Server (to be open sourced soon) running memcached and a Java sidecar appEC2 instance running memcachedElastiCache instance instance that can talk memcahce protocol (eg. Couchbase, MemcacheDB)


more...
No comment yet.