Cloud Storage, Di...
Follow
Find
4.8K views | +2 today
Cloud Storage, Distributed File System
Cloud Storage, 분산파일 시스템, 분산 Storage
Your new post is loading...
Your new post is loading...
Scooped by Steve Hyounggi Min
Scoop.it!

What you need to know about data center switching and packet buffering

What you need to know about data center switching and packet buffering | Cloud Storage, Distributed File System | Scoop.it
By Steve Chalmers, HP Networking Advanced Technology Group, and Ahmad
Zamer, HP Networking Global Product Marketing Choosing between
traditional, low-latency and deep-buffered data center network switches
requires you give far more attention to application behavior today than
it did just a few years ago. If you’re a network designer, you must look
past average bandwidth to understand whether there is a Hadoop-like
application which bursts data from many nodes to a single target to meet
its deadline, or traffic sensitivity to the slightest delay like High
Frequency Trading (HFT) and some supercomputer applications. There is no
longer a safe one-size-fits-all switch for your data centers. The shift
from gigabit to 10-gigabit server connections called for bandwidths so
high that switch chip designers had to choose between: On-chip buffers
which were fast, modest size and modest per-port cost, orExternal DRAM
buffers which were slow, very large, and more expensive per-port because
spending all those pins on DRAM interface means a switch chip has very
few portsEnter a class of switches optimized for low latency Demand for
low-latency communication between servers has been around for decades in
the supercomputer business. In the past few years, this demand has
become important to the HFTsegment of the financial services industry as
well. While generally the software path and NIC latency in the sending
and receiving nodes contribute more to latency than the switches
themselves, a class of switches optimized for low latency has emerged.
Low -latency switches are typically “cut-through” designs with minimal
packet processing; tables and buffer space are modest and strictly on
chip. InfiniBand has been the best-in-class low-latency fabric for over
a decade, but there are situations where a well-tuned but slightly
slower Ethernet based approach is preferred. Summarizing the technology
before moving on to applications, we have three switch design centers:
MainstreamDeep BufferLow LatencyBuffer spaceSmall and on chipGigabyte+,
stored in off chip DRAMVery small and on chipTable SizesModest and on
chipOn chip for Access switch, off chip in a core switchModest to very
small, always on chipPacketProcessingOn chip, single pass but can take
more time, do moreOn chip for Access switch. Core switch can do even
more.Focused: do a few things well, and do them quicklyTCP for end to
end flow controlWorks well for general workloads; DCTCP maybe betterBest
for handling incasts without packet dropsWants to run lossless and
generally can’t afford TCP retriesSwitch internalsHybridStore and
ForwardCut ThroughRisksIncast exceeds buffer capacityBuffer
bloatCongestion related performance collapse Here’s what we’re talking
about: a quick definition of switch-design terms Cut through: The switch
looks at the headers as soon as they arrive and starts sending the frame
to its destination immediately, even before the rest of the packet has
arrivedStore and Forward: The switch waits for the entire packet to
arrive and checks that it is not corrupt, before starting to send it on
its wayIncast: This is best explained with an example. Hadoop gets a
query and delegates a piece of it to each of tens of servers, giving
them all a deadline. As the deadline approaches, each of those servers
sends a burst of data to the server that will combine the results.
Because tens of servers are all sending to the same port at the same
time, these bursts need to be held in switch buffer space, queued for
that one port to receive. It is not unusual for such bursts to exceed
the available buffer space of a mainstream switch chip; if any packets
are dropped the TCP retries will miss the deadline and therefore be
excluded from the query results.Buffer bloat: if an application pushes
data into the network as fast as it can without any feedback from its
destination, a deep buffer switch will simply keep putting those packets
in its buffers. his is a waste of buffer space and tends to make the
network sluggish as other traffic is queued behind the “bloat.”DCTCP or
Data Center TCP: A Stanford/Microsoft research paper several years ago
proposes that TCP could be retuned to work significantly better within a
data center, using feedback on buffer consumption in switches to slow
senders much more rapidly than TCP can when it has to work at distances
and delays spanning the globe. The challenge here is that all of the
servers, and all of the switches, in a data center would have to support
and be configured for DCTCP. Any legacy devices using regular TCP would
get an unfair share of network bandwidth. So looking at application
behavior, the choice of switches becomes:
ApplicationBehaviorMainstreamDeep BufferLow LatencyNeeds extreme low
latencyPossibleNo. Storing and retrieving packets add to latency
BestIncast type behavior (large bursts to a single destination which may
not be dropped)Not recommended. BestPossible, but fabric must be
configured lossless and overprovisioned“Push” large amounts of data
without regard to receiverBest. Use small buffers and TCP drops to
regulate the senderNot recommended. This is the buffer bloat
case.Possible. Fabric must be tuned to handle this case to avoid
congestion and collapse of performanceIntend to use near 100% of
bandwidthNot recommended. Buffers likely to be exhausted resulting in
lower bandwidthBest when latency is not an issue; long queue enables
high utilizationPossible: best when low latency is also required as in
supercomputer cases HP offers a wide switch product line that satisfies
your needs for low latency, deep buffers and mainstream deployments
MainstreamDeep BufferLow LatencyProduct5900AF – gigabit5820 – 10
gigabit5830AF – gigabit5920AF – 10 gigabit5900AF – 10GbEHP offers
InfiniBand products with our BladeServers and other systems Do you have
a story to share about your own experience with packet buffering? Share
your story with our readers. Read my colleague’s blogs talking about the
data center today: Tell your data center story: Intelligent Resilient
Framework for scalable, reliable data centersSimplicity: what Ethernet
Virtual Interconnect brings to Data Center Interconnect>> Learn more
about HP Networking products and solutions.>> Follow HP Networking on
Twitter and Google+ | Join HPN LinkedIn Community | Like us HPN Facebook
more...
No comment yet.
Scooped by Steve Hyounggi Min
Scoop.it!

Elasticsearch - Devoxx France 2012 - English version

Elasticsearch presentation for Devoxx France 2012 English translation (feel free to correct my bad english ;-) ) French version is available here : http://www
more...
No comment yet.
Scooped by Steve Hyounggi Min
Scoop.it!

Battle of the giants: Apache Solr vs ElasticSearch

Slides from my talk during ApacheCon EU 2012 - "Battle of the giants: Apache Solr vs ElasticSearch". Video available at http://player.vimeo.com/video/55645629
more...
No comment yet.
Scooped by Steve Hyounggi Min
Scoop.it!

High Scalability - High Scalability - Tachyon - Fault Tolerant Distributed File System with 300 Times Higher Throughput than HDFS

High Scalability - High Scalability - Tachyon  - Fault Tolerant Distributed File System with 300 Times Higher Throughput than HDFS | Cloud Storage, Distributed File System | Scoop.it
Tachyon ( github ) is interesting new filesystem brought to by the folks at the UC Berkeley AMP ...
more...
Bryan Sung's curator insight, July 25, 2013 6:14 AM

interesting file system...

Scooped by Steve Hyounggi Min
Scoop.it!

[#HDFS-347] DFS read performance suboptimal when client co-located on nodes with data - ASF JIRA

[#HDFS-347] DFS read performance suboptimal when client co-located on nodes with data - ASF JIRA | Cloud Storage, Distributed File System | Scoop.it
Steve Hyounggi Min's insight:

Fast-path IO

more...
No comment yet.
Scooped by Steve Hyounggi Min
Scoop.it!

Data Deduplication Tactics With HDFS and MapReduce

Data Deduplication Tactics With HDFS and MapReduce | Cloud Storage, Distributed File System | Scoop.it
5 techniques and links to research papers about data deduplication using HDFS and MapReduce: Some of the common methods for data deduplication in storage architecture include hashing, binary comparison and delta differencing.
more...
No comment yet.
Scooped by Steve Hyounggi Min
Scoop.it!

SSD의 이해 - IDG Tech Report

SSD의 이해 - IDG Tech Report | Cloud Storage, Distributed File System | Scoop.it
PC를 켤 때면, 가만히 앉아 윈도우 시작 소리가 들릴 때까지의 기다리는 시간은 무척 길다. 이제는 스위치를 누른 뒤, 바로 앉아도 된다. 또한 귀를 자극하던 소음 또한 사라
more...
No comment yet.
Scooped by Steve Hyounggi Min
Scoop.it!

Why MapR dfs is better than HDFS? - MapR and Apache Hadoop

Why MapR dfs is better than HDFS? - MapR and Apache Hadoop | Cloud Storage, Distributed File System | Scoop.it
Hello guys: I have read mcsrivas ppt, it compare the namenode implementation between HDFS and MapR. I still do not understand why disttributed name node is better than the central one. Where can I get all file information?
Steve Hyounggi Min's insight:

MapR DFS

more...
No comment yet.
Rescooped by Steve Hyounggi Min from Cloud & Big Data Platform
Scoop.it!

Benchmark Tools – What I Use

Benchmark Tools – What I Use | Cloud Storage, Distributed File System | Scoop.it
IOmeter IOmeter is the primary tool I use to benchmark a disk subsystem, It could be SAN, NAS or DAS. Start with 4KB as it represents the default NTFS Unit Allocation Size. Next with 8KB because it...
Steve Hyounggi Min's insight:

File System Benchmark Tools - IOzone, IOmeter, HD Tune

more...
Steve Hyounggi Min's curator insight, December 17, 2012 5:44 AM

File System Benchmark Tools - IOzone, IOmeter, HD Tune

Scooped by Steve Hyounggi Min
Scoop.it!

Brief introduction on Hadoop,Dremel, Pig, FlumeJava and Cassandra

A Brief Discussion on: Hadoop MapReduce, Pig,JavaFlume,Cascading & Dremel Presented By: Somnath Mazumdar 29th Nov...
more...
No comment yet.
Scooped by Steve Hyounggi Min
Scoop.it!

DataStax introduces a Cassandra-based Hadoop distribution called Brisk | DBMS 2 : DataBase Management System Services

Cassandra company DataStax is introducing a Hadoop distribution called Brisk, for use cases that combine short-request and analytic processing.
more...
No comment yet.
Scooped by Steve Hyounggi Min
Scoop.it!

hello world » elasticsearch로 로그 검색 시스템 만들기

hello world » elasticsearch로 로그 검색 시스템 만들기 | Cloud Storage, Distributed File System | Scoop.it

elasticsearch는 Shay Banon이 Lucene을 바탕으로 개발한 분산 검색엔진입니다. 설치와 서버 확장이 매우 편리하기 때문에 개발하고 있는 시스템에 검색 기능이 필요하다면 elasticsearch를 적용하는 것을 권장하고 싶습니다. 분산 시스템이기 때문에 검색 대상 용량이 증가했을 때 대응하기가 무척 수월하다는 것이 장점입니다.

more...
No comment yet.
Scooped by Steve Hyounggi Min
Scoop.it!

Our Experience of Creating Large Scale Log Search System Using ElasticSearch | Architects Zone

Our Experience of Creating Large Scale Log Search System Using ElasticSearch | Architects Zone | Cloud Storage, Distributed File System | Scoop.it
This post comes from Lee Jae Ik at the CUBRID Blog. At NHN we have a service called NELO (NHN Error Log System) to manage and search logs pushed to the...
more...
No comment yet.
Scooped by Steve Hyounggi Min
Scoop.it!

Amazon Web Services Blog: Amazon S3: Multipart Upload

Amazon Web Services Blog: Amazon S3: Multipart Upload | Cloud Storage, Distributed File System | Scoop.it
Can I ask you some questions? Have you ever been forced to repeatedly try to upload a file across an unreliable network connection? In most cases there's no easy way to pick up from where you left off and you...
more...
No comment yet.
Scooped by Steve Hyounggi Min
Scoop.it!

Tachyon - File System Provides File Sharing at Memory-Speed Across Cluster Frameworks

Tachyon is a fault tolerant distributed file system enabling reliable file sharing at memory-speed across cluster frameworks, such as Spark and MapReduce. It achieves its high performance by leveraging lineage information and using memory aggressively. Tachyon caches working set files in memory, and enables different jobs/queries and frameworks to access cached files at memory speed. Thus, Tachyon avoids going to disk to load datasets that are frequently read.

more...
No comment yet.
Scooped by Steve Hyounggi Min
Scoop.it!

Main Page

Steve Hyounggi Min's insight:

Xyratex Advances Lustre® Initiative and Assumes Ownership of Related Assets

more...
No comment yet.
Scooped by Steve Hyounggi Min
Scoop.it!

What's New and Upcoming in HDFS

Great retrospective with many architecture details of the improvements added to HDFS in 2012 and what is planned for this year by Todd Lipcon. For a quick overview: • 2012: HDFS 2.0• HA (in 2...
more...
No comment yet.
Scooped by Steve Hyounggi Min
Scoop.it!

Because Hadoop isn't perfect: 8 ways to replace HDFS

Because Hadoop isn't perfect: 8 ways to replace HDFS | Cloud Storage, Distributed File System | Scoop.it
Hadoop is on its way to becomig the de facto platform for the next-generation of data-based applications, but it’s not without some flaws.
Steve Hyounggi Min's insight:

HDFS Compatible

more...
No comment yet.
Scooped by Steve Hyounggi Min
Scoop.it!

Storage: objects, blocks, and files - OpenStack Install and Deploy Manual  - Essex

Steve Hyounggi Min's insight:

Storage: objects, blocks, files

more...
No comment yet.
Scooped by Steve Hyounggi Min
Scoop.it!

러스터돌이의 정리창고 :: Lustre 와 HA 어떻게 할 것인가?

러스터돌이의 정리창고 :: Lustre 와 HA 어떻게 할 것인가? | Cloud Storage, Distributed File System | Scoop.it

제가 한동안 너무 바빴습니다. 아직도 뒷처리에 피X을 싸고 있지만 바쁜 와중에 뭔가 업데이트를 해야 될 것 같아서 가장 최근 Lustre HA 를 성공적으로 구축한 경험을 최대한 간단히 공유해 보려고 합니다.

Lustre 와 HA (High Availability) 를 어렵게 생각하시는 분이 너무 많습니다. 혹은 전혀 안된다거나, 아니면... 원래 잘 돼야 되는 거 아니냐는... 그런 분도 계시고 !!

more...
No comment yet.
Scooped by Steve Hyounggi Min
Scoop.it!

Introducing Apache Hadoop: The Modern Data Operating System - Stan...

Sophisticated data instrumentation and collection technologies are leading to unprecedented growth.
more...
No comment yet.