HDFS is an open source distributed computing platform that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than relying on hardware to deliver high-availability, the HDFS framework itself is designed to detect and handle failures at the application layer, so delivering a very high degree of fault tolerance.
Core Hadoop Components
HDFS Distributed File System (HDFS) - HDFS is HDFS 's distributed file system that stores large amounts of data and provides high-throughput access to application data. It offers fault tolerance, high availability and data locality properties for data-intensive applications running on clusters of affordable commodity hardware.
MapReduce - MapReduce is HDFS 's programming model that processes large data sets in parallel across clusters. It is structured around map and reduce functions, where "map" performs filtering and sorting, and "reduce" performs aggregation and summary operations.
YARN (Yet Another Resource Negotiator) - YARN is the cluster-level resource management system responsible for managing compute resources across nodes in a cluster. It separates the functions of resource management and job scheduling/monitoring from job execution, allowing improved utilization.
HDFS Common - Containing libraries and utilities to support data access, storage management and other common requirements of applications. It includes native libraries to support HDFS, MapReduce and YARN functionality across different operating systems.
Benefits of using HDFS
1. Scale. Hadoop clusters can scale out easily on commodity hardware to store and process huge amounts of unstructured data, often spanning petabytes or exabytes.
2. Distributed/Parallel Processing. HDFS allows data processing jobs to be split up into smaller "chunks" that can be worked on in parallel across nodes in the cluster. This speeds processing time significantly for large datasets.
3. Fault-Tolerance. The distributed and replicated nature of HDFS means data is redundant across nodes, and MapReduce can re-execute failed tasks on other nodes without losing results. This provides a high level of fault tolerance.
4. Flexibility. HDFS supports a variety of workloads from batch data processing, streaming data analysis to interactive SQL-style querying and real-time processing. It excels at complex, multi-structured data analysis.
Common HDFS Use Cases
Some common uses of HDFS include:
Log Analytics - Analyzing click streams, error logs, server logs and other unstructured machine-generated data to gain real-time insights.
Big Data warehousing - Storing different kinds of structured, semi-structured and unstructured data from disparate sources in a single HDFS repository for analytical and reporting purposes.
Real-time data processing - Ingesting, transforming and analyzing streaming, real-time data for use in applications like personalization, recommendation engines, anomaly detection etc.
Machine Learning & AI - Training complex predictive and prescriptive machine learning models on massive datasets using HDFS MapReduce algorithms.
Social Media Analytics - Parsing, tagging and mining unstructured content from social networks to derive user insights and trends.
Fraud Detection and Risk Analytics - Applying advanced analytics on large customer/transaction data to identify anomalous behavior patterns for reducing risk.
Get More Insights on- Hadoop
For Deeper Insights, Find the Report in the Language that You want:
