Spring for Apache Hadoop

Simplifies Apache Hadoop by providing a unified configuration model and easy to use APIs for using HDFS, MapReduce, Pig, and Hive. Also provides integration with other Spring ecosystem project such as Spring Integration and Spring Batch.

Quick Start
Fork me on GitHub

Introduction

Spring for Apache Hadoop simplifies developing Apache Hadoop by providing a unified configuration model and easy to use APIs for using HDFS, MapReduce, Pig, and Hive. It also provides integration with other Spring ecosystem project such as Spring Integration and Spring Batch enabling you to develop solutions for big data ingest/export and Hadoop workflow orchestration.

Check out the new book from O'Reilly Media Spring Data: Modern Data Access for Enterprise Java that contains several chapters on using Spring for Apache Hadoop. Sample code for the book is also available on github here.

Features

  • Support to create Hadoop applications that are configured using Dependency Injection and run as standard Java applications vs. using Hadoop command line utilities.
  • Create and configure applications that use Java MapReduce, Streaming, Hive, Pig, Cascading, or HBase
  • Extensions to Spring Batch to support creating Hadoop based workflows for any type of Hadoop Job or HDFS operation.
  • Script HDFS operations using any JVM based scripting language
  • DAO support (Template & Callbacks) for HBase
  • Cascading Taps for Spring & Spring Integration
  • Support for Hadoop Security

Versions and Distribution Support

Spring for Apache Hadoop supports a number of Apache releases as well as several commercial distributions.

  • Apache Hadoop 1.2.1 stable
  • Apache Hadoop 2.0.6-alpha
  • Apache Hadoop 2.2.0 stable
  • Cloudera CDH4
  • Hortonworks HDP 1.3
  • Pivotal HD 1.0 and 1.1

See the wiki for Maven build details.

The continuous integration builds for most supported versions can be seen here.

Quick Start

Download

The recommended way to get started using spring-hadoop in your project is with a dependency management system – the snippet below can be copied and pasted into your build. Need help? See our getting started guides on building with Maven and Gradle.