Spring XD

Spring XD is a unified, distributed, and extensible system for data ingestion, real time analytics, batch processing, and data export. The project's goal is to simplify the development of big data applications.

Quick Start
Fork me on GitHub


Spring XD is a unified, distributed, and extensible runtime for data ingestion, real time analytics, batch processing, and data export. The project's goal is to simplify the development of big data applications.

Big data applications share many characteristics with Enterprise Integration and Batch applications. Spring has provided proven solutions for building integration and batch applications for more than 8 years now via the Spring Integration and Spring Batch projects. Spring XD builds upon this foundation and provides a scalable, fault-tolerant runtime environment that is easily configured and assembled via a simple DSL. Application developers can focus on business logic as the developer friendly fixtures promote productivity and the DevOps can operationalize seamlessly as the orchestration is handled out-of-the-box.

The Spring ecosystem of projects provides an excellent foundation for building big data applications. The Spring XD project aims to build upon this foundation and provide a one stop shop solution for data-driven use-cases. This is in contrast to many other offerings that are more siloed and fragmented. One of the primary features is an out-of-the-box server that provides a consistent configuration model and runtime that spans traditional enterprise to big data use-cases.

You don't need to code anything up to get going: no build scripts, no IDE, no maven coordinates, no bundling or packaging. You can use a high level configuration DSL that will allow you to hit your head against the keyboard and get started quickly. However, if you choose to extend the platform (and we hope you will), Spring provides the foundation for extensibility.

Spring XD is designed with the goal to adapt to frameworks, standards and patterns that matter the most for enterprises. Adapting to Spring’s philosophy, Spring XD follows the mantra “simple things should be simple, complex things should be possible”.

For the curious, XD is an abbreviation for eXtreme Data.


  • Easy to adapt linux style pipe-and-filter semantics
  • Unified platform - Stream Processing and Batch Jobs
    • Off-Hadoop Batch Jobs
    • Hadoop Batch workflow orchestration
    • NoSQL Analytics
    • Scoring of Machine Learning algorithms
  • Runtime that provides critical non-functional requirements
    • Scalable, distributed, Fault-Tolerant
    • Portable runtime that runs where there is JVM. On prem DIY cluster, YARN, EC2, Mesos, Docker, Pivotal Cloud Foundry (coming soon)
  • Responsive UI
    • Stream and Batch Job listing
    • Create and deploy Batch Jobs
    • Historical snapshot of Batch Job executions
    • Container cluster view and management
    • Visualize pipeline message rates
  • Rich integration options
    • Functional programming model with Reactor, RxJava, and Spark Streaming
    • Kafka source, sink, and as message bus
    • Sqoop job
    • Spark job
  • Secured runtime for data pipelines
    • LDAP integration
    • File based security Auth
    • HTTPS/Basic Auth support
    • Kerberos support for HDFS sink
    • Role based access control (RBAC)
  • Easy to extend and integrate other technologies
  • No vendor lock-in

Quick Start


  • To get started, make sure your system has as a minimum Java JDK 6 or newer installed. Java JDK 7 is recommended.

Manual Installation

  • Download spring-xd-1.0.3.RELEASE.zip or spring-xd-1.1.0.M2.zip
  • Unzip the distribution. This will yield the installation directory spring-xd-1.0.3.RELEASE. All the commands below are executed from this directory, so change into it before proceeding
$ cd spring-xd-1.0.3.RELEASE
  • Set the environment variable XD_HOME to the installation directory <root-install-dir>\spring-xd\xd

Create a stream

  • Start-up the runtime: The single node option is the easiest to get started with. It runs everything you need in a single process. To start it, you just need to cd to the xd directory and run the following command: xd/bin>$ ./xd-singlenode
  • Start XD Shell: ./xd-shell
  • Create your first stream by typing: xd:> stream create --definition "time | log" --name ticktock --deploy