Build with Pivotal HD 3.0

If you are using Spring for Apache Hadoop with Pivotal HD 3.0 then you need to add the following repository to your Maven pom:

<repository>
  <id>spring-releases</id>
  <name>Spring Release Repository</name>
  <url>http://repo.spring.io/libs-release</url>
</repository>

Current: Spring for Apache Hadoop v2.2

If you build your own applications with Maven targeting the Pivotal HD 3.0 distribution then you can use the Hadoop artifacts that are now available in the Spring IO Release Repository. The version to use for the artifacts is 2.6.0.3.0.0.0-249.

Spring for Apache Hadoop v2.2 application using Maven

We have recently started publishing "flavors" of the Spring for Apache Hadoop project built against different Hadoop distributions. This allows for better transitive dependency resolution.

The dependency to use for the Pivotal HD 3.0 "flavor" of spring-data-hadoop is:

<dependency>
  <groupId>org.springframework.data</groupId>
  <artifactId>spring-data-hadoop</artifactId>
  <version>2.2.0.RELEASE-phd30</version>
</dependency>

If you are submitting YARN jobs from your application, then you should also add the following dependency to your applications pom:

<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
  <version>2.6.0.3.0.0.0-249</version>
  <scope>runtime</scope>
</dependency>

Note: The phd30 suffix for the spring-data-hadoop version will select the correct build that resolves the transitive dependencies for Pivotal HD 3.0.

Non-Spring Java application using Maven

With the repository mentioned above added and by using 2.6.0.3.0.0.0-249 as the version for any Hadoop dependencies the correct jars should be resolved.

Hive, Pig and HBase

If you are using Hive, Pig or HBase dependencies you should use the following versions in your Maven pom:

• Hive 0.14.0.3.0.0.0-249
• Pig 0.14.0.3.0.0.0-249
• HBase 0.98.4.3.0.0.0-249-hadoop2