Build with Pivotal HD 1.0

If you build your own applications with Maven targeting the Pivotal HD 1.0 distribution then you can use the Hadoop artifacts that are now available in the Spring IO Release Repository. The version to use for the artifacts is 2.0.2-alpha-gphd-2.0.1.0.

Spring for Apache Hadoop application using Maven

If you are using Spring for Apache Hadoop with Pivotal HD 1.0 then you need to add the following repository to your Maven pom:

<repository>
  <id>spring-releases</id>
  <name>Spring Release Repository</name>
  <url>http://repo.spring.io/libs-release</url>
</repository>

We have recently started publishing "flavors" of the Spring for Apache Hadoop project built against different Hadoop distributions. This allows for better transitive dependency resolution.

The dependency to use for the Pivotal HD 1.0 "flavor" of spring-data-hadoop is:

<dependency>
  <groupId>org.springframework.data</groupId>
  <artifactId>spring-data-hadoop</artifactId>
  <version>1.0.2.RELEASE-phd1</version>
</dependency>

If you are submitting YARN jobs from your application, then you should also add the following dependency to your applications pom:

<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
  <version>2.0.2-alpha-gphd-2.0.1.0</version>
  <scope>runtime</scope>
</dependency>

Note: The phd1 suffix for the spring-data-hadoop version will select the correct build that resolves the transitive dependencies for Pivotal HD 1.0.

Non-Spring Java application using Maven

Anyone using Maven to build a Java app with just plain Hadoop needs to add the following repository to the Maven pom:

<repository>
  <id>spring-releases</id>
  <name>Spring Release Repository</name>
  <url>http://repo.spring.io/libs-release</url>
</repository>

With this repository added and by using 2.0.2-alpha-gphd-2.0.1.0 as the version for any Hadoop dependencies the correct jars should be resolved.