Adding Apache Impala to TDP is not possible

[sergkudinov]

Apache Impala

Summary

I am starting this discussion just to track my work on analysis whether Apache Impala can be integrated to TDP or not. Shortly, I tried to build Impala with the required dependencies using TDP versions, and it didn't work. Therefore, Impala can't be integrated into TDP.

Presentation of the project

Apache Impala is the open source, native analytic database for Apache Hadoop.

Building Impala with TDP dependencies

It was tested building 3 different versions of Impala 3.2.0, 3.4.1 and 4.2.0 (latest) using theirs corresponding branches of the repository in a Ubuntu 18.04 virtual machine. These versions are chosen because they have close minor versions of their CDP dependencies in the default configuration, for example here.

Note, all the variables containing components versions have the CDP_ prefix. Only starting from Impala version 4.1.0 the 2 APACHE_ prefixed versions appeared: APACHE_HIVE_VERSION and APACHE_OZONE_VERSION.

Following the documentation, the required dependencies to build Impala are Apache Hadoop, Hive, HBase. I used these versions:

HADOOP_VERSION = 3.1.1-TDP-0.1.0-SNAPSHOT
HBASE_VERSION = 2.1.10-TDP-0.1.0-SNAPSHOT
HIVE_VERSION = 3.1.3-TDP-0.1.0-SNAPSHOT

While building I got incompatibility errors like below and all builds failed:

[ 65%] Building CXX object be/src/exec/CMakeFiles/Exec.dir/plan-root-sink.cc.o
/home/sergei/Impala/be/src/runtime/io/disk-io-mgr.cc: In lambda function:
/home/sergei/Impala/be/src/runtime/io/disk-io-mgr.cc:446:14: error: ‘hdfsPreadFully’ was not declared in this scope; did you mean ‘hdfsPread’?
  446 |       return hdfsPreadFully(hdfs_conn, remote_hdfs_file, offset_,
      |              ^~~~~~~~~~~~~~
      |              hdfsPread

...

[ 73%] Built target Runtime
CMakeFiles/Makefile2:1504: recipe for target 'CMakeFiles/notests_all_targets.dir/rule' failed
make[1]: *** [CMakeFiles/notests_all_targets.dir/rule] Error 2
Makefile:225: recipe for target 'notests_all_targets' failed
make: *** [notests_all_targets] Error 2

Conclusion

All my attempts to build Apache Impala using TDP components as dependencies have failed. I conclude that at the moment it can not be integrated into TDP.