What's the release policy for TDP ?

[PACordonnier]

TDP Release Policy

The goal of this discussion is to determine the release policy for the TDP Project

Discussions includes:

• What is considered a single TDP release
• What is the current state of the different software currently developed in TDP
• What is the lifecycle of a released TDP version
• What is considered a major, minor and build number for TDP and TDP software
• What are considered a SNAPSHOT, alpha, beta and release candidates

The conventions I will mention in this post are based upon Apache Software Foundation release policy as well as resources from the Semantic Versioning scheme (SemVer)

This discussion does not address the issue of distributing the releases (Maven repositories, Hadoop Components as .tar.gz etc.).

I will try my best to be as neutral as possible and wish that this post opens up the discussion to anyone wanting to contribute.

Regarding 'Release Candidates', beta, alpha or GA

As a basis for the rest of the discussion concerning releases, here is a reminder of what are "Releases Candidates" based on Wikipedia.

Other sources can have different definitions. Feel free to discuss this as well.

Release Candidate is a beta version with the potential to be a stable product, which is ready to release unless significant bugs emerge. In this stage of product stabilization, all product features have been designed, coded, and tested through one or more beta cycles with no known showstopper-class bugs. Wikipedia

Beta version are also available :

A beta phase generally begins when the software is feature complete but likely to contain several known or unknown bugs. Software in the beta phase will generally have many more bugs in it than completed software and speed or performance issues, and may still cause crashes or data loss. Wikipedia

Once Release Candidate are validated (how ?) they can be released as GA (General Availability).

Regarding what are 0.X.Z compared to a 1.0.0 release, consider reading the FAQ of Semver

Content of TDP

As of today, TDP is:

• Git forks of software from the Apache Hadoop ecosystems
  • In each of these forks, a branch labelled '*-TDP' has been created.
    • The base of these branches has been selected to correspond to a HDP 3.1.5 release
  • These softwares includes detailed build instructions
  • These softwares includes "backports" from later releases that have not been backported to this version by the Apache Foundation
  • These forks are called "TDP-Stack" in the rest of the document
• TDP Collections : a set of Ansible roles that configures, install and manages these Hadoop software on a set of machines. TDP-collections is built to be compatible with TDP-lib
• TDP Lib : A library and a CLI that provides capability to manage a DAG of Ansible playbooks as well as Ansible variables.
• TDP Server : A HTTP server which provides a REST API over TDP lib
• TDP UI : A webserver which provides a GUI over TDP Server

For convenience 'tdp-lib', 'tdp-server' and 'tdp-ui' are called 'tdp-manager' in the rest of this post.

Some satellites projects are also available:

• TDP Getting Started, TDP Vagrant, TDP prerequisites : helps developers setup environments for testing and/or developing on TDP
• TDP-extras, tdp-observability : extras projects that works the same as tdp-collection but does not focuses on core Hadoop software

I did not place 'tdp-collection' as 'manager' nor 'stack' on purpose and is dicussed below.

Currently all TDP projects are considered 0.1.0 (-SNAPSHOT for Java Projects i.e. Hadoop software)

Contributors wants to publish a public release in the coming weeks.

I think the main questions that needs to be addressed is:

"What do we want to publish in a 1.0 ?" and "Is it relevant to publish all of TDP as a single unit ?"

Moreover I think it raises the question about what components depends on each other and if some components share a common lifecycle.

The state of TDP-Stack

TDP-Stack is the forks of Apache Hadoop project on selected branch, with added build instructions as well as some backports from more recent branches.

Currently Hadoop software forked in TDP have a version number like: 3.1.1-TDP-0.1.0-SNAPSHOT (Apache Hadoop as example):

• 3.1.1 is the release name version made by the Apache Project and from which TDP has been based
• -TDP indicates that this version is managed by the TDP Project
• -0.1.0 is the version of the TDP Hadoop distribution
• -SNAPSHOT indicates that the 0.1.0 is still a WIP and that no release of this version has been made.

Few questions regarding this version notation:

• Keeping the original Apache version number helps keep tracks of the upstream code. Is it necessary / relevant ?
• What's the lifecycle of the TDP Version in this scenario ?
  • What's the first digit used for in this context ?
  • What's the second digit used for in this context ?
  • What's the third digit used for in this context ?
  • Is a build number needed ?
• Is TDP-stack considered 1.0 ?

Please consider Semantic Versioning scheme for such questions (or why you think it should not apply)

Build number

Build number is a suffix that can be added to the version of each component. Similarily to what HDP used to do, I think build number is a convenient way to track commits added to a specific version of a Hadoop component.

Example:

• hadoop 3.1.1-TDP-1.0.0-0 is released. -0 being the build number.
• A bug is found and needs a fix or a backport.
• Fix is added to 3.1.1-TDP-1.0.0-0. It is not a component update nor a distribution update, only a bugfix.
• Version is then bumped to 3.1.1-TDP-1.0.0-1
• A release note is published describing what fix are integrated in the new build number.

Questions :

• Should this build number be implemented for TDP-Stack once released ?
• Should this build number be the last digit of the TDP-Stack version ? (1.0.X)
  • If not, what should the last digit used for ?
• Should this build number be shared across all components in the TDP-Stack ?

Pros:

• Convenient way to make sure that every release contains every fix deemed necessary
• TDP-Stack can manage its own lifecycle when it comes to bugfix backported from Apache

Cons:

• A TDP administrator needs to keeps track on what is included in each build number if he requires a specific bugfix
• Every bugfix create a release for all components (if shared), even when no updates to the component is added.

Summary:

	TDP-1.0.0	TDP-1.0.0-1 (+1 backport to HDFS)	TDP-1.0.0-1	OR TDP-1.0.1	OR TDP-1.0.1
Hadoop-TDP	3.1.1-TDP-1.0.0	3.1.1-TDP-1.0.0-1	3.1.1-TDP-1.0.0-1	3.1.1-TDP-1.0.1	3.1.1-TDP-1.0.1
HBase-TDP	2.1.10-TDP-1.0.0	2.1.10-TDP-1.0.0-1	2.1.10-TDP-1.0.0-0	2.1.10-TDP-1.0.0	2.1.10-TDP-1.0.1
Hive-3.X.X-TDP	3.1.3-TDP-1.0.0	3.1.3-TDP-1.0.0-1	3.1.3-TDP-1.0.0-0	3.1.3-TDP-1.0.0	3.1.3-TDP-1.0.1

The state of TDP-collection

TDP-collection is a set a Ansible Roles which are capable of installing TDP-stack.

TDP-collection also provides playbooks to work on its own and expose through a topology its capability to install TDP-stack on a set of machines.

By exposing a deployment DAG, TDP-lib is able to work with TDP-collection.

TDP-collection has two sides:

• It is tightly coupled to one version of TDP-stack. If TDP-stack is upgrading, the collection needs to upgrade as well to add compatibility to this component.
  • Does a compatibility matrix required for TDP-collection and TDP-stack or do they work at the same pace ?
  • Is the lifecycle of collections actually the same as TDP-stack ?
• It also exposes a deployment DAG to make it usable to work with tdp-lib.
  • Is the lifecycle of collections actually the same as TDP-lib ?
  • If a version of tdp-lib breaks compatibility with collection, how is it managed ?
• Should the ansible roles in tdp-collection be separated from the DAG definition ?

These questions are relevant to consider TDP-collection part of TDP-stack, TDP-manager, a project on its own or should be split into one collection and one implementation of the tdp-lib DAG for this collection.

The state of TDP-manager

I call TDP Manager the projects 'tdp-lib', 'tdp-server' and 'tdp-ui' for convenience.

• Does the 3 components actually share the same lifecycle ?
  • How are managed breaking changes ?
• If TDP-UI requires a change to the underlying server and tdp-lib, how is it managed ?
  • Should everyting be managed in a cascading fashion (from lib to UI), with "Method not implemented" errors if versions are not synchronised properly.
• Can TDP-manager be considered 1.0 ?

TDP-Collections implements tdp-lib's interface (trough the DAG declaration). Can this interface be considered 1.0 ?

• If the answer is yes, it means that this interface won't evolve and breaks compatibility with tdp-collections
• If the answer is no, can tdp-collection be considered 1.0 if the interface it implements is not stable.

The state of TDP as a whole

(Kinda personal thoughts here)

Questioning the dependency between TDP project have me raised concerns about the relevance of releasing TDP as a whole.

If we would compare ourselves to HDP, Hortonworks never released both HDP and Ambari as a single unit. Instead they released HDP using its lifecycle and provided a support matrix indicating which Ambari version were able to deploy each HDP version.

Compatibility matrix

Compatibility matrix can be seen as followed:

	TDP Stack 1.0.0	TDP Stack 1.0.1	TDP Stack 1.1.0
TDP Collection 1.0	X	X
TDP Collection 1.1			X

	TDP COllection 1.0.0	TDP COllection 1.0.1	TDP Collection 1.1.0	Tdp Collection 2.0.0
TDP Lib 1.0	X	X
TDP Lib 1.1			X
TDP Lib 2.0				X

Update scenario

I think the best way to think of the different possibility offered by a release policy is to mentally try some scenarios. In each scenario, explain why the workflow is the alleviate the issue and how to deploy to a running environment.

• A bug is found is found in one component and require a backport.
• We want to bump Knox from 1.6.1 to 1.7.0 (no project depends on Knox)
• We want to bump Hadoop from 2.1 to 2.2 (requires rebuild to other project on the stack)
• We want to upgrade from Hadoop 3 to Hadoop 4 (breaking changes in all subprojects, I know that Hadoop 4 does not exist)
• We want to upgrade the UI on a running cluster hence no change to be made to TDP-Hadoop, depends on features from TDP-server and tdp-lib

Feel free to add any scenario that you think are worth discussing

[kpgtek]

I would think about the following scenarii:
After backporting and/or upgrading one or several components in TDP-Stack, we need to upgrade TDP-collection and/or TDP-manager, or we do not need to upgrade TDP-collection and TDP-manager.

[PaulFarault]

IMHO when the stack evolve, the collection must evolve (as it is based on the stack) but the manager shouldn't.

As PA said, the dependency is from the manager to the collection:

TDP-Collections implements tdp-lib's interface (trough the DAG declaration).

[kpgtek]

About the release policy for TDP, should we define frequencies for a minor release (for example for every 3 months or 6 months) and for a major release (for example for every 5 years)?

[PaulFarault]

What is TDP in the context of your question? The stack, stack + collection or stack + collection + manager?

[kpgtek]

To simplify, I would like to say "the stack".
I asked this question, because I did not find any information in "Apache Software Foundation release policy". Perharps this is not applicable.
For example, in PostgreSQL's world, this is defined in the versioning policy https://www.postgresql.org/support/versioning/.

[PaulFarault]

I'll try to answer questions that relate to the UI/Manager.

Does the 3 components actually share the same lifecycle ?

The UI depends on the server which depends on the lib. Hence, there is a long delay to bring changes from the lib to the UI and, we may want to improve the UI without changing the lib or the server.
If we keep the 3 projects separate, they should have 3 different lifecycle.

Identifying breaking changes is tricky for the UI:

• Lib: backwards incompatible changes on the lib CLI, the lib API (which is used by the server) or the interface with the collections (DAG declaration).
• Server: backwards incompatible changes on the REST API.
• UI: it could be a change in the project requirements (actually Java and Node.js if using the dev version), which won't happen so often.

A matrix could show which versions are compatible with each other. Even better, we should have a website that gives the correct server and UI versions to use based on a given lib version.

If TDP-UI requires a change to the underlying server and tdp-lib, how is it managed ?

Changes will always come from the server (or even the lib). We could investigate a "Method not implemented" errors as you said while some features are not implemented yet, or even not supporting the ui (or server) for versions of the server (or lib) that are too recent.

Can TDP-manager be considered 1.0 ?

No. I can't answer for the lib but the UI and server are definitely not ready for a v1. The server lacks stability and some features are still being integrated for the UI.

IMO, we should release the ui and the server (and maybe the lib) along with the stack/collection but in 0.1. It will allow us to continue the development and to practice the release system.

[gonzaloetjo]

Given your insight it makes sense that the lib, server, and ui components should each have their own lifecycle. But in this scenario, there would be no distinct release for tdp-manager.

Perhaps in the future, we could introduce a « manager » project; its release would manage the dependency matrix between lib, server, and ui components.

This approach would combine the simplicity of Ambari with the modularity intended for TDP

[PaulFarault]

True, if we look even further, the "manager" could even deploy other stacks than TDP 🚀

[gonzaloetjo]

Regarding the categorization of various components, I concur that TDP should organize them into distinct groups, as suggested by PA. It would be beneficial for us to determine what is the appropriate way to categorize and release these components, define their composition, and outline their respective lifecycles and roadmaps.

The categorization that @PACordonnier proposes in his explanation consists of:

• Stack: Own release
• Collections
  • tdp-collection (or core): Own release
  • tdp-collections-extras: follows Core
  • tdp-collections-observability
  • tdp-collections-prerequisites:
• Manager: Own release
  • Lib: Follows Collections Core
  • Server: Follows Lib
  • UI: Follows Server
  • CLI: Follows Server

With non-categorized components being:

• tdp-vagrant
• tdp-getting-started

I propose to discuss this or other suggested categorizations further below.

Edit: This particular discussion should focus on the categorization of these components, rather than determining their specific names, which will be addressed in a subsequent step.

[gonzaloetjo]

TDP-Collection: Addressing the question posed by @PACordonnier:

The inquiry is whether TDP-collection should be part of TDP-stack, TDP-manager, a standalone project, or divided into a separate collection and an implementation of the tdp-lib DAG for this collection.

I believe tdp-collection should be a standalone project, and maybe move some work it does to a separate product.
As it stands, tdp-collection includes:

• Main:
  • Roles for installing, configuring, and starting various core components.
  • Ansible playbooks.
  • Plugin - resolve.py and tdp_vars.py: merge and resolve the variables.
• Additional:
  • Meta playbooks: to launch tdp-stack without tdp-lib.
  • tdp_lib_dags. Required for tdp-lib.
  • tdp_vars_schema. Required for tdp-lib
  • tdp_default_variables. Required for a mock deployment (getting-started).
  • topology.ini. Required for a mock deployment (getting-started).

These components are designed to:

• Deploy standalone playbooks for TDP-stack installation.
• Expose requirements for tdp-lib (and by extension, tdp-manager).
• Expose requirements for for tdp-getting-started

In this context I believe tdp-collection should have it's own release, and possibly have some workload diverged somewhere else, more specifically the ones required for tdp-getting-started like default_variables and topology.ini.

[Pierrotws]

Hi,

Please found here my humble position on versioning in TDP projects:

A: TDP (TDP-stack) Versionning

We can find here explanations of HDP:
https://community.cloudera.com/t5/Support-Questions/Question-on-HDP-Versioning/m-p/154366

To resume, version is a.b.c.d_x with:

a. Major version, mapped on Hadoop
b. Minor version, contains component upgrade but in fact was also correlated with Hadoop minor version
c. Maintenance version, should have been cumulative patch only, but in fact could contain component upgrade
d. Patch version, Non-cumulative, per-client. In fact tailored-made following a potential P1 support outage
x. Build number, for pure tracability (rebuilding with or without any versioning change)

Considerations

TOSIT won't be accountable to deliver security patch.
So even if every contribution will be redistributed to the community,
TDP users shouldn't be impacted by TOSIT reviewing and delivering process,
and thus be able to version a security patch.

Build number is a deduplication mecanism which should not be used for versioning.

Proposition I : Mapped on HDP

a.d.c.d_x with:

a. Major version, contains component upgrade with known side-effect (change of Major version): Hadoop, Spark
b. Minor version, contains minor upgrade of component
c. Cumulative Patch
d. Reserved Patch version: TOSIT should not release any patch version, this is reserved for user-specific versioning (for immediate delivery)

Pros:

• Should be quite straightforward for any previous HDP user
• Allow to maintain multiples versions (we can deliver cumulative patch for multiples major/minor versions)

Cons:

• Some time-consuming discussions for minor vs major upgrade for "in-the-middle" situation

Note:
Major version can start with 1.x or 3.x if we plan to continue to expose Major Hadoop version.

Proposition II. Simplification

a.b.c_x with:

a. Major Version, contains ANY Major/minor component upgrade
b. Cumulative Maintenance Patch, can contains patch component upgrade (HBase 1.1.2 -> 1.1.3)
c. Reserved Patch Version

Pros:

• Very straightforward for TOSIT dev, no discussion on when to version major/minor

Cons:

• Users won't be able to do minor version upgrade only <-- Is it a use case for TOSIT users ?

B: TDP Collection

TDP should probably follow TDP versioning about Major/minor

In case of P1:
- P1: a.b.c:
a. Same Major version than TDP-stack
b. Minor version of TDP
c. Cumulative fix (contains fix of installation process)
- P2: a.b
a. Same Major version than TDP-stack
b. Cumulative fix

C: TDP-lib / TDP-server / TDP-ui (etc...) Versioning

TDP-lib, tdp-server, tdp-ui, etc...

Should follow simple a.b:
a. Major version, may contains breaking changes
b. minor version, should not contains any breaking changes

With following rules:

• TDP-lib should be released first
• Major versions always match -> breaking change in TDP-lib will be followed by major versioning in other components

Note:
We should start from 0.1 and reach 1.0 when:

• tdp-lib is fully decoupled from TDP project, allowing more generic and modular use
• Versioning of tdp-stack and tdp-lib & friends has been satisfaying for both installation and upgrade

[Pierrotws]

To answer conversation around TDP versioning and corresponding package versioning:

Solution 1: Package contains TDP-versioning

This is the solution shoosed by HDP.

TDP 3-digits: hbase-1.2.0-TDP-1.0.0

Pros:

• Anyone can identify both apache version but also which TDP version embeds it

Cons:
In case TDP version is upgrading but the package is not (in terms of source), full version will automatically change, which would probably imply to build unecessary package, or use a previously builded package with a wrong TDP version:

Example: hive-2.1.0-TDP-1.0.0 --- hive-2.1.0-TDP-1.1.0

Solution 2: Packages does not contains TDP-version

In that case, it should at least contains:

• Apache version number
• TOSIT (cumulative) patch number
• User-reserved non-cumulative patch number (non cumulative means the user may or may not reinclude previous patch in the next one, at his will)

Example (chronological steps):

1. TDP initially release vanilla apache hbase ==> TDP-1.0.0 & hbase-1.2.0-0.0
2. TOSIT validates bunch of security patches ==> TDP-1.1.0 & hbase-1.2.0-1.0
3. A specific user find a critical bug and patch ==> TDP-1.1.1 & hbase-1.2.0-1.1
4. TOSIT validates that bug and decide to include it ==> TDP-1.2.0 & hbase-1.2.0-2.0
5. Apache integrates this patch in new hbase release (1.2.1) ===> TDP-1.3.0 & hbase-1.2.1-0.0

Advantages

• package version change if and only if sources changes (except for the user that creates its custom patch, if TOSIT includes only his patch for next release. And yet only apply for the patched packages)
• any one can know if TOSIT has included cumulative patch, and if user has included custom patch

Cons:

• User cannot know from a package name which TDP version is concerned (but this info should easily be found somewhere else, like listing /opt/tdp/)

Possible path hierarchy in case of patch delivery on HBase following prod outage:
/opt/tdp/assets/hbase-1.2.0-0.0 dir #Apache vanilla
/opt/tdp/assets/hbase-1.2.0-1.0 dir #TOSIT cumulative patch 1
/opt/tdp/assets/hbase-1.2.0-1.1 dir # this is not released by TOSIT, but user-made
/opt/tdp/assets/hive-2.1.0-0.0 dir # Apache vanilla
/opt/tdp/1.0.0/hbase ---> /opt/tdp/assets/hbase-1.2.0-0.0
/opt/tdp/1.0.0/hive ---> /opt/tdp/assets/hive-2.1.0-0.0
/opt/tdp/1.1.0/hbase ---> /opt/tdp/assets/hbase-1.2.0-1.0
/opt/tdp/1.1.0/hive ---> /opt/tdp/assets/hive-2.1.0-0.0
/opt/tdp/1.1.1/hbase ---> /opt/tdp/assets/hbase-1.2.0-1.1
/opt/tdp/1.1.1/hive ---> /opt/tdp/assets/hive-2.1.0-0.0 (optional, as soon as "current" won't use TDP-1.1.1 for hive)
[...]
/opt/tdp/current/hbase ---> /opt/tdp/1.1.1/hbase
/opt/tdp/current/hive ---> /opt/tdp/1.1.0/hive OR /opt/tdp/1.1.1/hive, if any (symlink to same asset anyway)

tdp-select (not existing), on host, could be use for simple, immediate rollback. This utility would only update /opt/tdp/current/* symlinks.

[gonzaloetjo]

A thing I think we never mentioned is what we do with extra binaries (Kafka, etc).
Should there be a tdp-stack-extras? Should it be versioned?

[PACordonnier]

Discussions has been merged into a versioning manifest found here VERSIONING.md.

Thank you all for your inputs !