Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade YARN Application Timeline Service to v2 #623

Open
leopaul36 opened this issue Jan 5, 2023 · 2 comments · May be fixed by #745
Open

Upgrade YARN Application Timeline Service to v2 #623

leopaul36 opened this issue Jan 5, 2023 · 2 comments · May be fixed by #745
Labels
yarn

Comments

@leopaul36
Copy link
Member

We could upgrade YARN Application Timeline Service to v2. This versions runs in YARN and allows HA / distributed storage.

Docs: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html

See #612
@leopaul36 leopaul36 mentioned this issue Jan 5, 2023
Closed
@leopaul36 leopaul36 added the yarn label Jan 5, 2023
@dapicard
Copy link

Hi,
I'm working on this issue and I face a dependency version problem. Timeline-server V2 uses Hbase as its persitence backend so we have to create some tables using a tool named org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator.

This tool is built using Hbase 1.2.6 but we use HBase 2.1.10. So when I try to create the tables, I encounter a linking issue :

HADOOP_OPTS=-Djava.security.auth.login.config=/etc/hbase/conf/krb5JAASClient.conf HADOOP_CLASSPATH=/opt/tdp/hbase/lib/*:/opt/tdp/hadoop-3.1.1-TDP-0.1.0-SNAPSHOT/share/hadoop/yarn/timelineservice/* hadoop org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator -create -skipExistingTablell

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Admin.createTable(Lorg/apache/hadoop/hbase/HTableDescriptor;[[B)V
        at org.apache.hadoop.yarn.server.timelineservice.storage.entity.EntityTableRW.createTable(EntityTableRW.java:121)
        at org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator.createAllTables(TimelineSchemaCreator.java:307)
        at org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator.createAllSchemas(TimelineSchemaCreator.java:277)
        at org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator.main(TimelineSchemaCreator.java:146)
...

There is some solutions I can test :

  • try to update hbase-client to 2.1.10 in our version of hadoop timelineservice
  • create an "infra" Hbase at version 1.2.6 for timelineservice
  • try to use an hbase-client 1.2.6 with our 2.1.10 server.
  • other ideas I don't think of...

What should I do ?

@dapicard dapicard mentioned this issue Mar 26, 2023
Open
@dapicard
Copy link

I give a try to update hbase-client to 2.1.10 in the TOSIT hadoop package (see TOSIT-IO/hadoop#3) and the schema creation works fine. So I will finish to integrate Timeline Server v2 taking into consideration that hadoop will be built with HBase 2.1.10.

@dapicard dapicard linked a pull request Apr 14, 2023 that will close this issue
Draft
2 tasks
@dapicard dapicard mentioned this issue Jun 24, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
yarn
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

WIP: 612 yarn timeline v2 dapicard/tdp-collection
2 participants
@dapicard @leopaul36