Janus in Embedded mode with google bigtable as storage backend

[vinay1331]

I want to embed janus server in my JVM based app but with google bigtable as storage backend. Is it posssible? if Yes is there any documentation which i can refer to set it up.

[porunov]

Yes, it's possible as with any other storage backend.
See how to configure such connection here: https://docs.janusgraph.org/storage-backend/bigtable/
In your JVM you would typically do something like:

JanusGraph janusGraph = JanusGraphFactory.open(graphConfiguration);

Where graphConfiguration can be either file path or commons2 Configuration or BasicConfiguration or ReadConfiguration.

Notice, in case you run in embedded mode it's strongly recommended to run JanusGraph as close to your storage backends (bigtable) as possible as it will affect your performance.

[vinay1331]

Thank a ton for the help @porunov
I am able to connect to the big table but while adding an vertex getting an exception
Wrote claim for id block [1, 51) in PT0.388S => too slow, threshold is: PT0.3S
Is this related to performance you have mentioned and is there a way to increase this threshold for development on local box.

and also on startup i am getting below exception as well
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries. is this related to os may be windows ??

[porunov]

Wrote claim for id block [1, 51) in PT0.388S => too slow, threshold is: PT0.3S

This error could potentially be due to high latency but it doesn't look like that. It looks more like wrong timeouts and ids block configuration to me.

You could try increasing the ids allocation timeout: ids.renew-timeout = 120000.
Also, allocating block of 50 is quite small for most usecases. Probably you would like to increase it ids.block-size = 100000.

Nevertheless, when I was referring to high latency I was referring more to: #455 (comment)

Where I recommended to increase storage.lock.wait-time = 5000 or something like that because in case your latency you will see errors during communication between your storage backend and JanusGraph.

Low performance could be the result of high latency between your storage backend and JanusGraph. We would recommend to place JanusGraph (it doesn't matter if it is embadded or not) as close to storage backends as possible because JanusGraph usually performs many small requests instead of one big request. Thus, high latency will most likely result in bad overall performance. Usually you would place JanusGraph in the same private network as your storage backends or even on the same nodes as your storage nodes (it has it's own pros and cons).

java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries. is this related to os may be windows ??

This error is unrelated. It looks like you are running JanusGraph on Windows. If so, you need to download winutils.exe and locate it in accordance to your file system in bin folder in case you are running it via gremlin server script.

[vinay1331]

I am able to make it work with following configuration

storage.backend=hbase
storage.hbase.ext.hbase.client.connection.impl=com.google.cloud.bigtable.hbase2_x.BigtableConnection
storage.hbase.ext.google.bigtable.project.id=project-id
storage.hbase.ext.google.bigtable.instance.id=instance-id
cache.db-cache=false
ids.authority.wait-time=5000
ids.renew-timeout=120000
ids.block-size=100000
ids.renew-percentage=0.3
storage.lock.wait-time=5000

But it is not persisting the data in bigtable querying it back doesnot retrieve the data
graph.traversal().addV("Person").property("name", name).next()
using above to add vertex

Am i still missing any conf and in the getIDBlock method of ConsistentKeyIDAuthority class i am getting block size of only 50
And i am using janusgraph-all in pom is it right to include this as dependency so should i add specific required depenendcies which seems a lot in bigtable case

[porunov]

You need to commit the transaction where you created your data.
Probably graph.tx().commit() if you use default thread bounded transaction