Does SparkGraphComputer honor spark.deploy-mode?

[BrunoBerisso]

Hi all.

I'm running some experiments around this and as far as I can check it seems that the traversal always runs in spark.deploy-mode=client. Here is a Dockerfile I'm using to play with this: https://github.com/BrunoBerisso/JanusGraph-SparkGraphComputer

As far as I can understand, this makes sense but my experience with Spark is limited, and just wanted to check with the community in case I'm missing something.

Thanks!

[li-boxuan]

My experience with Spark is limited too and I personally only used spark.deploy-mode=client option before. Maybe the gremlin-user group is a more suitable place to ask.

[BrunoBerisso]

Yes, that makes sense. I'll ask the Tinkerpop folks.
Thanks!

[BrunoBerisso]

FYI, I reach out to the Gremlin user group and they weren't sure about this also so I end up building a little experiment here https://github.com/BrunoBerisso/JanusGraph-SparkGraphComputer and did some debugging.

Turns out that deploy-mode=cluster only works when using (some of the many forms of) spark-submit. So in short: submitting gremlin queries to a traversal source through SparkGraphComputer will always run in client mode unless specific actions are taken.