adds global configuration key to register gremlin traversal sources

[erickj]

Allows customizing the gremlin script engines traversal sources. This allows clients of the database to register tinkerpop traversal DSLs or customize the traversal strategies.

Motivation

Using tinkerpop DSLs and customizing traversal strategies.

Related issues

A list of issues either fixed, containing architectural discussions, otherwise relevant
for this Pull Request.

Additional Notes

Anything else we should know when reviewing?

Checklist

• I have run the build using mvn clean package command (some tests seem flaky, but unrelated)
• [x ] My unit tests cover both failure and success scenarios

[erickj]

@lvca please let me know what you think.

I find tinker pop DSLs to be one of the most useful features of the framework. Additionally, it's nice to have the ability to customize the graph travresal strategies for various reasons.

[lvca]

Thanks @erickj I think it's a cool feature to have integrated. Some questions in my review.

[lvca]

What does the annotation processor do? Does it browse all the jars in the classpath looking for an annotation?

[erickj]

it builds the SocialTraversalSource from the SocialTraversalDsl for the tests:

https://tinkerpop.apache.org/docs/current/reference/#gremlin-java-dsl

I'm not familiar very familiar with maven, but I added the tinkerpop-annotation artifact as a test-scoped dependency only. Will that be enough to not affect the release jar? Can I limit the annotationprocessor to just the test?

[lvca]

test only = good.

[lvca]

Perfect now, merged. Thanks!

@erickj it would be awesome if you could add other developers to use the Gremlin DSL. Maybe a paragraph in our docs?
@gramian WDYT?

[erickj]

sg, I'll send a pull request on the docs repo in the next couple of days

[lvca]

Thanks

[erickj]

@lvca I'm sorry to create extra work on this, but unfortunately I think this needs to get rolled back in its current form and rethought. Currently it is entirely broken in the GremlinLangScriptEngine, which was overlooked in testing.

While working on writing up some documentation for this last night I was inspecting the Tinkerpop sources for the java GremlinLangScriptEngine, and came across a surprising & AFAICT undocumented check, which does not affect the groovy engine. The gremlin lang g4 grammar explicitly defines its TRAVERSAL_ROOT token as the literal 'g'. This is then validated in the GremlinLangeScriptEngine#eval and throws an error if the g graph is not bound.

I went back to verify this and tried adding another test to the GremlinTest make sure to set java as the GREMLIN_ENGINE, sure enough I get this parse error:

Caused by: javax.script.ScriptException: org.apache.tinkerpop.gremlin.language.grammar.GremlinParserException: Failed to interpret Gremlin query: Query parsing failed at line 1, character position at 0, error message : mismatched input 'friends' expecting {EmptyStringLiteral, 'g'}
	at org.apache.tinkerpop.gremlin.jsr223.GremlinLangScriptEngine.eval(GremlinLangScriptEngine.java:116)
	at java.scripting/javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:231)
	at com.arcadedb.gremlin.ArcadeGremlin.executeStatement(ArcadeGremlin.java:215)
	at com.arcadedb.gremlin.ArcadeGremlin.executeStatement(ArcadeGremlin.java:201)
	at com.arcadedb.gremlin.ArcadeGremlin.execute(ArcadeGremlin.java:67)

Worse yet, if I try to just rename the binding to g in the test I have realized that the entire Gremlin.g4 grammar seems to have no support for DSLs at all, as each traveral method is explicitly tokenized in the grammar.

I can't find any info on this issue in the tinkerpop issue tracker or mailing lists, so I'm asking on their answers server to see if there is more info.

I think in the meantime this can:

a.) be rolled back entirely
b.) only allowed for GroovScriptEngine

a.) is probably better IMO

[lvca]

@erickj no worries, glad you caught up before the release. Usually the Gremlin Discord channel is the best way to get help and they are super responsive. You could link this issue (they are familiar with ArcadeDB). For now maybe it's better to rollback waiting for a solution that works in both cases.

[erickj]

Just to wrap up on this, I've had a brief conversation w/ Stephen Mallette on the tinkerpop discord about this

https://discord.com/channels/838910279550238720/1152253500705755258

tldr; from his POV, there is no plan to support DSLs in the java interpreter at any point in the near future, and "the two ScriptEngine implementations are not meant to have complete feature parity. " Overall the expectation of DSL usage is to pre-translate the query to bytecode on a client endpoint via something like https://github.com/apache/tinkerpop/blob/master/gremlin-javascript/src/main/javascript/gremlin-javascript/lib/process/translator.js then send the bytecode over HTTP to the server endpoint (aside, personally, I've found this approach too heavyweight in the past as you need to carry around your DSL & Traversals in multiple languages and need a javascript interpreter & HTTP server somewhere in the chain)

In the end this will most likely only ever be supported by the groovy interpreter.

Overall it's a bit disappointing to hear. I'm not sure on your thoughts on the general usefulness of this, but I think for the direction of the arcade project it would be difficult for users to start to understand script engine specific features, and would be too leaky for any user who wasn't well versed in the issue.

@lvca lmkwyt

[lvca]

Thanks for the detailed explanation. So @erickj are you currently using it with the Groovy Gremlin engine? What's your experience with the DSL and ArcadeDB? When do you find useful and when does it work properly?

[erickj]

I had been using the DSL prior to switching to ArcadeDB against an embedded tinkergraph where I was evaluating queries against the groovy gremlin engine. When I began testing against arcadedb I had thought I was using the java script engine, however I believe I understand what happened now. The DSL execucution throws a ScriptException, which when the GREMLIN_ENGINE is set to 'auto' gets caught, logs an error at FINE, and falls back to gremlin groovy. Because I never run logs at trace levels this error was hidden.

When it works well is in the graph schema design phase, where I design the indexes and queries in unison so that basic queries use the correct indices. It is also to cut down on boilerplate in queries in general for common or complex (but regular) queries. Finally, it makes queries easier to write overall. Since graph design phase is effectively an encapsulation and representation of a particular domain, keeping the query nomenclature in that same domain is a much more natural way for users to think than trying to map your question back to tinkerpop traversal methods.

Using the DSL on the write side is less effective IMO as the DSL construction becomes more onerous, and I've found using the tinkerpop mutation APIs to be unnecessary when compared to the native Arcadedb mutation APIs. Also, the mutation APIs aren't adhoc like queries are.

I'm currently working on a prototype to try and support the DSL in the GremlinLangScriptEngine which I'll go back and discuss with the tinkerpop devs when/if it actually works.