Possible Bug: Key size too large: 67087 > 65535 (Java::ComDatastaxOssDriverApiCoreServererrors::ServerError) on Scylla Backend

[mrckzgl]

I do currently test the Scylla Backend if it performs faster than BerkleyJE and encountered following problem when committing a transaction (this is JRuby):

app-1     | io.vavr.API$Match$Case0.apply(API.java:5135): Temporary failure in storage backend (Java::OrgJanusgraphDiskstorage::TemporaryBackendException)
app-1     | 	from io.vavr.API$Match.of(API.java:5092)
app-1     | 	from org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:131)
app-1     | 	from org.janusgraph.diskstorage.cql.function.mutate.CQLMutateManyUnloggedFunction.mutateMany(CQLMutateManyUnloggedFunction.java:79)
app-1     | 	from org.janusgraph.diskstorage.cql.CQLStoreManager.mutateMany(CQLStoreManager.java:354)
app-1     | 	from org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:84)
app-1     | 	from org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:109)
app-1     | 	from org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:106)
app-1     | 	from org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:66)
app-1     | 	from org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:52)
app-1     | 	from org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:106)
app-1     | 	from org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:161)
app-1     | 	from org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:224)
app-1     | 	from org.janusgraph.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:149)
app-1     | 	from org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:1009)
app-1     | 	from org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1619)
app-1     | 	from org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.doCommit(JanusGraphBlueprintsGraph.java:322)
app-1     | 	from org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:104)
app-1     | 	from org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.commit(JanusGraphBlueprintsGraph.java:300)
app-1     | 	from java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
app-1     | 	from java.base/java.lang.reflect.Method.invoke(Unknown Source)
app-1     | 	from org.jruby.dist/org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:300)
app-1     | 	from org.jruby.dist/org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:164)
app-1     | 	from org.jruby.dist/org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:32)
app-1     | 	from org.jruby.dist/org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:193)
app-1     | 	from org.jruby.dist/org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:346)
app-1     | 	from org.jruby.dist/org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:153)
app-1     | 	from org.jruby.dist/org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:118)
app-1     | 	from org.jruby.dist/org.jruby.runtime.InterpretedIRBlockBody.yieldDirect(InterpretedIRBlockBody.java:108)
app-1     | 	from org.jruby.dist/org.jruby.runtime.BlockBody.yield(BlockBody.java:114)
app-1     | 	from org.jruby.dist/org.jruby.runtime.Block.yieldNonArray(Block.java:228)
app-1     | 	from org.jruby.dist/org.jruby.RubyArray.collectArray(RubyArray.java:2798)
app-1     | 	from org.jruby.dist/org.jruby.RubyArray.map(RubyArray.java:2831)
app-1     | 	from org.jruby.dist/org.jruby.RubyArray$INVOKER$i$0$0$map.call(RubyArray$INVOKER$i$0$0$map.gen)
app-1     | 	from org.jruby.dist/org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:561)
app-1     | 	from org.jruby.dist/org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)
app-1     | 	from org.jruby.dist/org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:103)
app-1     | 	from org.jruby.dist/org.jruby.ir.instructions.CallBase.interpret(CallBase.java:545)
app-1     | 	from org.jruby.dist/org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:363)
app-1     | 	from org.jruby.dist/org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)
app-1     | 	from org.jruby.dist/org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:118)
app-1     | 	from org.jruby.dist/org.jruby.runtime.InterpretedIRBlockBody.commonYieldPath(InterpretedIRBlockBody.java:135)
app-1     | 	from org.jruby.dist/org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:176)
app-1     | 	from org.jruby.dist/org.jruby.runtime.BlockBody.yield(BlockBody.java:116)
app-1     | 	from org.jruby.dist/org.jruby.runtime.Block.yieldValues(Block.java:238)
app-1     | 	from org.jruby.dist/org.jruby.ir.runtime.IRRuntimeHelpers.yieldValues(IRRuntimeHelpers.java:504)
app-1     | 	from org.jruby.dist/org.jruby.ir.instructions.YieldInstr.interpret(YieldInstr.java:92)
app-1     | 	from org.jruby.dist/org.jruby.ir.interpreter.StartupInterpreterEngine.processOtherOp(StartupInterpreterEngine.java:155)
app-1     | 	from org.jruby.dist/org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:98)
app-1     | 	from org.jruby.dist/org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:118)
app-1     | 	from org.jruby.dist/org.jruby.runtime.InterpretedIRBlockBody.commonYieldPath(InterpretedIRBlockBody.java:135)
app-1     | 	from org.jruby.dist/org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)
app-1     | 	from org.jruby.dist/org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
app-1     | 	from org.jruby.dist/org.jruby.runtime.Block.call(Block.java:144)
app-1     | 	from org.jruby.dist/org.jruby.RubyProc.call(RubyProc.java:354)
app-1     | 	from org.jruby.dist/org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)
app-1     | 	from java.base/java.lang.Thread.run(Unknown Source)
app-1     | importers/lib/TaskQueueManager.rb:143:in 'start_next_item'Key size too large: 67087 > 65535 (Java::ComDatastaxOssDriverApiCoreServererrors::ServerError)

This did not occur on BerkleyJE. Is this a bug or some configuration problem or data structural problem?

best

EDIT: Related issue on scylla repo scylladb/scylladb#1736 indicates that the partition key is too large and for optimal performance should be much smaller than 64k. I wonder why this key is so large?

[porunov]

That's the default Scylla & Cassandra limitations. Key size should be <= 65535.
However, it's not clear why would you hit this limit, but a potential area you could investigate is check you don't set vertex ids to big strings. By default, JanusGraph uses Long vertex ids, but it also allows you to configure String vertex ids where you can specify any String as ID, but the size of the string should be limited to the one supported by the used storage backend. From documentation:

A string vertex ID can have arbitrary length as long as the underlying storage and index backends permit. Elasticsearch, for example, requires the length of the ID to be smaller than or equal to 512.

See documentation about custom vertex ids.

I would suggest any string vertex id to be less then 512 characters to avoid compatibility issues.

[mrckzgl]

Thanks alot! I am pretty sure I dont use custom vertex ids. Property values may get large, but property names shouldnt as well. I will investigate what insert will cause this.

[porunov]

Another thing to check is the combined size of the composite index key. Perhaps, you used multiple properties with large values in a composite index and it resulted in the encoded key larger than 65535. This should be double-checked, but I assume it could result in this issue as well.

[mrckzgl]

Yes, without double checking this is the case. I have a vertex centric edge property index, which values might get larger than 65k. Well, but I dont know what to do about this. I need that index ... :D

[mrckzgl]

I will try to truncate these properties as a workaround, it should be only a few entries affected, but I didnt know there is a limitation of value size for properties when indices are used ...

[mrckzgl]

Thanks alot, your intuition was right. Truncation of those values helped. Since we store (sometimes larger) text in these values and the composite indices only allow equality checks this is fine, as we need to use a mixed index searching in those anyway.

[porunov]

Thanks for verifying. We should probably improve documentation to clarify this limitation when people use composite indices with too big values.