TINKERPOP-3219 Expose serialization functions for alternative transport protocols in gremlin-go

[DR1N0]

Motivation

The current gremlin-go driver only supports WebSocket transport. However, many modern graph database deployments use alternative protocols like gRPC for better performance, streaming capabilities, and infrastructure compatibility.

The serialization logic (GraphBinary) in gremlin-go is currently internal/private, preventing developers from building custom transport implementations while maintaining Gremlin API compatibility.

JIRA: TINKERPOP-3219

Changes

This PR exposes existing internal serialization functions by making them public, bringing gremlin-go in line with other GLVs (Python, .NET, JavaScript) that already expose their serialization layers.

Functions Made Public

In request.go:

• MakeBytecodeRequest() - Creates a request from bytecode (formerly makeBytecodeRequest)
• MakeStringRequest() - Creates a request from string query (formerly makeStringRequest)

In serializer.go:

• SerializeMessage() - Serializes request to GraphBinary format (formerly serializeMessage)
• DeserializeMessage() - Deserializes GraphBinary to response (formerly deserializeMessage)
• And also the Serializer interface for completeness

Types Made Public

To enable usage of the serialization functions, the following types and their fields were made public:

Request handling:

• GraphBinarySerializer - The serializer implementation

Response handling:

• Response - Response structure with exported fields
• ResponseStatus - Status information (code, message, attributes)
• ResponseResult - Result data and metadata

This ensures users can create requests, serialize them, deserialize responses, and access the result data.

New Functions

• NewResultSet(requestID, results) - Creates ResultSet from collected results

Use Case

Enables building custom transport implementations for Gremlin queries:

1. gRPC Transport - Serialize Gremlin bytecode to GraphBinary, send over gRPC, deserialize responses
2. HTTP/2 - Use modern HTTP features with Gremlin API
3. Custom Protocols - Any protocol that can transport binary data

This is particularly valuable for:

• Production graph database deployments requiring gRPC
• Infrastructure that standardizes on specific protocols
• Performance-critical applications needing protocol flexibility

Example Usage

import (
    "github.com/apache/tinkerpop/gremlin-go/v3/driver/gremlingo"
)

// Create a traversal
g := gremlingo.NewGraphTraversalSource()
bytecode := g.V().HasLabel("person").Limit(10).Bytecode

// Build request with session support
req := gremlingo.MakeBytecodeRequest(bytecode, "g", "my-session-123")

// Serialize to GraphBinary
serializer := &gremlingo.GraphBinarySerializer{}
bytes, err := serializer.SerializeMessage(&req)
if err != nil {
    // handle error
}

// Send bytes over gRPC/HTTP/2/custom transport
// ... transport-specific code ...

// Deserialize response
resp, err := serializer.DeserializeMessage(responseBytes)
if err != nil {
    // handle error
}

// Access result data
result := resp.Result

Notes

This approach was chosen based on reviewer feedback suggesting consistency with other GLVs rather than adding a wrapper layer. The direct exposure of serialization functions is simpler and more maintainable.

[spmallette]

You mentioned that:

many modern graph database deployments use alternative protocols like gRPC

Could you elaborate on that? I'd be curious to know which ones are offering this that support TinkerPop.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (3.8-dev@ebdec97). Learn more about missing BASE report.

Additional details and impacted files

@@            Coverage Diff             @@
##             3.8-dev    #3285   +/-   ##
==========================================
  Coverage           ?   76.47%           
  Complexity         ?    14855           
==========================================
  Files              ?     1159           
  Lines              ?    71974           
  Branches           ?     8031           
==========================================
  Hits               ?    55039           
  Misses             ?    14006           
  Partials           ?     2929           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[DR1N0]

You mentioned that:

many modern graph database deployments use alternative protocols like gRPC

Could you elaborate on that? I'd be curious to know which ones are offering this that support TinkerPop.

That's an overstating description. gRPC isn't widespread yet. I'm working for eBay and we have an internal graph DB that is using gRPC for connection.
The problem I met was that MessageSerializer was exposed in Java, but not in go driver, forcing me to fork the whole gremlin-go to do the same thing that java client already did.

[andreachild]

It seems that the other driver GLVs already expose the serialization layer so go is an outlier. Perhaps instead of using an exposed wrapper we can just make the serializer public for consistency with the other GLVs.

[Cole-Greer]

+1 to Andrea's question above. Unless there are other justifications I'm not aware of, I would be more in favour of updating makeBytecodeRequest(), makeStringRequest(), serializeMessage(), and deserializeMessage() publicly accessible of wrapping them here. If we think that those methods don't provide a convenient interface for external usage, I would prefer to add dedicated public methods to graphBinary.go and request.go which are more usable.

[DR1N0]

Thanks for the feedback! The PR is updated:

1. Removed serializer_export.go
2. Capitalized makeBytecodeRequest/makeStringRequest in request.go
3. Capitalized serializeMessage/deserializeMessage in graphBinary.go
4. Capitalized some structs and their fields (see description)
5. Added comments explaining the alternative transport use case

[spmallette]

I'm working for eBay and we have an internal graph DB that is using gRPC for connection.

thanks, that's interesting. i've heard tell of the eBay graph. thanks for contributing. i'd actually like to hear more about this if you're willing to share. For instance, do you plug-in your own DriverRemoteConnection in some way to gRPC? Or, are queries sent in some other manner over gRPC?

[DR1N0]

Hi @spmallette !
While still keeping a websocket port for interactive console usage, our server side provides a gRPC port for production traffic, which has such constraints:

• Does not support session (conflicts with connection pooling and load balancing)
• Bytecode only (server requires "processor": "traversal", rejecting string queries likely for performance, security and stateless design)

gremlingo.DriverRemoteConnection is tightly coupled to websocket transport so we created a custom GrpcRemoteConnection. They look basically similar, take the SubmitByteCode function as an example:

1. We extract the bytecode from gremlin traversals (standard API)
2. Serialize with gremlingo.MakeBytecodeRequest and GraphBinarySerializer.SerializeMessage exposed in this PR
3. Send it over gRPC instead of websocket
4. Deserialize with the GraphBinarySerializer.DeserializeMessage exposed in this PR
5. Return standard ResultSet

[andreachild]

I am curious why you opted to create a new constuctor instead of changing an existing one to be public? Was there something lacking from the existing ones which made them not usable for your use cases?

[DR1N0]

The existing constructor signature is:

func newChannelResultSet(requestID string, container *synchronizedMap) ResultSet

The synchronizedMap parameter comes from connection.results and mainly serve for connection-based, concurrent request handling, which is not needed in my request-based use case.
Using newChannelResultSet would probably mean to make public addResult, synchronizedMap along with itself, and even something from connection.go, making everything complicated.
This new constructor enables Submitxxx to return a ResultSet from Response.ResponseResult.Data, which aligns with current interfaces

[Cole-Greer]

Hi @DR1N0,

Thanks for making all of the requested changes. All of the updates look good to me.

VOTE +1