colon `:` operator - initial implementation

[Samyak2]

Context

Discussion started here - #2 (comment) (see next few comments too).

The colon operator is a shorthand for variant_get. For example, variant_get(column, '$.field', 'int') can be written as column:field::int. The part after the colon is a JSON path.

Implementation

• The first major blocker was the fact that the parser did not understand : as an operator.
  • It is supported in the snowflake dialect of datafusion-sqlparser-rs, but datafusion uses the generic dialect.
  • Some initial exploration was done here: epic: Initial implementation of this crate #2 (comment)
  • I have a PR for this in the parser here: GenericDialect: support colon operator for JsonAccess apache/datafusion-sqlparser-rs#2124
• (Update: added on 1st March 2026) Make datafusion parse JsonAccess from sqlparser-rs as a binary expr: feat: parse JsonAccess as a binary operator, add Operator::Colon apache/datafusion#20628
• Use ExprPlanner to allow : as a custom operator from this crate. See: epic: Initial implementation of this crate #2 (comment)
  • Need to wait for datafusion v53 release for the above changes to come in. I have a draft PR here: wip: use ExprPlanner to convert colon op to variant_get #31
• The actual implementation of the operator - which would simply call the same function as variant_get.

References

• Databricks documentation on : - https://docs.databricks.com/aws/en/sql/language-manual/functions/colonsign (note: their colon works on JSON strings too, but we would be focusing on variant in this case)
• A blog post I had authored on variant: https://www.e6data.com/blog/faster-json-sql-variant-data-type

[Samyak2]

Update: apache/datafusion-sqlparser-rs#2124 is merged

[sdf-jkl]

Hi @Samyak2, how should we proceed with the next step?

[Samyak2]

The next step would be to use ExprPlanner to recognize the colon syntax. I'm working on that.
The blocker here is that datafusion v52 still uses an older v0.60 of datafusion-sqlparser-rs (which doesn't include apache/datafusion-sqlparser-rs#2124). The update was done recently (in apache/datafusion#20177) and should be available in the next v53 release of datafusion.

[sdf-jkl]

Thanks! Let's wait for the v53 release.

[Samyak2]

I tried using ExprPlanner from datafusion main to convert colon op to variant_get - #31

But I ran into a problem. sqlparser-rs exposes colon operator as a special JsonAccess method, not as a simple binary expr. I had to convert it in datafusion so that an ExprPlanner can intercept it. Please take a look: apache/datafusion#20628