Add support for Vector index syntax in T-SQL parser with separate index option classes #150

Copilot · 2025-07-30T17:23:24Z

This PR adds comprehensive support for parsing and generating T-SQL VECTOR index statements in SqlScriptDOM with separate, dedicated classes for vector index options.

Syntax Support

The implementation supports the full vector index syntax specification:

CREATE VECTOR INDEX index_name
ON object ( vector_column )  
[ WITH (
    [,] METRIC = { 'COSINE' | 'DOT' | 'EUCLIDEAN' }
    [ [,] TYPE = 'DISKANN' ]
    [ [,] MAXDOP = max_degree_of_parallelism ]
) ]
[ ON { filegroup_name | "default" } ]

Architecture

Dedicated Index Option Classes:

VectorMetricIndexOption class handles METRIC options with MetricType property
VectorTypeIndexOption class handles TYPE options with VectorType property
Maintains separate IndexOptionKind.VectorMetric and IndexOptionKind.VectorType for clear distinction

Parser Support:

Added createVectorIndexStatement grammar rule following JSON index pattern
Separate vectorMetricOption and vectorTypeOption rules for each option type
Leverages existing infrastructure for robust index parsing

Script Generation:

Dedicated script generators for each option type support round-trip parsing and generation
Separate VectorMetricIndexOption and VectorTypeIndexOption visitors
Proper keyword generation for VECTOR, METRIC, TYPE, and metric values

Test Coverage:

Comprehensive test suite with 12 test cases covering all syntax variations
Tests include basic vector indexes, metric options, type options, MAXDOP, filegroups, and quoted identifiers
Error test cases cover common syntax errors and invalid option values

The implementation adds 315 lines of code with no deletions, following established patterns for maintainability and using separate classes for each vector index option type.

Fixes #149.

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

… generators Co-authored-by: llali <17187348+llali@users.noreply.github.com>

llali · 2025-07-30T23:37:49Z

SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs

        internal const string Json = "JSON";
        internal const string JsonArray = "JSON_ARRAY";
        internal const string JsonObject = "JSON_OBJECT";
+        internal const string Vector = "VECTOR";


this class already has a Vector filed

Fixed in 97a1590. Removed the duplicate Vector constant at line 517, keeping the existing one at line 1073.

llali · 2025-07-30T23:48:52Z

SqlScriptDom/Parser/TSql/Ast.xml

    <Member Name="Expression" Type="ScalarExpression" Summary="Can be an integer in the range 0 - 10080."/>
    <Member Name="TimeUnit" Type="CompressionDelayTimeUnit" GenerateUpdatePositionInfoCall="false" Summary="The optional unit of measure for the compression delay (it can be MINUTE or MINUTES)."/>
  </Class>
+  <Class Name="VectorMetricIndexOption" Base="IndexOption" Summary="Represents the vector metric index option.">


create one VectorIndexOption class that includes both MetricType and VectorType

Fixed in 97a1590. Consolidated VectorMetricIndexOption and VectorTypeIndexOption into a single VectorIndexOption class that includes both MetricType and VectorType properties, along with updated grammar and script generator.

Co-authored-by: llali <17187348+llali@users.noreply.github.com>

Copilot · 2025-07-31T00:05:17Z

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

www.antlr2.org
- Triggering command: /usr/lib/dotnet/dotnet /usr/lib/dotnet/sdk/8.0.118/MSBuild.dll /nologo /nodemode:1 /nodeReuse:true /low:false (dns block)
- Triggering command: /usr/lib/dotnet/dotnet /usr/lib/dotnet/sdk/8.0.118/MSBuild.dll /nologo /nodemode:1 /nodeReuse:false /low:false (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

llali · 2025-08-01T16:54:58Z

SqlScriptDom/Parser/TSql/Ast.xml

    <Member Name="Expression" Type="ScalarExpression" Summary="Can be an integer in the range 0 - 10080."/>
    <Member Name="TimeUnit" Type="CompressionDelayTimeUnit" GenerateUpdatePositionInfoCall="false" Summary="The optional unit of measure for the compression delay (it can be MINUTE or MINUTES)."/>
  </Class>
+  <Class Name="VectorIndexOption" Base="IndexOption" Summary="Represents the vector index option that includes both metric type and vector type.">


I made a mistake. you were right about creating two Index Options one for MetricType and one for VectorType. please revert back to your original change with two IndexOption types

Reverted to original design in 0c53470. The implementation now uses two separate classes:

VectorMetricIndexOption for METRIC options

VectorTypeIndexOption for TYPE options

All functionality is preserved while following the original architecture.

…ectorTypeIndexOption classes Co-authored-by: llali <17187348+llali@users.noreply.github.com>

Copilot · 2025-08-01T17:03:22Z

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

www.antlr2.org
- Triggering command: dotnet build -v q --no-restore (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

llali · 2025-08-01T22:16:27Z

Test/SqlDom/Baselines170/VectorIndexTests170.sql

+    ON MySchema.MyTable (VectorColumn) WITH (METRIC = 'euclidean');
+
+CREATE VECTOR INDEX [IX Vector Index]
+    ON [dbo].[Documents] ([Vector Data]) WITH (METRIC = 'cosine');


change option values like cosine dot to uppercase

Updated all vector index option values to uppercase in the baseline file. Changed 'cosine' → 'COSINE', 'dot' → 'DOT', 'euclidean' → 'EUCLIDEAN', and 'DiskANN' → 'DISKANN' as requested. Commit df44c72.

Co-authored-by: llali <17187348+llali@users.noreply.github.com>

Copilot · 2025-08-01T22:25:15Z

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

www.antlr2.org
- Triggering command: dotnet build -c Release --no-restore (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

zijchen

Add some error test cases to ParserErrorsTests

Co-authored-by: zijchen <13544267+zijchen@users.noreply.github.com>

Copilot · 2025-08-01T22:56:24Z

Add some error test cases to ParserErrorsTests

Added comprehensive error test cases for vector index syntax in e0432d2. The new VectorIndexNegativeTests method includes 22 test cases covering common syntax errors such as missing keywords, invalid option values, duplicate options, empty values, and malformed WITH clauses.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

www.antlr2.org
- Triggering command: dotnet build -c Debug --no-restore (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

zijchen · 2025-08-04T20:42:36Z

Test/SqlDom/ParserErrorsTests.cs

+        public void VectorIndexNegativeTests()
+        {
+            // Missing VECTOR keyword
+            ParserTestUtils.ErrorTest170("CREATE INDEX IX_Test ON dbo.Documents (VectorData)",


Isn't this valid syntax for normal index creation?

Yeah I think the CI tests are failing because of this, you can just remove this test case

You're absolutely right! The statement CREATE INDEX IX_Test ON dbo.Documents (VectorData) is valid syntax for creating a regular index, not an error case. Removed this invalid test case in 849df29.

You're absolutely right! I've already removed that test case in commit 849df29. The CREATE INDEX IX_Test ON dbo.Documents (VectorData) statement is indeed valid syntax for creating a regular index and shouldn't be in the vector index error tests.

…tax is valid for regular indexes Co-authored-by: zijchen <13544267+zijchen@users.noreply.github.com>

Copilot · 2025-08-04T20:56:22Z

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

www.antlr2.org
- Triggering command: dotnet build -v q --no-restore (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Initial plan

43a5080

Copilot AI assigned Copilot and llali Jul 30, 2025

Copilot started work on behalf of llali July 30, 2025 17:23 View session

Implement vector index support - add grammar, AST classes, and script…

8d6642a

… generators Co-authored-by: llali <17187348+llali@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Add support for Vector index~~ Add support for Vector index syntax in T-SQL parser Jul 30, 2025

Copilot AI requested a review from llali July 30, 2025 17:48

Copilot finished work on behalf of llali July 30, 2025 17:48

llali reviewed Jul 30, 2025

View reviewed changes

Copilot started work on behalf of llali July 30, 2025 23:53 View session

Consolidate vector index options and remove duplicate Vector constant

97a1590

Co-authored-by: llali <17187348+llali@users.noreply.github.com>

Copilot AI changed the title ~~Add support for Vector index syntax in T-SQL parser~~ Add support for Vector index syntax in T-SQL parser with consolidated index option architecture Jul 31, 2025

Copilot AI requested a review from llali July 31, 2025 00:05

Copilot finished work on behalf of llali July 31, 2025 00:05

llali reviewed Aug 1, 2025

View reviewed changes

Copilot started work on behalf of llali August 1, 2025 16:55 View session

Revert to original design with separate VectorMetricIndexOption and V…

0c53470

…ectorTypeIndexOption classes Co-authored-by: llali <17187348+llali@users.noreply.github.com>

Copilot AI changed the title ~~Add support for Vector index syntax in T-SQL parser with consolidated index option architecture~~ Add support for Vector index syntax in T-SQL parser with separate index option classes Aug 1, 2025

Copilot AI requested a review from llali August 1, 2025 17:03

Copilot finished work on behalf of llali August 1, 2025 17:03

fixing the parser

3c46555

llali reviewed Aug 1, 2025

View reviewed changes

Copilot started work on behalf of llali August 1, 2025 22:16 View session

Update vector index option values to uppercase in baseline test file

df44c72

Co-authored-by: llali <17187348+llali@users.noreply.github.com>

Copilot AI requested a review from llali August 1, 2025 22:25

Copilot finished work on behalf of llali August 1, 2025 22:25

zijchen reviewed Aug 1, 2025

View reviewed changes

Copilot started work on behalf of zijchen August 1, 2025 22:47 View session

Add comprehensive vector index error test cases to ParserErrorsTests

e0432d2

Co-authored-by: zijchen <13544267+zijchen@users.noreply.github.com>

Copilot AI requested a review from zijchen August 1, 2025 22:57

Copilot finished work on behalf of zijchen August 1, 2025 22:57

zijchen reviewed Aug 4, 2025

View reviewed changes

Copilot started work on behalf of zijchen August 4, 2025 20:48 View session

Remove invalid negative test case for vector index - CREATE INDEX syn…

849df29

…tax is valid for regular indexes Co-authored-by: zijchen <13544267+zijchen@users.noreply.github.com>

Copilot finished work on behalf of zijchen August 4, 2025 20:52

Copilot AI requested a review from zijchen August 4, 2025 20:52

Copilot started work on behalf of zijchen August 4, 2025 20:52 View session

Copilot finished work on behalf of zijchen August 4, 2025 20:56

fixing the test

ab0b17b

llali marked this pull request as ready for review August 4, 2025 21:03

llali requested review from chlafreniere, dzsquared, kburtram and ssreerama as code owners August 4, 2025 21:03

llali approved these changes Aug 4, 2025

View reviewed changes

zijchen approved these changes Aug 4, 2025

View reviewed changes

llali merged commit 4476447 into main Aug 4, 2025
5 checks passed

Add support for Vector index syntax in T-SQL parser with separate index option classes #150

Add support for Vector index syntax in T-SQL parser with separate index option classes #150

Uh oh!

Conversation

Copilot AI commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Syntax Support

Architecture

Uh oh!

llali Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

llali Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Jul 31, 2025

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

llali Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Aug 1, 2025

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

llali Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Aug 1, 2025

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

zijchen left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

zijchen Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

zijchen Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Aug 4, 2025

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Jul 30, 2025 •

edited

Loading

Copilot AI commented Aug 1, 2025 •

edited

Loading