From b78d31f913c3b91bc3fa0d9d5fc1504d396120ca Mon Sep 17 00:00:00 2001 From: Mohit Agarwal Date: Wed, 3 Sep 2025 19:21:14 +0000 Subject: [PATCH 01/13] Merged PR 1790439: Remove unused variable in Antlr grammar file TSql170 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit # Pull Request Template for ScriptDom ## Description Remove unused local variable from _alterDbModifyAzureOptions_ grammar rule and ensure accurate token position tracking for WITH MANUAL_CUTOVER clause. Details: • Eliminated redundant _hasManualCutover_ variable. • Improved maintainability and clarity. • Ensured UpdateTokenInfo is called for position tracking. ## Code Change - [x] The [Common checklist](https://msdata.visualstudio.com/SQLToolsAndLibraries/_git/Common?path=/Templates/PR%20Checklist%20for%20SQLToolsAndLibraries.md&version=GBmain&_a=preview) has been reviewed and followed - [x] Code changes are accompanied by appropriate unit tests - [x] Identified and included SMEs needed to review code changes - [x] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=make-the-changes-in) here to make changes in the code ## Testing - [x] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=to-extend-the-tests-do-the-following%3A) here to add new tests for your feature Re-ran all existing tests to ensure the code changes are not impacting anything. All tests are passing successfully: ![image.png](https://msdata.visualstudio.com/c6789c20-b819-4bfd-9917-11471655156e/_apis/git/repositories/2247f543-55d8-45df-a9fe-23820ae656af/pullRequests/1790439/attachments/image.png) ## Documentation - [ ] Update relevant documentation in the [wiki](https://dev.azure.com/msdata/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki) and the README.md file ## Additional Information Please provide any additional information that might be helpful for the reviewers N/A. ---- #### AI description (iteration 1) #### PR Classification This pull request is a code cleanup that removes an unused variable and refactors manual cutover token handling in the TSql170 Antlr grammar file. #### PR Summary The changes remove the unused boolean variable in `SqlScriptDom/Parser/TSql/TSql170.g` and replace its assignment with a call to `UpdateTokenInfo` when processing the manual cutover identifier. - `SqlScriptDom/Parser/TSql/TSql170.g`: Removed the unnecessary declaration and assignment of the variable and added `UpdateTokenInfo` to update token information for manual cutover. Related work items: #4005325 --- SqlScriptDom/Parser/TSql/TSql170.g | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/SqlScriptDom/Parser/TSql/TSql170.g b/SqlScriptDom/Parser/TSql/TSql170.g index 89c95cb..3291d23 100644 --- a/SqlScriptDom/Parser/TSql/TSql170.g +++ b/SqlScriptDom/Parser/TSql/TSql170.g @@ -3064,17 +3064,14 @@ alterDbModify returns [AlterDatabaseStatement vResult = null] ; alterDbModifyAzureOptions returns [AlterDatabaseSetStatement vResult = FragmentFactory.CreateFragment()] -{ - bool hasManualCutover = false; -} : azureOptions[vResult, vResult.Options] ( With tManualCutover:Identifier { Match(tManualCutover, CodeGenerationSupporter.ManualCutover); - hasManualCutover = true; vResult.WithManualCutover = true; + UpdateTokenInfo(vResult, tManualCutover); } )? ; From 406149fc8782cf73b9103d0f1a9d237c4440f7d8 Mon Sep 17 00:00:00 2001 From: Urvashi Raj Date: Wed, 3 Sep 2025 23:50:11 +0000 Subject: [PATCH 02/13] Merged PR 1790607: Remove Unused Parameter form External Model # Pull Request Template for ScriptDom ## Description #### AI description (iteration 1) #### PR Classification Code cleanup: Removing an unused parameter from external model parser definitions. #### PR Summary This PR streamlines the T-SQL grammar by removing the unused parameter "encounteredOptions" from external model statement rules. - `SqlScriptDom/Parser/TSql/TSql170.g`: Removed `long encounteredOptions = 0;` from the `createExternalModelStatement` rule. - `SqlScriptDom/Parser/TSql/TSql170.g`: Removed `long encounteredOptions = 0;` from the `alterExternalModelStatement` rule. --- SqlScriptDom/Parser/TSql/TSql170.g | 2 -- 1 file changed, 2 deletions(-) diff --git a/SqlScriptDom/Parser/TSql/TSql170.g b/SqlScriptDom/Parser/TSql/TSql170.g index 3291d23..47db6c8 100644 --- a/SqlScriptDom/Parser/TSql/TSql170.g +++ b/SqlScriptDom/Parser/TSql/TSql170.g @@ -23927,7 +23927,6 @@ dropSecurityPolicyStatement returns [DropSecurityPolicyStatement vResult = Fragm createExternalModelStatement returns [CreateExternalModelStatement vResult = FragmentFactory.CreateFragment()] { Identifier vName; - long encounteredOptions = 0; } : tModel:Identifier vName = identifier { @@ -24089,7 +24088,6 @@ externalModelParameters[ExternalModelStatement vParent] alterExternalModelStatement returns [AlterExternalModelStatement vResult = FragmentFactory.CreateFragment()] { Identifier vName; - long encounteredOptions = 0; } : tModel:Identifier vName = identifier { From 2317343bb2f4ccff00d16729455ba29068be4d73 Mon Sep 17 00:00:00 2001 From: Zi Chen Date: Thu, 4 Sep 2025 21:44:30 +0000 Subject: [PATCH 03/13] Merged PR 1789319: Resend PR for vector type 2nd parameter Resending PR !1780528 which was reverted to unblock release Before submitting your pull request, please ensure you have completed the following: - [ ] The [Common checklist](https://msdata.visualstudio.com/SQLToolsAndLibraries/_git/Common?path=/Templates/PR%20Checklist%20for%20SQLToolsAndLibraries.md&version=GBmain&_a=preview) has been reviewed and followed - [ ] Code changes are accompanied by appropriate unit tests - [ ] Identified and included SMEs needed to review code changes - [ ] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=make-the-changes-in) here to make changes in the code - [ ] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=to-extend-the-tests-do-the-following%3A) here to add new tests for your feature - [ ] Update relevant documentation in the [wiki](https://dev.azure.com/msdata/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki) and the README.md file Please provide any additional information that might be helpful for the reviewers Reverting commit cd69b7806dfbd6a581dbe50779a07ac70047cee8 to unblock release Before submitting your pull request, please ensure you have completed the following: - [ ] The [Common checklist](https://msdata.visualstudio.com/SQLToolsAndLibraries/_git/Common?path=/Templates/PR%20Checklist%20for%20SQLToolsAndLibraries.md&version=GBmain&_a=preview) has been reviewed and followed - [ ] Code changes are accompanied by appropriate unit tests - [ ] Identified and included SMEs needed to review code changes - [ ] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=make-the-changes-in) here to make changes in the code - [ ] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=to-extend-the-tests-do-the-following%3A) here to add new tests for your feature - [ ] Update relevant documentation in the [wiki](https://dev.azure.com/msdata/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki) and the README.md file Please provide any additional information that might be helpful for the reviewers Revert cd69b7806dfbd6a581dbe50779a07ac70047cee8 ---- This PR temporarily reverts a previous commit to remove vector data type support. The pull request reverses vector-related changes by deleting files and code modification... --- SqlScriptDom/Parser/TSql/Ast.xml | 5 ++++ .../Parser/TSql/CodeGenerationSupporter.cs | 2 ++ SqlScriptDom/Parser/TSql/TSql170.g | 29 +++++++++++++++++++ ...qlScriptGeneratorVisitor.VectorDataType.cs | 24 +++++++++++++++ .../Baselines170/VectorFunctionTests170.sql | 2 +- .../VectorTypeSecondParameterTests170.sql | 17 +++++++++++ Test/SqlDom/Only170SyntaxTests.cs | 1 + .../VectorTypeSecondParameterTests170.sql | 11 +++++++ 8 files changed, 90 insertions(+), 1 deletion(-) create mode 100644 SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.VectorDataType.cs create mode 100644 Test/SqlDom/Baselines170/VectorTypeSecondParameterTests170.sql create mode 100644 Test/SqlDom/TestScripts/VectorTypeSecondParameterTests170.sql diff --git a/SqlScriptDom/Parser/TSql/Ast.xml b/SqlScriptDom/Parser/TSql/Ast.xml index c3de308..f4e1a59 100644 --- a/SqlScriptDom/Parser/TSql/Ast.xml +++ b/SqlScriptDom/Parser/TSql/Ast.xml @@ -271,6 +271,11 @@ Summary="The xml data type option."/> + + + + + diff --git a/SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs b/SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs index e59bb88..ff1b168 100644 --- a/SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs +++ b/SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs @@ -427,6 +427,8 @@ internal static class CodeGenerationSupporter internal const string FlushIntervalSecondsAlt = "DATA_FLUSH_INTERVAL_SECONDS"; internal const string Fn = "FN"; internal const string Float = "FLOAT"; + internal const string Float16 = "FLOAT16"; + internal const string Float32 = "FLOAT32"; internal const string For = "FOR"; internal const string ForceFailoverAllowDataLoss = "FORCE_FAILOVER_ALLOW_DATA_LOSS"; internal const string ForceScan = "FORCESCAN"; diff --git a/SqlScriptDom/Parser/TSql/TSql170.g b/SqlScriptDom/Parser/TSql/TSql170.g index 47db6c8..ca86c7c 100644 --- a/SqlScriptDom/Parser/TSql/TSql170.g +++ b/SqlScriptDom/Parser/TSql/TSql170.g @@ -30675,6 +30675,32 @@ xmlDataType [SchemaObjectName vName] returns [XmlDataTypeReference vResult = Fra )? ; +vectorDataType [SchemaObjectName vName] returns [VectorDataTypeReference vResult = FragmentFactory.CreateFragment()] +{ + vResult.Name = vName; + vResult.UpdateTokenInfo(vName); + + IntegerLiteral vDimension = null; + Identifier vBaseType = null; +} + : + ( LeftParenthesis vDimension=integer + { + vResult.Dimension = vDimension; + } + ( + Comma vBaseType=identifier + { + vResult.BaseType = vBaseType; + } + )? + tRParen:RightParenthesis + { + UpdateTokenInfo(vResult,tRParen); + } + ) + ; + scalarDataType returns [DataTypeReference vResult = null] { SchemaObjectName vName; @@ -30695,6 +30721,9 @@ scalarDataType returns [DataTypeReference vResult = null] ( {isXmlDataType}? vResult = xmlDataType[vName] + | + {typeOption == SqlDataTypeOption.Vector}? + vResult = vectorDataType[vName] | {typeOption != SqlDataTypeOption.None}? vResult = sqlDataTypeWithoutNational[vName, typeOption] diff --git a/SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.VectorDataType.cs b/SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.VectorDataType.cs new file mode 100644 index 0000000..c651659 --- /dev/null +++ b/SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.VectorDataType.cs @@ -0,0 +1,24 @@ +//------------------------------------------------------------------------------ +// +// Copyright (c) Microsoft Corporation. All rights reserved. +// +//------------------------------------------------------------------------------ + +namespace Microsoft.SqlServer.TransactSql.ScriptDom.ScriptGenerator +{ + partial class SqlScriptGeneratorVisitor + { + public override void ExplicitVisit(VectorDataTypeReference node) + { + GenerateIdentifier(CodeGenerationSupporter.Vector); + GenerateSymbol(TSqlTokenType.LeftParenthesis); + GenerateFragmentIfNotNull(node.Dimension); + if (node.BaseType != null) + { + GenerateSymbol(TSqlTokenType.Comma); + GenerateSpaceAndFragmentIfNotNull(node.BaseType); + } + GenerateSymbol(TSqlTokenType.RightParenthesis); + } + } +} diff --git a/Test/SqlDom/Baselines170/VectorFunctionTests170.sql b/Test/SqlDom/Baselines170/VectorFunctionTests170.sql index 2176223..9ab5f85 100644 --- a/Test/SqlDom/Baselines170/VectorFunctionTests170.sql +++ b/Test/SqlDom/Baselines170/VectorFunctionTests170.sql @@ -1,4 +1,4 @@ -DECLARE @qv AS VECTOR (1536) = AI_GENERATE_EMBEDDINGS(N'Pink Floyd music style' USE MODEL Ada2Embeddings); +DECLARE @qv AS VECTOR(1536) = AI_GENERATE_EMBEDDINGS(N'Pink Floyd music style' USE MODEL Ada2Embeddings); SELECT t.id, s.distance, diff --git a/Test/SqlDom/Baselines170/VectorTypeSecondParameterTests170.sql b/Test/SqlDom/Baselines170/VectorTypeSecondParameterTests170.sql new file mode 100644 index 0000000..c41e7f2 --- /dev/null +++ b/Test/SqlDom/Baselines170/VectorTypeSecondParameterTests170.sql @@ -0,0 +1,17 @@ +CREATE TABLE tbl ( + embedding VECTOR(1) +); + +CREATE TABLE tbl ( + embedding VECTOR(1, float32) +); + +CREATE TABLE tbl ( + embedding VECTOR(1, FLOAT16) +); + +DECLARE @embedding AS VECTOR(2); + +DECLARE @embedding AS VECTOR(2, FLOAT32); + +DECLARE @embedding AS VECTOR(2, float16); \ No newline at end of file diff --git a/Test/SqlDom/Only170SyntaxTests.cs b/Test/SqlDom/Only170SyntaxTests.cs index 949e22d..2d5a98c 100644 --- a/Test/SqlDom/Only170SyntaxTests.cs +++ b/Test/SqlDom/Only170SyntaxTests.cs @@ -27,6 +27,7 @@ public partial class SqlDomTests new ParserTest170("VectorFunctionTests170.sql", nErrors80: 1, nErrors90: 1, nErrors100: 3, nErrors110: 3, nErrors120: 3, nErrors130: 3, nErrors140: 3, nErrors150: 3, nErrors160: 3), new ParserTest170("SecurityStatementExternalModelTests170.sql", nErrors80: 2, nErrors90: 17, nErrors100: 17, nErrors110: 17, nErrors120: 17, nErrors130: 17, nErrors140: 17, nErrors150: 17, nErrors160: 17), new ParserTest170("CreateEventSessionNotLikePredicate.sql", nErrors80: 2, nErrors90: 1, nErrors100: 1, nErrors110: 1, nErrors120: 1, nErrors130: 0, nErrors140: 0, nErrors150: 0, nErrors160: 0), + new ParserTest170("VectorTypeSecondParameterTests170.sql", nErrors80: 2, nErrors90: 2, nErrors100: 2, nErrors110: 2, nErrors120: 2, nErrors130: 2, nErrors140: 2, nErrors150: 2, nErrors160: 2), new ParserTest170("RegexpLikeTests170.sql", nErrors80: 15, nErrors90: 15, nErrors100: 15, nErrors110: 18, nErrors120: 18, nErrors130: 18, nErrors140: 18, nErrors150: 18, nErrors160: 18) }; diff --git a/Test/SqlDom/TestScripts/VectorTypeSecondParameterTests170.sql b/Test/SqlDom/TestScripts/VectorTypeSecondParameterTests170.sql new file mode 100644 index 0000000..35351bf --- /dev/null +++ b/Test/SqlDom/TestScripts/VectorTypeSecondParameterTests170.sql @@ -0,0 +1,11 @@ +CREATE TABLE tbl (embedding VECTOR(1)) + +CREATE TABLE tbl (embedding VECTOR(1, float32)) + +CREATE TABLE tbl (embedding VECTOR(1, FLOAT16)) + +DECLARE @embedding VECTOR(2) + +DECLARE @embedding VECTOR(2, FLOAT32) + +DECLARE @embedding VECTOR(2, float16) \ No newline at end of file From 29d16af0a0e5365ca4ff9a121d312e8fe35c6718 Mon Sep 17 00:00:00 2001 From: Mateo Marenco Date: Tue, 9 Sep 2025 16:51:33 +0000 Subject: [PATCH 04/13] Merged PR 1792393: Add support for Optimized Locking database option As a result of the new [Optimized Locking](https://learn.microsoft.com/en-us/sql/relational-databases/performance/optimized-locking?view=sql-server-ver17) feature, new DDL has been added: `ALTER DATABASE MyDb SET OPTIMIZED_LOCKING = ON/OFF`. This PR adds this DDL to the grammar. - [X] The [Common checklist](https://msdata.visualstudio.com/SQLToolsAndLibraries/_git/Common?path=/Templates/PR%20Checklist%20for%20SQLToolsAndLibraries.md&version=GBmain&_a=preview) has been reviewed and followed - [X] Code changes are accompanied by appropriate unit tests - [X] Identified and included SMEs needed to review code changes - [X] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=make-the-changes-in) here to make changes in the code - [X] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=to-extend-the-tests-do-the-following%3A) here to add new tests for your feature - [X] Update relevant documentation in the [wiki](https://dev.azure.com/msdata/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki) and the README.md file Please provide any additional information that might be helpful for the reviewers ---- This pull request introduces a new feature by adding support for the Optimized Locking database option. The pull request implements the support for the Optimized Locking option in ALTER DATABASE statements by extending both the parser and code generation components. - `SqlScriptGeneratorVisitor.OptimizedLockingAlterDatabaseOption.cs`: Implements the visitor method to generate script text for the new option based on its state (ON/OFF). - `TSql170.g` and `TSqlFabricDW.g`: Add new grammar rules to parse the Optimized Locking option and integrate it into the database option parsing logic. - `Ast.xml`: Introduces a new AST class for the Optimized Locking database option. - `DatabaseOptionKind.cs` and `CodeGenerationSupporter.cs`: Update the enumeration and constant definitions to include the Optimized Locking option. --- SqlScriptDom/Parser/TSql/Ast.xml | 4 +++ .../Parser/TSql/CodeGenerationSupporter.cs | 1 + .../Parser/TSql/DatabaseOptionKind.cs | 3 +- SqlScriptDom/Parser/TSql/TSql170.g | 26 ++++++++++++++++ SqlScriptDom/Parser/TSql/TSqlFabricDW.g | 26 ++++++++++++++++ ...tor.OptimizedLockingAlterDatabaseOption.cs | 31 +++++++++++++++++++ .../Baselines170/OptimizedLockingTests170.sql | 5 +++ Test/SqlDom/Only170SyntaxTests.cs | 1 + .../TestScripts/OptimizedLockingTests170.sql | 5 +++ 9 files changed, 101 insertions(+), 1 deletion(-) create mode 100644 SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.OptimizedLockingAlterDatabaseOption.cs create mode 100644 Test/SqlDom/Baselines170/OptimizedLockingTests170.sql create mode 100644 Test/SqlDom/TestScripts/OptimizedLockingTests170.sql diff --git a/SqlScriptDom/Parser/TSql/Ast.xml b/SqlScriptDom/Parser/TSql/Ast.xml index f4e1a59..347befd 100644 --- a/SqlScriptDom/Parser/TSql/Ast.xml +++ b/SqlScriptDom/Parser/TSql/Ast.xml @@ -2525,6 +2525,10 @@ + + + + diff --git a/SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs b/SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs index ff1b168..590d390 100644 --- a/SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs +++ b/SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs @@ -720,6 +720,7 @@ internal static class CodeGenerationSupporter internal const string OperatorAudit = "OPERATOR_AUDIT"; internal const string Optimistic = "OPTIMISTIC"; internal const string Optimize = "OPTIMIZE"; + internal const string OptimizedLocking = "OPTIMIZED_LOCKING"; internal const string OptimizeForSequentialKey = "OPTIMIZE_FOR_SEQUENTIAL_KEY"; internal const string OptimizerQueue = "OPTIMIZER_QUEUE"; internal const string Order = "ORDER"; diff --git a/SqlScriptDom/Parser/TSql/DatabaseOptionKind.cs b/SqlScriptDom/Parser/TSql/DatabaseOptionKind.cs index c3118e2..84d3832 100644 --- a/SqlScriptDom/Parser/TSql/DatabaseOptionKind.cs +++ b/SqlScriptDom/Parser/TSql/DatabaseOptionKind.cs @@ -98,7 +98,8 @@ public enum DatabaseOptionKind Ledger = 68, ManualCutover = 69, - PerformCutover = 70 + PerformCutover = 70, + OptimizedLocking = 71 } #pragma warning restore 1591 diff --git a/SqlScriptDom/Parser/TSql/TSql170.g b/SqlScriptDom/Parser/TSql/TSql170.g index ca86c7c..af4b8a9 100644 --- a/SqlScriptDom/Parser/TSql/TSql170.g +++ b/SqlScriptDom/Parser/TSql/TSql170.g @@ -3244,6 +3244,8 @@ dbOptionStateItem[ref ulong encounteredOptions] returns [DatabaseOption vResult vResult = changeTrackingDbOption | {NextTokenMatches(CodeGenerationSupporter.AcceleratedDatabaseRecovery)}? vResult = acceleratedDatabaseRecoveryOption + | {NextTokenMatches(CodeGenerationSupporter.OptimizedLocking)}? + vResult = optimizedLockingOption | {NextTokenMatches(CodeGenerationSupporter.Containment)}? vResult = dbContainmentOption | {NextTokenMatches(CodeGenerationSupporter.Hadr)}? @@ -3557,6 +3559,30 @@ acceleratedDatabaseRecoveryOption returns [AcceleratedDatabaseRecoveryDatabaseOp ) ; +optimizedLockingOption returns [OptimizedLockingDatabaseOption vResult = FragmentFactory.CreateFragment()] + : tOptimizedLocking:Identifier + { + Match(tOptimizedLocking, CodeGenerationSupporter.OptimizedLocking); + vResult.OptionKind = DatabaseOptionKind.OptimizedLocking; + UpdateTokenInfo(vResult, tOptimizedLocking); + } + ( + (EqualsSign tOff:Off + { + vResult.OptionState = OptionState.Off; + UpdateTokenInfo(vResult, tOff); + } + ) + | + (EqualsSign tOn:On + { + vResult.OptionState = OptionState.On; + UpdateTokenInfo(vResult, tOn); + } + ) + ) + ; + changeTrackingOnOptions [ChangeTrackingDatabaseOption vParent] { bool autoCleanupEncountered = false; diff --git a/SqlScriptDom/Parser/TSql/TSqlFabricDW.g b/SqlScriptDom/Parser/TSql/TSqlFabricDW.g index 3f1cb10..9e1fea8 100644 --- a/SqlScriptDom/Parser/TSql/TSqlFabricDW.g +++ b/SqlScriptDom/Parser/TSql/TSqlFabricDW.g @@ -3221,6 +3221,8 @@ dbOptionStateItem[ref ulong encounteredOptions] returns [DatabaseOption vResult vResult = changeTrackingDbOption | {NextTokenMatches(CodeGenerationSupporter.AcceleratedDatabaseRecovery)}? vResult = acceleratedDatabaseRecoveryOption + | {NextTokenMatches(CodeGenerationSupporter.OptimizedLocking)}? + vResult = optimizedLockingOption | {NextTokenMatches(CodeGenerationSupporter.Containment)}? vResult = dbContainmentOption | {NextTokenMatches(CodeGenerationSupporter.Hadr)}? @@ -3534,6 +3536,30 @@ acceleratedDatabaseRecoveryOption returns [AcceleratedDatabaseRecoveryDatabaseOp ) ; +optimizedLockingOption returns [OptimizedLockingDatabaseOption vResult = FragmentFactory.CreateFragment()] + : tOptimizedLocking:Identifier + { + Match(tOptimizedLocking, CodeGenerationSupporter.OptimizedLocking); + vResult.OptionKind = DatabaseOptionKind.OptimizedLocking; + UpdateTokenInfo(vResult, tOptimizedLocking); + } + ( + (EqualsSign tOff:Off + { + vResult.OptionState = OptionState.Off; + UpdateTokenInfo(vResult, tOff); + } + ) + | + (EqualsSign tOn:On + { + vResult.OptionState = OptionState.On; + UpdateTokenInfo(vResult, tOn); + } + ) + ) + ; + changeTrackingOnOptions [ChangeTrackingDatabaseOption vParent] { bool autoCleanupEncountered = false; diff --git a/SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.OptimizedLockingAlterDatabaseOption.cs b/SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.OptimizedLockingAlterDatabaseOption.cs new file mode 100644 index 0000000..6ee34d9 --- /dev/null +++ b/SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.OptimizedLockingAlterDatabaseOption.cs @@ -0,0 +1,31 @@ +//------------------------------------------------------------------------------ +// +// Copyright (c) Microsoft Corporation. All rights reserved. +// +//------------------------------------------------------------------------------ +using Microsoft.SqlServer.TransactSql.ScriptDom; + +namespace Microsoft.SqlServer.TransactSql.ScriptDom.ScriptGenerator +{ + partial class SqlScriptGeneratorVisitor + { + public override void ExplicitVisit(OptimizedLockingDatabaseOption node) + { + System.Diagnostics.Debug.Assert(node.OptionKind == DatabaseOptionKind.OptimizedLocking); + GenerateIdentifier(CodeGenerationSupporter.OptimizedLocking); + GenerateSpace(); + + switch (node.OptionState) + { + case OptionState.Off: + GenerateSymbolAndSpace(TSqlTokenType.EqualsSign); + GenerateKeyword(TSqlTokenType.Off); + break; + case OptionState.On: + GenerateSymbolAndSpace(TSqlTokenType.EqualsSign); + GenerateKeyword(TSqlTokenType.On); + break; + } + } + } +} diff --git a/Test/SqlDom/Baselines170/OptimizedLockingTests170.sql b/Test/SqlDom/Baselines170/OptimizedLockingTests170.sql new file mode 100644 index 0000000..2e0d73d --- /dev/null +++ b/Test/SqlDom/Baselines170/OptimizedLockingTests170.sql @@ -0,0 +1,5 @@ +ALTER DATABASE db + SET OPTIMIZED_LOCKING = ON; + +ALTER DATABASE db + SET OPTIMIZED_LOCKING = OFF; diff --git a/Test/SqlDom/Only170SyntaxTests.cs b/Test/SqlDom/Only170SyntaxTests.cs index 2d5a98c..daa2e91 100644 --- a/Test/SqlDom/Only170SyntaxTests.cs +++ b/Test/SqlDom/Only170SyntaxTests.cs @@ -29,6 +29,7 @@ public partial class SqlDomTests new ParserTest170("CreateEventSessionNotLikePredicate.sql", nErrors80: 2, nErrors90: 1, nErrors100: 1, nErrors110: 1, nErrors120: 1, nErrors130: 0, nErrors140: 0, nErrors150: 0, nErrors160: 0), new ParserTest170("VectorTypeSecondParameterTests170.sql", nErrors80: 2, nErrors90: 2, nErrors100: 2, nErrors110: 2, nErrors120: 2, nErrors130: 2, nErrors140: 2, nErrors150: 2, nErrors160: 2), new ParserTest170("RegexpLikeTests170.sql", nErrors80: 15, nErrors90: 15, nErrors100: 15, nErrors110: 18, nErrors120: 18, nErrors130: 18, nErrors140: 18, nErrors150: 18, nErrors160: 18) + new ParserTest170("OptimizedLockingTests170.sql", nErrors80: 2, nErrors90: 2, nErrors100: 2, nErrors110: 2, nErrors120: 2, nErrors130: 2, nErrors140: 2, nErrors150: 2, nErrors160: 2), }; private static readonly ParserTest[] SqlAzure170_TestInfos = diff --git a/Test/SqlDom/TestScripts/OptimizedLockingTests170.sql b/Test/SqlDom/TestScripts/OptimizedLockingTests170.sql new file mode 100644 index 0000000..aba6b6f --- /dev/null +++ b/Test/SqlDom/TestScripts/OptimizedLockingTests170.sql @@ -0,0 +1,5 @@ +ALTER DATABASE db + SET OPTIMIZED_LOCKING = ON; + +ALTER DATABASE db + SET OPTIMIZED_LOCKING = OFF; \ No newline at end of file From 70b7add655544c0752421c33f727f19610b754b5 Mon Sep 17 00:00:00 2001 From: "GitOps (Git LowPriv)" Date: Thu, 11 Sep 2025 15:42:30 +0000 Subject: [PATCH 05/13] Update .NET SDK to latest patch version (8.0.414) --- global.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/global.json b/global.json index b812260..0f9441a 100644 --- a/global.json +++ b/global.json @@ -1,6 +1,6 @@ { "sdk": { - "version": "8.0.413", + "version": "8.0.414", "rollForward": "latestMajor" }, "msbuild-sdks": { From 2d24d2960df28e2df333fc7982639522fda01cff Mon Sep 17 00:00:00 2001 From: Leila Lali Date: Thu, 11 Sep 2025 16:11:31 +0000 Subject: [PATCH 06/13] Merged PR 1792484: Updating .md files MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Please provide a detailed description, include the link to the design specification or SQL feature document for the new TSQL syntaxes. Make sure to add links to the Github or DevDiv issue Before submitting your pull request, please ensure you have completed the following: - [ ] The [Common checklist](https://msdata.visualstudio.com/SQLToolsAndLibraries/_git/Common?path=/Templates/PR%20Checklist%20for%20SQLToolsAndLibraries.md&version=GBmain&_a=preview) has been reviewed and followed - [ ] Code changes are accompanied by appropriate unit tests - [ ] Identified and included SMEs needed to review code changes - [ ] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=make-the-changes-in) here to make changes in the code - [ ] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=to-extend-the-tests-do-the-following%3A) here to add new tests for your feature - [ ] Update relevant documentation in the [wiki](https://dev.azure.com/msdata/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki) and the README.md file Please provide any additional information that might be helpful for the reviewers Updating .md files ---- Documentation update. This pull request revises and expands the project’s documentation to clarify developer workflows and update build instructions for SqlScriptDOM. - `/github/copilot-instructions.md` now features detailed sections on key points, build/test workflows, and common pitfalls. - A new file, `/github/BUG_FIXING_GUIDE.md`, has been added to provide a step-by-step guide for bug fixing in SqlScriptDOM. - `/README.md` and `/CONTRIBUTING.md` have been updated to use repository-relative paths and improve instructions for modifying the DOM, parser, and script generator. --- CONTRIBUTING.md | 34 ++++++++++++++++++++++++---------- 1 file changed, 24 insertions(+), 10 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dbdf428..f4b000c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -117,23 +117,37 @@ Before sending a Pull Request, please do the following: ### Helpful notes for SQLDOM extensions -1. For changing the DOM classes, modify the XML file (the C# code is generated based on this during the build process) `$(EnlistmentRoot)\Source\SqlDom\SqlScriptDom\Parser\TSql\Ast.xml`. Change Ast.xml to put the class pieces on their appropriate statements. - 1. The build process is defined in `$(EnlistmentRoot)\Source\SqlDom\SqlScriptDom\SqlScriptDom.props` (Target Name="CreateAST") - 2. The generated files are dropped in `$(EnlistmentRoot)\obj\\\SqlScriptDom.csproj\` +1. For changing the DOM classes, modify the XML file (the C# code is generated based on this during the build process) `SqlScriptDom\Parser\TSql\Ast.xml`. Change Ast.xml to put the class pieces on their appropriate statements. + 1. The build process is defined in `SqlScriptDom\GenerateFiles.props` (Target Name="CreateAST") + 2. The generated files are dropped in `obj\SqlScriptDom\AnyCPU\\\Microsoft.SqlServer.TransactSql.ScriptDom.csproj\` + + Regenerating generated sources (what to run and when) + --------------------------------------------------- + When you modify `Source\SqlDom\SqlScriptDom\Parser\TSql\Ast.xml` or any `TSql<#>.g` grammar file, the C# parser and DOM sources are produced by MSBuild generation targets (for example `CreateAST`). These targets are invoked automatically during a normal build, so in most cases you can simply run: + + ```powershell + dotnet build Source\SqlDom\SqlScriptDom\Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug + ``` + + If you only want to run generation targets (no compile) or need more detailed generation logs, invoke the MSBuild targets directly: + + ```powershell + dotnet msbuild Source\SqlDom\SqlScriptDom\Microsoft.SqlServer.TransactSql.ScriptDom.csproj -t:GLexerParserCompile;GSqlTokenTypesCompile;CreateAST -p:Configuration=Debug + ``` + + Generated files are written into the `obj` folder for that project (for example `obj\SqlScriptDom\AnyCPU\\\Microsoft.SqlServer.TransactSql.ScriptDom.csproj\`). If antlr or related tools are missing, see `Directory.Build.props` for `AntlrLocation` and follow the repo guidance to supply the binaries. 2. For changing the parser, modify the .g file here: -`$(EnlistmentRoot)\Source\SqlDom\SqlScriptDom\Parser\TSql\TSql<#>.g` where # is the version (ie - 100, 120, 130). This will usually be the latest number if adding new grammar. Change the Tsql(xxx).g file to parse the new syntax. - 1. The build process is defined in `$(EnlistmentRoot)\Source\SqlDom\SqlScriptDom\SqlScriptDom.props` (Target Name="CreateAST") - 2. The generated files are dropped in `$(EnlistmentRoot)\obj\x86|x64\Debug|Release\sqlscriptdom.csproj\` +`SqlScriptDom\Parser\TSql\TSql<#>.g` where # is the version (ie - 100, 120, 130). This will usually be the latest number if adding new grammar. Change the Tsql(xxx).g file to parse the new syntax. + 1. The build process is defined in `SqlScriptDom\GenerateFiles.props` (Target Name="CreateAST") + 2. The generated files are dropped in `obj\SqlScriptDom\AnyCPU\\\Microsoft.SqlServer.TransactSql.ScriptDom.csproj\` -3. For changing the ScriptGenerator, modify the appropriate file (i.e. Visitor that accepts the modified DOM class) in here: `$(EnlistmentRoot)\Source\SqlDom\SqlScriptDom\ScriptDom\SqlServer\ScriptGenerator`. - 1. To add a new ScriptGenerator, you need to add the file to `$(EnlistmentRoot)\Source\SqlDom\SqlScriptDom\SqlScriptDom.props` +3. For changing the ScriptGenerator, modify the appropriate file (i.e. Visitor that accepts the modified DOM class) in here: `SqlScriptDom\ScriptDom\SqlServer\ScriptGenerator`. 1. Change The visitors SqlScriptGenerator.X to use the new piece from AST.XML 1. If you're adding syntax that's Azure-only or Standalone-only, implement appropriate Versioning Visitor for your constructs. -4. When adding/removing new files please add/remove an entry to/from `$(EnlistmentRoot)\Source\SqlDom\SqlScriptDom\SqlScriptDom.csproj` -5. To extend the tests do the following: +4. To extend the tests do the following: 1. Baselines# needs to be updated or added with the appropriate .sql file as expected results. 1. The Only#SyntaxTests.cs needs to be extended or added to specify the appropriate TestScripts script. 1. Positive tests go in Only#SyntaxTests.cs if adding new grammar. From 1e014f50158c2ae9b2097a4cba57df75d7771d00 Mon Sep 17 00:00:00 2001 From: Sicong Liu Date: Wed, 17 Sep 2025 19:31:53 +0000 Subject: [PATCH 07/13] Merged PR 1804708: [ai_generate_embeddings] PARAMETERS without parentheses This PR fixes a bug for AI_GENERATE_EMBEDDINGS when PARAMETERS comes without parentheses "SELECT AI_GENERATE_EMBEDDINGS(large_text USE MODEL MyAzureOpenAIModel PARAMETERS '{"dimensions" : 768 }')" Current behavior only allows PARAMETERS with parentheses. This PR adds syntax change along with tests. - [X] The [Common checklist](https://msdata.visualstudio.com/SQLToolsAndLibraries/_git/Common?path=/Templates/PR%20Checklist%20for%20SQLToolsAndLibraries.md&version=GBmain&_a=preview) has been reviewed and followed - [X] Code changes are accompanied by appropriate unit tests - [X] Identified and included SMEs needed to review code changes - [X] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=make-the-changes-in) here to make changes in the code - [X] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=to-extend-the-tests-do-the-following%3A) here to add new tests for your feature - [ ] Update relevant documentation in the [wiki](https://dev.azure.com/msdata/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki) and the README.md file Please provide any additional information that might be helpful for the reviewers First commit ---- Bug fix to correctly handle the ScriptDom PARAMETERS clause without parentheses. This pull request fixes the handling of the PARAMETERS clause in the AI_GENERATE_EMBEDDINGS function by conditionally emitting parentheses only when needed. - `SqlScriptGeneratorVisitor.AiGenerateEmbeddingsFunction.cs`: Modified the code to check if OptionalParameters is a StringLiteral and, if so, emit it without surrounding parentheses. - `TSql170.g`: Updated the grammar rules for aiGenerateEmbeddingsFunctionCall to accept either a parenthesized expression or a direct string literal. - Test files: Adjusted baseline and test scripts, including error expectations, to validate the new syntax support. Related work items: #4692491 --- SqlScriptDom/Parser/TSql/TSql170.g | 18 +++++++++++++----- ...torVisitor.AiGenerateEmbeddingsFunction.cs | 19 ++++++++++++++++--- .../AiGenerateEmbeddingsTests170.sql | 1 + Test/SqlDom/Only170SyntaxTests.cs | 2 +- .../AiGenerateEmbeddingsTests170.sql | 1 + 5 files changed, 32 insertions(+), 9 deletions(-) diff --git a/SqlScriptDom/Parser/TSql/TSql170.g b/SqlScriptDom/Parser/TSql/TSql170.g index af4b8a9..4addf6f 100644 --- a/SqlScriptDom/Parser/TSql/TSql170.g +++ b/SqlScriptDom/Parser/TSql/TSql170.g @@ -32153,6 +32153,7 @@ aiGenerateEmbeddingsFunctionCall ScalarExpression vInput; SchemaObjectName vModelName; ScalarExpression vParams = null; + StringLiteral vParamNode = null; } : tFunc:Identifier LeftParenthesis @@ -32164,8 +32165,7 @@ aiGenerateEmbeddingsFunctionCall { vResult.Input = vInput; } - - tUse:Use // your reserved keyword + tUse:Use { UpdateTokenInfo(vResult, tUse); } @@ -32184,10 +32184,18 @@ aiGenerateEmbeddingsFunctionCall tParams:Identifier { Match(tParams, CodeGenerationSupporter.Parameters); + UpdateTokenInfo(vResult, tParams); } - LeftParenthesis - vParams=expression - RightParenthesis + ( + LeftParenthesis + vParams=expression + RightParenthesis + | + vParamNode=stringLiteral + { + vParams = vParamNode; + } + ) { vResult.OptionalParameters = vParams; } diff --git a/SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.AiGenerateEmbeddingsFunction.cs b/SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.AiGenerateEmbeddingsFunction.cs index 0efdc60..30fbff1 100644 --- a/SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.AiGenerateEmbeddingsFunction.cs +++ b/SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.AiGenerateEmbeddingsFunction.cs @@ -36,9 +36,22 @@ public override void ExplicitVisit(AIGenerateEmbeddingsFunctionCall node) { GenerateSpace(); GenerateIdentifierWithoutCasing(CodeGenerationSupporter.Parameters); - GenerateSpaceAndSymbol(TSqlTokenType.LeftParenthesis); - GenerateFragmentIfNotNull(node.OptionalParameters); - GenerateSymbol(TSqlTokenType.RightParenthesis); + + // If OptionalParameters is a StringLiteral, emit without parentheses: + // PARAMETERS '' + // Otherwise, preserve the legacy form: + // PARAMETERS () + if (node.OptionalParameters is StringLiteral) + { + GenerateSpace(); + GenerateFragmentIfNotNull(node.OptionalParameters); + } + else + { + GenerateSpaceAndSymbol(TSqlTokenType.LeftParenthesis); + GenerateFragmentIfNotNull(node.OptionalParameters); + GenerateSymbol(TSqlTokenType.RightParenthesis); + } } // Emit closing parenthesis diff --git a/Test/SqlDom/Baselines170/AiGenerateEmbeddingsTests170.sql b/Test/SqlDom/Baselines170/AiGenerateEmbeddingsTests170.sql index 72493c7..4ff723b 100644 --- a/Test/SqlDom/Baselines170/AiGenerateEmbeddingsTests170.sql +++ b/Test/SqlDom/Baselines170/AiGenerateEmbeddingsTests170.sql @@ -3,6 +3,7 @@ SELECT AI_GENERATE_EMBEDDINGS(N'My Default Input Text' USE MODEL MyDefaultModel) SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL MyDefaultModel PARAMETERS (TRY_CONVERT (JSON, N'{}'))); SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL dbo.MyDefaultModel); SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL MyDatabase.dbo.MyDefaultModel); +SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL MyDefaultModel PARAMETERS '{"dimensions" : 768 }'); GO CREATE FUNCTION dbo.AI_GENERATE_EMBEDDINGS diff --git a/Test/SqlDom/Only170SyntaxTests.cs b/Test/SqlDom/Only170SyntaxTests.cs index daa2e91..9dc8976 100644 --- a/Test/SqlDom/Only170SyntaxTests.cs +++ b/Test/SqlDom/Only170SyntaxTests.cs @@ -20,7 +20,7 @@ public partial class SqlDomTests new ParserTest170("RegexpTests170.sql", nErrors80: 0, nErrors90: 0, nErrors100: 0, nErrors110: 0, nErrors120: 0, nErrors130: 0, nErrors140: 0, nErrors150: 0, nErrors160: 0), new ParserTest170("AiGenerateChunksTests170.sql", nErrors80: 19, nErrors90: 16, nErrors100: 15, nErrors110: 15, nErrors120: 15, nErrors130: 15, nErrors140: 15, nErrors150: 15, nErrors160: 15), new ParserTest170("JsonFunctionTests170.sql", nErrors80: 13, nErrors90: 8, nErrors100: 38, nErrors110: 38, nErrors120: 38, nErrors130: 38, nErrors140: 38, nErrors150: 38, nErrors160: 38), - new ParserTest170("AiGenerateEmbeddingsTests170.sql", nErrors80: 12, nErrors90: 9, nErrors100: 9, nErrors110: 9, nErrors120: 9, nErrors130: 9, nErrors140: 9, nErrors150: 9, nErrors160: 9), + new ParserTest170("AiGenerateEmbeddingsTests170.sql", nErrors80: 13, nErrors90: 10, nErrors100: 10, nErrors110: 10, nErrors120: 10, nErrors130: 10, nErrors140: 10, nErrors150: 10, nErrors160: 10), new ParserTest170("CreateExternalModelStatementTests170.sql", nErrors80: 2, nErrors90: 2, nErrors100: 2, nErrors110: 2, nErrors120: 2, nErrors130: 4, nErrors140: 4, nErrors150: 4, nErrors160: 4), new ParserTest170("AlterExternalModelStatementTests170.sql", nErrors80: 2, nErrors90: 2, nErrors100: 2, nErrors110: 2, nErrors120: 2, nErrors130: 5, nErrors140: 5, nErrors150: 5, nErrors160: 5), new ParserTest170("DropExternalModelStatementTests170.sql", nErrors80: 1, nErrors90: 1, nErrors100: 1, nErrors110: 1, nErrors120: 1, nErrors130: 1, nErrors140: 1, nErrors150: 1, nErrors160: 1), diff --git a/Test/SqlDom/TestScripts/AiGenerateEmbeddingsTests170.sql b/Test/SqlDom/TestScripts/AiGenerateEmbeddingsTests170.sql index 72493c7..4ff723b 100644 --- a/Test/SqlDom/TestScripts/AiGenerateEmbeddingsTests170.sql +++ b/Test/SqlDom/TestScripts/AiGenerateEmbeddingsTests170.sql @@ -3,6 +3,7 @@ SELECT AI_GENERATE_EMBEDDINGS(N'My Default Input Text' USE MODEL MyDefaultModel) SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL MyDefaultModel PARAMETERS (TRY_CONVERT (JSON, N'{}'))); SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL dbo.MyDefaultModel); SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL MyDatabase.dbo.MyDefaultModel); +SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL MyDefaultModel PARAMETERS '{"dimensions" : 768 }'); GO CREATE FUNCTION dbo.AI_GENERATE_EMBEDDINGS From 5caf450d022dc8ce411d823bfb7b423672b27e10 Mon Sep 17 00:00:00 2001 From: Sicong Liu Date: Wed, 24 Sep 2025 21:16:37 +0000 Subject: [PATCH 08/13] Merged PR 1808102: [AI_GENERATE_EMBEDDINGS] Additional Syntax Change MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit # Pull Request Template for ScriptDom ## Description Please provide a detailed description, include the link to the design specification or SQL feature document for the new TSQL syntaxes. Make sure to add links to the Github or DevDiv issue This PR updates the `aiGenerateEmbeddingsFunctionCall` grammar and the script generator to: - **Constrain model names** after `USE MODEL` to a **single token** (either an `Identifier` or one `QuotedIdentifier` like `[dbo.MyDefaultModel]`). Multi-part names (e.g., `dbo.MyDefaultModel`, `[dbo].[MyDefaultModel]`, `MyDb.dbo.Model`) now **fail at parse time**. - **Refine `PARAMETERS`** parsing to accept an **expression** with or without parentheses, but **reject** a **bare JSON string literal** (e.g., `PARAMETERS '{"a":1}'`). This is because the optional PARAMETERS can only be a MSJSON not varchar/nvarchar. ## What changed ### Grammar (`TSql170.g`) - **Model name**: - Replaced `schemaObjectThreePartName` with a **single-part** alternative: - `Identifier` **or** `QuotedIdentifier` (one token). - This allows `[dbo.MyDefaultModel]` (single token), but rejects dotted multi-part forms. - **PARAMETERS**: - Accept as **keyword** *or* **identifier** matched to `"Parameters"`. - Value can be either: - `PARAMETERS ( )` → wraps `` in a `ParenthesisExpression` to preserve parens. - `PARAMETERS ` → **guarded** to **forbid** a leading ASCII/Unicode **string literal** (blocks `PARAMETERS '{...}'`). - No support added for `=` (kept “same as before” by design). ### Script generator (`SqlScriptGeneratorVisitor.AIGenerateEmbeddingsFunction.cs`) - **Do not force parentheses** for `OptionalParameters`. Emit: - `PARAMETERS {space} ` - If the AST contains a `ParenthesisExpression`, it prints its own `(`…`)`. Bare expressions print without parens. Before submitting your pull request, please ensure you have completed the following: ## Code Change - [X] The [Common checklist](https://msdata.visualstudio.com/SQLToolsAndLibraries/_git/Common?path=/Templates/PR%20Checklist%20for%20SQLToolsAndLibraries.md&version=GBmain&_a=preview) has been reviewed and followed - [X] Code changes are accompanied by appropriate unit tests - [X] Identified and included SMEs needed to review code changes - [X] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=make-the-changes-in) here to make changes in the code ## Testing - [X] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=to-extend-the-tests-do-the-following%3A) here to add new tests for your feature ## Documentation - [ ] Update relevant documentation in the [wiki](https://dev.azure.com/msdata/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki) and the README.md file ## Additional Information Please ... --- SqlScriptDom/Parser/TSql/TSql170.g | 102 +++++++++++++++--- ...torVisitor.AiGenerateEmbeddingsFunction.cs | 18 +--- .../AiGenerateEmbeddingsTests170.sql | 9 +- Test/SqlDom/Only170SyntaxTests.cs | 2 +- Test/SqlDom/ParserErrorsTests.cs | 20 +++- .../AiGenerateEmbeddingsTests170.sql | 9 +- 6 files changed, 113 insertions(+), 47 deletions(-) diff --git a/SqlScriptDom/Parser/TSql/TSql170.g b/SqlScriptDom/Parser/TSql/TSql170.g index 4addf6f..27baeee 100644 --- a/SqlScriptDom/Parser/TSql/TSql170.g +++ b/SqlScriptDom/Parser/TSql/TSql170.g @@ -32153,7 +32153,7 @@ aiGenerateEmbeddingsFunctionCall ScalarExpression vInput; SchemaObjectName vModelName; ScalarExpression vParams = null; - StringLiteral vParamNode = null; + ScalarExpression vParamsInner; } : tFunc:Identifier LeftParenthesis @@ -32173,29 +32173,97 @@ aiGenerateEmbeddingsFunctionCall tModel:Identifier { Match(tModel, CodeGenerationSupporter.Model); - } - - vModelName=schemaObjectThreePartName - { - vResult.ModelName = vModelName; - } - - ( - tParams:Identifier + UpdateTokenInfo(vResult, tModel); + } + + // --- MODEL NAME: single-part only (strict) --------------------------------------------- + // We accept exactly **one identifier token** after "USE MODEL". + // + // Why: + // - Users may store model names that *visually* contain dots or spaces, e.g. [dbo.MyDefaultModel]. + // When bracket-delimited, the lexer emits this as a **single** token (QuotedIdentifier), so it’s OK. + // - True multipart names (db.schema.model) must be rejected here, so we do NOT consume any Dot tokens. + // + // Allowed (single token): + // USE MODEL MyDefaultModel -- Identifier + // USE MODEL [dbo.MyDefaultModel] -- QuotedIdentifier (one token; dot lives inside the brackets) + // + // Rejected (multipart): + // USE MODEL dbo.MyDefaultModel -- Identifier '.' Identifier (two tokens + Dot) + // USE MODEL [dbo].[MyDefaultModel] -- QuotedIdentifier '.' QuotedIdentifier + // + // Token notes: + // - Identifier : unquoted identifier; cannot contain spaces or '.'. + // - QuotedIdentifier : bracket-delimited; may contain spaces and '.' inside the brackets. + ( + vModelId:Identifier + { + vModelName = this.FragmentFactory.CreateFragment(); + vModelName.Identifiers.Add(this.CreateIdentifierFromToken(vModelId)); + vResult.ModelName = vModelName; + } + | + vModelQId:QuotedIdentifier { - Match(tParams, CodeGenerationSupporter.Parameters); - UpdateTokenInfo(vResult, tParams); + vModelName = this.FragmentFactory.CreateFragment(); + vModelName.Identifiers.Add(this.CreateIdentifierFromToken(vModelQId)); + vResult.ModelName = vModelName; } + ) + + // --- Optional PARAMETERS clause --------------------------------------------------------- + // Shape: [PARAMETERS ()] | [PARAMETERS ] + // Goals: + // 1) Accept a general **expression** as the PARAMETERS value. + // 2) Preserve user-written parentheses by constructing a ParenthesisExpression node + // for the "( )" variant so pretty-printing round-trips exactly. + // 3) **Reject** a bare JSON string literal (e.g., PARAMETERS '{...}'); callers must pass + // a parsed JSON expression (e.g., TRY_CONVERT(JSON, N'{}')). + // Notes: + // - Some builds tokenize PARAMETERS as a keyword; others as an Identifier. Support both. + ( ( - LeftParenthesis - vParams=expression - RightParenthesis + // PARAMETERS as a true keyword token. + tParamsKw:Parameters + { + UpdateTokenInfo(vResult, tParamsKw); + } | - vParamNode=stringLiteral + // PARAMETERS as an identifier token; enforce its text equals "Parameters". + tParams:Identifier { - vParams = vParamNode; + Match(tParams, CodeGenerationSupporter.Parameters); + UpdateTokenInfo(vResult, tParams); } ) + + // ---- Value of PARAMETERS ----------------------------------------------------------- + ( + // Variant A: user wrote parentheses around the value: PARAMETERS ( ) + // Build a ParenthesisExpression so the printer re-emits parens. + tLP:LeftParenthesis + vParamsInner=expression + tRP:RightParenthesis + { + ParenthesisExpression p = this.FragmentFactory.CreateFragment(); + p.Expression = vParamsInner; + vParams = p; + + // Attach LP/RP token info for accurate script generation. + UpdateTokenInfo(p, tLP); + UpdateTokenInfo(p, tRP); + } + | + // Variant B: bare expression without surrounding parentheses. + // Guardrail: Disallow a leading string literal so that + // PARAMETERS '{"dimensions":768}' + // is a **syntax error**, while + // PARAMETERS TRY_CONVERT(JSON, N'{}') + // is allowed. + // LA(1) is the next token type; block both ASCII ('...') and Unicode (N'...') strings. + { LA(1) != AsciiStringLiteral && LA(1) != UnicodeStringLiteral }? + vParams=expression + ) { vResult.OptionalParameters = vParams; } diff --git a/SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.AiGenerateEmbeddingsFunction.cs b/SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.AiGenerateEmbeddingsFunction.cs index 30fbff1..299a1c3 100644 --- a/SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.AiGenerateEmbeddingsFunction.cs +++ b/SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.AiGenerateEmbeddingsFunction.cs @@ -36,22 +36,8 @@ public override void ExplicitVisit(AIGenerateEmbeddingsFunctionCall node) { GenerateSpace(); GenerateIdentifierWithoutCasing(CodeGenerationSupporter.Parameters); - - // If OptionalParameters is a StringLiteral, emit without parentheses: - // PARAMETERS '' - // Otherwise, preserve the legacy form: - // PARAMETERS () - if (node.OptionalParameters is StringLiteral) - { - GenerateSpace(); - GenerateFragmentIfNotNull(node.OptionalParameters); - } - else - { - GenerateSpaceAndSymbol(TSqlTokenType.LeftParenthesis); - GenerateFragmentIfNotNull(node.OptionalParameters); - GenerateSymbol(TSqlTokenType.RightParenthesis); - } + GenerateSpace(); + GenerateFragmentIfNotNull(node.OptionalParameters); } // Emit closing parenthesis diff --git a/Test/SqlDom/Baselines170/AiGenerateEmbeddingsTests170.sql b/Test/SqlDom/Baselines170/AiGenerateEmbeddingsTests170.sql index 4ff723b..1728248 100644 --- a/Test/SqlDom/Baselines170/AiGenerateEmbeddingsTests170.sql +++ b/Test/SqlDom/Baselines170/AiGenerateEmbeddingsTests170.sql @@ -1,9 +1,10 @@ SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL MyDefaultModel); SELECT AI_GENERATE_EMBEDDINGS(N'My Default Input Text' USE MODEL MyDefaultModel); SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL MyDefaultModel PARAMETERS (TRY_CONVERT (JSON, N'{}'))); -SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL dbo.MyDefaultModel); -SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL MyDatabase.dbo.MyDefaultModel); -SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL MyDefaultModel PARAMETERS '{"dimensions" : 768 }'); +SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL MyDefaultModel PARAMETERS TRY_CONVERT (JSON, N'{}')); +SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL [My Default Model]); +SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL [dbo.MyDefaultModel]); +SELECT AI_GENERATE_EMBEDDINGS('' USE MODEL MyDefaultModel); GO CREATE FUNCTION dbo.AI_GENERATE_EMBEDDINGS @@ -36,7 +37,7 @@ GO CREATE VIEW dbo.MyEmbeddingView AS -SELECT AI_GENERATE_EMBEDDINGS(N'View Input' USE MODEL dbo.MyDefaultModel) AS EmbeddingResult; +SELECT AI_GENERATE_EMBEDDINGS(N'View Input' USE MODEL MyDefaultModel) AS EmbeddingResult; GO CREATE TABLE dbo.SimpleEmbeddingTable ( diff --git a/Test/SqlDom/Only170SyntaxTests.cs b/Test/SqlDom/Only170SyntaxTests.cs index 9dc8976..bc7676b 100644 --- a/Test/SqlDom/Only170SyntaxTests.cs +++ b/Test/SqlDom/Only170SyntaxTests.cs @@ -20,7 +20,7 @@ public partial class SqlDomTests new ParserTest170("RegexpTests170.sql", nErrors80: 0, nErrors90: 0, nErrors100: 0, nErrors110: 0, nErrors120: 0, nErrors130: 0, nErrors140: 0, nErrors150: 0, nErrors160: 0), new ParserTest170("AiGenerateChunksTests170.sql", nErrors80: 19, nErrors90: 16, nErrors100: 15, nErrors110: 15, nErrors120: 15, nErrors130: 15, nErrors140: 15, nErrors150: 15, nErrors160: 15), new ParserTest170("JsonFunctionTests170.sql", nErrors80: 13, nErrors90: 8, nErrors100: 38, nErrors110: 38, nErrors120: 38, nErrors130: 38, nErrors140: 38, nErrors150: 38, nErrors160: 38), - new ParserTest170("AiGenerateEmbeddingsTests170.sql", nErrors80: 13, nErrors90: 10, nErrors100: 10, nErrors110: 10, nErrors120: 10, nErrors130: 10, nErrors140: 10, nErrors150: 10, nErrors160: 10), + new ParserTest170("AiGenerateEmbeddingsTests170.sql", nErrors80: 14, nErrors90: 11, nErrors100: 11, nErrors110: 11, nErrors120: 11, nErrors130: 11, nErrors140: 11, nErrors150: 11, nErrors160: 11), new ParserTest170("CreateExternalModelStatementTests170.sql", nErrors80: 2, nErrors90: 2, nErrors100: 2, nErrors110: 2, nErrors120: 2, nErrors130: 4, nErrors140: 4, nErrors150: 4, nErrors160: 4), new ParserTest170("AlterExternalModelStatementTests170.sql", nErrors80: 2, nErrors90: 2, nErrors100: 2, nErrors110: 2, nErrors120: 2, nErrors130: 5, nErrors140: 5, nErrors150: 5, nErrors160: 5), new ParserTest170("DropExternalModelStatementTests170.sql", nErrors80: 1, nErrors90: 1, nErrors100: 1, nErrors110: 1, nErrors120: 1, nErrors130: 1, nErrors140: 1, nErrors150: 1, nErrors160: 1), diff --git a/Test/SqlDom/ParserErrorsTests.cs b/Test/SqlDom/ParserErrorsTests.cs index b793a14..3644819 100644 --- a/Test/SqlDom/ParserErrorsTests.cs +++ b/Test/SqlDom/ParserErrorsTests.cs @@ -7315,11 +7315,6 @@ public void GenerateEmbeddingsNegativeTest170() "SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' PARAMETERS (TRY_CONVERT(JSON, N'{}')))", new ParserErrorInfo(54, "SQL46010", "PARAMETERS")); - // PARAMETERS missing parentheses - ParserTestUtils.ErrorTest170( - "SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL MyModel PARAMETERS TRY_CONVERT(JSON, N'{}'))", - new ParserErrorInfo(83, "SQL46010", "TRY_CONVERT")); - // Invalid expression inside PARAMETERS (missing closing parenthesis) ParserTestUtils.ErrorTest170( "SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL MyModel PARAMETERS (TRY_CONVERT(JSON, N'{}')", @@ -7344,6 +7339,21 @@ public void GenerateEmbeddingsNegativeTest170() ParserTestUtils.ErrorTest170( "SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL NULL)", new ParserErrorInfo(64, "SQL46010", "NULL")); + + // Invalid model reference: schema-qualified name after USE MODEL + ParserTestUtils.ErrorTest170( + "SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL [dbo].[MyDefaultModel])", + new ParserErrorInfo(69, "SQL46010", ".")); + + // Invalid model reference: database + schema-qualified name after USE MODEL + ParserTestUtils.ErrorTest170( + "SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL [MyDatabase].[dbo].[MyDefaultModel])", + new ParserErrorInfo(76, "SQL46010", ".")); + + // Varchar Optional Parameters + ParserTestUtils.ErrorTest170( + "SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL MyDefaultModel PARAMETERS '{\"dimensions\" : 768 }')", + new ParserErrorInfo(90, "SQL46010", "'{\"dimensions\" : 768 }'")); } diff --git a/Test/SqlDom/TestScripts/AiGenerateEmbeddingsTests170.sql b/Test/SqlDom/TestScripts/AiGenerateEmbeddingsTests170.sql index 4ff723b..1728248 100644 --- a/Test/SqlDom/TestScripts/AiGenerateEmbeddingsTests170.sql +++ b/Test/SqlDom/TestScripts/AiGenerateEmbeddingsTests170.sql @@ -1,9 +1,10 @@ SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL MyDefaultModel); SELECT AI_GENERATE_EMBEDDINGS(N'My Default Input Text' USE MODEL MyDefaultModel); SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL MyDefaultModel PARAMETERS (TRY_CONVERT (JSON, N'{}'))); -SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL dbo.MyDefaultModel); -SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL MyDatabase.dbo.MyDefaultModel); -SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL MyDefaultModel PARAMETERS '{"dimensions" : 768 }'); +SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL MyDefaultModel PARAMETERS TRY_CONVERT (JSON, N'{}')); +SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL [My Default Model]); +SELECT AI_GENERATE_EMBEDDINGS('My Default Input Text' USE MODEL [dbo.MyDefaultModel]); +SELECT AI_GENERATE_EMBEDDINGS('' USE MODEL MyDefaultModel); GO CREATE FUNCTION dbo.AI_GENERATE_EMBEDDINGS @@ -36,7 +37,7 @@ GO CREATE VIEW dbo.MyEmbeddingView AS -SELECT AI_GENERATE_EMBEDDINGS(N'View Input' USE MODEL dbo.MyDefaultModel) AS EmbeddingResult; +SELECT AI_GENERATE_EMBEDDINGS(N'View Input' USE MODEL MyDefaultModel) AS EmbeddingResult; GO CREATE TABLE dbo.SimpleEmbeddingTable ( From 2d459134cad6f15e4fa9501f7cfa093d86bf6073 Mon Sep 17 00:00:00 2001 From: Lijun Ji Date: Thu, 25 Sep 2025 16:35:11 +0000 Subject: [PATCH 09/13] Merged PR 1814233: add OPTIMIZE_FOR_ARRAY_SEARCH option for Json index # Pull Request Template for ScriptDom ## Description Please provide a detailed description, include the link to the design specification or SQL feature document for the new TSQL syntaxes. Make sure to add links to the Github or DevDiv issue Before submitting your pull request, please ensure you have completed the following: ## Code Change - [ ] The [Common checklist](https://msdata.visualstudio.com/SQLToolsAndLibraries/_git/Common?path=/Templates/PR%20Checklist%20for%20SQLToolsAndLibraries.md&version=GBmain&_a=preview) has been reviewed and followed - [ ] Code changes are accompanied by appropriate unit tests - [ ] Identified and included SMEs needed to review code changes - [ ] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=make-the-changes-in) here to make changes in the code ## Testing - [ ] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=to-extend-the-tests-do-the-following%3A) here to add new tests for your feature ## Documentation - [ ] Update relevant documentation in the [wiki](https://dev.azure.com/msdata/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki) and the README.md file ## Additional Information Please provide any additional information that might be helpful for the reviewers This PR added OPTIMIZE_FOR_ARRAY_SEARCH option for Json index. A test case is added. ---- #### AI description (iteration 1) #### PR Classification This PR introduces a new feature by adding support for the OPTIMIZE_FOR_ARRAY_SEARCH option in JSON indexes. #### PR Summary The changes enable the OPTIMIZE_FOR_ARRAY_SEARCH option in ScriptDom by updating parser mappings, constants, and test cases to validate the new JSON index syntax. - `SqlScriptDom/Parser/TSql/IndexOptionKind.cs`: Added a new enum value for OptimizeForArraySearch. - `SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs`: Introduced a new constant for OPTIMIZE_FOR_ARRAY_SEARCH. - `SqlScriptDom/Parser/TSql/IndexOptionHelper.cs`: Mapped the new index option to support the updated feature. - Test files (`Test/SqlDom/TestScripts/JsonIndexTests170.sql`, `Test/SqlDom/Baselines170/JsonIndexTests170.sql`, `Test/SqlDom/Only170SyntaxTests.cs`): Updated to include and validate the new JSON index option. Related work items: #4709838 --- SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs | 1 + SqlScriptDom/Parser/TSql/IndexOptionHelper.cs | 2 ++ SqlScriptDom/Parser/TSql/IndexOptionKind.cs | 1 + Test/SqlDom/Baselines170/JsonIndexTests170.sql | 5 ++++- Test/SqlDom/Only170SyntaxTests.cs | 2 +- Test/SqlDom/TestScripts/JsonIndexTests170.sql | 6 +++++- 6 files changed, 14 insertions(+), 3 deletions(-) diff --git a/SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs b/SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs index 590d390..0a64b28 100644 --- a/SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs +++ b/SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs @@ -721,6 +721,7 @@ internal static class CodeGenerationSupporter internal const string Optimistic = "OPTIMISTIC"; internal const string Optimize = "OPTIMIZE"; internal const string OptimizedLocking = "OPTIMIZED_LOCKING"; + internal const string OptimizeForArraySearch = "OPTIMIZE_FOR_ARRAY_SEARCH"; internal const string OptimizeForSequentialKey = "OPTIMIZE_FOR_SEQUENTIAL_KEY"; internal const string OptimizerQueue = "OPTIMIZER_QUEUE"; internal const string Order = "ORDER"; diff --git a/SqlScriptDom/Parser/TSql/IndexOptionHelper.cs b/SqlScriptDom/Parser/TSql/IndexOptionHelper.cs index 1a0545f..4a739a1 100644 --- a/SqlScriptDom/Parser/TSql/IndexOptionHelper.cs +++ b/SqlScriptDom/Parser/TSql/IndexOptionHelper.cs @@ -46,6 +46,8 @@ private IndexOptionHelper() AddOptionMapping(IndexOptionKind.VectorMetric, CodeGenerationSupporter.Metric, SqlVersionFlags.TSql170AndAbove); AddOptionMapping(IndexOptionKind.VectorType, CodeGenerationSupporter.Type, SqlVersionFlags.TSql170AndAbove); + AddOptionMapping(IndexOptionKind.OptimizeForArraySearch, CodeGenerationSupporter.OptimizeForArraySearch, SqlVersionFlags.TSql170AndAbove); + } internal static readonly IndexOptionHelper Instance = new IndexOptionHelper(); diff --git a/SqlScriptDom/Parser/TSql/IndexOptionKind.cs b/SqlScriptDom/Parser/TSql/IndexOptionKind.cs index 7f2914a..ae69cb5 100644 --- a/SqlScriptDom/Parser/TSql/IndexOptionKind.cs +++ b/SqlScriptDom/Parser/TSql/IndexOptionKind.cs @@ -40,6 +40,7 @@ public enum IndexOptionKind XmlCompression = 23, VectorMetric = 24, VectorType = 25, + OptimizeForArraySearch = 26, } #pragma warning restore 1591 diff --git a/Test/SqlDom/Baselines170/JsonIndexTests170.sql b/Test/SqlDom/Baselines170/JsonIndexTests170.sql index f6010de..ff1b9d8 100644 --- a/Test/SqlDom/Baselines170/JsonIndexTests170.sql +++ b/Test/SqlDom/Baselines170/JsonIndexTests170.sql @@ -26,4 +26,7 @@ CREATE JSON INDEX [IX JSON Index] CREATE JSON INDEX IX_JSON_Complex ON dbo.Documents (Content) - FOR ('$.metadata.title', '$.content.sections[*].text', '$.tags[*]'); \ No newline at end of file + FOR ('$.metadata.title', '$.content.sections[*].text', '$.tags[*]'); + +CREATE JSON INDEX IX_JSON + ON dbo.Users (JsonData) WITH (OPTIMIZE_FOR_ARRAY_SEARCH = ON); \ No newline at end of file diff --git a/Test/SqlDom/Only170SyntaxTests.cs b/Test/SqlDom/Only170SyntaxTests.cs index bc7676b..bfb7b29 100644 --- a/Test/SqlDom/Only170SyntaxTests.cs +++ b/Test/SqlDom/Only170SyntaxTests.cs @@ -13,7 +13,7 @@ public partial class SqlDomTests private static readonly ParserTest[] Only170TestInfos = { new ParserTest170("RegexpTVFTests170.sql", nErrors80: 1, nErrors90: 1, nErrors100: 0, nErrors110: 0, nErrors120: 0, nErrors130: 0, nErrors140: 0, nErrors150: 0, nErrors160: 0), - new ParserTest170("JsonIndexTests170.sql", nErrors80: 2, nErrors90: 8, nErrors100: 8, nErrors110: 8, nErrors120: 8, nErrors130: 8, nErrors140: 8, nErrors150: 8, nErrors160: 8), + new ParserTest170("JsonIndexTests170.sql", nErrors80: 2, nErrors90: 9, nErrors100: 9, nErrors110: 9, nErrors120: 9, nErrors130: 9, nErrors140: 9, nErrors150: 9, nErrors160: 9), new ParserTest170("VectorIndexTests170.sql", nErrors80: 2, nErrors90: 12, nErrors100: 12, nErrors110: 12, nErrors120: 12, nErrors130: 12, nErrors140: 12, nErrors150: 12, nErrors160: 12), new ParserTest170("AlterDatabaseManualCutoverTests170.sql", nErrors80: 4, nErrors90: 4, nErrors100: 4, nErrors110: 4, nErrors120: 4, nErrors130: 4, nErrors140: 4, nErrors150: 4, nErrors160: 4), new ParserTest170("CreateColumnStoreIndexTests170.sql", nErrors80: 3, nErrors90: 3, nErrors100: 3, nErrors110: 3, nErrors120: 3, nErrors130: 0, nErrors140: 0, nErrors150: 0, nErrors160: 0), diff --git a/Test/SqlDom/TestScripts/JsonIndexTests170.sql b/Test/SqlDom/TestScripts/JsonIndexTests170.sql index 2a0f1ad..0c2aa8f 100644 --- a/Test/SqlDom/TestScripts/JsonIndexTests170.sql +++ b/Test/SqlDom/TestScripts/JsonIndexTests170.sql @@ -28,4 +28,8 @@ FOR ('$.data.attributes'); -- JSON index with complex path expressions CREATE JSON INDEX IX_JSON_Complex ON dbo.Documents (Content) -FOR ('$.metadata.title', '$.content.sections[*].text', '$.tags[*]'); \ No newline at end of file +FOR ('$.metadata.title', '$.content.sections[*].text', '$.tags[*]'); + +-- Create JSON index with OPTIMIZE_FOR_ARRAY_SEARCH option +CREATE JSON INDEX IX_JSON ON dbo.Users (JsonData) +WITH (OPTIMIZE_FOR_ARRAY_SEARCH = ON); \ No newline at end of file From 0d436a019207b41bf9f36fe531bf74fe7b39b1d4 Mon Sep 17 00:00:00 2001 From: Lijun Ji Date: Thu, 25 Sep 2025 20:42:57 +0000 Subject: [PATCH 10/13] Merged PR 1815158: Add test case for PAD_INDEX and DROP_EXISTING options for Json index # Pull Request Template for ScriptDom ## Description Please provide a detailed description, include the link to the design specification or SQL feature document for the new TSQL syntaxes. Make sure to add links to the Github or DevDiv issue Before submitting your pull request, please ensure you have completed the following: ## Code Change - [ ] The [Common checklist](https://msdata.visualstudio.com/SQLToolsAndLibraries/_git/Common?path=/Templates/PR%20Checklist%20for%20SQLToolsAndLibraries.md&version=GBmain&_a=preview) has been reviewed and followed - [ ] Code changes are accompanied by appropriate unit tests - [ ] Identified and included SMEs needed to review code changes - [ ] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=make-the-changes-in) here to make changes in the code ## Testing - [ ] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=to-extend-the-tests-do-the-following%3A) here to add new tests for your feature ## Documentation - [ ] Update relevant documentation in the [wiki](https://dev.azure.com/msdata/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki) and the README.md file ## Additional Information This PR added a test case for PAD_INDEX and DROP_EXISTING options for Json index. These options are shared options with other indexes, so only a test case is needed. ---- #### AI description (iteration 1) #### PR Classification This PR adds new test cases to validate the PAD_INDEX and DROP_EXISTING options for JSON indexes. #### PR Summary The changes extend the JSON index tests by including scenarios for the PAD_INDEX and DROP_EXISTING options. - `Test/SqlDom/TestScripts/JsonIndexTests170.sql`: Adds a JSON index creation statement with PAD_INDEX and DROP_EXISTING options. - `Test/SqlDom/Baselines170/JsonIndexTests170.sql`: Updates the baseline to include the new JSON index command with PAD_INDEX and DROP_EXISTING options. - `Test/SqlDom/Only170SyntaxTests.cs`: Adjusts the expected error counts for JSON index tests to reflect the new options. Related work items: #4711081 --- Test/SqlDom/Baselines170/JsonIndexTests170.sql | 5 ++++- Test/SqlDom/Only170SyntaxTests.cs | 2 +- Test/SqlDom/TestScripts/JsonIndexTests170.sql | 6 +++++- 3 files changed, 10 insertions(+), 3 deletions(-) diff --git a/Test/SqlDom/Baselines170/JsonIndexTests170.sql b/Test/SqlDom/Baselines170/JsonIndexTests170.sql index ff1b9d8..c6c72e2 100644 --- a/Test/SqlDom/Baselines170/JsonIndexTests170.sql +++ b/Test/SqlDom/Baselines170/JsonIndexTests170.sql @@ -29,4 +29,7 @@ CREATE JSON INDEX IX_JSON_Complex FOR ('$.metadata.title', '$.content.sections[*].text', '$.tags[*]'); CREATE JSON INDEX IX_JSON - ON dbo.Users (JsonData) WITH (OPTIMIZE_FOR_ARRAY_SEARCH = ON); \ No newline at end of file + ON dbo.Users (JsonData) WITH (OPTIMIZE_FOR_ARRAY_SEARCH = ON); + +CREATE JSON INDEX IX_JSON + ON dbo.Users (JsonData) WITH (PAD_INDEX = ON, DROP_EXISTING = ON); \ No newline at end of file diff --git a/Test/SqlDom/Only170SyntaxTests.cs b/Test/SqlDom/Only170SyntaxTests.cs index bfb7b29..a52a1ca 100644 --- a/Test/SqlDom/Only170SyntaxTests.cs +++ b/Test/SqlDom/Only170SyntaxTests.cs @@ -13,7 +13,7 @@ public partial class SqlDomTests private static readonly ParserTest[] Only170TestInfos = { new ParserTest170("RegexpTVFTests170.sql", nErrors80: 1, nErrors90: 1, nErrors100: 0, nErrors110: 0, nErrors120: 0, nErrors130: 0, nErrors140: 0, nErrors150: 0, nErrors160: 0), - new ParserTest170("JsonIndexTests170.sql", nErrors80: 2, nErrors90: 9, nErrors100: 9, nErrors110: 9, nErrors120: 9, nErrors130: 9, nErrors140: 9, nErrors150: 9, nErrors160: 9), + new ParserTest170("JsonIndexTests170.sql", nErrors80: 2, nErrors90: 10, nErrors100: 10, nErrors110: 10, nErrors120: 10, nErrors130: 10, nErrors140: 10, nErrors150: 10, nErrors160: 10), new ParserTest170("VectorIndexTests170.sql", nErrors80: 2, nErrors90: 12, nErrors100: 12, nErrors110: 12, nErrors120: 12, nErrors130: 12, nErrors140: 12, nErrors150: 12, nErrors160: 12), new ParserTest170("AlterDatabaseManualCutoverTests170.sql", nErrors80: 4, nErrors90: 4, nErrors100: 4, nErrors110: 4, nErrors120: 4, nErrors130: 4, nErrors140: 4, nErrors150: 4, nErrors160: 4), new ParserTest170("CreateColumnStoreIndexTests170.sql", nErrors80: 3, nErrors90: 3, nErrors100: 3, nErrors110: 3, nErrors120: 3, nErrors130: 0, nErrors140: 0, nErrors150: 0, nErrors160: 0), diff --git a/Test/SqlDom/TestScripts/JsonIndexTests170.sql b/Test/SqlDom/TestScripts/JsonIndexTests170.sql index 0c2aa8f..b5acf4a 100644 --- a/Test/SqlDom/TestScripts/JsonIndexTests170.sql +++ b/Test/SqlDom/TestScripts/JsonIndexTests170.sql @@ -32,4 +32,8 @@ FOR ('$.metadata.title', '$.content.sections[*].text', '$.tags[*]'); -- Create JSON index with OPTIMIZE_FOR_ARRAY_SEARCH option CREATE JSON INDEX IX_JSON ON dbo.Users (JsonData) -WITH (OPTIMIZE_FOR_ARRAY_SEARCH = ON); \ No newline at end of file +WITH (OPTIMIZE_FOR_ARRAY_SEARCH = ON); + +-- Create JSON index with PAD_INDEX and DROP_EXISTING options +CREATE JSON INDEX IX_JSON ON dbo.Users (JsonData) +WITH (PAD_INDEX = ON, DROP_EXISTING = ON); \ No newline at end of file From 18790bf5a38c28c2105056e9dfdc6824d039c012 Mon Sep 17 00:00:00 2001 From: Leila Lali Date: Thu, 25 Sep 2025 16:21:13 -0700 Subject: [PATCH 11/13] fixing merge issues --- Test/SqlDom/Only170SyntaxTests.cs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Test/SqlDom/Only170SyntaxTests.cs b/Test/SqlDom/Only170SyntaxTests.cs index a52a1ca..c0d7765 100644 --- a/Test/SqlDom/Only170SyntaxTests.cs +++ b/Test/SqlDom/Only170SyntaxTests.cs @@ -26,10 +26,10 @@ public partial class SqlDomTests new ParserTest170("DropExternalModelStatementTests170.sql", nErrors80: 1, nErrors90: 1, nErrors100: 1, nErrors110: 1, nErrors120: 1, nErrors130: 1, nErrors140: 1, nErrors150: 1, nErrors160: 1), new ParserTest170("VectorFunctionTests170.sql", nErrors80: 1, nErrors90: 1, nErrors100: 3, nErrors110: 3, nErrors120: 3, nErrors130: 3, nErrors140: 3, nErrors150: 3, nErrors160: 3), new ParserTest170("SecurityStatementExternalModelTests170.sql", nErrors80: 2, nErrors90: 17, nErrors100: 17, nErrors110: 17, nErrors120: 17, nErrors130: 17, nErrors140: 17, nErrors150: 17, nErrors160: 17), - new ParserTest170("CreateEventSessionNotLikePredicate.sql", nErrors80: 2, nErrors90: 1, nErrors100: 1, nErrors110: 1, nErrors120: 1, nErrors130: 0, nErrors140: 0, nErrors150: 0, nErrors160: 0), new ParserTest170("VectorTypeSecondParameterTests170.sql", nErrors80: 2, nErrors90: 2, nErrors100: 2, nErrors110: 2, nErrors120: 2, nErrors130: 2, nErrors140: 2, nErrors150: 2, nErrors160: 2), - new ParserTest170("RegexpLikeTests170.sql", nErrors80: 15, nErrors90: 15, nErrors100: 15, nErrors110: 18, nErrors120: 18, nErrors130: 18, nErrors140: 18, nErrors150: 18, nErrors160: 18) + new ParserTest170("RegexpLikeTests170.sql", nErrors80: 15, nErrors90: 15, nErrors100: 15, nErrors110: 18, nErrors120: 18, nErrors130: 18, nErrors140: 18, nErrors150: 18, nErrors160: 18), new ParserTest170("OptimizedLockingTests170.sql", nErrors80: 2, nErrors90: 2, nErrors100: 2, nErrors110: 2, nErrors120: 2, nErrors130: 2, nErrors140: 2, nErrors150: 2, nErrors160: 2), + new ParserTest170("CreateEventSessionNotLikePredicate.sql", nErrors80: 2, nErrors90: 1, nErrors100: 1, nErrors110: 1, nErrors120: 1, nErrors130: 0, nErrors140: 0, nErrors150: 0, nErrors160: 0) }; private static readonly ParserTest[] SqlAzure170_TestInfos = From ddff94d2f3ba0f5ab59b22d490ac8f716a8d0bac Mon Sep 17 00:00:00 2001 From: Leila Lali Date: Fri, 26 Sep 2025 01:15:01 +0000 Subject: [PATCH 12/13] Merged PR 1815666: Release notes for 170.121.2 # Pull Request Template for ScriptDom ## Description Please provide a detailed description, include the link to the design specification or SQL feature document for the new TSQL syntaxes. Make sure to add links to the Github or DevDiv issue Before submitting your pull request, please ensure you have completed the following: ## Code Change - [ ] The [Common checklist](https://msdata.visualstudio.com/SQLToolsAndLibraries/_git/Common?path=/Templates/PR%20Checklist%20for%20SQLToolsAndLibraries.md&version=GBmain&_a=preview) has been reviewed and followed - [ ] Code changes are accompanied by appropriate unit tests - [ ] Identified and included SMEs needed to review code changes - [ ] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=make-the-changes-in) here to make changes in the code ## Testing - [ ] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=to-extend-the-tests-do-the-following%3A) here to add new tests for your feature ## Documentation - [ ] Update relevant documentation in the [wiki](https://dev.azure.com/msdata/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki) and the README.md file ## Additional Information Please provide any additional information that might be helpful for the reviewers --- release-notes/170/170.121.2.md | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 release-notes/170/170.121.2.md diff --git a/release-notes/170/170.121.2.md b/release-notes/170/170.121.2.md new file mode 100644 index 0000000..d92500b --- /dev/null +++ b/release-notes/170/170.121.2.md @@ -0,0 +1,31 @@ +# Release Notes + +## Microsoft.SqlServer.TransactSql.ScriptDom 170.121.2 +This update brings the below changes over the previous release: + +### Target Platform Support + +* .NET Framework 4.7.2 (Windows x86, Windows x64) +* .NET 8 (Windows x86, Windows x64, Linux, macOS) +* .NET Standard 2.0+ (Windows x86, Windows x64, Linux, macOS) + +### Dependencies +* Updates .NET SDK to latest patch version 8.0.414 + +#### .NET Framework +#### .NET Core + +### New Features +* Adds support for 32-bit float vectors. +* Adds support for Optimized Locking database option. +* Adds OPTIMIZE_FOR_ARRAY_SEARCH option for Json index. +* Adds support for `JSON_QUERY` Functions. + +### Fixed +* Fixes parsing AI_GENERATE_EMBEDDINGS when PARAMETERS comes without parentheses +* Fixes https://github.com/microsoft/SqlScriptDOM/issues/163 +* Fixes https://github.com/microsoft/SqlScriptDOM/issues/160 +* Fixes https://github.com/microsoft/SqlScriptDOM/issues/153 +### Changes + +### Known Issues From 9c29a893dc9e063970324e5e871ef821e637ae62 Mon Sep 17 00:00:00 2001 From: Drew Skwiers-Koballa Date: Fri, 26 Sep 2025 10:24:35 -0700 Subject: [PATCH 13/13] replace smart apostrophe --- SqlScriptDom/Parser/TSql/TSql170.g | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SqlScriptDom/Parser/TSql/TSql170.g b/SqlScriptDom/Parser/TSql/TSql170.g index 27baeee..d746306 100644 --- a/SqlScriptDom/Parser/TSql/TSql170.g +++ b/SqlScriptDom/Parser/TSql/TSql170.g @@ -32181,7 +32181,7 @@ aiGenerateEmbeddingsFunctionCall // // Why: // - Users may store model names that *visually* contain dots or spaces, e.g. [dbo.MyDefaultModel]. - // When bracket-delimited, the lexer emits this as a **single** token (QuotedIdentifier), so it’s OK. + // When bracket-delimited, the lexer emits this as a **single** token (QuotedIdentifier), so it's OK. // - True multipart names (db.schema.model) must be rejected here, so we do NOT consume any Dot tokens. // // Allowed (single token):