Skip to content

Commit 6d8d8ff

Browse files
bewithgauravbewithgaurav
committed
Merged PR 6156: CHORE: OneBranch Pipelines - Build Pipeline
#### AI description (iteration 1) #### PR Classification This pull request implements a major migration and architectural overhaul that adds full OneBranch pipeline support, refactors and extends the low‑level DDBC bindings, improves connection/cursor behavior, and introduces comprehensive documentation and new CI workflow enhancements. #### PR Summary The changes refactor the core ODBC C++ bindings (adding support for SQL_SS_TIMESTAMPOFFSET, improved deferred (DAE) data handling, and better logging/error reporting) and update Python modules for enhanced thread safety, encoding/decoding, and output conversion. In addition, the PR adds a complete OneBranch pipeline structure with new job, step, and variable template files together with extensive documentation (analysis, architecture, transformation patterns, migration summary, and quick start) and updates to GitHub workflows for PR format and code coverage checks. - `mssql_python/pybind/ddbc_bindings.cpp` (and related header files): Added new wrappers (e.g. for SQL_SS_TIMESTAMPOFFSET), enhanced DAE streaming, logging and error handling improvements. - `OneBranchPipelines/` folder: New modular pipeline files with job templates for Windows, macOS, and Linux, step templates (malware scanning and ESRP code signing), and variable templates for OneBranch configuration. - `OneBranch_Learnings/`: New documentation files (README, Analysis, Architecture, Transformation Patterns, Migration Summary, Quick Start) outlining the migration strategy and best practices. - `.github/workflows/`: Updated CI workflows for PR code coverage and format checking. - `mssql_python/__init__.py` and supporting modules: Improvements in thread-safety, encoding settings, and global configuration. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #38066
1 parent 6fd72e6 commit 6d8d8ff

23 files changed

+6147
-0
lines changed

‎.config/CredScanSuppressions.json‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
{
2+
"tool": "Credential Scanner",
3+
"suppressions": [
4+
{
5+
"file": "tests/*",
6+
"justification": "Test projects contain sample credentials and should be skipped"
7+
},
8+
{
9+
"file": "benchmarks/*",
10+
"justification": "Benchmark code may include test connection strings"
11+
},
12+
{
13+
"file": "eng/*",
14+
"justification": "Engineering and pipeline configuration files"
15+
},
16+
{
17+
"file": "OneBranchPipelines/*",
18+
"justification": "OneBranch pipeline configuration files"
19+
}
20+
]
21+
}

‎.config/PolicheckExclusions.xml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
<PoliCheckExclusions>
2+
<!-- Each of these exclusions is a folder name - if \[name]\ exists in the file path, it will be skipped -->
3+
<Exclusion Type="FolderPathFull">
4+
tests|benchmarks|eng|OneBranchPipelines|examples|docs|build-artifacts|dist|__pycache__|myvenv|testenv
5+
</Exclusion>
6+
7+
<!-- The file(s) named with any of these specified file names will be skipped during the scan regardless which folder they are in -->
8+
<Exclusion Type="FileName">
9+
CHANGELOG.md|README.md|LICENSE|NOTICE.txt|ROADMAP.md|CODE_OF_CONDUCT.md|CONTRIBUTING.md|SECURITY.md|SUPPORT.md
10+
</Exclusion>
11+
</PoliCheckExclusions>

‎.config/SDL_CONFIGURATION.md‎

Lines changed: 210 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,210 @@
1+
# SDL Configuration - mssql-python
2+
3+
**Organization:** Microsoft ADO.Net Team
4+
**Project:** mssql-python
5+
**Instance:** https://sqlclientdrivers.visualstudio.com/
6+
**Date:** October 1, 2025
7+
8+
---
9+
10+
## 📋 Configuration Files Overview
11+
12+
### 1. TSA Options (`tsaoptions.json`)
13+
14+
**Purpose:** Configures Threat and Security Assessment (TSA) for automated security issue tracking.
15+
16+
**Configuration:**
17+
```json
18+
{
19+
"instanceUrl": "https://sqlclientdrivers.visualstudio.com/",
20+
"projectName": "ADO.Net",
21+
"areaPath": "ADO.Net",
22+
"iterationPath": "ADO.Net\\TSA\\mssql-python",
23+
"notificationAliases": ["SqlClient@microsoft.com"],
24+
"repositoryName": "mssql-python",
25+
"codebaseName": "mssql-python",
26+
"allTools": true,
27+
"template": "MSDATA_RevolutionR_Overloaded0",
28+
"language": "python",
29+
"includePathPatterns": "mssql_python/*, setup.py, requirements.txt",
30+
"excludePathPatterns": "tests/*, benchmarks/*, examples/*, docs/*"
31+
}
32+
```
33+
34+
**Key Points:**
35+
- ✅ Uses same ADO.Net project as SqlClient
36+
- ✅ Notifications go to SqlClient@microsoft.com
37+
- ✅ Scans only production code (excludes tests/docs/examples)
38+
- ✅ Python-specific language configuration
39+
- ✅ Uses MSDATA template for consistency
40+
41+
---
42+
43+
### 2. CredScan Suppressions (`CredScanSuppressions.json`)
44+
45+
**Purpose:** Suppresses false positives from credential scanning in non-production code.
46+
47+
**Excluded Paths:**
48+
- ✅ `tests/*` - Test code with sample credentials
49+
- ✅ `examples/*` - Example code with demo connection strings
50+
- ✅ `docs/*` - Documentation with sample data
51+
- ✅ `benchmarks/*` - Benchmark code with test credentials
52+
53+
**Why These Exclusions:**
54+
- Test and example code intentionally contains non-production credentials
55+
- Documentation shows sample connection strings for educational purposes
56+
- These paths don't contain actual secrets used in production
57+
58+
---
59+
60+
### 3. PoliCheck Exclusions (`PolicheckExclusions.xml`)
61+
62+
**Purpose:** Excludes specific directories and file types from politically incorrect term scanning.
63+
64+
**Exclusions:**
65+
66+
**Folders:**
67+
- `TESTS` - Test code
68+
- `BENCHMARKS` - Performance benchmarks
69+
- `EXAMPLES` - Example code
70+
- `DOCS` - Documentation
71+
- `BUILD-ARTIFACTS` - Build outputs
72+
- `DIST` - Distribution packages
73+
- `__PYCACHE__` - Python cache
74+
- `MYVENV` / `TESTENV` - Virtual environments
75+
76+
**File Types:**
77+
- `.YML` - Pipeline configuration files
78+
- `.MD` - Markdown documentation
79+
- `.SQL` - SQL scripts (may contain diverse terminology)
80+
- `.JSON` - Configuration files
81+
- `.TXT` - Text files
82+
- `.LOG` - Log files
83+
84+
**Specific Files:**
85+
- `CHANGELOG.MD` - Change log
86+
- `README.MD` - Repository documentation
87+
- `LICENSE` - License file
88+
- `NOTICE.TXT` - Legal notices
89+
- `ROADMAP.MD` - Project roadmap
90+
91+
**Why These Exclusions:**
92+
- Configuration and documentation files may reference diverse terminology
93+
- Test/example code may contain varied sample data
94+
- Build artifacts are auto-generated and temporary
95+
- Virtual environments contain third-party code
96+
97+
---
98+
99+
## 🔒 How SDL Tools Use These Configurations
100+
101+
### During OneBranch Pipeline Execution
102+
103+
1. **TSA (Threat and Security Assessment)**
104+
- Runs on **Official builds only**
105+
- Creates work items in ADO.Net project
106+
- Sends notifications to SqlClient@microsoft.com
107+
- Tracks security issues in iteration: `ADO.Net\TSA\mssql-python`
108+
109+
2. **CredScan (Credential Scanner)**
110+
- Scans all files except those in suppressions list
111+
- Flags potential credentials/secrets
112+
- Breaks build if credentials found (except suppressed paths)
113+
- Safe to have test credentials in excluded paths
114+
115+
3. **PoliCheck (Politically Incorrect Term Check)**
116+
- Scans source code for inappropriate terminology
117+
- Skips excluded folders and file types
118+
- Breaks build if violations found (except suppressed)
119+
- Focuses on production code only
120+
121+
---
122+
123+
## 📝 When to Update These Files
124+
125+
### Add to CredScan Suppressions When:
126+
- Adding new test directories with sample credentials
127+
- Creating examples with demo connection strings
128+
- Documentation includes sample authentication data
129+
130+
### Add to PoliCheck Exclusions When:
131+
- New documentation files reference technical terms flagged incorrectly
132+
- New test data files contain diverse terminology
133+
- Third-party code added that shouldn't be scanned
134+
135+
### Update TSA Options When:
136+
- Project organization changes (e.g., moved to different ADO project)
137+
- Notification distribution list changes
138+
- Area/iteration paths change in Azure DevOps
139+
- Want to include/exclude different paths from scanning
140+
141+
---
142+
143+
## ✅ Validation Checklist
144+
145+
Before running Official builds:
146+
147+
- [x] TSA configuration points to correct ADO.Net project
148+
- [x] Notification email (SqlClient@microsoft.com) is correct
149+
- [x] CredScan suppressions cover all test/example paths
150+
- [x] PoliCheck exclusions include all documentation/config files
151+
- [x] Include patterns focus on production code only
152+
- [x] Exclude patterns cover tests, docs, examples, benchmarks
153+
154+
---
155+
156+
## 🚨 Monitoring & Maintenance
157+
158+
### Weekly Tasks
159+
- Check TSA dashboard for new security issues
160+
- Review CredScan/PoliCheck warnings in pipeline logs
161+
- Address any legitimate security findings
162+
163+
### When Builds Break
164+
1. **CredScan breaks build:**
165+
- Check if it's a false positive (test/example code)
166+
- If yes: Add to CredScanSuppressions.json
167+
- If no: Remove the credential and use proper secrets management
168+
169+
2. **PoliCheck breaks build:**
170+
- Review the flagged term
171+
- If it's technical terminology: Add to PolicheckExclusions.xml
172+
- If it's in documentation/tests: Verify those paths are excluded
173+
- If legitimate issue: Update the code
174+
175+
3. **TSA creates work items:**
176+
- Review the security finding
177+
- Follow your team's security issue resolution process
178+
- Update suppressions only if truly false positive
179+
180+
---
181+
182+
## 📚 Related Documentation
183+
184+
- **OneBranch SDL Guide:** https://aka.ms/obpipelines/sdl
185+
- **CredScan Documentation:** Internal Microsoft docs
186+
- **PoliCheck Guidelines:** Internal Microsoft docs
187+
- **TSA Dashboard:** https://sqlclientdrivers.visualstudio.com/ → TSA work items
188+
189+
---
190+
191+
## 🎯 Key Differences from SqlClient Configuration
192+
193+
| Aspect | SqlClient | mssql-python |
194+
|--------|-----------|--------------|
195+
| **Language** | C# | Python |
196+
| **Repository** | SqlClient | mssql-python |
197+
| **Include Patterns** | src/Microsoft.Data.SqlClient/* | mssql_python/* |
198+
| **Exclude Patterns** | src/.../tests/* | tests/*, examples/*, benchmarks/* |
199+
| **TSA Iteration** | ADO.Net\TSA\SqlClient | ADO.Net\TSA\mssql-python |
200+
201+
**Note:** Both projects share:
202+
- ✅ Same ADO.Net project
203+
- ✅ Same notification alias (SqlClient@microsoft.com)
204+
- ✅ Same TSA template (MSDATA_RevolutionR_Overloaded0)
205+
- ✅ Similar exclusion philosophy (exclude tests/docs/examples)
206+
207+
---
208+
209+
_Last Updated: October 1, 2025_
210+
_Maintained by: mssql-python team_

‎.config/tsaoptions.json‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{
2+
"instanceUrl": "https://sqlclientdrivers.visualstudio.com/",
3+
"projectName": "mssql-python",
4+
"areaPath": "mssql-python",
5+
"iterationPath": "mssql-python",
6+
"notificationAliases": [
7+
"mssql-python@microsoft.com"
8+
],
9+
"repositoryName": "mssql-python",
10+
"codebaseName": "Microsoft Python Driver for SQL Server",
11+
"allTools": true,
12+
"includePathPatterns": "mssql_python/*, setup.py, requirements.txt",
13+
"excludePathPatterns": "tests/*, benchmarks/*, eng/*, OneBranchPipelines/*"
14+
}

0 commit comments

Comments
 (0)