Merged PR 6157: SYNC: Github main to ADO main

bewithgaurav · bewithgaurav · commit 6fd72e613565 · 2025-10-06T13:55:09.000Z
#### AI description  (iteration 1)
#### PR Classification
This PR implements API enhancements and extensive test/pipeline improvements to strengthen metadata, output conversion, encoding/decoding, and batch execution features.

#### PR Summary
- **`eng/pipelines/build-whl-pipeline.yml` &amp; `.github/workflows/pr-code-coverage.yml`:** Updated to build wheels and report code coverage across multi‑architecture Linux containers (manylinux, musllinux, Alpine, etc.).
- **`mssql_python/cursor.py`:** Improved metadata handling and result set processing with new helper methods and fallback descriptions.
- **`mssql_python/connection.py`:** Added a robust output converter API (including methods to add, get, remove, and clear converters) and enhanced `execute`/`batch_execute` with better error and timeout management.
- **Tests in `/tests/`:** Expanded coverage for connection, encoding/decoding, output converters, rowcount, transaction handling, and thread safety to meet DB‑API 2.0 requirements.
- **File deletion:** Removed `mssql_python/testing_ddbc_bindings.py` as its functionality has been integrated into the main modules.
&lt;!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --&gt;

Related work items: #39062
diff --git a/PyPI_Description.md b/PyPI_Description.md
@@ -39,12 +39,12 @@ PyBind11 provides:
 
 We are currently in **Public Preview**.
 
-## What's new in v0.12.0
+## What's new in v0.13.0
 
-- **Complex Data Type Support:** Added native support for DATETIMEOFFSET and UNIQUEIDENTIFIER data types with full round-trip handling, enabling seamless integration with Python's timezone-aware `datetime` objects and `uuid.UUID` types.
-- **Support for monetary or currency values data types:** Extended MONEY and SMALLMONEY support to `executemany` operations with proper NULL handling and decimal conversion for improved bulk financial data processing.
-- **Improved Database Metadata API:** Added `getinfo()` method with enhanced ODBC metadata retrieval, allowing users to query driver/data source information using ODBC info types.
-- **Data Processing Optimizations:** Removed aggressive datetime parsing to prevent incorrect type conversions and improve data integrity across diverse datetime formats and string data.
+- **Enhanced Batch Operations:** Complete support for UNIQUEIDENTIFIER and DATETIMEOFFSET in `executemany()` operations with automatic type inference, enabling efficient bulk inserts of complex data types including UUIDs and timezone-aware datetimes.
+- **Streaming Large Values:** Robust handling of large objects (NVARCHAR/VARCHAR/VARBINARY(MAX)) in `executemany()` with automatic Data-At-Execution detection and fallback, supporting streaming inserts and fetches for massive datasets.
+- **Improved Cursor Reliability:** Enhanced `cursor.rowcount` accuracy across all fetch operations, including proper handling of empty result sets and consistent behavior for SELECT, INSERT, and UPDATE operations.
+- **Critical Stability Fixes:** Resolved memory leaks with secure token buffer handling, fixed resource cleanup to prevent segmentation faults during Python shutdown, and corrected type inference bugs in batch operations.
 
 For more information, please visit the project link on Github: https://github.com/microsoft/mssql-python
 
diff --git a/mssql_python/cursor.py b/mssql_python/cursor.py
@@ -1613,6 +1613,7 @@ def executemany(self, operation: str, seq_of_parameters: list) -> None:
                 # Use auto-detection for columns without explicit types
                 column = [row[col_index] for row in seq_of_parameters] if hasattr(seq_of_parameters, '__getitem__') else []
                 sample_value, min_val, max_val = self._compute_column_type(column)
+
                 dummy_row = list(sample_row)
                 paraminfo = self._create_parameter_types_list(
                     sample_value, param_info, dummy_row, col_index, min_val=min_val, max_val=max_val
@@ -1724,6 +1725,10 @@ def fetchone(self) -> Union[None, Row]:
                 self.messages.extend(ddbc_bindings.DDBCSQLGetAllDiagRecords(self.hstmt))
             
             if ret == ddbc_sql_const.SQL_NO_DATA.value:
+                # No more data available
+                if self._next_row_index == 0 and self.description is not None:
+                    # This is an empty result set, set rowcount to 0
+                    self.rowcount = 0
                 return None
             
             # Update internal position after successful fetch
@@ -1732,6 +1737,8 @@ def fetchone(self) -> Union[None, Row]:
                 self._next_row_index += 1
             else:
                 self._increment_rownumber()
+
+            self.rowcount = self._next_row_index
             
             # Create and return a Row object, passing column name map if available
             column_map = getattr(self, '_column_name_map', None)
@@ -1774,6 +1781,12 @@ def fetchmany(self, size: int = None) -> List[Row]:
                 # advance counters by number of rows actually returned
                 self._next_row_index += len(rows_data)
                 self._rownumber = self._next_row_index - 1
+
+            # Centralize rowcount assignment after fetch
+            if len(rows_data) == 0 and self._next_row_index == 0:
+                self.rowcount = 0
+            else:
+                self.rowcount = self._next_row_index
             
             # Convert raw data to Row objects
             column_map = getattr(self, '_column_name_map', None)
@@ -1806,6 +1819,12 @@ def fetchall(self) -> List[Row]:
             if rows_data and self._has_result_set:
                 self._next_row_index += len(rows_data)
                 self._rownumber = self._next_row_index - 1
+
+            # Centralize rowcount assignment after fetch
+            if len(rows_data) == 0 and self._next_row_index == 0:
+                self.rowcount = 0
+            else:
+                self.rowcount = self._next_row_index
             
             # Convert raw data to Row objects
             column_map = getattr(self, '_column_name_map', None)
diff --git a/mssql_python/pybind/connection/connection.cpp b/mssql_python/pybind/connection/connection.cpp
@@ -173,16 +173,16 @@ SQLRETURN Connection::setAttribute(SQLINTEGER attribute, py::object value) {
     LOG("Setting SQL attribute");
     SQLPOINTER ptr = nullptr;
     SQLINTEGER length = 0;
+    std::string buffer; // to hold sensitive data temporarily
 
     if (py::isinstance<py::int_>(value)) {
         int intValue = value.cast<int>();
         ptr = reinterpret_cast<SQLPOINTER>(static_cast<uintptr_t>(intValue));
         length = SQL_IS_INTEGER;
     } else if (py::isinstance<py::bytes>(value) || py::isinstance<py::bytearray>(value)) {
-        static std::vector<std::string> buffers;
-        buffers.emplace_back(value.cast<std::string>());
-        ptr = const_cast<char*>(buffers.back().c_str());
-        length = static_cast<SQLINTEGER>(buffers.back().size());
+        buffer = value.cast<std::string>();  // stack buffer
+        ptr = buffer.data();
+        length = static_cast<SQLINTEGER>(buffer.size());
     } else {
         LOG("Unsupported attribute value type");
         return SQL_ERROR;
@@ -195,6 +195,11 @@ SQLRETURN Connection::setAttribute(SQLINTEGER attribute, py::object value) {
     else {
         LOG("Set attribute successfully");
     }
+    
+    // Zero out sensitive data if used
+    if (!buffer.empty()) {
+        std::fill(buffer.begin(), buffer.end(), static_cast<char>(0));
+    }
     return ret;
 }
 
diff --git a/mssql_python/pybind/ddbc_bindings.cpp b/mssql_python/pybind/ddbc_bindings.cpp
@@ -665,24 +665,67 @@ void HandleZeroColumnSizeAtFetch(SQLULEN& columnSize) {
 
 }  // namespace
 
+// Helper function to check if Python is shutting down or finalizing
+// This centralizes the shutdown detection logic to avoid code duplication
+static bool is_python_finalizing() {
+    try {
+        if (Py_IsInitialized() == 0) {
+            return true; // Python is already shut down
+        }
+        
+        py::gil_scoped_acquire gil;
+        py::object sys_module = py::module_::import("sys");
+        if (!sys_module.is_none()) {
+            // Check if the attribute exists before accessing it (for Python version compatibility)
+            if (py::hasattr(sys_module, "_is_finalizing")) {
+                py::object finalizing_func = sys_module.attr("_is_finalizing");
+                if (!finalizing_func.is_none() && finalizing_func().cast<bool>()) {
+                    return true; // Python is finalizing
+                }
+            }
+        }
+        return false;
+    } catch (...) {
+        std::cerr << "Error occurred while checking Python finalization state." << std::endl;
+        // Be conservative - don't assume shutdown on any exception
+        // Only return true if we're absolutely certain Python is shutting down
+        return false;
+    }
+}
+
 // TODO: Revisit GIL considerations if we're using python's logger
 template <typename... Args>
 void LOG(const std::string& formatString, Args&&... args) {
-    py::gil_scoped_acquire gil;  // <---- this ensures safe Python API usage
+    // Check if Python is shutting down to avoid crash during cleanup
+    if (is_python_finalizing()) {
+        return; // Python is shutting down or finalizing, don't log
+    }
+    
+    try {
+        py::gil_scoped_acquire gil;  // <---- this ensures safe Python API usage
 
-    py::object logger = py::module_::import("mssql_python.logging_config").attr("get_logger")();
-    if (py::isinstance<py::none>(logger)) return;
+        py::object logger = py::module_::import("mssql_python.logging_config").attr("get_logger")();
+        if (py::isinstance<py::none>(logger)) return;
 
-    try {
-        std::string ddbcFormatString = "[DDBC Bindings log] " + formatString;
-        if constexpr (sizeof...(args) == 0) {
-            logger.attr("debug")(py::str(ddbcFormatString));
-        } else {
-            py::str message = py::str(ddbcFormatString).format(std::forward<Args>(args)...);
-            logger.attr("debug")(message);
+        try {
+            std::string ddbcFormatString = "[DDBC Bindings log] " + formatString;
+            if constexpr (sizeof...(args) == 0) {
+                logger.attr("debug")(py::str(ddbcFormatString));
+            } else {
+                py::str message = py::str(ddbcFormatString).format(std::forward<Args>(args)...);
+                logger.attr("debug")(message);
+            }
+        } catch (const std::exception& e) {
+            std::cerr << "Logging error: " << e.what() << std::endl;
         }
+    } catch (const py::error_already_set& e) {
+        // Python is shutting down or in an inconsistent state, silently ignore
+        (void)e; // Suppress unused variable warning
+        return;
     } catch (const std::exception& e) {
-        std::cerr << "Logging error: " << e.what() << std::endl;
+        // Any other error, ignore to prevent crash during cleanup
+        (void)e; // Suppress unused variable warning
+        return;
     }
 }
 
@@ -993,17 +1036,26 @@ SQLSMALLINT SqlHandle::type() const {
  */
 void SqlHandle::free() {
     if (_handle && SQLFreeHandle_ptr) {
-        const char* type_str = nullptr;
-        switch (_type) {
-            case SQL_HANDLE_ENV:  type_str = "ENV"; break;
-            case SQL_HANDLE_DBC:  type_str = "DBC"; break;
-            case SQL_HANDLE_STMT: type_str = "STMT"; break;
-            case SQL_HANDLE_DESC: type_str = "DESC"; break;
-            default:              type_str = "UNKNOWN"; break;
+        // Check if Python is shutting down using centralized helper function
+        bool pythonShuttingDown = is_python_finalizing();
+        
+        // CRITICAL FIX: During Python shutdown, don't free STMT handles as their parent DBC may already be freed
+        // This prevents segfault when handles are freed in wrong order during interpreter shutdown
+        // Type 3 = SQL_HANDLE_STMT, Type 2 = SQL_HANDLE_DBC, Type 1 = SQL_HANDLE_ENV
+        if (pythonShuttingDown && _type == 3) {
+            _handle = nullptr; // Mark as freed to prevent double-free attempts
+            return;
         }
+        
+        // Always clean up ODBC resources, regardless of Python state
         SQLFreeHandle_ptr(_type, _handle);
         _handle = nullptr;
-        // Don't log during destruction - it can cause segfaults during Python shutdown
+        
+        // Only log if Python is not shutting down (to avoid segfault)
+        if (!pythonShuttingDown) {
+            // Don't log during destruction - even in normal cases it can be problematic
+            // If logging is needed, use explicit close() methods instead
+        }
     }
 }
 
@@ -2017,8 +2069,10 @@ SQLRETURN BindParameterArray(SQLHANDLE hStmt,
                     SQLGUID* guidArray = AllocateParamBufferArray<SQLGUID>(tempBuffers, paramSetSize);
                     strLenOrIndArray = AllocateParamBufferArray<SQLLEN>(tempBuffers, paramSetSize);
 
-                    static py::module_ uuid_mod = py::module_::import("uuid");
-                    static py::object uuid_class = uuid_mod.attr("UUID");
+                    // Get cached UUID class from module-level helper
+                    // This avoids static object destruction issues during Python finalization
+                    py::object uuid_class = py::module_::import("mssql_python.ddbc_bindings").attr("_get_uuid_class")();
+                    
                     for (size_t i = 0; i < paramSetSize; ++i) {
                         const py::handle& element = columnValues[i];
                         std::array<unsigned char, 16> uuid_bytes;
@@ -3851,6 +3905,14 @@ PYBIND11_MODULE(ddbc_bindings, m) {
     });
 
 
+    // Module-level UUID class cache
+    // This caches the uuid.UUID class at module initialization time and keeps it alive
+    // for the entire module lifetime, avoiding static destructor issues during Python finalization
+    m.def("_get_uuid_class", []() -> py::object {
+        static py::object uuid_class = py::module_::import("uuid").attr("UUID");
+        return uuid_class;
+    }, "Internal helper to get cached UUID class");
+
     // Add a version attribute
     m.attr("__version__") = "1.0.0";
     
diff --git a/setup.py b/setup.py
@@ -83,7 +83,7 @@ def finalize_options(self):
 
 setup(
     name='mssql-python',
-    version='0.12.0',
+    version='0.13.0',
     description='A Python library for interacting with Microsoft SQL Server',
     long_description=open('PyPI_Description.md', encoding='utf-8').read(),
     long_description_content_type='text/markdown',
diff --git a/tests/test_004_cursor.py b/tests/test_004_cursor.py
diff --git a/tests/test_005_connection_cursor_lifecycle.py b/tests/test_005_connection_cursor_lifecycle.py
