Skip to content

Commit 2b7f886

Browse files
jahnvi480jahnvi480
authored
FEAT: Adding getDecimalSeperator and setDecimalSeperator as global functions (#188)
### Work Item / Issue Reference <!-- IMPORTANT: Please follow the PR template guidelines below. For mssql-python maintainers: Insert your ADO Work Item ID below (e.g. AB#37452) For external contributors: Insert Github Issue number below (e.g. #149) Only one reference is required - either GitHub issue OR ADO Work Item. --> <!-- mssql-python maintainers: ADO Work Item --> > [AB#34907](https://sqlclientdrivers.visualstudio.com/c6d89619-62de-46a0-8b46-70b92a84d85e/_workitems/edit/34907) > [AB#34908](https://sqlclientdrivers.visualstudio.com/c6d89619-62de-46a0-8b46-70b92a84d85e/_workitems/edit/34908) ------------------------------------------------------------------- ### Summary This pull request adds support for configuring the decimal separator used when parsing and displaying NUMERIC/DECIMAL values in the `mssql_python` package. It introduces new Python and C++ APIs to control the decimal separator, ensures the separator is respected in string representations, and includes comprehensive tests to verify the new functionality. **Decimal separator configuration and propagation:** - Added `setDecimalSeparator` and `getDecimalSeparator` functions in `mssql_python/__init__.py` to allow users to set and retrieve the decimal separator character, with input validation and propagation to the C++ layer (`DDBCSetDecimalSeparator`). - Introduced a global `g_decimalSeparator` variable and the `DDBCSetDecimalSeparator` function in C++ (`ddbc_bindings.cpp` and `ddbc_bindings.h`) to store and update the separator, and exposed this setter to Python via pybind11. **Integration with data handling and display:** - Updated the C++ data fetching logic to use the configured decimal separator when converting NUMERIC/DECIMAL database values to Python `Decimal` objects, ensuring correct parsing and formatting. - Modified the Python `Row.__str__` method to use the current decimal separator when displaying decimal values, so string representations reflect user configuration. **Testing:** - Added and expanded tests in `tests/test_001_globals.py` and `tests/test_004_cursor.py` to cover the new decimal separator functionality, including validation, database operations, string formatting, and ensuring calculations are unaffected by the separator setting. These changes make the decimal separator fully configurable and ensure consistent behavior across both parsing and display of decimal values. --------- Co-authored-by: Jahnvi Thakkar <jathakkar@microsoft.com>
1 parent 5fe06f5 commit 2b7f886

File tree

7 files changed

+858
-29
lines changed

7 files changed

+858
-29
lines changed

‎mssql_python/__init__.py‎

Lines changed: 73 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@
44
This module initializes the mssql_python package.
55
"""
66
import threading
7+
import locale
8+
79
# Exceptions
810
# https://www.python.org/dev/peps/pep-0249/#exceptions
911

@@ -13,25 +15,90 @@
1315
paramstyle = "qmark"
1416
threadsafety = 1
1517

16-
_settings_lock = threading.Lock()
18+
# Initialize the locale setting only once at module import time
19+
# This avoids thread-safety issues with locale
20+
_DEFAULT_DECIMAL_SEPARATOR = "."
21+
try:
22+
# Get the locale setting once during module initialization
23+
_locale_separator = locale.localeconv()['decimal_point']
24+
if _locale_separator and len(_locale_separator) == 1:
25+
_DEFAULT_DECIMAL_SEPARATOR = _locale_separator
26+
except (AttributeError, KeyError, TypeError, ValueError):
27+
pass # Keep the default "." if locale access fails
1728

18-
# Create a settings object to hold configuration
1929
class Settings:
2030
def __init__(self):
2131
self.lowercase = False
32+
# Use the pre-determined separator - no locale access here
33+
self.decimal_separator = _DEFAULT_DECIMAL_SEPARATOR
2234

23-
# Create a global settings instance
35+
# Global settings instance
2436
_settings = Settings()
37+
_settings_lock = threading.Lock()
2538

26-
# Define the get_settings function for internal use
2739
def get_settings():
2840
"""Return the global settings object"""
2941
with _settings_lock:
3042
_settings.lowercase = lowercase
3143
return _settings
3244

33-
# Expose lowercase as a regular module variable that users can access and set
34-
lowercase = _settings.lowercase
45+
lowercase = _settings.lowercase # Default is False
46+
47+
# Set the initial decimal separator in C++
48+
from .ddbc_bindings import DDBCSetDecimalSeparator
49+
DDBCSetDecimalSeparator(_settings.decimal_separator)
50+
51+
# New functions for decimal separator control
52+
def setDecimalSeparator(separator):
53+
"""
54+
Sets the decimal separator character used when parsing NUMERIC/DECIMAL values
55+
from the database, e.g. the "." in "1,234.56".
56+
57+
The default is to use the current locale's "decimal_point" value when the module
58+
was first imported, or "." if the locale is not available. This function overrides
59+
the default.
60+
61+
Args:
62+
separator (str): The character to use as decimal separator
63+
64+
Raises:
65+
ValueError: If the separator is not a single character string
66+
"""
67+
# Type validation
68+
if not isinstance(separator, str):
69+
raise ValueError("Decimal separator must be a string")
70+
71+
# Length validation
72+
if len(separator) == 0:
73+
raise ValueError("Decimal separator cannot be empty")
74+
75+
if len(separator) > 1:
76+
raise ValueError("Decimal separator must be a single character")
77+
78+
# Character validation
79+
if separator.isspace():
80+
raise ValueError("Whitespace characters are not allowed as decimal separators")
81+
82+
# Check for specific disallowed characters
83+
if separator in ['\t', '\n', '\r', '\v', '\f']:
84+
raise ValueError(f"Control character '{repr(separator)}' is not allowed as a decimal separator")
85+
86+
# Set in Python side settings
87+
_settings.decimal_separator = separator
88+
89+
# Update the C++ side
90+
from .ddbc_bindings import DDBCSetDecimalSeparator
91+
DDBCSetDecimalSeparator(separator)
92+
93+
def getDecimalSeparator():
94+
"""
95+
Returns the decimal separator character used when parsing NUMERIC/DECIMAL values
96+
from the database.
97+
98+
Returns:
99+
str: The current decimal separator character
100+
"""
101+
return _settings.decimal_separator
35102

36103
# Import necessary modules
37104
from .exceptions import (

‎mssql_python/cursor.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
MONEY_MIN = decimal.Decimal('-922337203685477.5808')
2727
MONEY_MAX = decimal.Decimal('922337203685477.5807')
2828

29+
2930
class Cursor:
3031
"""
3132
Represents a database cursor, which is used to manage the context of a fetch operation.

‎mssql_python/pybind/ddbc_bindings.cpp‎

Lines changed: 67 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -2037,20 +2037,48 @@ SQLRETURN SQLGetData_wrap(SqlHandlePtr StatementHandle, SQLUSMALLINT colCount, p
20372037
ret = SQLGetData_ptr(hStmt, i, SQL_C_CHAR, numericStr, sizeof(numericStr), &indicator);
20382038

20392039
if (SQL_SUCCEEDED(ret)) {
2040-
if (indicator == SQL_NULL_DATA) {
2041-
row.append(py::none());
2042-
} else {
2043-
try {
2044-
std::string s(reinterpret_cast<const char*>(numericStr));
2045-
auto Decimal = py::module_::import("decimal").attr("Decimal");
2046-
row.append(Decimal(s));
2047-
} catch (const py::error_already_set& e) {
2048-
LOG("Error converting to Decimal: {}", e.what());
2049-
row.append(py::none());
2040+
try {
2041+
// Validate 'indicator' to avoid buffer overflow and fallback to a safe
2042+
// null-terminated read when length is unknown or out-of-range.
2043+
const char* cnum = reinterpret_cast<const char*>(numericStr);
2044+
size_t bufSize = sizeof(numericStr);
2045+
size_t safeLen = 0;
2046+
2047+
if (indicator > 0 && indicator <= static_cast<SQLLEN>(bufSize)) {
2048+
// indicator appears valid and within the buffer size
2049+
safeLen = static_cast<size_t>(indicator);
2050+
} else {
2051+
// indicator is unknown, zero, negative, or too large; determine length
2052+
// by searching for a terminating null (safe bounded scan)
2053+
for (size_t j = 0; j < bufSize; ++j) {
2054+
if (cnum[j] == '\0') {
2055+
safeLen = j;
2056+
break;
2057+
}
2058+
}
2059+
// if no null found, use the full buffer size as a conservative fallback
2060+
if (safeLen == 0 && bufSize > 0 && cnum[0] != '\0') {
2061+
safeLen = bufSize;
2062+
}
20502063
}
2064+
2065+
// Use the validated length to construct the string for Decimal
2066+
std::string numStr(cnum, safeLen);
2067+
2068+
// Create Python Decimal object
2069+
py::object decimalObj = py::module_::import("decimal").attr("Decimal")(numStr);
2070+
2071+
// Add to row
2072+
row.append(decimalObj);
2073+
} catch (const py::error_already_set& e) {
2074+
// If conversion fails, append None
2075+
LOG("Error converting to decimal: {}", e.what());
2076+
row.append(py::none());
20512077
}
2052-
} else {
2053-
LOG("Error retrieving data for column - {}, data type - {}, SQLGetData rc - {}",
2078+
}
2079+
else {
2080+
LOG("Error retrieving data for column - {}, data type - {}, SQLGetData return "
2081+
"code - {}. Returning NULL value instead",
20542082
i, dataType, ret);
20552083
row.append(py::none());
20562084
}
@@ -2560,11 +2588,24 @@ SQLRETURN FetchBatchData(SQLHSTMT hStmt, ColumnBuffers& buffers, py::list& colum
25602588
case SQL_DECIMAL:
25612589
case SQL_NUMERIC: {
25622590
try {
2563-
// Convert numericStr to py::decimal.Decimal and append to row
2564-
row.append(py::module_::import("decimal").attr("Decimal")(std::string(
2565-
reinterpret_cast<const char*>(
2566-
&buffers.charBuffers[col - 1][i * MAX_DIGITS_IN_NUMERIC]),
2567-
buffers.indicators[col - 1][i])));
2591+
// Convert the string to use the current decimal separator
2592+
std::string numStr(reinterpret_cast<const char*>(
2593+
&buffers.charBuffers[col - 1][i * MAX_DIGITS_IN_NUMERIC]),
2594+
buffers.indicators[col - 1][i]);
2595+
2596+
// Get the current separator in a thread-safe way
2597+
std::string separator = GetDecimalSeparator();
2598+
2599+
if (separator != ".") {
2600+
// Replace the driver's decimal point with our configured separator
2601+
size_t pos = numStr.find('.');
2602+
if (pos != std::string::npos) {
2603+
numStr.replace(pos, 1, separator);
2604+
}
2605+
}
2606+
2607+
// Convert to Python decimal
2608+
row.append(py::module_::import("decimal").attr("Decimal")(numStr));
25682609
} catch (const py::error_already_set& e) {
25692610
// Handle the exception, e.g., log the error and append py::none()
25702611
LOG("Error converting to decimal: {}", e.what());
@@ -3016,6 +3057,13 @@ void enable_pooling(int maxSize, int idleTimeout) {
30163057
});
30173058
}
30183059

3060+
// Thread-safe decimal separator setting
3061+
ThreadSafeDecimalSeparator g_decimalSeparator;
3062+
3063+
void DDBCSetDecimalSeparator(const std::string& separator) {
3064+
SetDecimalSeparator(separator);
3065+
}
3066+
30193067
// Architecture-specific defines
30203068
#ifndef ARCHITECTURE
30213069
#define ARCHITECTURE "win64" // Default to win64 if not defined during compilation
@@ -3102,6 +3150,8 @@ PYBIND11_MODULE(ddbc_bindings, m) {
31023150
py::arg("tableType") = std::wstring());
31033151
m.def("DDBCSQLFetchScroll", &SQLFetchScroll_wrap,
31043152
"Scroll to a specific position in the result set and optionally fetch data");
3153+
m.def("DDBCSetDecimalSeparator", &DDBCSetDecimalSeparator, "Set the decimal separator character");
3154+
31053155

31063156
// Add a version attribute
31073157
m.attr("__version__") = "1.0.0";

‎mssql_python/pybind/ddbc_bindings.h‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -413,3 +413,47 @@ inline std::wstring Utf8ToWString(const std::string& str) {
413413
return converter.from_bytes(str);
414414
#endif
415415
}
416+
417+
// Thread-safe decimal separator accessor class
418+
class ThreadSafeDecimalSeparator {
419+
private:
420+
std::string value;
421+
mutable std::mutex mutex;
422+
423+
public:
424+
// Constructor with default value
425+
ThreadSafeDecimalSeparator() : value(".") {}
426+
427+
// Set the decimal separator with thread safety
428+
void set(const std::string& separator) {
429+
std::lock_guard<std::mutex> lock(mutex);
430+
value = separator;
431+
}
432+
433+
// Get the decimal separator with thread safety
434+
std::string get() const {
435+
std::lock_guard<std::mutex> lock(mutex);
436+
return value;
437+
}
438+
439+
// Returns whether the current separator is different from the default "."
440+
bool isCustomSeparator() const {
441+
std::lock_guard<std::mutex> lock(mutex);
442+
return value != ".";
443+
}
444+
};
445+
446+
// Global instance
447+
extern ThreadSafeDecimalSeparator g_decimalSeparator;
448+
449+
// Helper functions to replace direct access
450+
inline void SetDecimalSeparator(const std::string& separator) {
451+
g_decimalSeparator.set(separator);
452+
}
453+
454+
inline std::string GetDecimalSeparator() {
455+
return g_decimalSeparator.get();
456+
}
457+
458+
// Function to set the decimal separator
459+
void DDBCSetDecimalSeparator(const std::string& separator);

‎mssql_python/row.py‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -91,7 +91,25 @@ def __iter__(self):
9191

9292
def __str__(self):
9393
"""Return string representation of the row"""
94-
return str(tuple(self._values))
94+
from decimal import Decimal
95+
from mssql_python import getDecimalSeparator
96+
97+
parts = []
98+
for value in self:
99+
if isinstance(value, Decimal):
100+
# Apply custom decimal separator for display
101+
sep = getDecimalSeparator()
102+
if sep != '.' and value is not None:
103+
s = str(value)
104+
if '.' in s:
105+
s = s.replace('.', sep)
106+
parts.append(s)
107+
else:
108+
parts.append(str(value))
109+
else:
110+
parts.append(repr(value))
111+
112+
return "(" + ", ".join(parts) + ")"
95113

96114
def __repr__(self):
97115
"""Return a detailed string representation for debugging"""

0 commit comments

Comments
 (0)