Skip to content

Commit 9343e83

Browse files
DifferentialOrangeDifferentialOrange
committed
types: support working with binary for Python 3
This is a breaking change. Before this patch, both bytes and str were encoded as mp_str. It was possible to work with utf and non-utf strings, but not with varbinary (mp_bin) [1]. This patch adds varbinary support for Python 3 by default. With Python 2 the behavior of the connector remains the same. Before this patch: * encoding="utf-8" (default) Python 3 -> Tarantool -> Python 3 str -> mp_str (string) -> str bytes -> mp_str (string) -> str mp_bin (varbinary) -> bytes * encoding=None Python 3 -> Tarantool -> Python 3 bytes -> mp_str (string) -> bytes str -> mp_str (string) -> bytes mp_bin (varbinary) -> bytes Using bytes as key was not supported by several methods (delete, update, select). After this patch: * encoding="utf-8" (default) Python 3 -> Tarantool -> Python 3 str -> mp_str (string) -> str bytes -> mp_bin (varbinary) -> bytes * encoding=None Python 3 -> Tarantool -> Python 3 bytes -> mp_str (string) -> bytes str -> mp_str (string) -> bytes mp_bin (varbinary) -> bytes Using bytes as key are now supported by all methods. Thus, encoding="utf-8" connection may be used to work with utf-8 strings and varbinary and encodine=None connection may be used to work with non-utf-8 strings. This patch does not add new restrictions (like "do not permit to use str in encoding=None mode because result may be confusing") to preserve current behavior (for example, using space name as str in schema get_space). 1. tarantool/tarantool#4201 Closes #105
1 parent 296c884 commit 9343e83

File tree

6 files changed

+338
-25
lines changed

6 files changed

+338
-25
lines changed

‎CHANGELOG.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,54 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1111
(PR #192).
1212

1313
### Changed
14+
- **Breaking**: change binary types encode/decode for Python 3
15+
to support working with varbinary (PR #211, #105).
16+
With Python 2 the behavior of the connector remains the same.
17+
18+
Before this patch:
19+
20+
* encoding="utf-8" (default)
21+
22+
| Python 3 | -> | Tarantool | -> | Python 3 |
23+
|----------|----|--------------------|----|----------|
24+
| str | -> | mp_str (string) | -> | str |
25+
| bytes | -> | mp_str (string) | -> | str |
26+
| | | mp_bin (varbinary) | -> | bytes |
27+
28+
* encoding=None
29+
30+
| Python 3 | -> | Tarantool | -> | Python 3 |
31+
|----------|----|--------------------|----|----------|
32+
| bytes | -> | mp_str (string) | -> | bytes |
33+
| str | -> | mp_str (string) | -> | bytes |
34+
| | | mp_bin (varbinary) | -> | bytes |
35+
36+
Using bytes as key was not supported by several methods (delete,
37+
update, select).
38+
39+
After this patch:
40+
41+
* encoding="utf-8" (default)
42+
43+
| Python 3 | -> | Tarantool | -> | Python 3 |
44+
|----------|----|--------------------|----|----------|
45+
| str | -> | mp_str (string) | -> | str |
46+
| bytes | -> | mp_bin (varbinary) | -> | bytes |
47+
48+
* encoding=None
49+
50+
| Python 3 | -> | Tarantool | -> | Python 3 |
51+
|----------|----|--------------------|----|----------|
52+
| bytes | -> | mp_str (string) | -> | bytes |
53+
| str | -> | mp_str (string) | -> | bytes |
54+
| | | mp_bin (varbinary) | -> | bytes |
55+
56+
Using bytes as key are now supported by all methods.
57+
58+
Thus, encoding="utf-8" connection may be used to work with
59+
utf-8 strings and varbinary and encodine=None connection
60+
may be used to work with non-utf-8 strings.
61+
1462
- Clarify license of the project (BSD-2-Clause) (PR #210, #197).
1563
- Migrate CI to GitHub Actions (PR #213, PR #216, #182).
1664
- Various improvements and fixes in README (PR #210, PR #215).

‎tarantool/request.py‎

Lines changed: 25 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
Request types definitions
55
'''
66

7+
import sys
78
import collections
89
import msgpack
910
import hashlib
@@ -84,8 +85,26 @@ def __init__(self, conn):
8485
# The option controls whether to pack binary (non-unicode)
8586
# string values as mp_bin or as mp_str.
8687
#
87-
# The default behaviour of the connector is to pack both
88-
# bytes and Unicode strings as mp_str.
88+
# The default behaviour of the Python 2 connector is to pack
89+
# both bytes and Unicode strings as mp_str.
90+
#
91+
# The default behaviour of the Python 3 connector (since
92+
# default encoding is "utf-8") is to pack bytes as mp_bin
93+
# and Unicode strings as mp_str. encoding=None mode must
94+
# be used to work with non-utf strings.
95+
#
96+
# encoding = 'utf-8'
97+
#
98+
# Python 3 -> Tarantool -> Python 3
99+
# str -> mp_str (string) -> str
100+
# bytes -> mp_bin (varbinary) -> bytes
101+
#
102+
# encoding = None
103+
#
104+
# Python 3 -> Tarantool -> Python 3
105+
# bytes -> mp_str (string) -> bytes
106+
# str -> mp_str (string) -> bytes
107+
# mp_bin (varbinary) -> bytes
89108
#
90109
# msgpack-0.5.0 (and only this version) warns when the
91110
# option is unset:
@@ -98,7 +117,10 @@ def __init__(self, conn):
98117
# just always set it for all msgpack versions to get rid
99118
# of the warning on msgpack-0.5.0 and to keep our
100119
# behaviour on msgpack-1.0.0.
101-
packer_kwargs['use_bin_type'] = False
120+
if conn.encoding is None or sys.version_info.major == 2:
121+
packer_kwargs['use_bin_type'] = False
122+
else:
123+
packer_kwargs['use_bin_type'] = True
102124

103125
self.packer = msgpack.Packer(**packer_kwargs)
104126

‎tarantool/utils.py‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,10 @@
66
if sys.version_info.major == 2:
77
string_types = (basestring, )
88
integer_types = (int, long)
9+
supported_types = integer_types + string_types + (float,)
10+
911
ENCODING_DEFAULT = None
12+
1013
if sys.version_info.minor < 6:
1114
binary_types = (str, )
1215
else:
@@ -17,10 +20,13 @@ def strxor(rhs, lhs):
1720
return "".join(chr(ord(x) ^ ord(y)) for x, y in zip(rhs, lhs))
1821

1922
elif sys.version_info.major == 3:
20-
binary_types = (bytes, )
21-
string_types = (str, )
22-
integer_types = (int, )
23+
binary_types = (bytes, )
24+
string_types = (str, )
25+
integer_types = (int, )
26+
supported_types = integer_types + string_types + binary_types + (float,)
27+
2328
ENCODING_DEFAULT = "utf-8"
29+
2430
from base64 import decodebytes as base64_decode
2531

2632
def strxor(rhs, lhs):
@@ -43,7 +49,7 @@ def check_key(*args, **kwargs):
4349
elif args[0] is None and kwargs['select']:
4450
return []
4551
for key in args:
46-
assert isinstance(key, integer_types + string_types + (float,))
52+
assert isinstance(key, supported_types)
4753
return list(args)
4854

4955

‎test/suites/__init__.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,11 +12,13 @@
1212
from .test_mesh import TestSuite_Mesh
1313
from .test_execute import TestSuite_Execute
1414
from .test_dbapi import TestSuite_DBAPI
15+
from .test_encoding import TestSuite_Encoding
1516

1617
test_cases = (TestSuite_Schema_UnicodeConnection,
1718
TestSuite_Schema_BinaryConnection,
1819
TestSuite_Request, TestSuite_Protocol, TestSuite_Reconnect,
19-
TestSuite_Mesh, TestSuite_Execute, TestSuite_DBAPI)
20+
TestSuite_Mesh, TestSuite_Execute, TestSuite_DBAPI,
21+
TestSuite_Encoding)
2022

2123
def load_tests(loader, tests, pattern):
2224
suite = unittest.TestSuite()

‎test/suites/lib/skip.py‎

Lines changed: 65 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,15 @@
11
import functools
22
import pkg_resources
33
import re
4+
import sys
45

5-
SQL_SUPPORT_TNT_VERSION = '2.0.0'
66

7-
8-
def skip_or_run_sql_test(func):
9-
"""Decorator to skip or run SQL-related tests depending on the tarantool
7+
def skip_or_run_test_tarantool(func, REQUIRED_TNT_VERSION, msg):
8+
"""Decorator to skip or run tests depending on the tarantool
109
version.
1110
12-
Tarantool supports SQL-related stuff only since 2.0.0 version. So this
13-
decorator should wrap every SQL-related test to skip it if the tarantool
14-
version < 2.0.0 is used for testing.
15-
16-
Also, it can be used with the 'setUp' method for skipping the whole test
17-
suite.
11+
Also, it can be used with the 'setUp' method for skipping
12+
the whole test suite.
1813
"""
1914

2015
@functools.wraps(func)
@@ -28,16 +23,69 @@ def wrapper(self, *args, **kwargs):
2823
).group()
2924

3025
tnt_version = pkg_resources.parse_version(self.tnt_version)
31-
sql_support_tnt_version = pkg_resources.parse_version(
32-
SQL_SUPPORT_TNT_VERSION
33-
)
26+
support_version = pkg_resources.parse_version(REQUIRED_TNT_VERSION)
3427

35-
if tnt_version < sql_support_tnt_version:
36-
self.skipTest(
37-
'Tarantool %s does not support SQL' % self.tnt_version
38-
)
28+
if tnt_version < support_version:
29+
self.skipTest('Tarantool %s %s' % (self.tnt_version, msg))
3930

4031
if func.__name__ != 'setUp':
4132
func(self, *args, **kwargs)
4233

4334
return wrapper
35+
36+
37+
def skip_or_run_test_python_major(func, REQUIRED_PYTHON_MAJOR, msg):
38+
"""Decorator to skip or run tests depending on the Python major
39+
version.
40+
41+
Also, it can be used with the 'setUp' method for skipping
42+
the whole test suite.
43+
"""
44+
45+
@functools.wraps(func)
46+
def wrapper(self, *args, **kwargs):
47+
if func.__name__ == 'setUp':
48+
func(self, *args, **kwargs)
49+
50+
major = sys.version_info.major
51+
if major != REQUIRED_PYTHON_MAJOR:
52+
self.skipTest('Python %s connector %s' % (major, msg))
53+
54+
if func.__name__ != 'setUp':
55+
func(self, *args, **kwargs)
56+
57+
return wrapper
58+
59+
60+
def skip_or_run_sql_test(func):
61+
"""Decorator to skip or run SQL-related tests depending on the
62+
tarantool version.
63+
64+
Tarantool supports SQL-related stuff only since 2.0.0 version.
65+
So this decorator should wrap every SQL-related test to skip it if
66+
the tarantool version < 2.0.0 is used for testing.
67+
"""
68+
69+
return skip_or_run_test_tarantool(func, '2.0.0', 'does not support SQL')
70+
71+
72+
def skip_or_run_varbinary_test(func):
73+
"""Decorator to skip or run VARBINARY-related tests depending on
74+
the tarantool version.
75+
76+
Tarantool supports VARBINARY type only since 2.2.1 version.
77+
See https://github.com/tarantool/tarantool/issues/4201
78+
"""
79+
80+
return skip_or_run_test_tarantool(func, '2.2.1',
81+
'does not support VARBINARY type')
82+
83+
84+
def skip_or_run_mp_bin_test(func):
85+
"""Decorator to skip or run mp_bin-related tests depending on
86+
the Python version.
87+
88+
Python 2 connector do not support mp_bin.
89+
"""
90+
91+
return skip_or_run_test_python_major(func, 3, 'does not support mp_bin')

0 commit comments

Comments
 (0)