Release 3.1

.gitignore

@@ -249,3 +249,5 @@ multiColSplay/**
 singleColSplay/**
 sym
 symsEnumsSplay/**
+
+src/pykx/pykx_init.q_

LICENSE.txt

@@ -1,4 +1,7 @@
-This work is dual licensed under "Apache 2.0" and the "Software License for q.so", users are required to abide by the terms of both licenses in their entirety. The terms of these licenses are included below
+All files contained within this repository are not covered by a single license. The following outlines the differences.
+
+1. All files and folders contained within the source code directory 'src/pykx/q.so/' are licensed under the terms of the 'Software License for q.so' which are included below
+2. All other files within this repository are licensed under the "Apache 2.0" license include below
 
 ***********************************************************************************
                                   Apache 2.0

README.md

@@ -95,24 +95,26 @@ PyKX depends on the following third-party Python packages:
 
 - `pandas>=1.2, <2.0; python_version=='3.8'`
 - `pandas>=1.2, <=2.2.3; python_version>'3.8'`
-- `numpy~=1.22, <2.0; python_version<'3.11'`
-- `numpy~=1.23, <2.0; python_version=='3.11'`
-- `numpy~=1.26, <2.0; python_version=='3.12'`
+- `numpy~=1.22; python_version<'3.11'`
+- `numpy~=1.23; python_version=='3.11'`
+- `numpy~=1.26; python_version>'3.11'`
 - `pytz>=2022.1`
 - `toml~=0.10.2`
 - `dill>=0.2.0`
 - `requests>=2.25.0`
 
 They are installed automatically by `pip` when PyKX is installed.
 
-PyKX also has an optional Python dependency of `pyarrow>=3.0.0`, which can be included by installing the `pyarrow` extra, e.g. `pip install pykx[pyarrow]`
+PyKX also has an optional Python dependency of `pyarrow>=3.0.0, <19.0.0`, which can be included by installing the `pyarrow` extra, e.g. `pip install pykx[pyarrow]`
 
 When using PyKX with KX Dashboards users will be required to install `ast2json~=0.3` this can be installed using the `dashboards` extra, e.g. `pip install pykx[dashboards]`
 
 When using PyKX Streaming users may require the ability to stop processes initialized in a now unavailable process to facilitate this PyKX can make use of `psutil` this can be installed using the `streaming` extra, e.g. `pip install pykx[streaming]`
 
 When using Streamlit users will be required to install `streamlit~=1.28` this can be installed using the `streamlit` extra, e.g. `pip install pykx[streamlit]`
 
+When attempting to convert data to/from PyTorch users will be required to install `torch>2.1` this can be installed using the `torch` extra, e.g. `pip install pykx[torch]`
+
 **Warning:** Trying to use the `pa` conversion methods of `pykx.K` objects or the `pykx.toq.from_arrow` method when PyArrow is not installed (or could not be imported without error) will raise a `pykx.PyArrowUnavailable` exception.
 
 #### Optional Non-Python Dependencies

conda-recipe/conda_build_config.yaml

@@ -4,3 +4,4 @@ python:
     - 3.10
     - 3.11
     - 3.12
+    - 3.13

conda-recipe/meta.yaml

@@ -18,18 +18,25 @@ requirements:
     - python
     - setuptools>=68.0
     - setuptools_scm[toml]>=8.0.0
-    - cython==3.0.0
-    - numpy==1.26  # [py==312]
-    - numpy==1.22.*  # [py<312]
+    - cython==3.0.*
+    - numpy==2.*  # [py==313]
+    - numpy==2.0.*  # [py==312]
+    - numpy==2.0.*  # [py==311]
+    - numpy==2.0.*  # [py==310]
+    - numpy==2.0.*  # [py==39]
+    - numpy==1.22.*  # [py==38]
+    - numpy==1.20.*  # [py==37]
     - tomli>=2.0.1
     - wheel>=0.36
     - sysroot_linux-64 # [linux64]
 
   run:
     - python
-    - numpy>=1.22,<2.0
+    - numpy>=1.20  # [py==37]
+    - numpy>=1.22 # [py>37]
     - pandas>=1.2, <=2.2.3 # [py>38]
     - pandas<2.0 # [py==38]
+    - pyarrow>=3.0.0, <19.0.0
     - pytz>=2022.1
     - toml>=0.10.2
     - dill>=0.2.0

docs/api/pykx-execution/q.md

@@ -112,7 +112,7 @@ pykx.List(pykx.q('
 
 Restricted evaluation of a parse tree.
 
-Behaves similar to [`eval`](#eval) except the evaluation is blocked from modifying values or global state.
+Behaves similar to [`eval`](#eval) except the evaluation is blocked from modifying state  for any handle context other than 0.
 
 ```python
 >>> pykx.q.reval(pykx.q.parse(b'til 10'))

docs/beta-features/index.md

@@ -1,8 +1,13 @@
+---
+title: PyKX Beta Features
+description: PyKX features in beta status
+date: January 2025
+author: KX Systems, Inc.,
+tags: PyKX, beta features, 
+---
 # Beta Features
 
-!!! "Note"
-
-	There are currently no active features in beta status, the following page outlines broadly the concept of beta features within PyKX and how it is managed today
+_This page provides an overview of PyKX Beta Features, including what they are, how to enable them, and what features are available._
 
 ## What is a Beta Feature?
 
@@ -12,14 +17,14 @@ Feedback on Beta Feature development is incredibly helpful and helps to determin
 
 ## How do I enable Beta Features?
 
-Within PyKX beta features are enabled through the use of a configuration/environment variable `PYKX_BETA_FEATURES`, within a Python session users can set this prior to importing PyKX as shown below, note that when enabled you will be able to see what features are in beta through access of `kx.beta_features`:
+Enable PyKX beta features using the `#!python PYKX_BETA_FEATURES` configuration/environment variable. Set this before importing PyKX in a Python session, as shown below, to view available beta features through `#!python kx.beta_features`:
 
 ```python
 >>> import os
 >>> os.environ['PYKX_BETA_FEATURES'] = 'True'
 >>> import pykx as kx
 >>> kx.beta_features
-[]
+['PyTorch Conversions']
 ```
 
 Alternatively you can set beta features to be available at all times by adding `PYKX_BETA_FEATURES` to your `.pykx-config` file as outlined [here](../user-guide/configuration.md#configuration-file). An example of a configuration making use of this is as follows:
@@ -34,12 +39,12 @@ PYKX_BETA_FEATURES='true'
 
 ## What Beta Features are available?
 
-As mentioned above the list of available features to a user is contained within the `beta_features` property, for users with these features available you can get access to this information as follows within a Python session
+As mentioned above, the `beta_features` property contains the list of available features. You can retrieve this information in a Python session as follows:
 
 ```python
 >>> import pykx as kx
 >>> kx.beta_features
-[]
+['PyTorch Conversions']
 ```
 
-There are currently no active features in beta status. This page will be updated when new beta features are added at a future point in time.
+1. [`PyTorch Conversions`](torch.md): Allow users to convert numeric type PyKX vectors and N-Dimensional lists to PyTorch Tensor objects.

docs/beta-features/torch.md

@@ -0,0 +1,110 @@
+---
+title: PyTorch Conversions
+description: PyTorch Tensor Conversions beta feature in PyKX
+date: January 2025
+author: KX Systems, Inc.,
+tags: PyKX,  PyTorch Tensor
+---
+# PyTorch Conversions
+
+_This page provides an overview of PyTorch Conversions, a beta feature in PyKX._
+
+!!! Warning
+
+	This functionality is provided as a Beta Feature and is subject to change. To enable this functionality for testing please follow the configuration instructions [here](../user-guide/configuration.md) setting `PYKX_BETA_FEATURES='true'`.
+
+## Introduction
+
+Commonly used in the development of complex machine learning algorithms, PyTorch is a machine learning library based on the Torch library and is used in applications such as computer vision and natural language processing. Originally developed by Meta AI it is now widely used in the open-source community for algorithm development.
+
+This beta feature allows PyKX users to convert PyKX Vector/List objects into their PyTorch [Tensor](https://pytorch.org/docs/stable/tensors.html) equivalents.
+
+## Requirements and limitations
+
+
+Before you run this functionality, first you must install `torch>2.1` in your local Python session, by using the following command:
+
+```bash
+pip install pykx[torch]
+```
+
+## Functional walkthrough
+
+This walkthrough demonstrates the following steps:
+
+1. Convert a PyKX Vector object to a Tensor object.
+1. Convert a PyKX List object to a Tensor object.
+1. Convert a Tensor object to a PyKX equivalent object.
+
+### Vector to Tensor
+
+Use the `*.pt()` methods to convert PyKX numeric data representations to Tensor objects. In the example below we convert PyKX numeric types to their PyTorch Tensor equivalents:
+
+```python
+>>> import os
+>>> os.environ['PYKX_BETA_FEATURES'] = 'True'
+>>> import pykx as kx
+>>> svec = kx.q('1 2 3h')
+>>> lvec = kx.q('1 2 3j')
+>>> rvec = kx.q('1 2 3e')
+>>> fvec = kx.q('1 2 3f')
+>>> svec.pt()
+tensor([1, 2, 3], dtype=torch.int16)
+>>> lvec.pt()
+tensor([1, 2, 3])
+>>> rvec.pt()
+tensor([1., 2., 3.])
+>>> fvec.pt()
+tensor([1., 2., 3.], dtype=torch.float64)
+```
+
+In particular note in the above that the data types are converted to their Tensor size equivalent.
+
+### List to Tensor
+
+To convert PyKX List objects to Tensors, two criteria must be met:
+
+1. The `#!python pykx.List` contains only data of a single type.
+1. The `#!python pykx.List` is an N-Dimensional regularly shaped/rectangular structure.
+
+By default conversions to a `#!python torch.Tensor` object test for these criteria and it throws an error if they are not met as follows:
+
+```python
+>>> import os
+>>> os.environ['PYKX_BETA_FEATURES'] = 'True'
+>>> import pykx as kx
+>>> kx.q('(1 2;2 3f)').pt()
+TypeError: Data must be a singular type "rectangular" matrix
+```
+
+A working example of this is as follows:
+
+```python
+>>> kx.q('100 100 100#1000000?1f').pt()
+tensor([[[0.3928, 0.5171, 0.5160,  ..., 0.3410, 0.8618, 0.5549],
+         [0.0617, 0.2858, 0.6685,  ..., 0.9234, 0.4016, 0.5619],
+         [0.7249, 0.8112, 0.2087,  ..., 0.3187, 0.1873, 0.8416],
+         ...,
+       dtype=torch.float64)
+```
+
+Having to pre-compute the shape of the data can slow down the processing of large matrices. To avoid this, if you already know the final shape of the tensor, you can specify it using the `#!python reshape` keyword in advance.
+
+```python
+>>> kx.q('100 100 100#1000000?1f').pt(reshape=[100, 100, 100])
+tensor([[[0.3928, 0.5171, 0.5160,  ..., 0.3410, 0.8618, 0.5549],
+         [0.0617, 0.2858, 0.6685,  ..., 0.9234, 0.4016, 0.5619],
+         [0.7249, 0.8112, 0.2087,  ..., 0.3187, 0.1873, 0.8416],
+         ...,
+       dtype=torch.float64)
+```
+
+While not clear from the above for particularly complex nested `#!python pykx.List` objects setting the data shape can provide significant performance boosts:
+
+```python
+lst = kx.q('100 100 100 100#100000000?1f')
+%timeit lst.pt()
+# 1.22 s ± 24.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
+%timeit lst.pt(reshape=[100, 100, 100, 100])
+# 265 ms ± 4.96 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
+```

docs/examples/jupyter-integration.ipynb

@@ -19,7 +19,9 @@
     "1. [Execute against Embedded q](#3-execute-against-embedded-q)\n",
     "1. [SQL interface](#4-sql-interface)\n",
     "1. [q namespaces](#5-q-namespaces)\n",
-    "1. [(Advanced) q over IPC](#6-advanced-q-over-ipc)"
+    "1. [q over IPC](#6-q-over-ipc)\n",
+    "1. [q first mode](#7-q-first-mode)\n",
+    "1. [Saving code blocks](#8-saving-code-blocks)"
    ]
   },
   {
@@ -157,94 +159,6 @@
     "([] a: 1 2 3)"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "cda0d38a",
-   "metadata": {},
-   "source": [
-    "####  Executing against an external q process over IPC\n",
-    "\n",
-    "Connection information can also be included after the `%%q` to connect to a remote `q` process over\n",
-    "IPC.\n",
-    "\n",
-    "Here is the list of currently supported connection parameters.\n",
-    "If they specify a type a second value is expected to follow them to be used as the parameter.\n",
-    "If no type follows them they can be used as a stand alone flag.\n",
-    "\n",
-    "```\n",
-    "--host: A string object denoting the host to connect to\n",
-    "--port: An int object denoting the port to connect over\n",
-    "--user: A str object denoting the username to use when connecting\n",
-    "--password: A str object denoting the password to use when connecting\n",
-    "--timeout: A float object denoting the time in seconds before the query\n",
-    "      times out, defaults to no timeout\n",
-    "--nolarge: Disable messages over 2GB being sent / received\n",
-    "--tls: Use a tls connection\n",
-    "--unix: Use a unix connection\n",
-    "--reconnection_attempts: An int object denoting how many\n",
-    "      reconnection attempts to make\n",
-    "--noctx: Disable the context interface\n",
-    "```\n",
-    "\n",
-    "Connect to a q server running on `localhost` at port `5001` as `user` using password `password`\n",
-    "and disable the context interface."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "1faca1e1",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%%q --host localhost --port 5000 --user user --pass password --noctx\n",
-    "til 10"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "f046ebb6",
-   "metadata": {},
-   "source": [
-    "All connection arguments are optional with the exception of the `--port` argument. If `--host` is not provided `localhost` will be used as the default host."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "615d7d2e",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%%q --port 5000\n",
-    "tab:([]a:1000?1000; b:1000?500.0; c:1000?`AAPL`MSFT`GOOG);"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "d756f342",
-   "metadata": {},
-   "source": [
-    "It is possible to execute `q` code spanning multiple lines."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "c739a80a",
-   "metadata": {
-    "scrolled": true
-   },
-   "outputs": [],
-   "source": [
-    "%%q --port 5000\n",
-    "afunc: {[x; y]\n",
-    "  x + y  \n",
-    "  };\n",
-    "afunc[0; 1]\n",
-    "afunc[2; 3]"
-   ]
-  },
   {
    "cell_type": "markdown",
    "id": "2905895e",
@@ -313,7 +227,7 @@
    "id": "52ca850e",
    "metadata": {},
    "source": [
-    "##  6. (Advanced) q over IPC\n",
+    "##  6. q over IPC\n",
     "\n",
     "After `%%q` you can include connection information, if you wish to connect to a remote `q` process over IPC. \n",
     "\n",
@@ -386,7 +300,7 @@
    "source": [
     "%%q --port 5000\n",
     "afunc: {[x; y]\n",
-    "  x + y  \n",
+    "  x + y\n",
     "  };\n",
     "afunc[0; 1]\n",
     "afunc[2; 3]"
@@ -408,7 +322,7 @@
    "id": "607997ac-e7d7-4cc9-a06f-aa1cd3d742ce",
    "metadata": {},
    "source": [
-    "#### q first mode\n",
+    "##  7. q first mode\n",
     "q first mode can be enabled by importing PyKX after setting the environment variable `PYKX_JUPYTERQ` to `true`, or at runtime use:"
    ]
   },
@@ -533,7 +447,7 @@
    "id": "080dd085-e54f-478c-a305-eac9f23db020",
    "metadata": {},
    "source": [
-    "#### Saving code blocks\n",
+    "##  8. Saving code blocks\n",
     "The `--save` feature allows user to save code in a cell as a q file.\n",
     "\n",
     "To use this feature, include `--save` followed by the `path` of the file.\n",