Skip to content

Commit 8a1bb48

Browse files
xia-mcxia-mc
committed
Spilt Unsafe pyi
Add Benchmark asm Add more apis for ASM Add more apis for Unsafe
1 parent 6dc0311 commit 8a1bb48

24 files changed

Lines changed: 19924 additions & 11735 deletions

‎.idea/PyFastUtil.iml‎

Lines changed: 9 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎.idea/inspectionProfiles/Project_Default.xml‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎README.md‎

Lines changed: 132 additions & 92 deletions
Original file line numberDiff line numberDiff line change
@@ -12,182 +12,222 @@ English | [简体中文](./README_CN.md)
1212

1313
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
1414
[![Issues](https://img.shields.io/github/issues/xia-mc/PyFastUtil)](https://img.shields.io/github/issues/xia-mc/PyFastUtil)
15-
![Version](https://img.shields.io/badge/CPython-3.9_or_later-blue)
15+
![Version](https://img.shields.io/badge/CPython-3.9_and_later-blue)
1616
[![Build](https://img.shields.io/github/actions/workflow/status/xia-mc/PyFastUtil/python-package.yml)](https://github.com/xia-mc/PyFastUtil/actions)
1717

1818
</div>
1919

2020
## Introduction
2121

22-
**PyFastUtil** is a high-performance utility library for Python, inspired by the popular [FastUtil](https://fastutil.di.unimi.it/) library in Java. However, **PyFastUtil is not a Python binding of FastUtil library**, but a **complete re-implementation** from scratch, designed to bring the same efficiency and functionality to Python.
23-
24-
> **Note**: PyFastUtil is still in its **early development phase**. The codebase is under active development and is not yet ready for production use.
25-
> We are working hard to make it feature-complete and thoroughly tested before releasing it on PyPI.
22+
**PyFastUtil** is a high-performance utility library for Python, inspired by Java's [FastUtil](https://fastutil.di.unimi.it/) library. However, **PyFastUtil is NOT a Python binding for FastUtil**. It is a library built from the ground up, designed to bring C-like efficiency and functionality to CPython.
2623

2724
### Features
2825

29-
- Implements all corresponding Python data structure interfaces, while providing significant performance improvements through targeted optimizations. Users can choose the most suitable data structure for their specific needs.
30-
- Fully implemented in C/C++, leveraging hardware-specific optimizations such as SIMD to maximize the performance of data structures.
31-
- Offers efficient Python bindings for certain C APIs, allowing advanced users to perform low-level, "unsafe" operations when needed.
26+
- Implements all corresponding Python data structure interfaces while significantly improving performance through targeted optimizations. Users can choose the most suitable data structure type based on their needs.
27+
- Fully implemented in C/C++ with hardware-level optimizations such as SIMD, aiming to maximize the performance of data structures.
28+
- Provides efficient Python bindings for some C APIs, allowing advanced users to perform "unsafe" low-level operations.
29+
- Enables inline assembly, allowing advanced users to dynamically generate, invoke, and destroy C functions at runtime.
3230

33-
### Performance Benchmark
31+
### Benchmark
3432

35-
> **Note**: For some very fast O(1) operations (such as `pop` and `extend`), PyFastUtil may perform slightly worse than Python's native implementation due to the non-negligible overhead of CPython's C extension calls. We are working on optimizing this.
33+
> **Note**: For extremely fast \(O(1)\) operations (e.g., `pop`, `extend`), PyFastUtil's performance may be slightly inferior to Python's native implementation due to the unavoidable overhead of calling C extensions from CPython. We are actively working to optimize this.
3634
3735
> CPU: AMD Ryzen 7 5700G (AVX2)
3836
>
3937
> Windows 11 23H2, Python 3.12, MSVC 19.41.34120
4038
41-
#### Specialized List Benchmark (e.g., `IntArrayList`)
39+
#### Type-Specialized List (e.g., `IntArrayList`)
4240

4341
```text
42+
Preparing data...
4443
---Python list & IntArrayList Benchmark---
4544
Batch size: 10000
4645
Repeat: 3
4746
48-
Python list sort time: 1.31 ms
49-
PyFastUtil IntArrayList sort time: 0.03 ms
50-
PyFastUtil speed of Python list (sort): 4139.240 %
47+
Python list init time: 0.02 ms
48+
PyFastUtil IntArrayList init time: 0.08 ms
49+
PyFastUtil speed of Python list (init): 29.788 %
50+
51+
Python list copy time: 0.02 ms
52+
PyFastUtil IntArrayList copy time: 0.00 ms
53+
PyFastUtil speed of Python list (copy): 766.102 %
54+
55+
Python list to_python time: 0.02 ms
56+
PyFastUtil IntArrayList to_python time: 0.27 ms
57+
PyFastUtil speed of Python list (to_python): 5.821 %
58+
59+
Python list sequential_access time: 0.00 ms
60+
PyFastUtil IntArrayList sequential_access time: 0.00 ms
61+
PyFastUtil speed of Python list (sequential_access): 126.667 %
62+
63+
Python list random_access time: 0.39 ms
64+
PyFastUtil IntArrayList random_access time: 0.62 ms
65+
PyFastUtil speed of Python list (random_access): 62.423 %
66+
67+
Python list sort time: 1.29 ms
68+
PyFastUtil IntArrayList sort time: 0.04 ms
69+
PyFastUtil speed of Python list (sort): 3344.483 %
5170
52-
Python list append time: 0.44 ms
53-
PyFastUtil IntArrayList append time: 0.43 ms
54-
PyFastUtil speed of Python list (append): 102.314 %
71+
Python list append time: 0.23 ms
72+
PyFastUtil IntArrayList append time: 0.32 ms
73+
PyFastUtil speed of Python list (append): 72.517 %
5574
56-
Python list insert time: 71.75 ms
57-
PyFastUtil IntArrayList insert time: 10.68 ms
58-
PyFastUtil speed of Python list (insert): 671.948 %
75+
Python list insert time: 70.74 ms
76+
PyFastUtil IntArrayList insert time: 10.93 ms
77+
PyFastUtil speed of Python list (insert): 647.349 %
5978
60-
Python list pop time: 0.34 ms
61-
PyFastUtil IntArrayList pop time: 0.40 ms
62-
PyFastUtil speed of Python list (pop): 85.341 %
79+
Python list pop time: 0.29 ms
80+
PyFastUtil IntArrayList pop time: 0.35 ms
81+
PyFastUtil speed of Python list (pop): 85.143 %
6382
64-
Python list remove time: 5.13 ms
65-
PyFastUtil IntArrayList remove time: 2.55 ms
66-
PyFastUtil speed of Python list (remove): 201.138 %
83+
Python list remove time: 5.50 ms
84+
PyFastUtil IntArrayList remove time: 2.54 ms
85+
PyFastUtil speed of Python list (remove): 216.749 %
6786
68-
Python list contains time: 536.74 ms
69-
PyFastUtil IntArrayList contains time: 15.29 ms
70-
PyFastUtil speed of Python list (contains): 3511.205 %
87+
Python list contains time: 258.02 ms
88+
PyFastUtil IntArrayList contains time: 2.61 ms
89+
PyFastUtil speed of Python list (contains): 9896.599 %
7190
72-
Python list index time: 414.75 ms
73-
PyFastUtil IntArrayList index time: 3.85 ms
74-
PyFastUtil speed of Python list (index): 10764.189 %
91+
Python list index time: 438.12 ms
92+
PyFastUtil IntArrayList index time: 3.77 ms
93+
PyFastUtil speed of Python list (index): 11618.038 %
7594
76-
Python list extend time: 0.09 ms
77-
PyFastUtil IntArrayList extend time: 0.20 ms
78-
PyFastUtil speed of Python list (extend): 44.228 %
95+
Python list extend time: 0.08 ms
96+
PyFastUtil IntArrayList extend time: 0.14 ms
97+
PyFastUtil speed of Python list (extend): 58.433 %
7998
99+
Python list reverse time: 0.00 ms
100+
PyFastUtil IntArrayList reverse time: 0.00 ms
101+
PyFastUtil speed of Python list (reverse): 378.948 %
80102
81-
Avg speed of PyFastUtil compared to Python list: 2439.950 %
103+
104+
Avg speed of PyFastUtil compared to Python list: 1950.647 %
82105
```
83106

84-
#### Generic List Benchmark (e.g., `ObjectArrayList`)
107+
#### Generic Type List (e.g., `ObjectArrayList`)
85108

86109
```text
110+
Preparing data...
87111
---Python list & ObjectArrayList Benchmark---
88112
Batch size: 10000
89113
Repeat: 3
90114
91-
Python list sort time: 125.56 ms
92-
PyFastUtil ObjectArrayList sort time: 116.61 ms
93-
PyFastUtil speed of Python list (sort): 107.668 %
115+
Python list init time: 0.06 ms
116+
PyFastUtil ObjectArrayList init time: 0.05 ms
117+
PyFastUtil speed of Python list (init): 129.484 %
94118
95-
Python list append time: 0.30 ms
96-
PyFastUtil ObjectArrayList append time: 0.35 ms
97-
PyFastUtil speed of Python list (append): 86.724 %
119+
Python list copy time: 0.07 ms
120+
PyFastUtil ObjectArrayList copy time: 0.03 ms
121+
PyFastUtil speed of Python list (copy): 268.376 %
98122
99-
Python list insert time: 81.55 ms
100-
PyFastUtil ObjectArrayList insert time: 23.95 ms
101-
PyFastUtil speed of Python list (insert): 340.471 %
123+
Python list to_python time: 0.03 ms
124+
PyFastUtil ObjectArrayList to_python time: 0.03 ms
125+
PyFastUtil speed of Python list (to_python): 99.325 %
102126
103-
Python list pop time: 0.44 ms
104-
PyFastUtil ObjectArrayList pop time: 0.32 ms
105-
PyFastUtil speed of Python list (pop): 139.226 %
127+
Python list sequential_access time: 0.00 ms
128+
PyFastUtil ObjectArrayList sequential_access time: 0.00 ms
129+
PyFastUtil speed of Python list (sequential_access): 139.130 %
106130
107-
Python list remove time: 5.25 ms
108-
PyFastUtil ObjectArrayList remove time: 5.14 ms
109-
PyFastUtil speed of Python list (remove): 102.170 %
131+
Python list random_access time: 0.22 ms
132+
PyFastUtil ObjectArrayList random_access time: 0.31 ms
133+
PyFastUtil speed of Python list (random_access): 71.053 %
110134
111-
Python list contains time: 1140.28 ms
112-
PyFastUtil ObjectArrayList contains time: 16.37 ms
113-
PyFastUtil speed of Python list (contains): 6966.576 %
135+
Python list sort time: 1039.91 ms
136+
PyFastUtil ObjectArrayList sort time: 1003.55 ms
137+
PyFastUtil speed of Python list (sort): 103.623 %
114138
115-
Python list index time: 866.88 ms
116-
PyFastUtil ObjectArrayList index time: 6.77 ms
117-
PyFastUtil speed of Python list (index): 12803.132 %
139+
Python list append time: 0.27 ms
140+
PyFastUtil ObjectArrayList append time: 0.33 ms
141+
PyFastUtil speed of Python list (append): 81.091 %
118142
119-
Python list extend time: 0.12 ms
120-
PyFastUtil ObjectArrayList extend time: 0.15 ms
121-
PyFastUtil speed of Python list (extend): 80.975 %
143+
Python list insert time: 72.43 ms
144+
PyFastUtil ObjectArrayList insert time: 20.82 ms
145+
PyFastUtil speed of Python list (insert): 347.893 %
122146
147+
Python list pop time: 0.36 ms
148+
PyFastUtil ObjectArrayList pop time: 0.27 ms
149+
PyFastUtil speed of Python list (pop): 130.323 %
123150
124-
Avg speed of PyFastUtil compared to Python list: 2578.368 %
125-
```
151+
Python list remove time: 5.17 ms
152+
PyFastUtil ObjectArrayList remove time: 5.17 ms
153+
PyFastUtil speed of Python list (remove): 100.102 %
126154
127-
## Installation
155+
Python list contains time: 665.64 ms
156+
PyFastUtil ObjectArrayList contains time: 3.77 ms
157+
PyFastUtil speed of Python list (contains): 17634.553 %
128158
129-
Currently, **PyFastUtil** is not officially available on PyPI. However, you can install it from **Test PyPI** or build it from the source code.
159+
Python list index time: 985.46 ms
160+
PyFastUtil ObjectArrayList index time: 6.80 ms
161+
PyFastUtil speed of Python list (index): 14501.919 %
130162
131-
### Option 1: Install from Test PyPI
163+
Python list extend time: 0.08 ms
164+
PyFastUtil ObjectArrayList extend time: 0.12 ms
165+
PyFastUtil speed of Python list (extend): 73.264 %
132166
133-
You can install the latest pre-release version of **PyFastUtil** from Test PyPI:
167+
Python list reverse time: 0.00 ms
168+
PyFastUtil ObjectArrayList reverse time: 0.00 ms
169+
PyFastUtil speed of Python list (reverse): 106.000 %
134170
135-
```bash
136-
pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pyfastutil
171+
172+
Avg speed of PyFastUtil compared to Python list: 2413.296 %
137173
```
138174

139-
> **Note**: Test PyPI is a separate package repository used for testing purposes. Packages hosted there may not be as stable as those on the official PyPI.
175+
## Installation
140176

141-
---
177+
### Install PyFastUtil from PyPI:
178+
```shell
179+
pip install PyFastUtil
180+
```
142181

143-
### Option 2: Build from source
182+
### Or, build from source
144183

145-
If you'd prefer to build the project from the source code, follow these steps:
184+
If you'd like to build the project from source, follow these steps:
146185

147186
1. Clone the repository:
148-
```bash
187+
```shell
149188
git clone https://github.com/yourusername/PyFastUtil.git
150189
cd PyFastUtil
151190
```
152191

153192
2. Build the project:
154193
- On **Windows**:
155-
```bash
156-
build.cmd
194+
```shell
195+
./build.cmd
157196
```
158197
- On **Linux/macOS**:
159-
```bash
198+
```shell
160199
./build.sh
161200
```
162201

163-
> **Note**: Future releases will be available on the official PyPI repository once the project is feature-complete and well-tested.
202+
> **Note**: The project will be officially released on PyPI once all features are complete and thoroughly tested.
164203

165204
## License
166205

167-
This project is licensed under the **Apache License 2.0**. See the [LICENSE](LICENSE) file for more details.
206+
This project is licensed under the **Apache License 2.0**. For more details, see the [LICENSE](LICENSE) file.
168207

169208
## Roadmap
170209

171-
- [ ] Implement core data structures (e.g., fast lists, maps, sets).
172-
- [x] Numpy support.
173-
- [ ] Provide SIMD and inline assembly bindings for Python.
174-
- [ ] Thorough testing and benchmarking.
175-
- [ ] PyPI release.
210+
- [ ] Implement `int`, `float`, and `double` ArrayList and LinkedList.
211+
- [x] Add Numpy support.
212+
- [x] Provide bindings for SIMD utility functions.
213+
- [x] Provide raw AVX512 bindings.
214+
- [ ] Perform comprehensive testing and benchmarking.
215+
- [x] Publish to PyPI.
176216

177-
## Contributing
217+
## Contribution
178218

179-
Contributions are welcome! Feel free to submit issues or pull requests. Please note that the project is in an early stage, and we appreciate any feedback or suggestions.
219+
Contributions are welcome! Feel free to submit issues or pull requests. Please note that the project is in its early stages, and we greatly appreciate any feedback or suggestions.
180220

181221
## Acknowledgements
182222

183-
This project includes code from the following amazing open-source projects:
223+
This project is partially based on the following excellent open-source projects:
184224

185-
- [CPython](https://github.com/python/cpython): The official Python interpreter, licensed under the [Python Software Foundation License](https://docs.python.org/3/license.html).
186-
- [C++ Standard Library (STL)](https://en.cppreference.com/w/cpp): The C++ Standard Library, which provides essential data structures, algorithms, and utilities, licensed under the [ISO C++ Standard](https://isocpp.org/).
225+
- [CPython](https://github.com/python/cpython): The official implementation of the Python interpreter, licensed under the [Python Software Foundation License](https://docs.python.org/3/license.html).
226+
- [C++ Standard Library (STL)](https://en.cppreference.com/w/cpp): The standard library for C++, providing essential data structures, algorithms, and utilities, licensed under the [ISO C++ Standard](https://isocpp.org/).
187227
- [cpp-TimSort](https://github.com/timsort/cpp-TimSort): A C++ implementation of the TimSort algorithm, licensed under the [MIT License](https://github.com/timsort/cpp-TimSort/blob/master/LICENSE).
188-
- [ankerl::unordered_dense](https://github.com/martinus/unordered_dense): A fast and memory-efficient hash map implementation for modern C++, licensed under the [MIT License](https://github.com/martinus/unordered_dense/blob/main/LICENSE).
189-
- [qReverse](https://github.com/Wunkolo/qreverse): A high-performance, architecture-optimized array reversal algorithm, distributed under the [MIT License](https://github.com/martinus/unordered_dense/blob/main/LICENSE).
228+
- [ankerl::unordered_dense](https://github.com/martinus/unordered_dense): A modern, high-performance, low-memory hash table implementation in C++, licensed under the [MIT License](https://github.com/martinus/unordered_dense/blob/main/LICENSE).
229+
- [qReverse](https://github.com/Wunkolo/qreverse): A high-performance, architecture-optimized array reversal algorithm, released under the [MIT License](https://github.com/martinus/unordered_dense/blob/main/LICENSE).
190230

191231
Special thanks to the contributors of these open-source projects!
192232

193-
The "mascot" image is provided by the artist [kokola](https://x.com/kokola10032) and is used with permission. You can find the original image on [X](https://x.com/kokola10032/status/1812480707643506704).
233+
The mascot image of the project was created by the artist [kokola](https://x.com/kokola10032) and is used with permission. You can view the original artwork on [X](https://x.com/kokola10032/status/1812480707643506704).

0 commit comments

Comments
 (0)