Skip to content

Commit 35531cb

Browse files
committed
merge master (after merging onnx upgrade to 1.9)
2 parents 4e9c1a0 + b04ec83 commit 35531cb

File tree

8 files changed

+52
-43
lines changed

8 files changed

+52
-43
lines changed

.circleci/config.yml

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@ commands:
101101
- checkout-all
102102
- restore_cache:
103103
keys:
104-
- v1.2.5-dependencies-{{ checksum "get_deps.sh" }}-cpu
104+
- v1.2.5-deps-{{ checksum "get_deps.sh" }}-cpu
105105
# If no exact match is found will get dependencies from source
106106
- setup-build-system
107107
- run:
@@ -112,7 +112,7 @@ commands:
112112
- save_cache:
113113
paths:
114114
- deps
115-
key: v1.2.5-dependencies-{{ checksum "get_deps.sh" }}-cpu
115+
key: v1.2.5-deps-{{ checksum "get_deps.sh" }}-cpu
116116
- run:
117117
name: Build
118118
command: make -C opt all SHOW=1
@@ -151,7 +151,7 @@ commands:
151151
- checkout-all
152152
- restore_cache:
153153
keys:
154-
- v1-dependencies-{{ checksum "get_deps.sh" }}-cpu
154+
- v1.2.5-deps-{{ checksum "get_deps.sh" }}-cpu
155155
# If no exact match is found will get dependencies from source
156156
- setup-build-system
157157
- run:
@@ -178,7 +178,7 @@ commands:
178178
- checkout-all
179179
- restore_cache:
180180
keys:
181-
- v1.2.5-dependencies-{{ checksum "get_deps.sh" }}-gpu
181+
- v1.2.5-deps-{{ checksum "get_deps.sh" }}-gpu
182182
- relocate-docker-storage
183183
- run:
184184
name: Build
@@ -189,7 +189,7 @@ commands:
189189
- save_cache:
190190
paths:
191191
- deps
192-
key: v1.2.5-dependencies-{{ checksum "get_deps.sh" }}-gpu
192+
key: v1.2.5-deps-{{ checksum "get_deps.sh" }}-gpu
193193
- run:
194194
name: Test
195195
command: |
@@ -239,7 +239,7 @@ jobs:
239239
- checkout-all
240240
- restore_cache:
241241
keys:
242-
- v1.2.5-dependencies-{{ checksum "get_deps.sh" }}-<<parameters.osnick>>-<<parameters.target>>
242+
- v1-dependencies-{{ checksum "get_deps.sh" }}-<<parameters.osnick>>-<<parameters.target>>
243243
- setup-automation
244244

245245
# since we run in parallel, we need to generate docker files with different suffixes hence the DOCKER_SUFFIX
@@ -260,7 +260,7 @@ jobs:
260260
- save_cache:
261261
paths:
262262
- deps
263-
key: v1.2.5-dependencies-{{ checksum "get_deps.sh" }}-<<parameters.osnick>>-<<parameters.target>>
263+
key: v1.2.5-deps-{{ checksum "get_deps.sh" }}-<<parameters.osnick>>-<<parameters.target>>
264264
- persist_to_workspace:
265265
root: bin/
266266
paths:
@@ -275,7 +275,7 @@ jobs:
275275
- checkout-all
276276
- restore_cache:
277277
keys:
278-
- v1.2.5-dependencies-{{ checksum "get_deps.sh" }}-cpu
278+
- v1.2.5-deps-{{ checksum "get_deps.sh" }}-cpu
279279
# If no exact match is found will get dependencies from source
280280
- setup-build-system
281281
- run:
@@ -309,7 +309,7 @@ jobs:
309309
- checkout-all
310310
- restore_cache:
311311
keys:
312-
- v1.2.5-dependencies-{{ checksum "get_deps.sh" }}-cpu
312+
- v1.2.5-deps-{{ checksum "get_deps.sh" }}-cpu
313313
# If no exact match is found will get dependencies from source
314314
- setup-build-system
315315
- run:

docs/developer-backends.md

Lines changed: 13 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -2,59 +2,46 @@
22

33
This document describes how a backend for RedisAI can be built, from this repository. It highlights the supported compilation devices on a per-backend basis, and highlights the tools and commands required. Unless indicated otherwise, a backend is compiled in a docker, which is responsible for the configuration and installation of all tools required for a given backend on a per-platform basis.
44

5-
To follow these instructions, this repository must be cloned with all of its submodules (i.e *git clone --recursive https://github.com/RedisLabsModules/redisai*)
5+
To follow these instructions, this repository must be cloned with all of its submodules (i.e *git clone --recursive https://github.com/redisai/redisai*)
66

7-
GNU Make is used as a runner for the dockerfile generator. Python is the language used for the generator script, and jinja is the templating library used to create the docker file from the template.
7+
GNU Make is used as a runner for the dockerfile generator. Python is the language used for the generator script, and jinja is the templating library used to create the docker file from a template *dockerfile.tmpl* that can be found in the directory of a given backend listed below.
88

99
## Tools
1010

11-
Buiding the backends requires installation of the following tools:
11+
Building the backends requires installation of the following tools:
1212

1313
1. gnu make
1414
1. python (3.0 or higher)
1515
1. docker
16-
1. jinja2 jinja is used to generate the platform dockerfile from a *dockerfile.tmpl* that can be found in the directory of a given backend listed below.
16+
1. jinja2
1717

18-
On ubuntu bionic these can be installed by running:
18+
On ubuntu bionic these can be installed by running the following steps, to install python3, create a virtual environment, and install the jinja templating dependency. Replace */path/to/venv* with your desired virtualenv location.
1919

20-
* sudo apt install python3 python3-dev make docker
21-
* pip install --user jinja
20+
```
21+
sudo apt install python3 python3-dev make docker
22+
python3 -m venv /path/to/venv
23+
source /path/to/venv/bin/activate
24+
pip install jinja
25+
```
2226

2327
-------
2428

2529
## Backends
2630

2731
### onnxruntime
2832

33+
We build Onnxruntime library with DISABLE_EXTERNAL_INITIALIZERS=ON build flag. This means that loading ONNX models that use external files to store the initial (usually very large) values of the model's operations, is invalid. That is, initializers values must be part of the serialized model, which is also the standard use case.
34+
2935
**Compilation target devices:**
3036

3137
1. x86\_64 bit linux systems
3238

3339
1. x86\_64 bit linux systems with a GPU
3440

35-
1. jetson devices
36-
3741
**Directory:** opt/build/onnxruntime
3842

3943
**Build options:**
4044

4145
1. To build run *make*
4246

4347
1. To build with GPU support on x86\_64 run *make GPU=1*
44-
45-
1. Should you want to build multiple targets from a shared directory, run *make DOCKER_SUFFIX=<yoursuffix>* on your target system. For example, if building on an arm and x64 workload, from a shared directory run:
46-
* On x86: make DOCKER\_SUFFIX=x86\_64
47-
48-
* On arm: make DOCKER\_SUFFIX=arm
49-
50-
### tensorflow
51-
52-
**Compilation target devices:**
53-
54-
1. x86\_64 bit linux system
55-
56-
**Directory:** opt/build/tensorflow
57-
58-
**Build options:**
59-
60-
1. To build run *make*

opt/build/backends.rules

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,10 @@
1+
# PRODUCT var is the backend name, without any other configuration (set in each backend's corresponding makefile)
2+
13
# default is x64
24
ARCH=$(shell ${READIES}/bin/platform --arch)
35
OS=$(shell ${READIES}/bin/platform --os)
46
REDIS_CMAKE_ARCH=x86_64
7+
DOCKER_ORG=redislabs
58

69
# jetson
710
ifeq ($(shell test -e /usr/share/doc/nvidia-l4t-jetson-io && echo -n yes),yes)
@@ -46,3 +49,17 @@ build:
4649

4750
publish:
4851
@aws s3 cp ${BACKEND_NAME} s3://$(S3_URL)/ --acl public-read
52+
53+
# ---------------------------------------------------------------------------------------------------
54+
define HELP
55+
make build> # build the backend, tagging the docker image so that the files can be copied out of it
56+
OSNICK=<> # optional base operating system (xenial, bionic, etc)
57+
REDIS_CUDA_VERSION=<> # optional cuda version to override
58+
DOCKER_SUFFIX=<> # optional suffix for the generated dockerfile
59+
GPU=1 # if set, build the GPU
60+
make publish > # upload the generated artifacts to s3 same
61+
GPU=1 # if set, upload the GPU artifact (defaults to cpu)
62+
endef
63+
# ---------------------------------------------------------------------------------------------------
64+
include ${READIES}/mk/help.defs
65+
include ${READIES}/mk/help.rules

opt/build/dockerparts/apt.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,10 +9,10 @@ RUN add-apt-repository ppa:ubuntu-toolchain-r/test
99
RUN apt-get update -qq
1010
RUN DEBIAN_NONINTERACTIVE=1 apt-get install -y curl wget tar git patch \
1111
build-essential libcurl4-openssl-dev libssl-dev libatlas-base-dev zlib1g-dev \
12-
python3.6 python3-pip python3-dev python3-numpy \
12+
python3.7 python3-pip python3-dev python3-numpy \
1313
gcc-7 g++-7
1414

15-
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.6 60
15+
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 60
1616
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 60 --slave /usr/bin/g++ g++ /usr/bin/g++-7
1717

1818
RUN python3 -m pip install --upgrade pip setuptools wheel

opt/build/onnxruntime/Makefile

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,18 +2,17 @@ REDIS_ONNX_VERSION?=1.9.0
22
REDIS_ONNX_REPO?=https://github.com/microsoft/onnxruntime
33

44
PRODUCT=onnxruntime
5-
DOCKER_ORG=redislabs
65
VERSION=${REDIS_ONNX_VERSION}
76
REDIS_CUDA_VERSION=11.0-cudnn8
87

8+
# anything prefixed with REDIS, and exported is available in the docker build
99
export REDIS_ONNX_VERSION
1010
export REDIS_ONNX_REPO
1111
export REDIS_CUDA_VERSION
1212

1313
OSNICK=xenial
1414

1515
ROOT=.
16-
1716
READIES=${ROOT}/../../readies
1817

1918
include ../backends.rules

opt/build/onnxruntime/dockerfile.tmpl

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,8 +30,8 @@ ARG ONNXRUNTIME_VER={{REDIS_ONNX_VERSION}}
3030
# build
3131
WORKDIR /build
3232
{% if REDIS_GPU is defined %}
33-
{% set BUILDTYPE="MinSizeRel" %}
34-
{% set BUILDARGS="--use_cuda --cudnn_home /usr/local/cuda --cuda_home /usr/local/cuda" %}
33+
{% set BUILDTYPE="Release" %}
34+
{% set BUILDARGS="--use_cuda --cudnn_home /usr/local/cuda --cuda_home /usr/local/cuda --cmake_extra_defines onnxruntime_DISABLE_EXTERNAL_INITIALIZERS=ON"%}
3535
{% else %}
3636
{% set BUILDTYPE="Release" %}
3737
{% set BUILDARGS="--cmake_extra_defines onnxruntime_DISABLE_EXTERNAL_INITIALIZERS=ON" %}

opt/redis_valgrind.sup

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,13 @@
2626
obj:*/libtorch_cpu.so*
2727
}
2828

29+
{
30+
ignore_unversioned_libs
31+
Memcheck:Overlap
32+
...
33+
obj:*/libonnxruntime.so*
34+
}
35+
2936
{
3037
ignore_unversioned_libs
3138
Memcheck:Leak

tests/flow/tests_onnx.py

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -435,8 +435,7 @@ def test_forbidden_external_initializers(env):
435435
if not TEST_ONNX:
436436
env.debugPrint("skipping {} since TEST_ONNX=0".format(sys._getframe().f_code.co_name), force=True)
437437
return
438-
if DEVICE != 'CPU':
439-
return
438+
440439
con = get_connection(env, '{1}')
441440

442441
# move the external initializer to the redis' current dir (tests/flow/logs)

0 commit comments

Comments
 (0)