Skip to content

developer.md

Latest commit

 

History

History
214 lines (165 loc) · 8.75 KB

developer.md

File metadata and controls

214 lines (165 loc) · 8.75 KB

Developer notes

Cloning the project

git clone --recursive https://github.com/RediSearch/RediSearch.git

Dev container

We have provided a dev container based on ubuntu:latest docker image with all the dependencies (Redis, build systems, rust, Python virtual environment) installed. Just clone the repo and open it in the dev container. It will take a few minutes for the first time to download and install the dependencies. If you would like to build it on your machine without a docker, proceed to the next sections.

Installing prerequisites

Build dependencies

To build and test RediSearch you need to install several packages, depending on the underlying OS. The following OSes are supported and tested in our CI:

  • Ubuntu 18.04
  • Ubuntu 20.04
  • Ubuntu 22.04
  • Ubuntu 24.04
  • Debian linux 11
  • Debian linux 12
  • Rocky linux 8
  • Rocky linux 9
  • Amazon linux 2
  • Amazon linux 2023
  • Mariner 2.0
  • Azure linux 3
  • MacOS
  • Alpine linux 3

For installing the prerequisites you can take the following approaches:

  1. Install the dependencies manually - check our install script at ".install/install_script.sh" for the list of dependencies. In general, the common installations are:

    • rust (latest stable version)

    • cmake >= 3.25.1

    • boost == 1.88.0 you can skip this dependency so our CMake script will have it for you, but this has build time penalty.

    • build-essential (on Debian/Ubuntu) or equivalent build tools on other systems

    • python3 and python3-pip (for running tests)

    • openssl-devel / libssl-dev (for secure connections)

    • Ubuntu 18.04: GCC 10 (not default, installed via PPA)

    • Ubuntu 20.04: GCC 10 (not default, installed via PPA)

    • Ubuntu 22.04: GCC 12 (not default, PPA not required)

    • Ubuntu 24.04: Default GCC is sufficient

    • Debian 11: Default GCC is sufficient

    • Debian 12: Default GCC is sufficient

    • Rocky Linux 8: GCC 13 (not default, installed via gcc-toolset-13-gcc and gcc-toolset-13-gcc-c++)

    • Rocky Linux 9: GCC 13 (not default, installed via gcc-toolset-13-gcc and gcc-toolset-13-gcc-c++)

    • Amazon Linux 2: GCC 11 (not default, installed via Amazon's SCL)

    • Amazon Linux 2023: Default GCC is sufficient

    • Mariner 2.0: Default GCC is sufficient

    • Azure Linux 3: Default GCC is sufficient

    • MacOS: Install clang-18 via brew

    • Alpine Linux 3: Default GCC is sufficient

  • you can find the dependencies in each OS script under the .install directory.

  1. Use our CI installation scripts to install the dependencies - you can find the scripts under the .install directory.

    cd .install
./install_script.sh sudo

    Note that this will install various packages on your system using the native package manager (sudo is not required in a Docker environment and the script will fail calling it so).

Redis

You will not be able to run and test your code without Redis, since you need to load the module. You can build it from source and install it as described in redis GitHub page.

RedisJSON

Some of our behavioral tests require RedisJSON to be present. Our testing framework will clone and build RedisJSON for you.

Building from source

To build RediSearch from source, you need to run the following commands:

make build

Running Redis with RediSearch

To run Redis with RediSearch, you need to load the module. You can do this by running the following command:

make run

Testing

Running the tests

To run the tests, you need to execute the following commands:

make test

For running specific tests you can use the following commands:

  • C tests, located in tests/ctests, run by make c-tests.
  • C++ tests (enabled by GTest), located in tests/cpptests, run by make cpp-tests.
  • Python behavioral tests (enabled by RLTest), located in tests/pytests, run by make pytest.

Test prerequisites

To run the python behavioral tests you need to install the following dependencies:

  • python test requirements listed in pyproject.toml
  • If you want to execute RediSearch+RedisJSON behavioral tests you need to install rust, which is required to compile the RedisJSON module

if you don't want to install those manually, you can use the CI installation scripts to install the dependencies.

.install/test_deps/common_installations.sh

Note those scripts will create a python virtual environment called venv and install the required dependencies in it.

RedisJSON

Some of our behavioral tests require RedisJSON to be present. You can skip the RedisJSON tests by setting the REJSON=0 in the command

make pytest REJSON=0

If you have RedisJSON module already built on your machine you can specify the path to it by setting the REJSON_PATH variable.

make pytest REJSON_PATH=/path/to/redisjson.so

If the module does not exist in the specified path, our testing framework will clone and build RedisJSON for you. You can specify the branch of RedisJSON you want to use by setting the REJSON_BRANCH variable.

make pytest REJSON_BRANCH=branch

Sanitizers

Currently address sanitizer is supported. To run the tests with address sanitizer you can use the following command:

make build test SAN=address

Debug

To build the code with debug symbols, you can use the following commands:

make DEBUG=1

Logging

C code uses RedisModule_Log to log messages while the Rust side uses the standard tracing crate, see below.

The tracing_redismodule crate provides a bridge between the two logging systems. It implements a tracing subscriber emitting traces and logs to the RedisModule logging system. This subscriber is automatically registered when the RediSearch module is loaded by the Redis server.

Logging from Rust

The recommended way for Rust code to emit logs is through tracing since it allows us to produce structured logging output, but also generate performance data through spans. To record events to the redis log you can use the macros provided by [tracing]:

tracing::trace!("This is the most verbose");
tracing::debug!("This is the second most verbose");
tracing::info!("This is the third most verbose");
tracing::warn!("This is the fourth most verbose");
tracing::error!("This is the fifth most verbose");

By default, only error, warn, and info events are emitted but you can set the RUST_LOG environment variable to customize the behaviour of the system:

# enables all verbositity levels from all sources
RUST_LOG=trace 
# enables all verbositity levels from all sources EXCEPT the result_processor crate which is fully disabled.
RUST_LOG=trace,result_processor=off

These directives can be chained and support quite deep configuration. For details, see the tracing_subscriber documentation.

Note, the verbosity levels defined by tracing are NOT the same as the ones used by redis. The mapping is like so:

  • trace => LOGLEVEL_DEBUG
  • debug => LOGLEVEL_VERBOSE
  • info => LOGLEVEL_VERBOSE
  • warn and error => LOGLEVEL_WARNING

By default the log output will be colored to help with reading. The system already attempts to be smart about turning it off (e.g. in CI) but if you manually need to disable/enable output coloring set the RUST_LOG_STYLE environment variable. It supports the following values:

  • RUST_LOG_STYLE=never never print color codes
  • RUST_LOG_STYLE=always always print color codes
  • RUST_LOG_STYLE=auto automatically detect when to enable or disable color codes (default)

Rust Tests

tracing_redismodule is not meant to be used in Rust tests. Instead, the redis_mock crate re-implements the RedisModule_Log so logs from C are emitted using tracing.

Tests can then use the test-log crate to easily initialize tracing and receive logs from both C and Rust.

#[test_log::test]
fn some_test() {
}

The log level can be configured by setting the RUST_LOG environment variable when running the tests:

$ RUST_LOG=debug cargo test some_test -- --nocapture

By default, test-log sets this level to info, but this can be overridden in the test itself if its output is too verbose.

#[test_log::test]
#[test_log(default_log_filter = "error")]
fn some_test() {
}