Skip to content

Illegal instruction error when running 'midnight-node wizards generate-keys' on CPU without AVX support #568

@sp213

Description

@sp213

Context & versions:

Environment:

  • OS: Linux 5.15.0-164-generic
  • Docker: Running midnight-node in container
  • Midnight Node Image: midnightnetwork/midnight-node:0.12.0
  • Midnight Node Version: midnight-node 0.12.0
  • CPU: Intel(R) Pentium(R) Silver N6005 @ 2.00GHz
  • CPU Architecture: x86_64 (amd64)

CPU Flags Analysis:
The CPU does NOT support AVX/AVX2 instruction sets. Available flags include: sse, sse2, ssse3, sse4_1, sse4_2, but notably missing avx or avx2.

Configuration:

Steps to reproduce:

  1. Enter the midnight-node container using ./midnight-shell.sh
  2. Run the command: ./midnight-node wizards generate-keys
  3. The wizard starts successfully and begins generating keys
  4. When it reaches the "Inserting Cross-chain (ecdsa) key" step, the error occurs

Full command sequence:

$ ./midnight-shell.sh
root@a0a994c65ce1:/# ./midnight-node wizards generate-keys

Actual behavior:

The generate-keys wizard fails with an "Illegal instruction (core dumped)" error when executing the key insert command. The error occurs specifically during the key insertion phase, not during key generation.

Error Output:

This 🧙 wizard will generate the following keys and save them to your node's keystore:
→  an ECDSA Cross-chain key
→  an ED25519 Grandpa key
→  an SR25519 Aura key
It will also generate a network key for your node if needed.

🛠️ Loaded node base path from config (pc-resources-config.json): ./data

⚙️ Generating Cross-chain (ecdsa) key
running external command: /midnight-node key generate --scheme ecdsa --output-type json
💾 Inserting Cross-chain (ecdsa) key
running external command: /midnight-node key insert --base-path ./data --scheme ecdsa --key-type crch --suri 'author breeze tiny usage bamboo deny fuel daring fluid final dune hollow'
command output: Illegal instruction (core dumped)
Running executable failed with status exit status: 132
Error: Application(Failed to run command
...

Exit Code: 132 (SIGILL - Illegal instruction)

Additional Testing:

  • midnight-node key generate --scheme ecdsa --output-type json works fine
  • midnight-node key insert --base-path ./data --scheme ecdsa --key-type crch --suri 'test' fails with "Illegal instruction"

This confirms that the issue is specific to the key insert subcommand, which likely uses CPU instructions (AVX/AVX2) not supported by the processor.

Expected behavior:

The generate-keys wizard should complete successfully and generate all required keys (ECDSA Cross-chain, ED25519 Grandpa, SR25519 Aura, and network key) regardless of whether the CPU supports AVX instruction sets.

Root Cause Analysis:

The midnight-node binary appears to be compiled with optimizations that require AVX (Advanced Vector Extensions) instruction set support. The Intel Pentium Silver N6005 processor, while being a modern x86_64 CPU, does not support AVX instructions (which require Intel Sandy Bridge / AMD Bulldozer or newer architectures).

Impact:
This affects users running midnight-node on:

  • Older x86_64 CPUs without AVX support
  • Some low-power/embedded x86_64 processors
  • Virtual machines with limited CPU feature exposure

Suggested Solutions:

  1. Provide a non-AVX build variant: Compile a version of midnight-node that doesn't require AVX instructions, targeting a broader range of x86_64 CPUs (e.g., using -march=x86-64 instead of -march=native or -mavx)

  2. Runtime CPU feature detection: Implement runtime detection of CPU capabilities and use fallback code paths when AVX is not available

  3. Documentation update: Add CPU requirements to the documentation, specifying that AVX support is required

  4. Docker image variants: Provide separate Docker images labeled with CPU requirements (e.g., midnight-node:0.12.0-avx and midnight-node:0.12.0-compat)

Additional Information:

System Information:

# Host CPU info
model name: Intel(R) Pentium(R) Silver N6005 @ 2.00GHz
flags: ... sse4_1 sse4_2 ... (no avx/avx2)

# Container info
Architecture: amd64
Image: midnightnetwork/midnight-node:0.12.0

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions