WIP: Add a new remote execution system that is integrated in the cuopt engine

[tmckayus]

Description

This change adds a new remote execution system which will ultimately replace the cuopt Python server. In this new implementation, remote solves are identical to local solves except that the env vars CUOPT_REMOTE_HOST and CUOPT_REMOTE_PORT are set to indicate a remote server.

The remote server code is included and can be built optionally with "build.sh cuopt_remote_server". It does not build by default.

Internal code paths have been modified to use data_model_view_t with non-owning pointers to store problems and solutions as GPU memory or CPU memory, the former for local solves and the latter for remote solves.

Some markdown files have been added that describe the changes:

1. REMOTE_SOLVE_GUIDE.md (Basic understanding)
2. REMOTE_SOLVE_MODES.md (Protocol details)
3. remote_solve_architecture.md (Deep architecture)
4. SOLUTION_MEMORY_ARCHITECTURE.md (Memory management)
5. SERIALIZATION_PLUGIN_GUIDE.md (If extending)

Checklist

Need to:

evaluate the need for new tests
review the available markdowns and make sure the feature is well-documented

Summary by CodeRabbit

• New Features

  • Remote solve platform: sync & async solves, job submit/status/result, cancellation, streaming logs, pluggable serializers (Protobuf & MsgPack), CLI and C/Python client helpers, and lazy GPU submodule loading.

• Documentation

  • Added comprehensive remote-solve, serialization-plugin, solution-memory, and architecture guides.

• Tests

  • New unit/integration tests and example scripts exercising remote workflows, cancellation, and client-server interactions.

• Chores

  • New remote-server build target, Protobuf dependency and installer, packaging/build updates, and CI install step.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[tmckayus]

/ok to test afd25ed

[tmckayus]

/ok to test 28ef558

[coderabbitai]

Actionable comments posted: 2

🤖 Fix all issues with AI agents

In @conda/environments/all_cuda-130_arch-x86_64.yaml.orig:
- Around line 1-82: The file all_cuda-130_arch-x86_64.yaml.orig should be
deleted from the commit because it represents an unsupported CUDA 13.0
environment and is missing the libprotobuf dependency; remove the file entirely
from the branch (git rm --cached/--force as appropriate) and commit that
deletion, then verify no other generated env files reference CUDA 13.0 or lack
libprotobuf so the supported CUDA 12.9 and 13.1 variants remain intact.

In @conda/recipes/libcuopt/recipe.yaml:
- Line 65: The run requirements for the libcuopt package are missing
libprotobuf, causing runtime link failures because cpp/CMakeLists.txt links
libcuopt.so against protobuf::libprotobuf; update the recipe.yaml run section
for the libcuopt output (the run list around the current entries pinned at the
run block) to include libprotobuf alongside libboost-devel and librmm (i.e.,
insert a run entry "libprotobuf" in the run dependencies list so consumers of
libcuopt get protobuf at runtime).

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between afd25ed and 28ef558.

📒 Files selected for processing (7)

• conda/environments/all_cuda-129_arch-aarch64.yaml
• conda/environments/all_cuda-129_arch-x86_64.yaml
• conda/environments/all_cuda-130_arch-x86_64.yaml.orig
• conda/environments/all_cuda-131_arch-aarch64.yaml
• conda/environments/all_cuda-131_arch-x86_64.yaml
• conda/recipes/libcuopt/recipe.yaml
• dependencies.yaml

🔇 Additional comments (5)

conda/environments/all_cuda-131_arch-aarch64.yaml (1)

35-35: LGTM - libprotobuf dependency correctly added.

The addition aligns with the dependencies.yaml source file change and supports the new protobuf-based remote solve functionality.

conda/environments/all_cuda-129_arch-x86_64.yaml (1)

35-35: LGTM - Consistent with other environment files.

conda/environments/all_cuda-131_arch-x86_64.yaml (1)

35-35: LGTM - Consistent with other environment files.

dependencies.yaml (1)

296-305: LGTM - libprotobuf correctly added to build_cpp dependencies.

The placement under build_cpp.common with output_types: conda is appropriate for the C++ build dependency needed for protobuf-based remote serialization. This follows the existing pattern of other build dependencies in this section.

conda/environments/all_cuda-129_arch-aarch64.yaml (1)

35-35: LGTM!

The libprotobuf dependency addition is consistent with the PR's protobuf-based remote solve serialization requirements. The alphabetical placement is correct.

[coderabbitai]

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Check if the non-.orig version exists
echo "=== Checking for existing all_cuda-130_arch-x86_64.yaml ==="
fd -t f "all_cuda-130_arch-x86_64.yaml" conda/environments/

echo ""
echo "=== Checking libprotobuf presence in all environment files ==="
for f in conda/environments/all_cuda-*.yaml; do
  if [ -f "$f" ]; then
    echo -n "$f: "
    grep -q "libprotobuf" "$f" && echo "HAS libprotobuf" || echo "MISSING libprotobuf"
  fi
done

echo ""
echo "=== Listing any .orig files that may be artifacts ==="
fd -e orig . conda/environments/

echo ""
echo "=== Checking git status and history for .orig files ==="
cd conda/environments && git status --short | grep orig || echo "No .orig files in git status"

Repository: NVIDIA/cuopt

Length of output: 694

---

🏁 Script executed:

#!/bin/bash
# Check if .orig file is tracked in git
echo "=== Checking if .orig file is tracked in git ==="
git ls-files | grep "all_cuda-130_arch-x86_64.yaml.orig"

echo ""
echo "=== Checking git log for this file ==="
git log --oneline -- conda/environments/all_cuda-130_arch-x86_64.yaml.orig | head -5

echo ""
echo "=== List all all_cuda*.yaml* files to see naming pattern ==="
ls -la conda/environments/all_cuda*.yaml*

Repository: NVIDIA/cuopt

Length of output: 804

---

🏁 Script executed:

#!/bin/bash
echo "=== Check all environment files in the directory ==="
ls -1 conda/environments/all_cuda*.yaml* | sort

echo ""
echo "=== Check what's in the git commit that added libprotobuf ==="
git show --stat 28ef558 | head -50

echo ""
echo "=== Check the full diff for the .orig file in that commit ==="
git show 28ef558 -- conda/environments/all_cuda-130_arch-x86_64.yaml.orig | head -100

Repository: NVIDIA/cuopt

Length of output: 5691

---

🏁 Script executed:

#!/bin/bash
echo "=== Check if dependencies.yaml mentions CUDA 130 ==="
grep -A 10 "130" dependencies.yaml | head -30

echo ""
echo "=== Count how many env variants are defined for each CUDA version in dependencies.yaml ==="
grep -E "cuda-130|cuda-129|cuda-131" dependencies.yaml | wc -l

Repository: NVIDIA/cuopt

Length of output: 202

---

🏁 Script executed:

#!/bin/bash
echo "=== Search for CUDA version patterns in dependencies.yaml ==="
grep -i "cuda" dependencies.yaml | head -20

echo ""
echo "=== Check the structure of dependencies.yaml ==="
head -100 dependencies.yaml

echo ""
echo "=== Look for arch and cuda variants ==="
grep -E "arch|cuda" dependencies.yaml | grep -i "x86_64\|aarch64\|130\|129\|131" | head -20

Repository: NVIDIA/cuopt

Length of output: 3306

---

🏁 Script executed:

#!/bin/bash
echo "=== Check git history for the .orig file ==="
git log --all --oneline -- conda/environments/all_cuda-130_arch-x86_64.yaml.orig

echo ""
echo "=== Check if there are any references to CUDA 130 in the codebase ==="
rg "130" conda/ --type yaml | head -20

echo ""
echo "=== Verify that CUDA 129 and 131 both have both arch variants ==="
echo "CUDA 129:"
ls -1 conda/environments/all_cuda-129* | sort

echo ""
echo "CUDA 131:"
ls -1 conda/environments/all_cuda-131* | sort

Repository: NVIDIA/cuopt

Length of output: 504

---

Delete the all_cuda-130_arch-x86_64.yaml.orig file—it should not be committed.

This file represents an unsupported CUDA version. The dependencies.yaml only defines CUDA versions 12.9 and 13.1, which generate environment files for CUDA 129 and 131. CUDA 130 is not in the supported matrix and has no corresponding aarch64 variant (unlike the 129 and 131 versions which both have x86_64 and aarch64 pairs).

Additionally, this file is missing the libprotobuf dependency that was added across all other environment files in this PR. It appears to be a leftover from the file regeneration process and should be removed entirely rather than renamed.

🤖 Prompt for AI Agents

In @conda/environments/all_cuda-130_arch-x86_64.yaml.orig around lines 1 - 82,
The file all_cuda-130_arch-x86_64.yaml.orig should be deleted from the commit
because it represents an unsupported CUDA 13.0 environment and is missing the
libprotobuf dependency; remove the file entirely from the branch (git rm
--cached/--force as appropriate) and commit that deletion, then verify no other
generated env files reference CUDA 13.0 or lack libprotobuf so the supported
CUDA 12.9 and 13.1 variants remain intact.

[coderabbitai]

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

# Find the recipe.yaml file and check its structure
find . -name "recipe.yaml" -path "*libcuopt*" 2>/dev/null

Repository: NVIDIA/cuopt

Length of output: 92

---

🏁 Script executed:

# Look for CMakeLists.txt files that might contain protobuf references
find . -name "CMakeLists.txt" 2>/dev/null | head -20

Repository: NVIDIA/cuopt

Length of output: 799

---

🏁 Script executed:

# Search for any protobuf references in the repository
rg -l "protobuf" --type yaml --type cmake 2>/dev/null | head -20

Repository: NVIDIA/cuopt

Length of output: 367

---

🏁 Script executed:

# Read the recipe.yaml file to see the structure and all requirements
cat -n ./conda/recipes/libcuopt/recipe.yaml

Repository: NVIDIA/cuopt

Length of output: 8706

---

🏁 Script executed:

# Check cpp/CMakeLists.txt for protobuf linking details
rg -n "protobuf" ./cpp/CMakeLists.txt -A 3 -B 1

Repository: NVIDIA/cuopt

Length of output: 380

---

🏁 Script executed:

# Also check the main source CMakeLists for protobuf
rg -n "protobuf" ./cpp/src/CMakeLists.txt -A 3 -B 1

Repository: NVIDIA/cuopt

Length of output: 38

---

🏁 Script executed:

# Search for how protobuf is found/configured in CMakeLists
rg -n "find_package.*protobuf" ./cpp/CMakeLists.txt -i -A 5

Repository: NVIDIA/cuopt

Length of output: 342

---

🏁 Script executed:

# Check if there's any mention of static protobuf linking
rg -n "protobuf.*STATIC\|Protobuf_USE_STATIC_LIBS\|Protobuf.*static" ./cpp/CMakeLists.txt -i

Repository: NVIDIA/cuopt

Length of output: 38

---

🏁 Script executed:

# Look for the full context of the protobuf link in libcuopt
rg -n "target_link_libraries.*cuopt" ./cpp/CMakeLists.txt -B 10 -A 10 | head -60

Repository: NVIDIA/cuopt

Length of output: 1934

---

libprotobuf missing from libcuopt run requirements — add at line 161.

While libprotobuf is correctly added to build and host requirements, it must also be in the run requirements for the libcuopt output package. The CMakeLists.txt dynamically links libcuopt.so against protobuf::libprotobuf (line 370 in cpp/CMakeLists.txt), which means consumers of libcuopt require protobuf at runtime. Without this dependency, binaries and libraries linking to libcuopt.so will fail with missing symbol errors.

Add libprotobuf to the run requirements at line 161:

      run:
        - ${{ pin_compatible("cuda-version", upper_bound="x", lower_bound="x") }}
        - ${{ pin_subpackage("libmps-parser", exact=True) }}
        - libboost-devel
        - libprotobuf
        - librmm =${{ minor_version }}

🤖 Prompt for AI Agents

In @conda/recipes/libcuopt/recipe.yaml at line 65, The run requirements for the
libcuopt package are missing libprotobuf, causing runtime link failures because
cpp/CMakeLists.txt links libcuopt.so against protobuf::libprotobuf; update the
recipe.yaml run section for the libcuopt output (the run list around the current
entries pinned at the run block) to include libprotobuf alongside libboost-devel
and librmm (i.e., insert a run entry "libprotobuf" in the run dependencies list
so consumers of libcuopt get protobuf at runtime).

[tmckayus]

/ok to test 3b4e741

[tmckayus]

TODO: take another pass and replace client protocol with gRPC

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@ci/utils/install_protobuf.sh`:
- Around line 9-25: Pin Protobuf to a compatible version across all places:
update dependencies.yaml to require a specific Protobuf range (e.g. "protobuf:
>=3.20,<4.0"), update CMakeLists.txt's find_package invocation to include the
minimum version (e.g. change find_package(Protobuf REQUIRED) to
find_package(Protobuf 3.20 REQUIRED)), and change this install script
(install_protobuf.sh) to install that pinned version via the package manager
(adjust the dnf/apt package names or add version qualifiers / use apt-get
install protobuf-compiler=3.20.* or the equivalent distro-specific
package/version) and also ensure conda environment specs are updated to the same
version range; make sure error branches still instruct manual install if the
exact version is unavailable.

🧹 Nitpick comments (2)

ci/utils/install_protobuf.sh (2)

14-17: Consider cleaning up apt cache to reduce image size.

In CI/container environments, cleaning up the apt cache after installation reduces the final image size.

Suggested improvement

     elif [[ "$ID" == "ubuntu" ]]; then
         echo "Detected Ubuntu. Installing Protobuf via apt..."
         apt-get update
-        apt-get install -y libprotobuf-dev protobuf-compiler
+        apt-get install -y --no-install-recommends libprotobuf-dev protobuf-compiler
+        rm -rf /var/lib/apt/lists/*
     else

---

11-21: Consider adding Debian support.

Since Debian is closely related to Ubuntu and uses the same package manager, you might want to support it explicitly. This would help if the CI environment or developer machines use Debian.

Suggested improvement

     if [[ "$ID" == "rocky" ]]; then
         echo "Detected Rocky Linux. Installing Protobuf via dnf..."
         dnf install -y protobuf-devel protobuf-compiler
-    elif [[ "$ID" == "ubuntu" ]]; then
-        echo "Detected Ubuntu. Installing Protobuf via apt..."
+    elif [[ "$ID" == "ubuntu" || "$ID" == "debian" ]]; then
+        echo "Detected $ID. Installing Protobuf via apt..."
         apt-get update
         apt-get install -y libprotobuf-dev protobuf-compiler
     else

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 3b4e741 and 0dcd54a.

📒 Files selected for processing (2)

• ci/build_wheel_libcuopt.sh
• ci/utils/install_protobuf.sh

🔇 Additional comments (2)

ci/utils/install_protobuf.sh (1)

1-6: Good shell script setup.

The strict mode (set -euo pipefail) ensures the script fails fast on errors, which is appropriate for CI scripts.

ci/build_wheel_libcuopt.sh (1)

20-22: LGTM!

The protobuf installation step is correctly placed in the dependency installation sequence, following the established pattern used by other dependencies (Boost, TBB, cuDSS). The script's set -euo pipefail ensures any installation failure will abort the build.

_{✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.}

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash

# Check if there are version constraints specified elsewhere in the project
echo "=== Checking dependency files for protobuf version constraints ==="
rg "protobuf" --type yaml --type txt -g '!*.pb.*' -n

echo ""
echo "=== Checking CMake files for protobuf version requirements ==="
rg "find_package.*[Pp]rotobuf" -A2 -B2 -n

echo ""
echo "=== Searching for CMakeLists.txt files ==="
fd "CMakeLists.txt" -x cat -n {}

echo ""
echo "=== Checking for .proto files ==="
fd "\.proto$" -x cat -n {}

echo ""
echo "=== Checking requirements.txt or similar Python dependency files ==="
fd -E node_modules "(requirements.*\.txt|setup\.py|pyproject\.toml|poetry\.lock|Pipfile)" -x cat -n {}

echo ""
echo "=== Checking for docker or container files ==="
fd "(Dockerfile|\.dockerignore)" -x cat -n {}

Repository: NVIDIA/cuopt

Length of output: 50369

---

🏁 Script executed:

#!/bin/bash

echo "=== Checking dependencies.yaml for protobuf version constraints ==="
rg -A5 "depends_on_protobuf|libprotobuf" dependencies.yaml

echo ""
echo "=== Checking conda environment files for libprotobuf version specs ==="
fd "all_cuda.*\.yaml" conda/environments/ -x grep -A1 -B1 "libprotobuf" {}

echo ""
echo "=== Checking conda recipe for libprotobuf ==="
grep -A2 -B2 "libprotobuf" conda/recipes/libcuopt/recipe.yaml

echo ""
echo "=== Checking if protobuf has version specifications anywhere ==="
rg "protobuf.*==" -n

Repository: NVIDIA/cuopt

Length of output: 1147

---

🏁 Script executed:

#!/bin/bash

echo "=== Looking for .proto file to understand protobuf usage ==="
find . -name "*.proto" -type f -exec cat -n {} +

echo ""
echo "=== Checking what's in linear_programming/utilities/protobuf_serializer.cu ==="
fd "protobuf_serializer" -x head -30 {}

Repository: NVIDIA/cuopt

Length of output: 14312

---

Add protobuf version constraints to ensure compatibility across distributions.

Protobuf version mismatches between build and runtime environments can cause serialization/deserialization issues. Currently, protobuf has no version pinning in dependencies.yaml, conda environments, or CMakeLists.txt (find_package(Protobuf REQUIRED) lacks version specification), while the install script installs without version constraints. Given that this PR introduces protobuf-based serialization for remote solving, add explicit version constraints to dependencies.yaml, CMakeLists.txt, and update the install script to match the pinned version.

🤖 Prompt for AI Agents

In `@ci/utils/install_protobuf.sh` around lines 9 - 25, Pin Protobuf to a
compatible version across all places: update dependencies.yaml to require a
specific Protobuf range (e.g. "protobuf: >=3.20,<4.0"), update CMakeLists.txt's
find_package invocation to include the minimum version (e.g. change
find_package(Protobuf REQUIRED) to find_package(Protobuf 3.20 REQUIRED)), and
change this install script (install_protobuf.sh) to install that pinned version
via the package manager (adjust the dnf/apt package names or add version
qualifiers / use apt-get install protobuf-compiler=3.20.* or the equivalent
distro-specific package/version) and also ensure conda environment specs are
updated to the same version range; make sure error branches still instruct
manual install if the exact version is unavailable.

[rgsl888prabhu]

/ok to test 0dcd54a

[rgsl888prabhu]

/ok to test de8daf7

[rgsl888prabhu]

/ok to test 7660217

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@cpp/CMakeLists.txt`:
- Around line 621-624: Wrap the
add_subdirectory(src/linear_programming/utilities/serializers) call with the
BUILD_REMOTE_SERVER conditional so the serializers plugin is only built when
BUILD_REMOTE_SERVER is enabled; locate the existing add_subdirectory invocation
in CMakeLists and move or surround it with the if(BUILD_REMOTE_SERVER) ...
endif() block (keeping it near other remote-server related entries and
references such as cuopt_remote_server.cpp) to prevent building the msgpack
serializer when remote server support is disabled.

🧹 Nitpick comments (2)

cpp/CMakeLists.txt (2)

234-238: Prefer target-specific includes over global include_directories().

Using global include_directories() adds paths to all targets in this directory, which can cause unintended include pollution. Modern CMake best practice is to use target_include_directories() on specific targets that need the paths.

Since multiple targets need the generated proto headers (cuopt, cuopt_remote_server), consider either:

1. Adding the paths explicitly to each target that needs them, or
2. Creating an interface library for the proto headers

♻️ Suggested approach using target-specific includes

 # Protocol Buffers for remote solve serialization
 find_package(Protobuf REQUIRED)
-include_directories(SYSTEM ${Protobuf_INCLUDE_DIRS})
-include_directories(${CMAKE_CURRENT_BINARY_DIR})
+# Protobuf includes added to targets via target_include_directories below

Then add to the cuopt target (around line 305-319):

target_include_directories(cuopt SYSTEM PRIVATE ${Protobuf_INCLUDE_DIRS})
target_include_directories(cuopt PRIVATE ${CMAKE_CURRENT_BINARY_DIR})

And similarly for cuopt_remote_server.

---

575-581: Consider using PRIVATE for executable link dependencies.

For executable targets, the PUBLIC/PRIVATE distinction on target_link_libraries has no practical effect on the executable itself. Using PRIVATE is the conventional choice and more clearly expresses intent.

♻️ Suggested change

   target_link_libraries(cuopt_remote_server
-    PUBLIC
-    cuopt
-    OpenMP::OpenMP_CXX
     PRIVATE
+    cuopt
+    OpenMP::OpenMP_CXX
     protobuf::libprotobuf
   )

   target_link_libraries(cuopt_cancel_job
-    PUBLIC
-    cuopt
+    PRIVATE cuopt
   )

Also applies to: 609-612

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 1a2227b and 7660217.

📒 Files selected for processing (1)

• cpp/CMakeLists.txt

🧰 Additional context used

🧠 Learnings (11)

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.{cu,cuh,cpp,hpp,h} : Check that hard-coded GPU device IDs and resource limits are made configurable; abstract multi-backend support for different CUDA versions

Applied to files:

• cpp/CMakeLists.txt

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.{cu,cuh,cpp,hpp,h} : Verify error propagation from CUDA to user-facing APIs is complete; ensure CUDA errors are caught and mapped to meaningful user error codes

Applied to files:

• cpp/CMakeLists.txt

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.{cu,cuh} : Every CUDA kernel launch and memory operation must have error checking with CUDA_CHECK or equivalent verification

Applied to files:

• cpp/CMakeLists.txt

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.{cu,cpp,hpp,h} : Avoid inappropriate use of exceptions in performance-critical GPU operation paths; prefer error codes or CUDA error checking for latency-sensitive code

Applied to files:

• cpp/CMakeLists.txt

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.{cu,cuh} : Avoid reinventing functionality already available in Thrust, CCCL, or RMM libraries; prefer standard library utilities over custom implementations

Applied to files:

• cpp/CMakeLists.txt

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.cu : Detect inefficient GPU kernel launches with low occupancy or poor memory access patterns; optimize for coalesced memory access and minimize warp divergence in hot paths

Applied to files:

• cpp/CMakeLists.txt

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.{cu,cuh,cpp,hpp,h} : Verify correct problem size checks before expensive GPU/CPU operations; prevent resource exhaustion on oversized problems

Applied to files:

• cpp/CMakeLists.txt

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.{cu,cuh,cpp,hpp,h} : Ensure race conditions are absent in multi-GPU code and multi-threaded server implementations; verify proper synchronization of shared state

Applied to files:

• cpp/CMakeLists.txt

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.{cu,cuh,cpp,hpp,h} : Track GPU device memory allocations and deallocations to prevent memory leaks; ensure cudaMalloc/cudaFree balance and cleanup of streams/events

Applied to files:

• cpp/CMakeLists.txt

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.{cu,cuh,cpp,hpp,h} : For concurrent CUDA operations (barriers, async operations), explicitly create and manage dedicated streams instead of reusing the default stream; document stream lifecycle

Applied to files:

• cpp/CMakeLists.txt

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.{cu,cuh,cpp,hpp,h} : Refactor code duplication in solver components (3+ occurrences) into shared utilities; for GPU kernels, use templated device functions to avoid duplication

Applied to files:

• cpp/CMakeLists.txt

🔇 Additional comments (3)

cpp/CMakeLists.txt (3)

142-147: LGTM!

Reasonable workaround for GCC's known false-positive -Warray-bounds diagnostics from external headers like Protobuf, while preserving -Werror for other warnings.

---

239-253: Proto generation setup looks correct.

The custom command is properly configured with DEPENDS for rebuild tracking and VERBATIM for safe argument handling.

Minor observation: PROTO_HDRS is defined but unused. Consider adding it to the cuopt target's sources or target_sources() to ensure proper dependency tracking for the header file, though in practice the .cc file inclusion typically suffices.

---

264-267: LGTM!

The protobuf sources are correctly added to the cuopt library, and the linking arrangement is appropriate:

• mps_parser_static embeds the parser symbols directly
• protobuf::libprotobuf as PRIVATE keeps it an implementation detail

Also applies to: 375-377

_{✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.}

[coderabbitai]

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Check if serializers are used outside of remote solve code paths

# Find references to serializer in the codebase excluding the serializers directory itself
echo "=== References to serializer outside serializers directory ==="
rg -n "serializer" --type cpp --type cmake -g '!**/serializers/**' -g '!**/test**' | head -100

echo ""
echo "=== Check if serializers CMakeLists.txt has its own guards ==="
fd -t f "CMakeLists.txt" --path "*/serializers/*" -exec cat {} \;

echo ""
echo "=== Check the context around line 621-624 in CMakeLists.txt ==="
rg -B 30 -A 5 "add_subdirectory.*serializers" cpp/CMakeLists.txt

Repository: NVIDIA/cuopt

Length of output: 7539

---

🏁 Script executed:

#!/bin/bash
# Get serializers CMakeLists.txt
echo "=== Serializers CMakeLists.txt ==="
find . -path "*/serializers/CMakeLists.txt" -exec cat {} \;

echo ""
echo "=== Check BUILD_REMOTE_SERVER conditional structure in main CMakeLists.txt ==="
# Search for BUILD_REMOTE_SERVER conditions to see the structure
rg -n "BUILD_REMOTE_SERVER|add_subdirectory.*serializers" cpp/CMakeLists.txt | tail -50

echo ""
echo "=== Check protobuf_serializer.cu location and context ==="
fd -t f "protobuf_serializer.cu" -exec head -20 {} \;

echo ""
echo "=== Check if protobuf_serializer is conditioned in src/linear_programming/CMakeLists.txt ==="
rg -B 5 -A 5 "protobuf_serializer" cpp/src/linear_programming/CMakeLists.txt

Repository: NVIDIA/cuopt

Length of output: 2658

---

Move the serializers subdirectory inside the BUILD_REMOTE_SERVER conditional.

The add_subdirectory(src/linear_programming/utilities/serializers) call builds the msgpack serializer plugin, which is only used by the remote server. All serializer references in the codebase are confined to cuopt_remote_server.cpp. Building this optional plugin when remote server support is disabled is wasteful and should be gated behind BUILD_REMOTE_SERVER.

🤖 Prompt for AI Agents

In `@cpp/CMakeLists.txt` around lines 621 - 624, Wrap the
add_subdirectory(src/linear_programming/utilities/serializers) call with the
BUILD_REMOTE_SERVER conditional so the serializers plugin is only built when
BUILD_REMOTE_SERVER is enabled; locate the existing add_subdirectory invocation
in CMakeLists and move or surround it with the if(BUILD_REMOTE_SERVER) ...
endif() block (keeping it near other remote-server related entries and
references such as cuopt_remote_server.cpp) to prevent building the msgpack
serializer when remote server support is disabled.

[coderabbitai]

Actionable comments posted: 2

🤖 Fix all issues with AI agents

In `@cpp/CMakeLists.txt`:
- Around line 266-269: The cuopt library is currently built with ${PROTO_SRCS}
always appended which breaks builds when Protobuf is disabled; ensure PROTO_SRCS
is defined to an empty value when the protobuf/remote-server branch is not taken
(e.g., set(PROTO_SRCS "" ) or guard with if(NOT DEFINED PROTO_SRCS)
set(PROTO_SRCS "")) so the add_library(cuopt SHARED ${CUOPT_SRC_FILES}
${PROTO_SRCS}) line works unconditionally; update the CMake logic that defines
PROTO_SRCS (the conditional that generates/sets it) to provide this default.
- Around line 236-255: Wrap the Protobuf dependency and proto generation in an
if(BUILD_REMOTE_SERVER) ... endif() guard so find_package(Protobuf REQUIRED),
the add_custom_command that produces PROTO_SRCS/PROTO_HDRS, and setting
PROTO_FILE/PROTO_SRCS/PROTO_HDRS only run when BUILD_REMOTE_SERVER is ON; remove
the unconditional include_directories(SYSTEM ${Protobuf_INCLUDE_DIRS}) and
include_directories(${CMAKE_CURRENT_BINARY_DIR}) and instead call
target_include_directories(<the target that consumes PROTO_SRCS/PROTO_HDRS>
PRIVATE ${CMAKE_CURRENT_BINARY_DIR} SYSTEM ${Protobuf_INCLUDE_DIRS}) so only the
target that uses the generated headers gets the includes.

♻️ Duplicate comments (1)

cpp/CMakeLists.txt (1)

623-626: Move the serializers subdirectory inside the BUILD_REMOTE_SERVER conditional.

This was flagged in a previous review. The add_subdirectory(src/linear_programming/utilities/serializers) builds the msgpack serializer plugin, which is only used by the remote server.

♻️ Suggested fix

-# ##################################################################################################
-# - Pluggable Serializers --------------------------------------------------------------------------
-# Build optional serializer plugins (e.g., msgpack)
-add_subdirectory(src/linear_programming/utilities/serializers)

Move this inside the if(BUILD_REMOTE_SERVER AND NOT BUILD_LP_ONLY) block above (before line 621).

🧹 Nitpick comments (2)

cpp/CMakeLists.txt (2)

376-379: Protobuf linkage should be conditional.

The protobuf::libprotobuf is linked unconditionally. If the Protobuf dependency is made conditional on BUILD_REMOTE_SERVER, this linkage should also be guarded:

♻️ Suggested refactor

   PRIVATE
   ${CUOPT_PRIVATE_CUDA_LIBS}
   cuopt::mps_parser_static  # Static link - symbols embedded in libcuopt.so
-  protobuf::libprotobuf
+  $<$<BOOL:${BUILD_REMOTE_SERVER}>:protobuf::libprotobuf>
   )

---

559-621: Remote server build section is well-structured.

The BUILD_REMOTE_SERVER guard and RUNPATH configuration are appropriate. Minor suggestions:

1. Redundant include directories (lines 567-576): For executables, PUBLIC include directories are not meaningful since nothing links against them. The same paths appear in both PRIVATE and PUBLIC.

2. Executable linkage (lines 577-583, 611-614): For executables, PUBLIC vs PRIVATE linkage is semantically equivalent. Consider using only PRIVATE for clarity.

♻️ Optional cleanup

   target_include_directories(cuopt_remote_server
     PRIVATE
     "${CMAKE_CURRENT_SOURCE_DIR}/src"
     "${CMAKE_CURRENT_SOURCE_DIR}/include"
     "${CMAKE_CURRENT_SOURCE_DIR}/libmps_parser/include"
     "${CMAKE_CURRENT_BINARY_DIR}"
-    PUBLIC
     "$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>"
     "$<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}/include>"
   )
   target_link_libraries(cuopt_remote_server
-    PUBLIC
-    cuopt
-    OpenMP::OpenMP_CXX
     PRIVATE
+    cuopt
+    OpenMP::OpenMP_CXX
     protobuf::libprotobuf
   )

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 7660217 and 235323c.

📒 Files selected for processing (1)

• cpp/CMakeLists.txt

🧰 Additional context used

🧠 Learnings (12)

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.{cu,cuh,cpp,hpp,h} : Check that hard-coded GPU device IDs and resource limits are made configurable; abstract multi-backend support for different CUDA versions

Applied to files:

• cpp/CMakeLists.txt

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.{cpp,hpp,h} : Prevent thread-unsafe use of global and static variables; use proper mutex/synchronization in server code accessing shared solver state

Applied to files:

• cpp/CMakeLists.txt

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.{cu,cuh,cpp,hpp,h} : Ensure race conditions are absent in multi-GPU code and multi-threaded server implementations; verify proper synchronization of shared state

Applied to files:

• cpp/CMakeLists.txt

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.{cu,cuh,cpp,hpp,h} : Refactor code duplication in solver components (3+ occurrences) into shared utilities; for GPU kernels, use templated device functions to avoid duplication

Applied to files:

• cpp/CMakeLists.txt

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.{cu,cuh,cpp,hpp,h} : Verify error propagation from CUDA to user-facing APIs is complete; ensure CUDA errors are caught and mapped to meaningful user error codes

Applied to files:

• cpp/CMakeLists.txt

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.{cu,cuh} : Every CUDA kernel launch and memory operation must have error checking with CUDA_CHECK or equivalent verification

Applied to files:

• cpp/CMakeLists.txt

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.{cu,cpp,hpp,h} : Avoid inappropriate use of exceptions in performance-critical GPU operation paths; prefer error codes or CUDA error checking for latency-sensitive code

Applied to files:

• cpp/CMakeLists.txt

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.{cu,cuh} : Avoid reinventing functionality already available in Thrust, CCCL, or RMM libraries; prefer standard library utilities over custom implementations

Applied to files:

• cpp/CMakeLists.txt

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.cu : Detect inefficient GPU kernel launches with low occupancy or poor memory access patterns; optimize for coalesced memory access and minimize warp divergence in hot paths

Applied to files:

• cpp/CMakeLists.txt

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.{cu,cuh,cpp,hpp,h} : Verify correct problem size checks before expensive GPU/CPU operations; prevent resource exhaustion on oversized problems

Applied to files:

• cpp/CMakeLists.txt

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.{cu,cuh,cpp,hpp,h} : Track GPU device memory allocations and deallocations to prevent memory leaks; ensure cudaMalloc/cudaFree balance and cleanup of streams/events

Applied to files:

• cpp/CMakeLists.txt

📚 Learning: 2025-11-25T10:20:49.822Z

Learnt from: CR
Repo: NVIDIA/cuopt PR: 0
File: .github/.coderabbit_review_guide.md:0-0
Timestamp: 2025-11-25T10:20:49.822Z
Learning: Applies to **/*.{cu,cuh,cpp,hpp,h} : For concurrent CUDA operations (barriers, async operations), explicitly create and manage dedicated streams instead of reusing the default stream; document stream lifecycle

Applied to files:

• cpp/CMakeLists.txt

🔇 Additional comments (1)

cpp/CMakeLists.txt (1)

142-149: LGTM: Pragmatic warning suppression for third-party headers.

The GCC-specific suppression of -Warray-bounds and -Wstringop-overread is a reasonable approach to avoid false positives from Protobuf headers while maintaining strict warnings for project code.

_{✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.}

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Protobuf is unconditionally required even when remote server is disabled.

The find_package(Protobuf REQUIRED) and proto generation occur unconditionally, adding a mandatory build dependency for all users even when BUILD_REMOTE_SERVER=OFF. This increases build complexity and external dependencies for users who only need local solves.

Consider making Protobuf conditional on BUILD_REMOTE_SERVER:

♻️ Suggested refactor

-# Protocol Buffers for remote solve serialization
-find_package(Protobuf REQUIRED)
-include_directories(SYSTEM ${Protobuf_INCLUDE_DIRS})
-include_directories(${CMAKE_CURRENT_BINARY_DIR})
-
-# Generate C++ code from .proto file
-set(PROTO_FILE "${CMAKE_CURRENT_SOURCE_DIR}/src/linear_programming/utilities/cuopt_remote.proto")
-set(PROTO_SRCS "${CMAKE_CURRENT_BINARY_DIR}/cuopt_remote.pb.cc")
-set(PROTO_HDRS "${CMAKE_CURRENT_BINARY_DIR}/cuopt_remote.pb.h")
-
-add_custom_command(
-  OUTPUT "${PROTO_SRCS}" "${PROTO_HDRS}"
-  COMMAND ${Protobuf_PROTOC_EXECUTABLE}
-  ARGS --cpp_out ${CMAKE_CURRENT_BINARY_DIR}
-       --proto_path ${CMAKE_CURRENT_SOURCE_DIR}/src/linear_programming/utilities
-       ${PROTO_FILE}
-  DEPENDS ${PROTO_FILE}
-  COMMENT "Generating C++ code from cuopt_remote.proto"
-  VERBATIM
-)
+# Protocol Buffers for remote solve serialization (optional)
+set(PROTO_SRCS "")
+if(BUILD_REMOTE_SERVER)
+  find_package(Protobuf REQUIRED)
+  
+  set(PROTO_FILE "${CMAKE_CURRENT_SOURCE_DIR}/src/linear_programming/utilities/cuopt_remote.proto")
+  set(PROTO_SRCS "${CMAKE_CURRENT_BINARY_DIR}/cuopt_remote.pb.cc")
+  set(PROTO_HDRS "${CMAKE_CURRENT_BINARY_DIR}/cuopt_remote.pb.h")
+
+  add_custom_command(
+    OUTPUT "${PROTO_SRCS}" "${PROTO_HDRS}"
+    COMMAND ${Protobuf_PROTOC_EXECUTABLE}
+    ARGS --cpp_out ${CMAKE_CURRENT_BINARY_DIR}
+         --proto_path ${CMAKE_CURRENT_SOURCE_DIR}/src/linear_programming/utilities
+         ${PROTO_FILE}
+    DEPENDS ${PROTO_FILE}
+    COMMENT "Generating C++ code from cuopt_remote.proto"
+    VERBATIM
+  )
+endif()

Additionally, prefer target_include_directories() over global include_directories() for better encapsulation.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	# Protocol Buffers for remote solve serialization
	find_package(Protobuf REQUIRED)
	include_directories(SYSTEM ${Protobuf_INCLUDE_DIRS})
	include_directories(${CMAKE_CURRENT_BINARY_DIR})
	# Generate C++ code from .proto file
	set(PROTO_FILE "${CMAKE_CURRENT_SOURCE_DIR}/src/linear_programming/utilities/cuopt_remote.proto")
	set(PROTO_SRCS "${CMAKE_CURRENT_BINARY_DIR}/cuopt_remote.pb.cc")
	set(PROTO_HDRS "${CMAKE_CURRENT_BINARY_DIR}/cuopt_remote.pb.h")
	add_custom_command(
	OUTPUT "${PROTO_SRCS}" "${PROTO_HDRS}"
	COMMAND ${Protobuf_PROTOC_EXECUTABLE}
	ARGS --cpp_out ${CMAKE_CURRENT_BINARY_DIR}
	--proto_path ${CMAKE_CURRENT_SOURCE_DIR}/src/linear_programming/utilities
	${PROTO_FILE}
	DEPENDS ${PROTO_FILE}
	COMMENT "Generating C++ code from cuopt_remote.proto"
	VERBATIM
	)
	# Protocol Buffers for remote solve serialization (optional)
	set(PROTO_SRCS "")
	if(BUILD_REMOTE_SERVER)
	find_package(Protobuf REQUIRED)
	set(PROTO_FILE "${CMAKE_CURRENT_SOURCE_DIR}/src/linear_programming/utilities/cuopt_remote.proto")
	set(PROTO_SRCS "${CMAKE_CURRENT_BINARY_DIR}/cuopt_remote.pb.cc")
	set(PROTO_HDRS "${CMAKE_CURRENT_BINARY_DIR}/cuopt_remote.pb.h")
	add_custom_command(
	OUTPUT "${PROTO_SRCS}" "${PROTO_HDRS}"
	COMMAND ${Protobuf_PROTOC_EXECUTABLE}
	ARGS --cpp_out ${CMAKE_CURRENT_BINARY_DIR}
	--proto_path ${CMAKE_CURRENT_SOURCE_DIR}/src/linear_programming/utilities
	${PROTO_FILE}
	DEPENDS ${PROTO_FILE}
	COMMENT "Generating C++ code from cuopt_remote.proto"
	VERBATIM
	)
	endif()

🤖 Prompt for AI Agents

In `@cpp/CMakeLists.txt` around lines 236 - 255, Wrap the Protobuf dependency and
proto generation in an if(BUILD_REMOTE_SERVER) ... endif() guard so
find_package(Protobuf REQUIRED), the add_custom_command that produces
PROTO_SRCS/PROTO_HDRS, and setting PROTO_FILE/PROTO_SRCS/PROTO_HDRS only run
when BUILD_REMOTE_SERVER is ON; remove the unconditional
include_directories(SYSTEM ${Protobuf_INCLUDE_DIRS}) and
include_directories(${CMAKE_CURRENT_BINARY_DIR}) and instead call
target_include_directories(<the target that consumes PROTO_SRCS/PROTO_HDRS>
PRIVATE ${CMAKE_CURRENT_BINARY_DIR} SYSTEM ${Protobuf_INCLUDE_DIRS}) so only the
target that uses the generated headers gets the includes.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Proto sources compiled into main library unconditionally.

The ${PROTO_SRCS} is added to the cuopt library regardless of whether remote server support is enabled. If Protobuf setup is made conditional per the previous comment, ensure PROTO_SRCS defaults to empty so this line works correctly in both cases.

🤖 Prompt for AI Agents

In `@cpp/CMakeLists.txt` around lines 266 - 269, The cuopt library is currently
built with ${PROTO_SRCS} always appended which breaks builds when Protobuf is
disabled; ensure PROTO_SRCS is defined to an empty value when the
protobuf/remote-server branch is not taken (e.g., set(PROTO_SRCS "" ) or guard
with if(NOT DEFINED PROTO_SRCS) set(PROTO_SRCS "")) so the add_library(cuopt
SHARED ${CUOPT_SRC_FILES} ${PROTO_SRCS}) line works unconditionally; update the
CMake logic that defines PROTO_SRCS (the conditional that generates/sets it) to
provide this default.

[tmckayus]

closing, superseded by #799