Skip to content

CTS_ANALYSIS.md

Latest commit

 

History

History
191 lines (134 loc) · 6.24 KB

CTS_ANALYSIS.md

File metadata and controls

191 lines (134 loc) · 6.24 KB

OpenVX Conformance Test Suite (CTS) Analysis Report

Date: 2025-04-05
Test Version: VxTests 1.3 (VCS version: 45722c3, Release config)
Total Test Cases: 69
Total Tests: 15,277

Executive Summary

The CTS run encountered a segmentation fault during the Graph.GraphFactory test, preventing completion of the full 15,277 tests. Before the crash, 27 tests were executed with:

  • 14 tests PASSED
  • 12 tests FAILED
  • 1 test CRASHED (segfault)

The failures reveal a critical memory management issue with reference counting throughout the OpenVX implementation.

Test Results Breakdown

Tests Passed (14)

Test Case Test Name
GraphBase AllocateUserKernelId
GraphBase AllocateUserKernelLibraryId
GraphBase RegisterUserStructWithName
GraphBase GetUserStructNameByEnum
GraphBase GetUserStructEnumByName
GraphBase vxQueryNodeBase
GraphBase vxReleaseNodeBase
GraphBase vxRemoveNodeBase
GraphBase vxSetNodeAttributeBase
Logging Cummulative
SmokeTestBase vxLoadKernels
SmokeTestBase vxUnloadKernels
SmokeTestBase vxGetStatus
TargetBase vxSetNodeTargetBase

Tests Failed (12)

All failures share the same root cause: dangling references / incorrect reference counting.

Test Case Test Name Failure Pattern
GraphBase vxCreateGraph 1 dangling ref
GraphBase vxIsGraphVerifiedBase 1 dangling ref
GraphBase vxQueryGraph 1 dangling ref
GraphBase vxReleaseGraph 1 dangling ref
GraphBase vxReplicateNodeBase 1 dangling ref
SmokeTestBase vxReleaseReferenceBase 2 dangling refs
SmokeTestBase vxSetReferenceName 1 dangling ref
SmokeTestBase vxQueryReference 1 dangling ref
SmokeTestBase vxRetainReferenceBase Reference count mismatch (43 vs 42)
TargetBase vxCreateContext Massive dangling refs (4294967254)
TargetBase vxReleaseContext Massive dangling refs (4294967254)
Graph TwoNodes 1 dangling ref

Tests Crashed (1)

Test Case Test Name Issue
Graph GraphFactory Segmentation fault during test execution

Failure Analysis

Primary Issue: Reference Counting

Error Pattern:

FAILED at /home/simon/.openclaw/workspace/rustvx/OpenVX-cts/test_engine/test_utils.c:733
	Expected: 0 == dangling_refs_count
	Actual: 0 != N

The test framework tracks all OpenVX objects created during a test and expects them to be properly released. The failures indicate that objects are not being properly released.

Critical Finding: Context Underflow

The TargetBase.vxCreateContext and TargetBase.vxReleaseContext tests show:

Actual: 0 != 4294967254

This value (4294967254 = 0xFFFFFF6E) strongly suggests an unsigned integer underflow in the reference counting logic. This happens when:

  1. A reference is released more times than it was retained
  2. The reference count decrements below zero
  3. In unsigned arithmetic, this wraps around to a very large number

Reference Retention Mismatch

SmokeTestBase.vxRetainReferenceBase shows:

Expected: num_refs4 == num_refs1
Actual: 43 != 42

This confirms the reference counting is inconsistent - the number of references increased unexpectedly.

Recommendations for Fixes

Priority 1: Fix Reference Count Underflow

Location: Likely in vxReleaseContext or the reference tracking system

Issue: The massive underflow value (4294967254) indicates context reference counting is broken.

Action:

  1. Audit all vxRetainReference and vxReleaseReference calls in the context code
  2. Add debug assertions to catch underflow before it happens
  3. Consider using signed integers for internal reference counts with bounds checking

Priority 2: Fix Graph Object Leaks

Tests Affected: GraphBase.vxCreateGraph, GraphBase.vxReleaseGraph, Graph.TwoNodes

Issue: Graph objects are not being fully released.

Action:

  1. Review graph creation/destruction paths
  2. Ensure all internal graph resources (nodes, parameters) are properly freed
  3. Check for circular references between nodes and graphs

Priority 3: Fix Reference Naming Leak

Test Affected: SmokeTestBase.vxSetReferenceName

Issue: Setting a reference name causes a leak.

Action:

  1. Check if the name string is properly freed when the reference is released
  2. Verify that updating a name frees the old name

Priority 4: Fix Retain/Release Balance

Test Affected: SmokeTestBase.vxRetainReferenceBase

Issue: The retain operation itself may be creating extra references.

Action:

  1. Review the vxRetainReference implementation
  2. Ensure it doesn't inadvertently create additional internal references

Priority 5: Investigate GraphFactory Crash

Test Affected: Graph.GraphFactory

Issue: Segmentation fault during graph factory test.

Action:

  1. Run with gdb to get a backtrace: gdb --args ./bin/vx_test_conformance --filter=Graph.GraphFactory
  2. Check for null pointer dereferences in the factory code
  3. Verify parameter validation in node creation

Next Steps

  1. Fix reference counting - This is blocking most tests
  2. Re-run CTS - After fixes, run full suite to completion
  3. Add memory debugging - Consider running with Valgrind: valgrind --leak-check=full ./bin/vx_test_conformance
  4. Unit test individual components - Test reference counting in isolation before full CTS

Test Categories (Full List)

Based on --list_tests, the 15,277 tests are organized into 69 test cases covering:

  • GraphBase (14 tests) - Graph creation, management, verification
  • Logging (1 test) - Debug logging functionality
  • SmokeTestBase (7 tests) - Basic API smoke tests
  • TargetBase (3 tests) - Target/context management
  • Graph (~100+ tests) - Graph execution, performance, kernels
  • Vision functions (~15,000+ tests) - Individual vision kernel conformance
    • Color conversion, filtering, geometric transforms
    • Feature detection (Harris, FAST corners)
    • Optical flow, image pyramids
    • Arithmetic operations
    • And many more...

Report generated from CTS run on /home/simon/.openclaw/workspace/rustvx/OpenVX-cts/build