fix: skip airt_* kwargs for attacks that don't support them#44
Merged
Conversation
Root cause: _build_attack_params() unconditionally injected airt_assessment_id, airt_goal_category, and airt_target_model into every generated attack call. 12 of 46 SDK attacks (rainbow, gptfuzzer, autodan, renellm, beast, drattack, deep_inception, hopskipjump, simba, nes, zoo, multimodal) don't accept these kwargs and raise TypeError. This caused campaigns with mixed attack types to crash partway through, creating multiple assessments instead of one and leaving the platform with missing attack data. Fix: - Add _ATTACKS_WITHOUT_AIRT_KWARGS set (determined via SDK introspection) - _build_attack_params() checks canonical_name before adding airt_* kwargs - _generate_transform_study() same conditional logic - Category template uses runtime inspect.signature() check since attack functions are dispatched dynamically Tested: campaign with tap+rainbow+beast generates correct kwargs per attack Existing tests pass.
All 46 SDK attacks now accept airt_assessment_id, airt_goal_category, and airt_target_model kwargs (added in dreadnode-tiger#1693). The code-gen passes them unconditionally to every attack — no more _ATTACKS_WITHOUT_AIRT_KWARGS guard needed. This ensures all attacks in a campaign have proper OTEL span linkage to the assessment in ClickHouse, so the platform shows correct findings counts, ASR, and severity for every attack type. Flow for '10 attacks + 5 transforms': - 1 Assessment created - 10 assessment.run() calls (one per attack type) - Each attack gets transforms=[t1,t2,t3,t4,t5] applied - Each attack gets airt_assessment_id for platform span linkage - Platform sees 10 attack spans, each linked to the assessment - Metrics: overall_asr = (C+H+M+L findings) / 10 total attacks Depends on: dreadnode-tiger#1693 (SDK airt_* kwargs)
rdheekonda
force-pushed
the
fix/airt-kwargs-compat
branch
from
2823dc2 to
65fd559
Compare
…aigns Two bugs that caused failures when running large campaigns: 1. execute_workflow didn't pass platform credentials (DREADNODE_SERVER, DREADNODE_API_KEY, etc.) to the workflow subprocess. When the agent called execute_workflow manually after an auto-execute timeout, the workflow script couldn't connect to the platform. Fixed by adding _resolve_platform_env() that reads from saved profile (same logic as attack_runner's auto-execute). 2. Timeout caps were too low for multi-attack campaigns: - execute_workflow: 600s max → 3600s (1 hour) - _auto_execute_workflow: 540s default → 3600s - _call_runner: 660s → 3660s A 12-attack campaign with 10 iterations each easily takes 20+ min. The old 540s cap caused timeouts on the first run, forcing the user to manually re-execute (which then hit Bug 1).
The previous _resolve_platform_env() had two issues: 1. Early return on DREADNODE_LLM_BASE: In TUI mode, the runtime sets LLM proxy vars but NOT server/org/workspace/project. The old code returned early when LLM vars were set, skipping the profile read. Result: subprocess had no org/workspace/project scope, so assessments landed in the wrong project or failed to register. 2. Raw YAML parsing: Manually parsed ~/.dreadnode/config.yaml instead of using the SDK's UserConfig.read(). This missed workspace/project switches done via /workspace in the TUI (which updates the profile object but the YAML key names may differ). Fix: Always read the active profile via UserConfig (which respects dn login, /workspace, /profile switches). Use setdefault() so explicit env vars (sandbox mode) still take precedence.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When running a campaign with all 12 attack types against a target, 7 of 12 attacks crash with:
This affects: rainbow, gptfuzzer, autodan, renellm, beast, drattack, deep_inception (and hopskipjump, simba, nes, zoo, multimodal).
Impact
Root Cause
_build_attack_params()inattack_runner.pyunconditionally injectsairt_assessment_id,airt_goal_category, andairt_target_modelinto every generated attack call. However, only 34/46 SDK attacks accept these kwargs — the other 12 raise TypeError.Fix
_ATTACKS_WITHOUT_AIRT_KWARGSset — 12 attack functions that don't accept airt_* params (verified via SDKinspect.signature())_build_attack_params()— conditionally adds airt_* kwargs only for supported attacks_generate_transform_study()— same conditional logicinspect.signature()check since attack functions are dispatched dynamicallyTesting
python3 -cverification: campaign with tap+rainbow+beast generates correct kwargs per attack (tap gets airt_*, rainbow/beast don't)Note for SDK team
Ideally all 46 attacks should accept airt_* kwargs for platform span linkage. The 12 attacks without support are the older/specialized ones. A follow-up SDK PR to add
**kwargsor explicit airt_* params to all attacks would remove the need for this workaround.