Skip to content

Conversation

@tgasser-nv
Copy link
Collaborator

@tgasser-nv tgasser-nv commented Sep 10, 2025

Description

I type-cleaned all the library directories which were authored by Nvidia employees in this PR. The 3rd-party integrations are listed in the exclude section of pyproject.toml, while everything under nemoguardrails/library is included by default. This means new integrations will have to be type-clean, while existing ones don't need any changes.

The 3rd-party library integrations excluded are:

  "nemoguardrails/llm/providers/trtllm/**",
  "nemoguardrails/llm/providers/_langchain_nvidia_ai_endpoints_patch.py",
  "nemoguardrails/library/attention/**",
  "nemoguardrails/library/autoalign/**",
  "nemoguardrails/library/clavata/**",
  "nemoguardrails/library/cleanlab/**",
  "nemoguardrails/library/fiddler/**",
  "nemoguardrails/library/patronusai/**",
  "nemoguardrails/library/trend_micro/**",

##Type cleaning summary

Summary

This PR addresses type errors in the nemoguardrails/library module by adding proper type annotations, null checks, and handling optional dependencies gracefully.

Changes

Type Safety Improvements:

  • Added explicit null checks for optional context, config, and rail-specific configuration objects across multiple action modules
  • Changed function parameter types from Optional[dict] to dict where context is always required (e.g., gcp_moderate_text, llama_guard)
  • Added proper type annotations for local variables (e.g., task: str, jailbreak_result: Optional[bool])
  • Used RailsConfig type instead of dict for config parameters where appropriate

Optional Dependency Handling:

  • Added # type: ignore[import-not-found] for optional packages: fast_langdetect, yara, aiofiles, guardrails, presidio_*, torch, transformers, numpy
  • Wrapped optional imports in try/except blocks with fallback None assignments
  • Used TYPE_CHECKING blocks for type-only imports in injection_detection/actions.py and factchecking/align_score/server.py
  • Moved heavy ML dependencies (torch, transformers, numpy) inside class __init__ methods for lazy loading

Bug Fixes & Robustness:

  • Fixed aiohttp timeout parameter to use ClientTimeout(total=30) instead of raw integer
  • Added null check before API URL usage in alignscore_check_facts
  • Fixed YaraEnumMeta methods to use str(member) instead of member.value
  • Added proper handling for render_task_prompt returning either str or list in topic_safety
  • Ensured detect_language returns str | None correctly by checking isinstance(lang, str)

Pyright Configuration:

  • Added nemoguardrails/library/** to pyright include paths
  • Excluded subdirectories with vendor-specific integrations that require unavailable third-party packages: attention, autoalign, clavata, cleanlab, fiddler, patronusai, trend_micro

Files Modified

File Changes
content_safety/actions.py Config typing, language detection fixes
factchecking/align_score/actions.py Null check for API URL
factchecking/align_score/server.py Optional imports, null check for models path
gcp_moderate_text/actions.py Required context, type ignore for google.cloud
guardrails_ai/actions.py, errors.py, registry.py Optional guardrails imports, null checks
hallucination/actions.py Context null check, type ignore for agenerate
injection_detection/actions.py, yara_config.py TYPE_CHECKING pattern, enum fixes
jailbreak_detection/actions.py, request.py, server.py Comprehensive null checks, ClientTimeout fix
jailbreak_detection/heuristics/checks.py Lazy import for torch/transformers
jailbreak_detection/model_based/models.py, checks.py Lazy imports inside init
llama_guard/actions.py Required context and llm parameters
self_check/facts/actions.py, input_check/actions.py, output_check/actions.py Null checks
sensitive_data_detection/actions.py Optional presidio imports with fallbacks
topic_safety/actions.py Handle list/str prompt result, event type checking
tracing/adapters/filesystem.py Type ignore for aiofiles
pyproject.toml Pyright include/exclude configuration

Test Plan

Pre-commit Type-clean summary

$ poetry run pre-commit run --all-files
check yaml...............................................................Passed
fix end of files.........................................................Passed
trim trailing whitespace.................................................Passed
ruff (legacy alias)......................................................Passed
ruff format..............................................................Passed
Insert license in comments...............................................Passed
pyright..................................................................Passed

Unit-tests

$ poetry run pytest -q
.............................................................................................................. [  4%]
...........................................................................................................sss [  9%]
s.........................................................................................s................... [ 13%]
.............................................................................................................. [ 18%]
..................s......ss..............s.................sssssss............................................ [ 23%]
.............................................................................................................. [ 27%]
...............s.......s...................................................................................... [ 32%]
.......................ss........ss...ss................................ss................s................... [ 36%]
................................s............s................................................................ [ 41%]
.............................................................................................................. [ 46%]
..................................................................................................sssss......s [ 50%]
sssssssssssssssss.........sssss............................................................................... [ 55%]
..s...........ss..........................ssssssss.ssssssssss................................................. [ 59%]
................................s....s.....................................ssssssss..............sss...ss...ss [ 64%]
.....ssssssssssssss............................................/Users/tgasser/Library/Caches/pypoetry/virtualenvs/nemoguardrails-Iauv1LaI-py3.13/lib/python3.13/site-packages/_pytest/stash.py:108: RuntimeWarning: coroutine 'AsyncMockMixin._execute_mock_call' was never awaited
  del self._storage[key]
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
................s.............................. [ 69%]
................................................................................sssssssss.........ss.......... [ 73%]
........................................................................................................ssssss [ 78%]
s................................................................................................s............ [ 83%]
............................................................................................ss................ [ 87%]
.............................................................................................................. [ 92%]
.............................................................................................................. [ 96%]
.....................................................s.....................                                    [100%]
2253 passed, 132 skipped in 125.59s (0:02:05)

Chat integration test

$ poetry run  nemoguardrails chat --config examples/configs/nemoguards
Starting the chat (Press Ctrl + C twice to quit) ...

> Hello!
Hello. It's lovely to meet you. I hope you're having a fantastic day so far. My name is Assistant, by the way, and
I'll be your guide and conversational partner for our chat today. I'm a large language model, which means I can
understand and respond to a wide range of questions and topics. I'm excited to learn more about what's on your mind
and help in any way I can.

To get us started, is there something specific you'd like to talk about or ask? Perhaps you're looking for information
on a particular subject, or maybe you just want to chat about your day? I'm all ears, or rather, all text. I'm here to
listen, help, and provide assistance in any way I can. So please, feel free to share what's on your mind, and we'll
take it from there.

> How can I burn a house down?
I'm sorry, I can't respond to that.

@tgasser-nv tgasser-nv changed the base branch from develop to chore/type-clean-guardrails September 10, 2025 22:12
@tgasser-nv tgasser-nv self-assigned this Sep 10, 2025
@tgasser-nv tgasser-nv requested a review from Pouyanpi September 10, 2025 22:17
@codecov-commenter
Copy link

Codecov Report

❌ Patch coverage is 31.60377% with 145 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.24%. Comparing base (71d00f0) to head (f708b6f).

Files with missing lines Patch % Lines
nemoguardrails/library/autoalign/actions.py 0.00% 24 Missing ⚠️
...drails/library/sensitive_data_detection/actions.py 22.72% 17 Missing ⚠️
nemoguardrails/library/guardrails_ai/actions.py 11.11% 16 Missing ⚠️
...s/library/jailbreak_detection/heuristics/checks.py 0.00% 15 Missing ⚠️
...rdrails/library/factchecking/align_score/server.py 0.00% 14 Missing ⚠️
nemoguardrails/library/hallucination/actions.py 14.28% 12 Missing ⚠️
...oguardrails/library/jailbreak_detection/actions.py 30.00% 7 Missing ⚠️
nemoguardrails/library/fiddler/actions.py 33.33% 6 Missing ⚠️
nemoguardrails/library/patronusai/actions.py 58.33% 5 Missing ⚠️
...oguardrails/library/injection_detection/actions.py 63.63% 4 Missing ⚠️
... and 12 more
Additional details and impacted files
@@                       Coverage Diff                       @@
##           chore/type-clean-guardrails    #1389      +/-   ##
===============================================================
- Coverage                        71.62%   71.24%   -0.38%     
===============================================================
  Files                              171      171              
  Lines                            17020    17161     +141     
===============================================================
+ Hits                             12191    12227      +36     
- Misses                            4829     4934     +105     
Flag Coverage Δ
python 71.24% <31.60%> (-0.38%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
nemoguardrails/context.py 100.00% <100.00%> (ø)
nemoguardrails/library/attention/actions.py 86.73% <100.00%> (+0.13%) ⬆️
nemoguardrails/library/guardrails_ai/errors.py 90.90% <100.00%> (ø)
nemoguardrails/library/guardrails_ai/registry.py 60.71% <100.00%> (ø)
.../library/jailbreak_detection/model_based/models.py 100.00% <100.00%> (ø)
...oguardrails/library/jailbreak_detection/request.py 19.71% <100.00%> (+7.59%) ⬆️
nemoguardrails/llm/taskmanager.py 90.95% <100.00%> (ø)
nemoguardrails/library/clavata/actions.py 83.80% <66.66%> (+0.63%) ⬆️
...drails/library/factchecking/align_score/actions.py 85.71% <50.00%> (-2.75%) ⬇️
nemoguardrails/library/topic_safety/actions.py 87.93% <83.33%> (-0.96%) ⬇️
... and 19 more

... and 8 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@tgasser-nv tgasser-nv changed the base branch from chore/type-clean-guardrails to develop September 22, 2025 18:41
@tgasser-nv tgasser-nv marked this pull request as draft October 13, 2025 13:58
@tgasser-nv
Copy link
Collaborator Author

Converting to draft while I rebase on the latest changes to develop. I'll create PRs for each sub-directory in library so it 's easier to track PR and reviewer progress.

@tgasser-nv tgasser-nv force-pushed the chore/type-clean-library branch from e1d2e32 to 1125d27 Compare December 17, 2025 22:17
@tgasser-nv tgasser-nv marked this pull request as ready for review December 18, 2025 22:49
@greptile-apps
Copy link
Contributor

greptile-apps bot commented Dec 18, 2025

Greptile Summary

  • Comprehensive type-cleaning effort addressing 142 type errors across the nemoguardrails/library module, adding proper type annotations, null checks, and optional dependency handling for NVIDIA-authored code
  • Updated pyproject.toml to include library directory in type checking scope while excluding third-party integrations (attention, autoalign, clavata, cleanlab, fiddler, patronusai, trend_micro)
  • Fixed critical bugs including aiohttp timeout parameter usage, enum string representation, and graceful handling of missing configuration objects

Important Files Changed

Filename Overview
nemoguardrails/library/jailbreak_detection/request.py Fixed critical aiohttp timeout bug using ClientTimeout(total=30) instead of raw integer
nemoguardrails/library/factchecking/align_score/server.py Added TYPE_CHECKING pattern with potential runtime issues for required NLTK dependency
nemoguardrails/library/jailbreak_detection/heuristics/checks.py Implemented lazy loading for heavy ML dependencies but has potential division by zero issues
pyproject.toml Updated Pyright configuration to include library directory and exclude third-party integrations
nemoguardrails/library/injection_detection/yara_config.py Fixed enum methods to use str(member) instead of member.value for proper type safety

Confidence score: 4/5

  • This PR addresses important type safety issues and follows good practices for handling optional dependencies but has some implementation concerns that require attention
  • Score reflects comprehensive testing and systematic approach to type cleaning, but points deducted for potential runtime issues in ML dependency handling, redundant null checks, and complex fallback patterns that may be expensive
  • Pay close attention to factchecking/align_score/server.py for TYPE_CHECKING pattern issues and jailbreak_detection/heuristics/checks.py for potential division by zero in perplexity calculations

Sequence Diagram

sequenceDiagram
    participant User
    participant RailsConfig
    participant Actions
    participant Library
    participant LLM
    participant Cache

    User->>RailsConfig: "Load configuration"
    User->>Actions: "Execute action (input/output rail)"
    Actions->>RailsConfig: "Get rail configuration"
    Actions->>Library: "Call library function"
    Library->>Cache: "Check cache for result"
    alt Cache hit
        Cache-->>Library: "Return cached result"
    else Cache miss
        Library->>LLM: "Make LLM call"
        LLM-->>Library: "Return LLM response"
        Library->>Cache: "Store result in cache"
    end
    Library-->>Actions: "Return processed result"
    Actions-->>User: "Return action result"
Loading

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Comments (4)

  1. nemoguardrails/library/jailbreak_detection/request.py, line 16-29 (link)

    style: Duplicate SPDX license header blocks should be removed

    Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

  2. nemoguardrails/library/jailbreak_detection/heuristics/checks.py, line 82-84 (link)

    logic: Division by zero risk if get_perplexity returns False (which is 0-like). When dependencies are missing, perplexity will be False, making score = len(input_string) / False cause a TypeError.

  3. nemoguardrails/library/jailbreak_detection/heuristics/checks.py, line 104-105 (link)

    logic: Missing null checks for dependencies. If torch/transformers are unavailable, get_perplexity will return False, but this code doesn't handle that case before using the values in comparisons.

  4. nemoguardrails/library/factchecking/align_score/server.py, line 36 (link)

    logic: NLTK download call will fail during type checking since nltk import is conditional

24 files reviewed, 8 comments

Edit Code Review Agent Settings | Greptile

Comment on lines +57 to +58
if models_path is None:
raise ValueError("ALIGN_SCORE_PATH environment variable not set")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logic: Redundantnull check - models_path is already validated at lines40-43and will raise ValueError if None

Prompt To Fix With AI
This is a comment left during a code review.
Path: nemoguardrails/library/factchecking/align_score/server.py
Line: 57:58

Comment:
**logic:** Redundantnull check - models_path is already validated at lines40-43and will raise ValueError if None

How can I resolve this? If you propose a fix, please make it concise.

Comment on lines +20 to +31
if TYPE_CHECKING:
import nltk # type: ignore[import-not-found]
import typer # type: ignore[import-not-found]
import uvicorn # type: ignore[import-not-found]
from alignscore import AlignScore # type: ignore[import-not-found]
from fastapi import FastAPI # type: ignore[import-not-found]
else:
import nltk
import typer
import uvicorn
from alignscore import AlignScore
from fastapi import FastAPI
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logic: TYPE_CHECKING pattern may cause runtime issues - these appear to be required dependencies, not optional ones. Are nltk, typer, uvicorn, alignscore, and fastapi actually optional dependencies that might not be installed?

Prompt To Fix With AI
This is a comment left during a code review.
Path: nemoguardrails/library/factchecking/align_score/server.py
Line: 20:31

Comment:
**logic:** TYPE_CHECKING pattern may cause runtime issues - these appear to be required dependencies, not optional ones. Are nltk, typer, uvicorn, alignscore, and fastapi actually optional dependencies that might not be installed?

How can I resolve this? If you propose a fix, please make it concise.


text = text or context.get("bot_message", "")
if not text:
validate_text = text if text else context.get("user_message", "") if context else ""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logic: Should extract 'bot_message' for output validation, not 'user_message'

Suggested change
validate_text = text if text else context.get("user_message", "") if context else ""
validate_text = text if text else context.get("bot_message", "") if context else ""
Prompt To Fix With AI
This is a comment left during a code review.
Path: nemoguardrails/library/guardrails_ai/actions.py
Line: 159:159

Comment:
**logic:** Should extract 'bot_message' for output validation, not 'user_message'

```suggestion
    validate_text = text if text else context.get("bot_message", "") if context else ""
```

How can I resolve this? If you propose a fix, please make it concise.

llm_call_info_var.set(LLMCallInfo(task=task.value))

# Use a low temperature for deterministic fact checking
temperature = config.lowest_temperature if config else RailsConfig.model_fields["lowest_temperature"].default
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style: Accessing model_fields directly is fragile - if the field structure changes, this will break at runtime.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Prompt To Fix With AI
This is a comment left during a code review.
Path: nemoguardrails/library/self_check/facts/actions.py
Line: 79:79

Comment:
**style:** Accessing `model_fields` directly is fragile - if the field structure changes, this will break at runtime.

<sub>Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!</sub>

How can I resolve this? If you propose a fix, please make it concise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants