Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions backend/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,8 @@ bash test-preflight.sh # Verify env
bash test.sh # Run all tests (CI source of truth)
```

On Windows, use `powershell -NoProfile -ExecutionPolicy Bypass -File .\test-preflight.ps1` for the preflight check, then `powershell -NoProfile -ExecutionPolicy Bypass -File .\test.ps1` to run the same pytest list from `test.sh` with `PYTHONUTF8=1`.

**New test files must be added to `test.sh`** or they won't run in CI.

Pre-mock heavy deps before importing the module under test. Use `patch.object(target_module, "func")` not string-based `patch("module.func")` — the string form silently patches the wrong reference if the function was already imported. When modules construct objects at import time, use lazy getters to avoid triggering heavy init in tests.
Expand Down
28 changes: 28 additions & 0 deletions backend/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,34 @@ This README provides a quick setup guide for the Omi backend. For a comprehensiv
deactivate
```

## Running Backend Tests on Windows

From PowerShell, run the Windows preflight before the backend test suite:

```powershell
cd backend
powershell -NoProfile -ExecutionPolicy Bypass -File .\test-preflight.ps1
```

If you have multiple Python versions installed, pass the Python 3.11 executable explicitly:

```powershell
powershell -NoProfile -ExecutionPolicy Bypass -File .\test-preflight.ps1 -Python C:\Path\To\Python311\python.exe
```

The preflight checks Python, pytest, formatter availability, key backend imports, optional integration environment
variables, Redis CLI connectivity, and whether every test referenced by `test.sh` exists.

After the preflight passes, run the Windows test wrapper:

```powershell
powershell -NoProfile -ExecutionPolicy Bypass -File .\test.ps1
```

`test.ps1` reads the pytest commands from `test.sh`, sets `PYTHONUTF8=1` for Windows source reads, and skips the
fair-use integration tests when `redis-cli` is not available. To preview the pytest commands without running them, add
`-List`.

## Additional Resources

- [Full Backend Setup Documentation](https://docs.omi.me/developer/backend/Backend_Setup)
Expand Down
264 changes: 264 additions & 0 deletions backend/test-preflight.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,264 @@
param(
[string]$Python
)

$ErrorActionPreference = "Stop"

$RootDir = Split-Path -Parent $MyInvocation.MyCommand.Path
Set-Location $RootDir

$script:PassCount = 0
$script:WarnCount = 0
$script:FailCount = 0
$script:PythonCommand = @()

function Write-Ok {
param([string]$Message)
Write-Host " [OK] $Message" -ForegroundColor Green
$script:PassCount += 1
}

function Write-Warn {
param([string]$Message)
Write-Host " [WARN] $Message" -ForegroundColor Yellow
$script:WarnCount += 1
}

function Write-Bad {
param([string]$Message)
Write-Host " [FAIL] $Message" -ForegroundColor Red
$script:FailCount += 1
}

function Set-PythonCommand {
if ($Python) {
$script:PythonCommand = @($Python)
return
}

if (Get-Command python -ErrorAction SilentlyContinue) {
$script:PythonCommand = @("python")
return
}

if (Get-Command py -ErrorAction SilentlyContinue) {
$script:PythonCommand = @("py", "-3")
return
}
}

function Invoke-Python {
param([string[]]$Arguments)

$exe = $script:PythonCommand[0]
$prefix = @()
if ($script:PythonCommand.Count -gt 1) {
$prefix = $script:PythonCommand[1..($script:PythonCommand.Count - 1)]
}

$oldErrorActionPreference = $ErrorActionPreference
$ErrorActionPreference = "Continue"
try {
& $exe @prefix @Arguments
} finally {
$ErrorActionPreference = $oldErrorActionPreference
}
}

function Test-PythonImport {
param([string]$ModuleName)

Invoke-Python @("-c", "import $ModuleName") *> $null
return $LASTEXITCODE -eq 0
}

function Get-EnvValue {
param([string]$Name)
return [Environment]::GetEnvironmentVariable($Name)
}

function Check-Env {
param(
[string]$Name,
[string]$Description
)

if (Get-EnvValue $Name) {
Write-Ok "$Name ($Description)"
} else {
Write-Warn "$Name not set ($Description)"
}
}

Write-Host "Tools:"
Set-PythonCommand

if ($script:PythonCommand.Count -eq 0) {
Write-Bad "python not found (install Python 3.11 or pass -Python <path>)"
} else {
$pythonVersion = (Invoke-Python @("--version") 2>&1) -join " "
if ($LASTEXITCODE -eq 0) {
if ($pythonVersion -match "Python 3\.11\.") {
Write-Ok $pythonVersion
} else {
Write-Warn "$pythonVersion detected; backend expects Python 3.11"
}
Comment on lines +101 to +105

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Version mismatch exits with hard failure, blocking non-3.11 Windows contributors

The bash counterpart (test-preflight.sh) reports whatever Python version is present as "ok" without any version gate. This script uses Write-Bad for any Python that isn't 3.11.x, which causes exit code 1 and the message "Fix failures above before running test.sh" for a developer running Python 3.12 or 3.13 — even if their environment is otherwise fully functional. Using Write-Warn here would match the bash behavior (advisory rather than blocking) and avoid unnecessarily gating Windows contributors on an exact minor version.

Suggested change
if ($pythonVersion -match "Python 3\.11\.") {
Write-Ok $pythonVersion
} else {
Write-Bad "$pythonVersion detected; backend expects Python 3.11"
}
if ($pythonVersion -match "Python 3\.11\.") {
Write-Ok $pythonVersion
} else {
Write-Warn "$pythonVersion detected; backend expects Python 3.11"
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 3d942f839: the Python version mismatch now uses Write-Warn instead of Write-Bad, so non-3.11 Windows users are not blocked by the preflight. Re-ran on this Windows machine with Python 3.10.6: 10 passed, 10 warnings, 0 failed, exit code 0.

} else {
Write-Bad "python command failed"
}
}

if ($script:PythonCommand.Count -gt 0) {
$pytestVersion = (Invoke-Python @("-m", "pytest", "--version") 2>&1) -join " "
if ($LASTEXITCODE -eq 0) {
Write-Ok $pytestVersion
} else {
Write-Bad "pytest not installed (pip install pytest)"
}
}

if (Get-Command black -ErrorAction SilentlyContinue) {
Write-Ok "black (formatter)"
} elseif ($script:PythonCommand.Count -gt 0) {
Invoke-Python @("-m", "black", "--version") *> $null
if ($LASTEXITCODE -eq 0) {
Write-Ok "python -m black (formatter)"
} else {
Write-Warn "black not installed; pre-commit formatting will fail (pip install black)"
}
} else {
Write-Warn "black not checked because python was not found"
}

Write-Host ""
Write-Host "Python packages:"

$missingPackages = @()
if ($script:PythonCommand.Count -gt 0) {
foreach ($pkg in @("pydantic", "fastapi", "firebase_admin", "google.cloud.firestore", "redis", "deepgram_sdk", "openpipe")) {
if (Test-PythonImport $pkg) {
Write-Ok $pkg
} else {
$missingPackages += $pkg
Write-Warn "$pkg not importable"
}
}
}

if ($missingPackages.Count -gt 0) {
Write-Host " -> Run: pip install -r requirements.txt" -ForegroundColor Yellow
}

Write-Host ""
Write-Host "Env vars (unit tests):"

if (Get-EnvValue "ENCRYPTION_SECRET") {
Write-Ok "ENCRYPTION_SECRET (set in env)"
} else {
Write-Ok "ENCRYPTION_SECRET (set by test.sh; no action needed)"
}

Write-Host ""
Write-Host "Env vars (integration - optional):"

Check-Env "OPENAI_API_KEY" "LLM calls; some integration tests skip without it"
Check-Env "DEEPGRAM_API_KEY" "STT streaming and pre-recorded transcription"
Check-Env "ADMIN_KEY" "admin endpoint tests"
Check-Env "REDIS_DB_HOST" "Redis connection (default: localhost)"
Check-Env "REDIS_DB_PASSWORD" "Redis auth"
Check-Env "GOOGLE_APPLICATION_CREDENTIALS" "Firebase/Firestore integration tests"

Write-Host ""
Write-Host "Services:"

$redisCli = Get-Command redis-cli -ErrorAction SilentlyContinue
if ($redisCli) {
$redisHost = Get-EnvValue "REDIS_DB_HOST"
if (-not $redisHost) {
$redisHost = "localhost"
}
$redisPort = Get-EnvValue "REDIS_DB_PORT"
if (-not $redisPort) {
$redisPort = "6379"
}

$redisArgs = @("-h", $redisHost, "-p", $redisPort)
$redisPassword = Get-EnvValue "REDIS_DB_PASSWORD"
$redisArgs += "ping"

$previousRedisCliAuth = [Environment]::GetEnvironmentVariable("REDISCLI_AUTH")
if ($redisPassword) {
$env:REDISCLI_AUTH = $redisPassword
}

try {
& $redisCli.Source @redisArgs *> $null
} finally {
if ($null -eq $previousRedisCliAuth) {
Remove-Item Env:REDISCLI_AUTH -ErrorAction SilentlyContinue
} else {
$env:REDISCLI_AUTH = $previousRedisCliAuth
}
}

if ($LASTEXITCODE -eq 0) {
Write-Ok "Redis ($redisHost`:$redisPort) connected"
} else {
Write-Warn "Redis ($redisHost`:$redisPort) not reachable (integration tests may fail)"
}
} else {
Write-Warn "redis-cli not installed; cannot check Redis connectivity"
}

Write-Host ""
Write-Host "Test files:"

$unitTests = Get-ChildItem -Path "tests/unit" -Filter "test_*.py" -File -ErrorAction SilentlyContinue
if ($unitTests.Count -gt 0) {
Write-Ok "$($unitTests.Count) unit test files found"
} else {
Write-Bad "No unit test files found in tests/unit/"
}

$missingTests = @()
if (Test-Path "test.sh") {
Comment on lines +219 to +224

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 test.sh flag-stripping regex may silently drop test references if flags change

The pattern \s+-v\s*$ strips only a trailing -v. Lines in test.sh that carry additional pytest flags (e.g., --tb=short, -x) won't be stripped correctly, leaving the flag text as part of the resolved path, which will then fail Test-Path and be reported as a missing file. The bash counterpart uses sed 's/ -v//' (not anchored) which has the same narrow assumption. Both are fine today, but worth noting if test.sh flags ever change.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 3d942f839: the test.sh validation now tokenizes each pytest line and only keeps tokens matching tests/*, so future pytest flags are ignored instead of being resolved as paths. Re-ran the Windows preflight: 10 passed, 10 warnings, 0 failed, exit code 0.

$testRefs = Get-Content "test.sh" |
ForEach-Object {
$line = $_.Trim()
if ($line -match "^pytest\s+") {
$line -split "\s+" | Where-Object { $_ -like "tests/*" }
}
}

foreach ($testRef in $testRefs) {
if (-not (Test-Path $testRef)) {
$missingTests += $testRef
}
}

if ($missingTests.Count -gt 0) {
Write-Bad "test.sh references missing files: $($missingTests -join ', ')"
} else {
Write-Ok "All test.sh references resolve to existing files"
}
} else {
Write-Bad "test.sh not found"
}

Write-Host ""
Write-Host "----------------------------------------"
$total = $script:PassCount + $script:WarnCount + $script:FailCount
Write-Host " $script:PassCount passed $script:WarnCount warnings $script:FailCount failed ($total checks)"

if ($script:FailCount -gt 0) {
Write-Host " Fix failures above before running test.sh" -ForegroundColor Red
exit 1
}

if ($script:WarnCount -gt 0) {
Write-Host " Warnings are optional; unit tests should still pass" -ForegroundColor Yellow
exit 0
}

Write-Host " All clear; ready to run test.sh" -ForegroundColor Green
exit 0
59 changes: 59 additions & 0 deletions backend/test.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
param(
[string]$Python = "python",
[switch]$List
)

$ErrorActionPreference = "Stop"

$RootDir = Split-Path -Parent $MyInvocation.MyCommand.Path
Set-Location $RootDir

$env:ENCRYPTION_SECRET = "omi_ZwB2ZNqB2HHpMK6wStk7sTpavJiPTFg7gXUHnc4tFABPU6pZ2c2DKgehtfgi4RZv"
$env:PYTHONUTF8 = "1"

if (-not (Test-Path "test.sh")) {
Write-Error "test.sh not found"
}

$pytestCommands = Get-Content "test.sh" |
Where-Object { $_ -match "^pytest\s+tests/unit/" } |
ForEach-Object { $_.Trim() }

if ($pytestCommands.Count -eq 0) {
Write-Error "No pytest commands found in test.sh"
}

if ($List) {
$pytestCommands | ForEach-Object { Write-Output $_ }
exit 0
}

foreach ($command in $pytestCommands) {
$arguments = $command -split "\s+"
Write-Host "> $Python -m $($arguments -join ' ')" -ForegroundColor Cyan
& $Python -m @arguments
if ($LASTEXITCODE -ne 0) {
exit $LASTEXITCODE
}
}

$redisCli = Get-Command redis-cli -ErrorAction SilentlyContinue
if ($redisCli) {
& $redisCli.Source ping *> $null
if ($LASTEXITCODE -eq 0) {
foreach ($integrationTest in @(
"tests/integration/test_fair_use_live.py",
"tests/integration/test_fair_use_api.py"
)) {
Write-Host "> $Python -m pytest $integrationTest -v" -ForegroundColor Cyan
& $Python -m pytest $integrationTest -v
if ($LASTEXITCODE -ne 0) {
exit $LASTEXITCODE
}
}
} else {
Write-Host "SKIP: fair-use integration tests (Redis not available)"
}
} else {
Write-Host "SKIP: fair-use integration tests (redis-cli not available)"
}
Loading