Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
277 changes: 275 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,59 @@ keyprobe --huggingface <key>
keyprobe --replicate <key>
```

### Requirement Flags

Validate specific capabilities or models with **fuzzy matching** and **short aliases**:

```sh
# Capability validation
keyprobe --require-streaming <key> # Real-time streaming
keyprobe --require-embeddings <key> # Vector embeddings
keyprobe --require-chat <key> # Chat completion
keyprobe --require-assistants <key> # AI assistants
keyprobe --require-dall-e <key> # Image generation

# Model validation
keyprobe --require-model "gpt-4o" <key> # Specific model access
keyprobe --require-model "claude-3-5-sonnet" <key>

# Combined validation
keyprobe --require-streaming --require-model "gpt-4o" <key>

# Provider-specific
keyprobe --anthropic --require-files <key> # Anthropic files API
keyprobe --openai --require-whisper <key> # OpenAI speech-to-text

# Fuzzy matching works too!
keyprobe --require "streamig" <key> # Matches "Streaming"
keyprobe --require "embedings" <key> # Matches "Embeddings"
```

### Common Use Cases

```sh
# Real-time applications
keyprobe --require-streaming sk-ant-api03-...

# Vector search & RAG
keyprobe --require-embeddings sk-proj-...

# Advanced reasoning models
keyprobe --require-model "gpt-4o" sk-proj-...

# Production deployment
keyprobe --require-chat --require-streaming sk-proj-...

# Multi-modal AI
keyprobe --require-dall-e --require-whisper sk-proj-...

# AI assistants & agents
keyprobe --require-assistants sk-proj-...

# Spelling mistakes work too!
keyprobe --require "streamig" sk-ant-api03-...
```

## Output

**CLI (default)**
Expand Down Expand Up @@ -84,12 +137,36 @@ keyprobe --output json <key>
}
```


## ⚡ Short Aliases
No quotes needed for common capabilities:

```bash
keyprobe --require-streaming <key> # --require "Streaming"
keyprobe --require-embeddings <key> # --require "Embeddings"
keyprobe --require-chat <key> # --require "Messages API (chat)"
keyprobe --require-batch <key> # --require "Batch API"
keyprobe --require-files <key> # --require "Files API (beta)"
keyprobe --require-assistants <key> # --require "Assistants API"
keyprobe --require-dall-e <key> # --require "DALL-E 3 (image gen)"
keyprobe --require-whisper <key> # --require "Whisper (speech-to-text)"
keyprobe --require-fine-tuning <key> # --require "Fine-tuning"
```


## Exit codes

| Code | Meaning |
|------|---------|
| `0` | Key is valid |
| `1` | Key is invalid or an error occurred |
| `0` | Key is valid and all requirements met |
| `1` | Key is invalid, error occurred, or requirements failed |

**Exit code 1 scenarios:**
- API key is invalid or expired
- Required capability is not available (`--require`)
- Required model is not accessible (`--require-model`)
- Network errors or API failures
- Invalid command-line arguments

## Supported providers

Expand All @@ -108,6 +185,202 @@ keyprobe --output json <key>
| DeepSeek | 64-char key |
| Cohere | `sk-cohere-` or 40-char key |

## GitHub Actions Integration

KeyProbe integrates seamlessly with GitHub Actions for automated API key validation in CI/CD pipelines with support for capability and model requirements.

### Quick Setup

1. **Add your API key to GitHub Secrets:**
- Go to your repository → Settings → Secrets and variables → Actions
- Add a new secret named `LLM_API_KEY` with your API key value

2. **Copy this workflow to `.github/workflows/keyprobe.yml`:**
```yaml
name: KeyProbe API Key Validation

on:
workflow_dispatch:
inputs:
required_capability:
description: 'Required capability (e.g., "Streaming", "Embeddings")'
required: false
type: string
default: ''
required_model:
description: 'Required model (e.g., "gpt-4o", "claude-3-sonnet")'
required: false
type: string
default: ''
push:
branches: [ main, master ]
pull_request:
branches: [ main, master ]

jobs:
validate:
runs-on: ubuntu-latest
timeout-minutes: 5

steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: '1.21'

- name: Build KeyProbe CLI
run: |
git clone https://github.com/datumbrain/keyprobe.git keyprobe-temp
cd keyprobe-temp
go build -o ../keyprobe .
cd ..
rm -rf keyprobe-temp

- name: Safety Check - Verify API Key Secret
run: |
# Fail early if LLM_API_KEY secret is missing or empty
if [ -z "$LLM_API_KEY" ]; then
echo "❌ LLM_API_KEY secret is missing or empty"
echo "Please add LLM_API_KEY to your repository secrets"
exit 1
fi
echo "✅ API key secret is present"
env:
LLM_API_KEY: ${{ secrets.LLM_API_KEY }}

- name: Validate API Key and Requirements
run: |
# Build KeyProbe command with optional requirement flags
CMD="./keyprobe"

# Add --require flag if capability is specified
if [ -n "${{ github.event.inputs.required_capability }}" ]; then
CMD="$CMD --require '${{ github.event.inputs.required_capability }}'"
echo "🔍 Checking for required capability: ${{ github.event.inputs.required_capability }}"
fi

# Add --require-model flag if model is specified
if [ -n "${{ github.event.inputs.required_model }}" ]; then
CMD="$CMD --require-model '${{ github.event.inputs.required_model }}'"
echo "🔍 Checking for required model: ${{ github.event.inputs.required_model }}"
fi

# Add API key
CMD="$CMD '$LLM_API_KEY'"

echo "🚀 Running command: $CMD"
echo ""

# Execute validation - KeyProbe will exit with code 1 if:
# - API key is invalid
# - Required capability is not available
# - Required model is not accessible
if eval "$CMD"; then
echo ""
echo "✅ All validations passed successfully"
else
echo ""
echo "❌ Validation failed - workflow will exit with error"
exit 1
fi
env:
LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
```

### Usage Examples

**Basic validation (runs automatically on pushes/PRs):**
- Workflow validates the API key stored in `LLM_API_KEY` secret
- If invalid, the workflow fails and blocks merges

**Manual validation with capability requirement:**
1. Go to Actions → KeyProbe API Key Validation → Run workflow
2. Enter "Streaming" in the "Required capability" field
3. Leave "Required model" empty
4. Click "Run workflow"
5. Workflow validates the key and streaming capability

**Manual validation with model requirement:**
1. Go to Actions → KeyProbe API Key Validation → Run workflow
2. Leave "Required capability" empty
3. Enter "gpt-4o" in the "Required model" field
4. Click "Run workflow"
5. Workflow validates the key and GPT-4o access

**Manual validation with both requirements:**
1. Go to Actions → KeyProbe API Key Validation → Run workflow
2. Enter "Streaming" in "Required capability"
3. Enter "gpt-4o" in "Required model"
4. Click "Run workflow"
5. Workflow validates key, streaming capability, and GPT-4o access

### Common CI/CD Scenarios

```yaml
# Example: Require streaming for real-time features (using short alias)
- name: Validate Streaming Capability
run: |
./keyprobe --require-streaming "$LLM_API_KEY"

# Example: Require specific model for production
- name: Validate Production Model
run: |
./keyprobe --require-model "gpt-4o" "$LLM_API_KEY"

# Example: Comprehensive validation with short aliases
- name: Validate All Requirements
run: |
./keyprobe --require-streaming --require-model "gpt-4o" "$LLM_API_KEY"

# Example: Multiple capabilities validation
- name: Validate Multiple Capabilities
run: |
./keyprobe --require-chat --require-embeddings --require-assistants "$LLM_API_KEY"

# Example: Provider-specific validation
- name: Validate Anthropic Features
run: |
./keyprobe --anthropic --require-streaming --require-files "$LLM_API_KEY"
```

### Requirements

- No external dependencies required - workflow uses KeyProbe's built-in requirement flags
- KeyProbe handles all validation logic internally
- Optional JSON validation step for detailed reporting

### How Failure Works

KeyProbe uses **exit codes** to automatically control workflow behavior:

- **Exit code 0**: Key is valid and all requirements met → Workflow continues ✅
- **Exit code 1**: Key is invalid, requirements failed, or error occurred → Workflow fails ❌

**Exit code 1 scenarios:**
- API key is invalid or expired
- Required capability is not available (`--require`)
- Required model is not accessible (`--require-model`)
- Network errors or API failures
- Invalid command-line arguments

**CI/CD Integration:**
- Failed validation blocks deployments and pull request merges
- No additional logic needed - GitHub Actions respects exit codes automatically
- Clear error messages show exactly what failed (key validity, missing capability, or model access)
- JSON validation step provides detailed diagnostics for troubleshooting

### Production Features

- **Robust validation**: Uses KeyProbe's built-in requirement flags with proper error handling
- **Flexible requirements**: Support for both capability and model validation
- **Security**: API keys only accessed via GitHub Secrets
- **Detailed reporting**: Optional JSON validation for troubleshooting
- **Manual control**: Optional requirement checking for specific deployment scenarios
- **Clean integration**: Works seamlessly with existing CI/CD pipelines

## License

MIT
Loading