Skip to content

Commit 688454c

Browse files
viadezo1erviadezo1er
authored
Cli install sdk evals (#85)
### bt setup instrument: interactive mode, language selection, and scoped permissions Adds three new flags to bt setup instrument and wires them end-to-end through agent invocation and task generation. ```--interactive / -i``` opens the agent in its interactive TUI (Claude Code, etc.) so the user can review and approve each tool use. ```--yolo``` runs the agent in the background with bypassPermissions — no approval prompts. ```--language <LANG>``` restricts instrumentation to specific languages (python, typescript, go, java, ruby, csharp); repeatable; omit to let the agent auto-detect. #### Run-mode prompt (interactive terminal, no flags) When none of the above flags are passed and the terminal is interactive, the user is asked how to run the agent. Background mode uses acceptEdits with --allowedTools scoped to the package managers for the selected language(s) only (e.g. uv for Python, npm/yarn/pnpm for TypeScript, dotnet for C#). Interactive TUI mode opens the agent's terminal UI. #### Language selection prompt A multi-select prompt is shown between the workflow and run-mode prompts. Selecting "All languages" (the default) lets the agent auto-detect; selecting specific languages also narrows the background tool allowlist.
1 parent b9cc7d1 commit 688454c

10 files changed

Lines changed: 2007 additions & 134 deletions

File tree

‎skills/sdk-install/braintrust-url-formats.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Braintrust URL Formats
2+
3+
## App Links (Current Format)
4+
5+
### Experiments
6+
7+
`https://www.braintrust.dev/app/{org}/p/{project}/experiments/{experiment_name}?r={root_span_id}&s={span_id}`
8+
9+
### Datasets
10+
11+
`https://www.braintrust.dev/app/{org}/p/{project}/datasets/{dataset_name}?r={root_span_id}`
12+
13+
### Project Logs
14+
15+
`https://www.braintrust.dev/app/{org}/p/{project}/logs?r={root_span_id}&s={span_id}`
16+
17+
## Legacy Object URLs
18+
19+
`https://www.braintrust.dev/app/object?object_type=...&object_id=...&id=...`
20+
21+
## URL Parameters
22+
23+
| Parameter | Description |
24+
| --------- | --------------------------------------------------------- |
25+
| r | The root_span_id - identifies a trace |
26+
| s | The span_id - identifies a specific span within the trace |
27+
| id | Legacy parameter for root_span_id in object URLs |
28+
29+
## Notes
30+
31+
- The `r=` parameter is always the root_span_id
32+
- For logs and experiments, use `s=` to reference a specific span within a trace

‎skills/sdk-install/csharp.md‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
# C# SDK Install
2+
3+
Reference guide for installing the Braintrust C# SDK.
4+
5+
- SDK repo: https://github.com/braintrustdata/braintrust-sdk-dotnet
6+
- NuGet: https://www.nuget.org/packages/Braintrust.Sdk
7+
- Requires .NET 8.0+
8+
9+
## Find the latest version of the SDK
10+
11+
Look up the latest version from NuGet **without installing anything**. Do not guess -- use a read-only query so the environment stays unchanged until you pin the exact version.
12+
13+
```bash
14+
dotnet package search Braintrust.Sdk --exact-match
15+
```
16+
17+
Then install that exact version:
18+
19+
### .NET CLI
20+
21+
```bash
22+
dotnet add package Braintrust.Sdk --version <VERSION>
23+
```
24+
25+
### Or add to .csproj
26+
27+
```xml
28+
<ItemGroup>
29+
<PackageReference Include="Braintrust.Sdk" Version="<VERSION>" />
30+
</ItemGroup>
31+
```
32+
33+
## Initialize the SDK
34+
35+
```csharp
36+
using Braintrust.Sdk;
37+
using Braintrust.Sdk.Config;
38+
39+
var apiKey = Environment.GetEnvironmentVariable("BRAINTRUST_API_KEY");
40+
Braintrust? braintrust = null;
41+
System.Diagnostics.ActivitySource? activitySource = null;
42+
43+
if (!string.IsNullOrEmpty(apiKey))
44+
{
45+
// Set the project name in code (do NOT require an env var for project name).
46+
var config = BraintrustConfig.Of(
47+
("BRAINTRUST_API_KEY", apiKey),
48+
("BRAINTRUST_DEFAULT_PROJECT_NAME", "my-project")
49+
);
50+
51+
braintrust = Braintrust.Get(config);
52+
activitySource = braintrust.GetActivitySource();
53+
}
54+
```
55+
56+
`Braintrust.Get(config)` is the main entry point. The SDK requires an API key to be present, so initialize Braintrust conditionally and run the application normally when `BRAINTRUST_API_KEY` is missing. `GetActivitySource()` returns the `System.Diagnostics.ActivitySource` used to create spans.
57+
58+
## Install instrumentation
59+
60+
The C# SDK instruments LLM clients by wrapping them. Only instrument clients that are actually present in the project.
61+
62+
### OpenAI (`OpenAI` NuGet package)
63+
64+
```bash
65+
dotnet add package OpenAI
66+
```
67+
68+
Create an instrumented OpenAI client:
69+
70+
```csharp
71+
using Braintrust.Sdk;
72+
using Braintrust.Sdk.Config;
73+
using Braintrust.Sdk.Instrumentation.OpenAI;
74+
75+
var btApiKey = Environment.GetEnvironmentVariable("BRAINTRUST_API_KEY");
76+
var openAIApiKey = Environment.GetEnvironmentVariable("OPENAI_API_KEY");
77+
78+
if (!string.IsNullOrEmpty(btApiKey) && !string.IsNullOrEmpty(openAIApiKey))
79+
{
80+
var config = BraintrustConfig.Of(
81+
("BRAINTRUST_API_KEY", btApiKey),
82+
("BRAINTRUST_DEFAULT_PROJECT_NAME", "my-project")
83+
);
84+
var braintrust = Braintrust.Get(config);
85+
var activitySource = braintrust.GetActivitySource();
86+
var client = BraintrustOpenAI.WrapOpenAI(activitySource, openAIApiKey);
87+
88+
// Optional: create a root activity so you can generate a permalink.
89+
using var activity = activitySource.StartActivity("braintrust-openai-example");
90+
91+
var chatClient = client.GetChatClient("gpt-5-mini");
92+
var response = await chatClient.CompleteChatAsync(
93+
new ChatMessage[]
94+
{
95+
new SystemChatMessage("You are a helpful assistant."),
96+
new UserChatMessage("What is the capital of France?")
97+
}
98+
);
99+
100+
if (activity != null)
101+
{
102+
var projectUri = await braintrust.GetProjectUriAsync();
103+
var url = $"{projectUri}/logs?r={activity.TraceId}&s={activity.SpanId}";
104+
Console.WriteLine($"View your data in Braintrust: {url}");
105+
}
106+
}
107+
```
108+
109+
### Custom spans
110+
111+
For business logic that isn't an LLM call, create spans manually with the `ActivitySource`:
112+
113+
```csharp
114+
using (var activity = activitySource.StartActivity("my-operation"))
115+
{
116+
activity?.SetTag("some.attribute", "value");
117+
// LLM calls inside here are automatically nested under this span
118+
}
119+
```
120+
121+
## Run the application
122+
123+
Try to figure out how to run the application from the project structure:
124+
125+
- **dotnet run**: `dotnet run` or `dotnet run --project path/to/Project.csproj`
126+
- **ASP.NET**: `dotnet run` (typically starts Kestrel)
127+
- **Published app**: `dotnet path/to/app.dll`
128+
- **Visual Studio / Rider**: run from IDE
129+
130+
If you can't determine how to run the app, ask the user.
131+
132+
## Generate a permalink (required)
133+
134+
The installer must produce a permalink to the emitted trace/logs in its final output.
135+
136+
In .NET, the most reliable permalink can be generated from the root Activity's TraceId/SpanId:
137+
138+
```csharp
139+
if (braintrust != null && activitySource != null)
140+
{
141+
using var activity = activitySource.StartActivity("braintrust-install-verify");
142+
if (activity != null)
143+
{
144+
// Perform a real operation that triggers LLM spans / instrumentation here.
145+
146+
var projectUri = await braintrust.GetProjectUriAsync();
147+
var url = $"{projectUri}/logs?r={activity.TraceId}&s={activity.SpanId}";
148+
Console.WriteLine($"View your data in Braintrust: {url}");
149+
}
150+
}
151+
```
152+
153+
The final assistant response must include the printed URL.
154+
155+
If the SDK-generated URL is not available, construct the permalink manually using the URL format documented in `braintrust-url-formats.md` as described in the agent task (Step 5).

‎skills/sdk-install/go.md‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# Go SDK Install
2+
3+
Reference guide for installing the Braintrust Go SDK.
4+
5+
- SDK repo: https://github.com/braintrustdata/braintrust-sdk-go
6+
- pkg.go.dev: https://pkg.go.dev/github.com/braintrustdata/braintrust-sdk-go
7+
- Requires Go 1.22+
8+
9+
## Install the SDK
10+
11+
```bash
12+
go get github.com/braintrustdata/braintrust-sdk-go
13+
```
14+
15+
## Initialize the SDK
16+
17+
Every Go project needs OpenTelemetry setup and a Braintrust client.
18+
19+
```go
20+
package main
21+
22+
import (
23+
"context"
24+
"log"
25+
26+
"github.com/braintrustdata/braintrust-sdk-go"
27+
"go.opentelemetry.io/otel"
28+
"go.opentelemetry.io/otel/sdk/trace"
29+
)
30+
31+
func main() {
32+
ctx := context.Background()
33+
34+
tp := trace.NewTracerProvider()
35+
defer tp.Shutdown(ctx)
36+
otel.SetTracerProvider(tp)
37+
38+
_, err := braintrust.New(tp, braintrust.WithProject("my-project"))
39+
if err != nil {
40+
log.Fatal(err)
41+
}
42+
}
43+
```
44+
45+
`braintrust.New` reads `BRAINTRUST_API_KEY` from the environment automatically.
46+
47+
## Install instrumentation
48+
49+
The Go SDK uses [Orchestrion](https://github.com/DataDog/orchestrion) to automatically inject tracing at compile time -- no wrapper code needed in the application.
50+
51+
**1. Install orchestrion:**
52+
53+
```bash
54+
go install github.com/DataDog/orchestrion@v1.6.1
55+
```
56+
57+
**2. Create `orchestrion.tool.go` in the project root:**
58+
59+
To instrument all supported providers:
60+
61+
```go
62+
//go:build tools
63+
64+
package main
65+
66+
import (
67+
_ "github.com/DataDog/orchestrion"
68+
_ "github.com/braintrustdata/braintrust-sdk-go/trace/contrib/all"
69+
)
70+
```
71+
72+
Or import only the integrations the project actually uses:
73+
74+
```go
75+
//go:build tools
76+
77+
package main
78+
79+
import (
80+
_ "github.com/DataDog/orchestrion"
81+
_ "github.com/braintrustdata/braintrust-sdk-go/trace/contrib/anthropic" // anthropic-sdk-go
82+
_ "github.com/braintrustdata/braintrust-sdk-go/trace/contrib/genai" // Google GenAI
83+
_ "github.com/braintrustdata/braintrust-sdk-go/trace/contrib/github.com/sashabaranov/go-openai" // sashabaranov/go-openai
84+
_ "github.com/braintrustdata/braintrust-sdk-go/trace/contrib/langchaingo" // LangChainGo
85+
_ "github.com/braintrustdata/braintrust-sdk-go/trace/contrib/openai" // openai-go
86+
)
87+
```
88+
89+
**3. Build with orchestrion:**
90+
91+
```bash
92+
orchestrion go build ./...
93+
```
94+
95+
Or set GOFLAGS to use orchestrion automatically:
96+
97+
```bash
98+
export GOFLAGS="-toolexec='orchestrion toolexec'"
99+
go build ./...
100+
```
101+
102+
After this, LLM client calls are automatically traced with no code changes.
103+
104+
### Supported providers
105+
106+
Orchestrion supports these providers (import the corresponding `trace/contrib/` package in `orchestrion.tool.go`):
107+
108+
| Provider | Import path |
109+
| ---------------------- | --------------------------------------------------------------------------------------------- |
110+
| OpenAI (`openai-go`) | `github.com/braintrustdata/braintrust-sdk-go/trace/contrib/openai` |
111+
| Anthropic | `github.com/braintrustdata/braintrust-sdk-go/trace/contrib/anthropic` |
112+
| Google GenAI / Gemini | `github.com/braintrustdata/braintrust-sdk-go/trace/contrib/genai` |
113+
| LangChainGo | `github.com/braintrustdata/braintrust-sdk-go/trace/contrib/langchaingo` |
114+
| sashabaranov/go-openai | `github.com/braintrustdata/braintrust-sdk-go/trace/contrib/github.com/sashabaranov/go-openai` |
115+
| All of the above | `github.com/braintrustdata/braintrust-sdk-go/trace/contrib/all` |
116+
117+
## Run the application
118+
119+
Try to figure out how to run the application from the project structure:
120+
121+
- **go run**: `go run .` or `go run ./cmd/myapp`
122+
- **Orchestrion**: `orchestrion go run .`
123+
- **Makefile**: check for `run`, `serve`, or similar targets
124+
- **Docker**: check for a `Dockerfile`
125+
126+
If you can't determine how to run the app, ask the user.
127+
128+
## Generate a permalink (required)
129+
130+
Follow the permalink generation steps in the agent task (Step 5). Use the value passed to `braintrust.WithProject(...)` as the project name.

0 commit comments

Comments
 (0)