diff --git a/docs.json b/docs.json index a87eb70f66..9f73c9cd5a 100644 --- a/docs.json +++ b/docs.json @@ -830,6 +830,7 @@ "pages": [ "serverless-training", "serverless-training/prerequisites", + "serverless-training/quickstart", "serverless-training/usage-limits", { "group": "Serverless RL", diff --git a/serverless-training.mdx b/serverless-training.mdx index 6ab3a3475b..2efab02bbb 100644 --- a/serverless-training.mdx +++ b/serverless-training.mdx @@ -18,7 +18,7 @@ Serverless Training is ideal for tasks like: Serverless Training trains low-rank adapters (LoRAs) to specialize a model for your specific task. This extends the original model's capabilities with on-the-job experience. W&B automatically stores the LoRAs you train as artifacts in your account. You can also save them locally or to a third party for backup. Serverless Inference also automatically hosts models that you train through Serverless Training. -See the ART [quickstart](https://art.openpipe.ai/getting-started/quick-start) or [Google Colab notebook](https://colab.research.google.com/github/openpipe/art-notebooks/blob/main/examples/2048/2048.ipynb) to get started. +To get started, train a customer support agent in the [quickstart](/serverless-training/quickstart). ## Why Serverless Training? diff --git a/serverless-training/cookbooks/train-support-agent.ipynb b/serverless-training/cookbooks/train-support-agent.ipynb new file mode 100644 index 0000000000..d51a5fcaa7 --- /dev/null +++ b/serverless-training/cookbooks/train-support-agent.ipynb @@ -0,0 +1,470 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Quickstart: Train a customer support agent with Serverless RL\n", + "\n", + "This notebook is the runnable companion to the [Serverless Training quickstart](https://docs.wandb.ai/serverless-training/quickstart). You'll use [Serverless RL](https://docs.wandb.ai/serverless-training) to train a LoRA adapter for a customer support agent that searches a product knowledge base and answers customer questions, then send inference requests to the model you trained.\n", + "\n", + "The agent supports a fictional smart thermostat. The same pattern applies to any multi-turn agentic task where the model uses tools to gather context before answering: customer support, agentic RAG, deep research, or internal help desks. You don't need labeled answers or a hand-written reward function \u2014 [RULER](https://art.openpipe.ai/fundamentals/ruler) uses an LLM judge to rank the agent's attempts against each other, and Serverless RL uses those rankings to update the LoRA.\n", + "\n", + "**Prerequisites**\n", + "- A [W&B account](https://wandb.ai) and [API key](https://wandb.ai/settings), plus a W&B project \u2014 see the [prerequisites](https://docs.wandb.ai/serverless-training/prerequisites).\n", + "- An OpenAI API key for the RULER judge (any [LiteLLM-supported model](https://docs.litellm.ai/docs/providers) works as an alternative).\n", + "\n", + "Training is free during the public preview; you pay only for inference usage and artifact storage. See [Usage information and limits](https://docs.wandb.ai/serverless-training/usage-limits)." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Installation" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "%pip install -q openpipe-art" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Environment variables\n", + "\n", + "Set your W&B API key (used for training, inference, and logging) and your OpenAI API key (used by the RULER judge)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "import os\n", + "from getpass import getpass\n", + "\n", + "if not os.environ.get(\"WANDB_API_KEY\"):\n", + " os.environ[\"WANDB_API_KEY\"] = getpass(\"Enter your W&B API key: \")\n", + "if not os.environ.get(\"OPENAI_API_KEY\"):\n", + " os.environ[\"OPENAI_API_KEY\"] = getpass(\"Enter your OpenAI API key (RULER judge): \")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Register a trainable model\n", + "\n", + "Declare a trainable model and register it with the `ServerlessBackend`. The backend sends inference and training requests to the W&B training cluster, which autoscales to match your job's demand. `base_model` must be one of the [available models](https://docs.wandb.ai/serverless-training/available-models)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "import art\n", + "from art.serverless.backend import ServerlessBackend\n", + "\n", + "model = art.TrainableModel(\n", + " name=\"support-agent-001\",\n", + " project=\"support-agent\",\n", + " base_model=\"OpenPipe/Qwen3-14B-Instruct\",\n", + ")\n", + "\n", + "backend = ServerlessBackend()\n", + "await model.register(backend)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Create the agent's environment\n", + "\n", + "The agent answers questions using a small inline product knowledge base and a single search tool. In a real application, the tool would query your documentation index, help desk, or vector store." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "KNOWLEDGE_BASE = [\n", + " {\n", + " \"id\": \"kb-01\",\n", + " \"title\": \"Pairing the thermostat with the mobile app\",\n", + " \"content\": \"To pair the thermostat, open the mobile app, tap Add Device, and hold the thermostat's center button for five seconds until the display shows a pairing code. Enter the code in the app. Pairing requires Bluetooth and a 2.4 GHz Wi-Fi network.\",\n", + " },\n", + " {\n", + " \"id\": \"kb-02\",\n", + " \"title\": \"Blank screen after installation\",\n", + " \"content\": \"A blank screen usually means the thermostat isn't receiving power. Confirm the C-wire (common wire) is connected at both the thermostat base and the HVAC control board. If your system has no C-wire, install the included power adapter.\",\n", + " },\n", + " {\n", + " \"id\": \"kb-03\",\n", + " \"title\": \"Wi-Fi disconnects and offline status\",\n", + " \"content\": \"If the thermostat shows as offline, confirm your router broadcasts a 2.4 GHz network; the thermostat doesn't support 5 GHz-only networks. Move the router within 10 meters if the signal indicator shows one bar, then restart the thermostat from Settings > Restart.\",\n", + " },\n", + " {\n", + " \"id\": \"kb-04\",\n", + " \"title\": \"Battery and power adapter\",\n", + " \"content\": \"The thermostat has a backup battery that keeps settings for 48 hours during a power outage. The battery charges from the C-wire or power adapter. A battery icon on the display means the thermostat is running on backup power and will shut down soon.\",\n", + " },\n", + " {\n", + " \"id\": \"kb-05\",\n", + " \"title\": \"Creating temperature schedules\",\n", + " \"content\": \"Create schedules in the app under Schedule > New. Each schedule supports up to eight temperature changes per day. Schedules sync to the thermostat within one minute. Manual temperature changes override the schedule until the next scheduled change.\",\n", + " },\n", + " {\n", + " \"id\": \"kb-06\",\n", + " \"title\": \"Temperature reading seems wrong\",\n", + " \"content\": \"If the displayed temperature differs from the room temperature, check that the thermostat isn't in direct sunlight or near a heat source. You can apply a calibration offset of up to \u00b15 degrees in Settings > Temperature Offset.\",\n", + " },\n", + " {\n", + " \"id\": \"kb-07\",\n", + " \"title\": \"Warranty and returns\",\n", + " \"content\": \"The thermostat includes a two-year limited warranty covering manufacturing defects. Returns are accepted within 60 days of purchase with proof of purchase. Warranty claims are handled through the app under Support > Start a Claim.\",\n", + " },\n", + " {\n", + " \"id\": \"kb-08\",\n", + " \"title\": \"Updating firmware\",\n", + " \"content\": \"Firmware updates install automatically overnight when the thermostat is connected to Wi-Fi. To update manually, go to Settings > About > Check for Updates. The display shows a progress bar during updates; don't cut power while an update is in progress.\",\n", + " },\n", + " {\n", + " \"id\": \"kb-09\",\n", + " \"title\": \"Vacation mode\",\n", + " \"content\": \"Vacation mode holds an energy-saving temperature while you're away and resumes your normal schedule when you return. Enable it in the app under Modes > Vacation and set a start and end date. Geofencing can end vacation mode automatically when your phone arrives home.\",\n", + " },\n", + " {\n", + " \"id\": \"kb-10\",\n", + " \"title\": \"Compatible HVAC systems\",\n", + " \"content\": \"The thermostat supports most 24V heating and cooling systems, including gas, oil, electric, forced air, and heat pumps with auxiliary heat. It doesn't support high-voltage (120/240V) baseboard heaters. Use the compatibility checker in the app before installing.\",\n", + " },\n", + "]\n", + "\n", + "\n", + "def search_kb(keywords: list[str]) -> list[dict]:\n", + " \"\"\"Return knowledge base articles that match any of the given keywords.\"\"\"\n", + " results = []\n", + " for article in KNOWLEDGE_BASE:\n", + " text = f\"{article['title']} {article['content']}\".lower()\n", + " if any(keyword.lower() in text for keyword in keywords):\n", + " results.append(article)\n", + " return results\n", + "\n", + "\n", + "SEARCH_TOOL = {\n", + " \"type\": \"function\",\n", + " \"function\": {\n", + " \"name\": \"search_kb\",\n", + " \"description\": \"Search the product knowledge base for articles matching any of the given keywords.\",\n", + " \"parameters\": {\n", + " \"type\": \"object\",\n", + " \"properties\": {\n", + " \"keywords\": {\n", + " \"type\": \"array\",\n", + " \"items\": {\"type\": \"string\"},\n", + " \"description\": \"Keywords to search for.\",\n", + " }\n", + " },\n", + " \"required\": [\"keywords\"],\n", + " },\n", + " },\n", + "}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Define the questions the agent trains on. RULER compares the agent's attempts against each other, so the questions don't need labeled answers." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "TRAINING_QUESTIONS = [\n", + " \"My thermostat's screen is completely blank after I installed it. What should I check?\",\n", + " \"How do I pair the thermostat with my phone?\",\n", + " \"The app says my thermostat is offline but my other devices are fine on Wi-Fi.\",\n", + " \"Can I make the thermostat follow a different temperature at night?\",\n", + " \"The thermostat says it's 75 degrees but my room thermometer says 71. Can I fix that?\",\n", + " \"How long does the battery last if the power goes out?\",\n", + " \"I'm going out of town for two weeks. How do I keep my energy bill down?\",\n", + " \"Does this work with my baseboard heaters?\",\n", + " \"How do I update the thermostat's software?\",\n", + " \"My thermostat is six months old and the display flickers. Is this covered?\",\n", + " \"I changed the temperature by hand. Why did it change back an hour later?\",\n", + " \"Do I need a C-wire to install the thermostat?\",\n", + "]\n", + "\n", + "HELD_OUT_QUESTION = \"I set up vacation mode but I'm coming home early. Will the house be cold when I arrive?\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Define a rollout\n", + "\n", + "A rollout is one complete attempt by the agent to handle a scenario. The rollout function sends the question to the model, executes any tool calls it makes, and returns the full interaction as an `art.Trajectory`. The client points at your model's Serverless Training inference endpoint, so every rollout uses the latest LoRA weights as training progresses. `temperature=1` keeps attempts diverse \u2014 RULER needs variation within a group to produce a useful ranking." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "import json\n", + "\n", + "from openai import AsyncOpenAI\n", + "\n", + "MAX_TURNS = 5\n", + "\n", + "SYSTEM_PROMPT = (\n", + " \"You are a customer support agent for a smart thermostat company. \"\n", + " \"Use the search_kb tool to find relevant knowledge base articles before answering. \"\n", + " \"Answer the customer's question accurately and concisely based on the articles you find, \"\n", + " \"and include the steps the customer should take. If the knowledge base doesn't cover the \"\n", + " \"question, say so instead of guessing.\"\n", + ")\n", + "\n", + "\n", + "async def rollout(model: art.Model, question: str) -> art.Trajectory:\n", + " trajectory = art.Trajectory(\n", + " messages_and_choices=[\n", + " {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n", + " {\"role\": \"user\", \"content\": question},\n", + " ],\n", + " tools=[SEARCH_TOOL],\n", + " reward=0.0,\n", + " )\n", + "\n", + " client = AsyncOpenAI(\n", + " base_url=model.inference_base_url,\n", + " api_key=model.inference_api_key,\n", + " )\n", + "\n", + " for _ in range(MAX_TURNS):\n", + " response = await client.chat.completions.create(\n", + " model=model.get_inference_name(),\n", + " messages=trajectory.messages(),\n", + " tools=trajectory.tools,\n", + " temperature=1,\n", + " )\n", + " choice = response.choices[0]\n", + " trajectory.messages_and_choices.append(choice)\n", + "\n", + " # No tool calls means the agent gave its final answer\n", + " if not choice.message.tool_calls:\n", + " break\n", + "\n", + " for tool_call in choice.message.tool_calls:\n", + " arguments = json.loads(tool_call.function.arguments)\n", + " results = search_kb(**arguments)\n", + " trajectory.messages_and_choices.append(\n", + " {\n", + " \"role\": \"tool\",\n", + " \"tool_call_id\": tool_call.id,\n", + " \"name\": tool_call.function.name,\n", + " \"content\": json.dumps(results),\n", + " }\n", + " )\n", + "\n", + " return trajectory" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Score trajectories with RULER\n", + "\n", + "Instead of writing a reward function, pass each group of trajectories to RULER. An LLM judge compares the attempts for the same question against each other and scores them from 0 to 1. Because the RL algorithm normalizes scores within each group, only the relative ranking matters.\n", + "\n", + "Verify that RULER's rankings make sense for your task before you train on them \u2014 `debug=True` prints the judge's reasoning." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "from art.rewards import ruler_score_group\n", + "\n", + "test_group = art.TrajectoryGroup(\n", + " [await rollout(model, TRAINING_QUESTIONS[0]) for _ in range(3)]\n", + ")\n", + "\n", + "judged_group = await ruler_score_group(test_group, \"openai/gpt-5.4\", debug=True)\n", + "\n", + "for trajectory in sorted(judged_group.trajectories, key=lambda t: t.reward, reverse=True):\n", + " print(f\"Score: {trajectory.reward:.3f}\")\n", + " print(f\"Answer: {trajectory.messages()[-1]['content'][:120]}\\n\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Run the training loop\n", + "\n", + "Each training step gathers a batch of trajectory groups, scores them with RULER, and sends the scored trajectories to the W&B training cluster, which updates your LoRA adapter and saves a checkpoint. This configuration runs six training steps.\n", + "\n", + "While the loop runs, open your `support-agent` project at [wandb.ai](https://wandb.ai) to watch reward and training metrics update in real time. Each step also saves a LoRA checkpoint as an [artifact](https://docs.wandb.ai/models/artifacts) in your project." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "from art.rewards import ruler_score_group\n", + "from art.utils import iterate_dataset\n", + "\n", + "GROUPS_PER_STEP = 4 # Questions per training step\n", + "TRAJECTORIES_PER_GROUP = 4 # Attempts per question, compared by RULER\n", + "NUM_EPOCHS = 2 # Passes over the training questions\n", + "LEARNING_RATE = 1e-5\n", + "\n", + "training_iterator = iterate_dataset(\n", + " TRAINING_QUESTIONS,\n", + " groups_per_step=GROUPS_PER_STEP,\n", + " num_epochs=NUM_EPOCHS,\n", + " initial_step=await model.get_step(),\n", + ")\n", + "\n", + "for batch in training_iterator:\n", + " print(f\"Training step {batch.step} (epoch {batch.epoch})\")\n", + "\n", + " groups = [\n", + " art.TrajectoryGroup(\n", + " rollout(model, question) for _ in range(TRAJECTORIES_PER_GROUP)\n", + " )\n", + " for question in batch.items\n", + " ]\n", + "\n", + " finished_groups = await art.gather_trajectory_groups(\n", + " groups,\n", + " pbar_desc=\"gather\",\n", + " max_exceptions=TRAJECTORIES_PER_GROUP * len(batch.items),\n", + " )\n", + "\n", + " judged_groups = [\n", + " await ruler_score_group(group, \"openai/gpt-5.4\")\n", + " for group in finished_groups\n", + " ]\n", + "\n", + " result = await backend.train(\n", + " model,\n", + " judged_groups,\n", + " learning_rate=LEARNING_RATE,\n", + " )\n", + "\n", + " await model.log(\n", + " judged_groups,\n", + " metrics=result.metrics,\n", + " step=result.step,\n", + " split=\"train\",\n", + " )\n", + "\n", + " print(f\"Completed training step {result.step}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Use your trained model\n", + "\n", + "Your model's endpoint continues to serve the latest trained weights, so the same rollout function now runs against your trained LoRA. Try the held-out question:" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "result = await rollout(model, HELD_OUT_QUESTION)\n", + "print(result.messages()[-1][\"content\"])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Every checkpoint is also deployed automatically for inference, so you can call your trained model from any OpenAI-compatible client using a `wandb-artifact` model reference and the Serverless Training API. Replace `[YOUR-ENTITY]` with your W&B entity (team) name, and choose the checkpoint step you want to serve. For details, see [Use your trained models](https://docs.wandb.ai/serverless-training/use-trained-models)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "from openai import OpenAI\n", + "\n", + "client = OpenAI(\n", + " base_url=\"https://api.training.wandb.ai/v1\",\n", + " api_key=os.environ[\"WANDB_API_KEY\"],\n", + ")\n", + "\n", + "response = client.chat.completions.create(\n", + " model=\"wandb-artifact:///[YOUR-ENTITY]/support-agent/support-agent-001:step6\",\n", + " messages=[\n", + " {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n", + " {\"role\": \"user\", \"content\": HELD_OUT_QUESTION},\n", + " ],\n", + ")\n", + "print(response.choices[0].message.content)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Next steps\n", + "\n", + "You trained a LoRA adapter with Serverless RL and served it without managing any infrastructure. To keep going:\n", + "\n", + "- **Scale up the task**: Point the search tool at your real documentation or help desk data, expand the question set, and increase the number of epochs and validation checks. The [ART\u00b7E notebook](https://colab.research.google.com/github/openpipe/art-notebooks/blob/main/examples/art-e.ipynb) shows a larger version of this pattern.\n", + "- **Warm up with SFT**: If you have curated example conversations, fine-tune on them first with [Serverless SFT](https://docs.wandb.ai/serverless-training/sft), then apply RL for further refinement.\n", + "- **Manage storage**: Each checkpoint counts toward your artifact storage. Delete low-performing checkpoints with the [ART SDK](https://art.openpipe.ai/features/checkpoint-deletion).\n", + "- **Go deeper on ART**: See the [ART documentation](https://art.openpipe.ai) for advanced training configuration, validation splits, and additional examples." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python" + }, + "colab": { + "provenance": [] + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/serverless-training/prerequisites.mdx b/serverless-training/prerequisites.mdx index 13aedd7ab9..771bfc2382 100644 --- a/serverless-training/prerequisites.mdx +++ b/serverless-training/prerequisites.mdx @@ -26,5 +26,5 @@ Create a project in your W&B account to track usage, record training metrics, an After you complete the prerequisites, try the following resources: +* Train your first LoRA in the [quickstart](/serverless-training/quickstart). * Check the [API reference](/serverless-training/api-reference) to learn about available endpoints. -* Try the [ART quickstart](https://art.openpipe.ai/getting-started/quick-start). diff --git a/serverless-training/quickstart.mdx b/serverless-training/quickstart.mdx new file mode 100644 index 0000000000..632a9379f5 --- /dev/null +++ b/serverless-training/quickstart.mdx @@ -0,0 +1,378 @@ +--- +title: "Quickstart: Train a support agent" +description: "Train your first LoRA with Serverless RL by teaching a customer support agent to search a knowledge base and answer product questions." +keywords: ["Serverless RL quickstart", "train a LoRA", "RULER", "ART framework", "customer support agent", "tool calling"] +--- + +import { ColabLink } from '/snippets/_includes/colab-link.mdx'; +import { GitHubLink } from '/snippets/_includes/github-source-link.mdx'; + +
+ + +
+ +This quickstart shows you how to train your first LoRA adapter with [Serverless Training](/serverless-training). You'll use [Serverless RL](/serverless-training/usage) to train a customer support agent that searches a product knowledge base and answers customer questions, then you'll send inference requests to the model you trained. + +The agent in this quickstart supports a fictional smart thermostat. The same pattern applies to any multi-turn agentic task where the model uses tools to gather context before answering: customer support, agentic RAG, deep research, or internal help desks. You don't need labeled answers or a hand-written reward function. Instead, [RULER](https://art.openpipe.ai/fundamentals/ruler) (Relative Universal LLM-Elicited Rewards) uses an LLM judge to rank the agent's attempts against each other, and Serverless RL uses those rankings to update a LoRA adapter for the base model. + +## What you'll learn + +This quickstart shows you how to: + +- Register a trainable model with the W&B training cluster. +- Define a tool-using rollout for a support agent. +- Score the agent's attempts with RULER instead of a hand-written reward function. +- Run an RL training loop that produces LoRA checkpoints. +- Send inference requests to your trained model. + +Serverless Training provisions and autoscales the GPUs for you: inference requests during rollouts run on [Serverless Inference](/inference), and training steps run on managed training GPUs. Training is free during the public preview; you pay only for inference usage and artifact storage. For details, see [Usage information and limits](/serverless-training/usage-limits). + +## Prerequisites + +Before you begin, complete the [Serverless Training prerequisites](/serverless-training/prerequisites): a W&B account, an API key, and a project. + +You also need the following: + +- Python 3.10 or later, running in an environment that supports `async`/`await` at the top level, such as a Jupyter notebook. To run the code as a script instead, wrap it in a function and call it with `asyncio.run()`. +- An OpenAI API key. RULER uses an LLM as a judge to rank trajectories; this example uses an OpenAI model as the judge. To use a different judge, pass any [LiteLLM-supported model](https://docs.litellm.ai/docs/providers) to `ruler_score_group`. + +Install the [OpenPipe ART](https://art.openpipe.ai/getting-started/about) framework, the open source RL training client that Serverless Training uses: + +```bash +pip install openpipe-art +``` + +Then set your credentials as environment variables. Replace `[YOUR-WANDB-API-KEY]` and `[YOUR-OPENAI-API-KEY]` with your keys: + +```python lines +import os + +os.environ["WANDB_API_KEY"] = "[YOUR-WANDB-API-KEY]" # Used for training, inference, and logging +os.environ["OPENAI_API_KEY"] = "[YOUR-OPENAI-API-KEY]" # Used by the RULER judge +``` + +## Register a trainable model + +Declare a trainable model and register it with the `ServerlessBackend`. The backend sends inference and training requests to the W&B training cluster, which autoscales to match your job's demand. Registration also sets up metric logging to your W&B project. + +```python lines +import art +from art.serverless.backend import ServerlessBackend + +model = art.TrainableModel( + name="support-agent-001", + project="support-agent", + base_model="OpenPipe/Qwen3-14B-Instruct", +) + +backend = ServerlessBackend() +await model.register(backend) +``` + +`base_model` must be one of the [available models](/serverless-training/available-models). This example uses a 14B model optimized for building agents with fine-tuning. + +## Create the agent's environment + +The agent answers questions using a small product knowledge base and a single search tool. This example defines the knowledge base inline so the quickstart is self-contained; in a real application, the tool would query your documentation index, help desk, or vector store. + +```python lines expandable +KNOWLEDGE_BASE = [ + { + "id": "kb-01", + "title": "Pairing the thermostat with the mobile app", + "content": "To pair the thermostat, open the mobile app, tap Add Device, and hold the thermostat's center button for five seconds until the display shows a pairing code. Enter the code in the app. Pairing requires Bluetooth and a 2.4 GHz Wi-Fi network.", + }, + { + "id": "kb-02", + "title": "Blank screen after installation", + "content": "A blank screen usually means the thermostat isn't receiving power. Confirm the C-wire (common wire) is connected at both the thermostat base and the HVAC control board. If your system has no C-wire, install the included power adapter.", + }, + { + "id": "kb-03", + "title": "Wi-Fi disconnects and offline status", + "content": "If the thermostat shows as offline, confirm your router broadcasts a 2.4 GHz network; the thermostat doesn't support 5 GHz-only networks. Move the router within 10 meters if the signal indicator shows one bar, then restart the thermostat from Settings > Restart.", + }, + { + "id": "kb-04", + "title": "Battery and power adapter", + "content": "The thermostat has a backup battery that keeps settings for 48 hours during a power outage. The battery charges from the C-wire or power adapter. A battery icon on the display means the thermostat is running on backup power and will shut down soon.", + }, + { + "id": "kb-05", + "title": "Creating temperature schedules", + "content": "Create schedules in the app under Schedule > New. Each schedule supports up to eight temperature changes per day. Schedules sync to the thermostat within one minute. Manual temperature changes override the schedule until the next scheduled change.", + }, + { + "id": "kb-06", + "title": "Temperature reading seems wrong", + "content": "If the displayed temperature differs from the room temperature, check that the thermostat isn't in direct sunlight or near a heat source. You can apply a calibration offset of up to ±5 degrees in Settings > Temperature Offset.", + }, + { + "id": "kb-07", + "title": "Warranty and returns", + "content": "The thermostat includes a two-year limited warranty covering manufacturing defects. Returns are accepted within 60 days of purchase with proof of purchase. Warranty claims are handled through the app under Support > Start a Claim.", + }, + { + "id": "kb-08", + "title": "Updating firmware", + "content": "Firmware updates install automatically overnight when the thermostat is connected to Wi-Fi. To update manually, go to Settings > About > Check for Updates. The display shows a progress bar during updates; don't cut power while an update is in progress.", + }, + { + "id": "kb-09", + "title": "Vacation mode", + "content": "Vacation mode holds an energy-saving temperature while you're away and resumes your normal schedule when you return. Enable it in the app under Modes > Vacation and set a start and end date. Geofencing can end vacation mode automatically when your phone arrives home.", + }, + { + "id": "kb-10", + "title": "Compatible HVAC systems", + "content": "The thermostat supports most 24V heating and cooling systems, including gas, oil, electric, forced air, and heat pumps with auxiliary heat. It doesn't support high-voltage (120/240V) baseboard heaters. Use the compatibility checker in the app before installing.", + }, +] + + +def search_kb(keywords: list[str]) -> list[dict]: + """Return knowledge base articles that match any of the given keywords.""" + results = [] + for article in KNOWLEDGE_BASE: + text = f"{article['title']} {article['content']}".lower() + if any(keyword.lower() in text for keyword in keywords): + results.append(article) + return results + + +SEARCH_TOOL = { + "type": "function", + "function": { + "name": "search_kb", + "description": "Search the product knowledge base for articles matching any of the given keywords.", + "parameters": { + "type": "object", + "properties": { + "keywords": { + "type": "array", + "items": {"type": "string"}, + "description": "Keywords to search for.", + } + }, + "required": ["keywords"], + }, + }, +} +``` + +Next, define the questions the agent trains on. RULER compares the agent's attempts against each other, so the questions don't need labeled answers: + +```python lines +TRAINING_QUESTIONS = [ + "My thermostat's screen is completely blank after I installed it. What should I check?", + "How do I pair the thermostat with my phone?", + "The app says my thermostat is offline but my other devices are fine on Wi-Fi.", + "Can I make the thermostat follow a different temperature at night?", + "The thermostat says it's 75 degrees but my room thermometer says 71. Can I fix that?", + "How long does the battery last if the power goes out?", + "I'm going out of town for two weeks. How do I keep my energy bill down?", + "Does this work with my baseboard heaters?", + "How do I update the thermostat's software?", + "My thermostat is six months old and the display flickers. Is this covered?", + "I changed the temperature by hand. Why did it change back an hour later?", + "Do I need a C-wire to install the thermostat?", +] + +HELD_OUT_QUESTION = "I set up vacation mode but I'm coming home early. Will the house be cold when I arrive?" +``` + +## Define a rollout + +A rollout is one complete attempt by the agent to handle a scenario. The rollout function sends the question to the model, executes any tool calls it makes, and returns the full interaction as an `art.Trajectory`. During training, Serverless RL runs many rollouts in parallel against your model's current LoRA weights. + +```python lines expandable +import json + +from openai import AsyncOpenAI + +MAX_TURNS = 5 + +SYSTEM_PROMPT = ( + "You are a customer support agent for a smart thermostat company. " + "Use the search_kb tool to find relevant knowledge base articles before answering. " + "Answer the customer's question accurately and concisely based on the articles you find, " + "and include the steps the customer should take. If the knowledge base doesn't cover the " + "question, say so instead of guessing." +) + + +async def rollout(model: art.Model, question: str) -> art.Trajectory: + trajectory = art.Trajectory( + messages_and_choices=[ + {"role": "system", "content": SYSTEM_PROMPT}, + {"role": "user", "content": question}, + ], + tools=[SEARCH_TOOL], + reward=0.0, + ) + + client = AsyncOpenAI( + base_url=model.inference_base_url, + api_key=model.inference_api_key, + ) + + for _ in range(MAX_TURNS): + response = await client.chat.completions.create( + model=model.get_inference_name(), + messages=trajectory.messages(), + tools=trajectory.tools, + temperature=1, + ) + choice = response.choices[0] + trajectory.messages_and_choices.append(choice) + + # No tool calls means the agent gave its final answer + if not choice.message.tool_calls: + break + + for tool_call in choice.message.tool_calls: + arguments = json.loads(tool_call.function.arguments) + results = search_kb(**arguments) + trajectory.messages_and_choices.append( + { + "role": "tool", + "tool_call_id": tool_call.id, + "name": tool_call.function.name, + "content": json.dumps(results), + } + ) + + return trajectory +``` + +A few details matter here: + +- The `AsyncOpenAI` client points at your model's Serverless Training inference endpoint (`model.inference_base_url`), so every rollout uses the latest LoRA weights as training progresses. +- `temperature=1` keeps the agent's attempts diverse. RULER needs variation within a group of attempts to produce a useful ranking. +- The trajectory records the entire interaction, including tool calls and tool results, so training can credit the behavior that led to a good answer. + +## Score trajectories with RULER + +Instead of writing a reward function, pass each group of trajectories to RULER. An LLM judge compares the attempts for the same question against each other and scores them from 0 to 1 based on how well each one achieves the goal described in the system prompt. Because the RL algorithm normalizes scores within each group, only the relative ranking matters. + +Verify that RULER works before you start the training loop: + +```python lines +from art.rewards import ruler_score_group + +test_group = art.TrajectoryGroup( + [await rollout(model, TRAINING_QUESTIONS[0]) for _ in range(3)] +) + +judged_group = await ruler_score_group(test_group, "openai/gpt-5.4", debug=True) + +for trajectory in sorted(judged_group.trajectories, key=lambda t: t.reward, reverse=True): + print(f"Score: {trajectory.reward:.3f}") + print(f"Answer: {trajectory.messages()[-1]['content'][:120]}\n") +``` + +The `debug=True` flag prints the judge's reasoning so you can confirm the rankings make sense for your task before you train on them. + +## Run the training loop + +Each training step gathers a batch of trajectory groups, scores them with RULER, and sends the scored trajectories to the W&B training cluster, which updates your LoRA adapter and saves a checkpoint. Metrics for each step log to your W&B project automatically. + +```python lines expandable +from art.rewards import ruler_score_group +from art.utils import iterate_dataset + +GROUPS_PER_STEP = 4 # Questions per training step +TRAJECTORIES_PER_GROUP = 4 # Attempts per question, compared by RULER +NUM_EPOCHS = 2 # Passes over the training questions +LEARNING_RATE = 1e-5 + +training_iterator = iterate_dataset( + TRAINING_QUESTIONS, + groups_per_step=GROUPS_PER_STEP, + num_epochs=NUM_EPOCHS, + initial_step=await model.get_step(), +) + +for batch in training_iterator: + print(f"Training step {batch.step} (epoch {batch.epoch})") + + groups = [ + art.TrajectoryGroup( + rollout(model, question) for _ in range(TRAJECTORIES_PER_GROUP) + ) + for question in batch.items + ] + + finished_groups = await art.gather_trajectory_groups( + groups, + pbar_desc="gather", + max_exceptions=TRAJECTORIES_PER_GROUP * len(batch.items), + ) + + judged_groups = [ + await ruler_score_group(group, "openai/gpt-5.4") + for group in finished_groups + ] + + result = await backend.train( + model, + judged_groups, + learning_rate=LEARNING_RATE, + ) + + await model.log( + judged_groups, + metrics=result.metrics, + step=result.step, + split="train", + ) + + print(f"Completed training step {result.step}") +``` + +This configuration runs six training steps. While the loop runs, open your `support-agent` project at [wandb.ai](https://wandb.ai) to watch reward and training metrics update in real time. Each step also saves a LoRA checkpoint as an [artifact](/models/artifacts) in your project. + + +To trace every message in each rollout, including the judge's rankings, add [Weave](/weave) to your project: call `weave.init(model.project)` before the training loop and decorate `rollout` with `@weave.op`. Weave records each step of the training loop so you can inspect exactly how your agent's behavior changes during training. + + +## Use your trained model + +Compare the agent's behavior on the held-out question. Your model's endpoint continues to serve the latest trained weights, so the same rollout function now runs against your trained LoRA: + +```python lines +result = await rollout(model, HELD_OUT_QUESTION) +print(result.messages()[-1]["content"]) +``` + +Every checkpoint is also deployed automatically for inference, so you can call your trained model from any OpenAI-compatible client using a `wandb-artifact` model reference and the Serverless Training API: + +```python lines +from openai import OpenAI + +client = OpenAI( + base_url="https://api.training.wandb.ai/v1", + api_key=os.environ["WANDB_API_KEY"], +) + +response = client.chat.completions.create( + model="wandb-artifact:///[YOUR-ENTITY]/support-agent/support-agent-001:step6", + messages=[ + {"role": "system", "content": SYSTEM_PROMPT}, + {"role": "user", "content": HELD_OUT_QUESTION}, + ], +) +print(response.choices[0].message.content) +``` + +Replace `[YOUR-ENTITY]` with your W&B entity (team) name, and choose the checkpoint step you want to serve. To see the checkpoints your training run produced, open the **Artifacts** tab of your project. For details on constructing endpoints for trained models, see [Use your trained models](/serverless-training/use-trained-models). + +## Next steps + +You trained a LoRA adapter with Serverless RL and served it without managing any infrastructure. To keep going: + +- **Scale up the task**: Point the search tool at your real documentation or help desk data, expand the question set, and increase the number of epochs and validation checks. The [ART·E notebook](https://colab.research.google.com/github/openpipe/art-notebooks/blob/main/examples/art-e.ipynb) shows a larger version of this pattern, including periodic validation and checkpoint cleanup. +- **Warm up with SFT**: If you have curated example conversations, fine-tune on them first with [Serverless SFT](/serverless-training/sft), then apply RL for further refinement. +- **Manage storage**: Each checkpoint counts toward your artifact storage. Delete low-performing checkpoints with the [ART SDK](https://art.openpipe.ai/features/checkpoint-deletion). +- **Go deeper on ART**: See the [ART documentation](https://art.openpipe.ai) for advanced training configuration, validation splits, and additional examples. diff --git a/serverless-training/usage.mdx b/serverless-training/usage.mdx index 416173364f..5d069f38ad 100644 --- a/serverless-training/usage.mdx +++ b/serverless-training/usage.mdx @@ -6,6 +6,6 @@ keywords: ["Colab notebook", "ART quickstart", "post-training tutorial"] Use Serverless RL through [OpenPipe's ART framework](https://art.openpipe.ai/getting-started/about) and the [Serverless Training API](/serverless-training/api-reference). -To start using Serverless RL, you must satisfy the [prerequisites](/serverless-training/prerequisites) to use W&B tools, and then go through the ART [quickstart](https://art.openpipe.ai/getting-started/quick-start). +To start using Serverless RL, you must satisfy the [prerequisites](/serverless-training/prerequisites) to use W&B tools, and then train your first LoRA in the [quickstart](/serverless-training/quickstart). -For code examples and workflows, see the [Google Colab notebook](https://colab.research.google.com/github/openpipe/art-notebooks/blob/main/examples/2048/2048.ipynb). +For more code examples and workflows, see the ART [quickstart](https://art.openpipe.ai/getting-started/quick-start) and [Google Colab notebook](https://colab.research.google.com/github/openpipe/art-notebooks/blob/main/examples/2048/2048.ipynb).