diff --git a/README.md b/README.md index 600c8f63..5462f6eb 100644 --- a/README.md +++ b/README.md @@ -78,7 +78,7 @@ Download the encoderfile CLI tool to build your own model binaries: curl -fsSL https://raw.githubusercontent.com/mozilla-ai/encoderfile/main/install.sh | sh ``` -> **Note for Windows users:** Pre-built binaries are not available for Windows. Please see our guide on [building from source](https://mozilla-ai.github.io/encoderfile/latest/reference/building/) for instructions on building from source. +> **Note for Windows users:** Pre-built binaries are not available for Windows. Please see our guide on [building from source](https://mozilla-ai.github.io/encoderfile/reference/building/) for instructions on building from source. Move the binary to a location in your PATH: ```bash @@ -92,7 +92,7 @@ mv encoderfile ~/.local/bin/ ### Option 2: Build CLI Tool from Source -See our guide on [building from source](https://mozilla-ai.github.io/encoderfile/latest/reference/building/) for detailed instructions on building the CLI tool from source. +See our guide on [building from source](https://mozilla-ai.github.io/encoderfile/reference/building/) for detailed instructions on building the CLI tool from source. Quick build: ```bash @@ -310,16 +310,16 @@ Run as a Model Context Protocol server: ## 📚 Documentation -- **[Getting Started Guide](https://mozilla-ai.github.io/encoderfile/latest/getting-started/)** - Step-by-step tutorial -- **[Building Guide](https://mozilla-ai.github.io/encoderfile/latest/reference/building/)** - Build encoderfiles from ONNX models -- **[CLI Reference](https://mozilla-ai.github.io/encoderfile/latest/reference/cli/)** - Complete command-line documentation -- **[API Reference](https://mozilla-ai.github.io/encoderfile/latest/reference/api-reference/)** - REST, gRPC, and MCP API docs +- **[Getting Started Guide](https://mozilla-ai.github.io/encoderfile/getting-started/)** - Step-by-step tutorial +- **[Building Guide](https://mozilla-ai.github.io/encoderfile/reference/building/)** - Build encoderfiles from ONNX models +- **[CLI Reference](https://mozilla-ai.github.io/encoderfile/reference/cli/)** - Complete command-line documentation +- **[API Reference](https://mozilla-ai.github.io/encoderfile/reference/api-reference/)** - REST, gRPC, and MCP API docs ## 🛠️ Building Custom Encoderfiles Once you have the `encoderfile` CLI tool installed, you can build binaries from any compatible HuggingFace model. -See our guide on [building from source](https://mozilla-ai.github.io/encoderfile/latest/reference/building/) for detailed instructions including: +See our guide on [building from source](https://mozilla-ai.github.io/encoderfile/reference/building/) for detailed instructions including: - How to export models to ONNX format - Configuration file options diff --git a/docs/cookbooks/mcp-integration.md b/docs/cookbooks/mcp-integration.md new file mode 100644 index 00000000..ef28f0b8 --- /dev/null +++ b/docs/cookbooks/mcp-integration.md @@ -0,0 +1,42 @@ +# Using `encoderfile` from agents + +The [Model Context Protocol](https://www.anthropic.com/news/model-context-protocol) (MCP) introduced by Anthropic has proven to be a popular method for providing an AI agent with access to a variety of tools. [This Huggingface blog post](https://huggingface.co/blog/Kseniase/mcp) has a nice explanation of MCP. + +In the following example we will use Mozilla's own [`any-agent`](https://github.com/mozilla-ai/any-agent) and [`any-llm`](https://github.com/mozilla-ai/any-llm) packages to build a small agent that leverages on capabilities provided by a test encoderfile. + + +## Build the custom encoderfile and start the server + +We will use the existing test config to build an encoderfile using one of the test models by Mozilla.ai. It will detect Personally Identifiable Information (PII) and tag it accordingly, using tags like `B-SURNAME` for, well, surnames, and `O` for non-PII tokens. As we will see, even if the output consists of logits and tags, the underlying LLM is usually robust enough to focus only on the tags and act appropriately. + +```sh +curl -fsSL https://raw.githubusercontent.com/mozilla-ai/encoderfile/main/install.sh | sh +encoderfile build -f test_config.yml +``` + +After building it, we only need to set it up in MCP mode so it will listen to requests. By default it will bind to all interfaces, using port 9100. + +```sh +my-model-2.encoderfile mcp +``` + +## Install Dependencies + +For this test, we will need the `any-agent` and `any-llm` Python packages: + +```sh +pip install any-agent +pip install any-llm-sdk[mistral] +``` + +## Write the agent + +Now we will write an agent with the appropriate prompt. We instruct the agent to use the provided tool, since the current description is fairly generic, and not use metadata that it might consider useful but is not documented anywhere in the tool itself. We will also instruct it to replace only surnames to showcase that the tags can be extracted appropriately: + +```python +--8<-- "examples/agent_mcp_integration/agent_test.py" +``` + +After some struggling with the call conventions, the LLM finally obtains the information from the encoderfile and acts accordingly: + +> `My name is Javier [REDACTED]` diff --git a/docs/index.md b/docs/index.md index cad0e2df..88465192 100644 --- a/docs/index.md +++ b/docs/index.md @@ -28,7 +28,7 @@ While **Llamafile** focuses on generative models, **Encoderfile** is purpose-bui * **Protocol Agnostic:** Runs as a REST API, gRPC microservice, CLI tool, or MCP Server out of the box. * **Compliance-Friendly:** Deterministic and offline-safe, making it ideal for strict security boundaries. -> **Note for Windows users:** Pre-built binaries are not available for Windows. Please see our guide on [building from source](https://mozilla-ai.github.io/encoderfile/latest/reference/building/) for instructions on building from source. +> **Note for Windows users:** Pre-built binaries are not available for Windows. Please see our guide on [building from source](https://mozilla-ai.github.io/encoderfile/reference/building/) for instructions on building from source. ## Use Cases @@ -48,7 +48,7 @@ Encoderfile supports encoder-only transformers for: - **Token Classification** - Named Entity Recognition, PII detection - **Sentence Embeddings** - Semantic search, clustering -See our guide on [building from source](https://mozilla-ai.github.io/encoderfile/latest/reference/building/) for detailed instructions on building the CLI tool from source. +See our guide on [building from source](https://mozilla-ai.github.io/encoderfile/reference/building/) for detailed instructions on building the CLI tool from source. Generation models (GPT, T5) are not supported. See [CLI Reference](reference/cli.md) for complete model type details. diff --git a/docs/reference/api-reference.md b/docs/reference/api-reference.md index 827416db..5693c8bf 100644 --- a/docs/reference/api-reference.md +++ b/docs/reference/api-reference.md @@ -652,7 +652,7 @@ Encoderfile supports Model Context Protocol, allowing integration with MCP-compa ### Connection Details - **Endpoint:** `/mcp` -- **Transport:** HTTP-based MCP protocol +- **Transport:** HTTP-based MCP protocol (Streamable HTTP only) - **Port:** Same as HTTP server (default: 8080) ### MCP Tools diff --git a/examples/agent_mcp_integration/agent_test.py b/examples/agent_mcp_integration/agent_test.py new file mode 100644 index 00000000..00fb31d3 --- /dev/null +++ b/examples/agent_mcp_integration/agent_test.py @@ -0,0 +1,66 @@ +import os +import shutil +from getpass import getpass + +if "MISTRAL_API_KEY" not in os.environ: + print("MISTRAL_API_KEY not found in environment!") + api_key = getpass("Please enter your MISTRAL_API_KEY: ") + os.environ["MISTRAL_API_KEY"] = api_key + print("MISTRAL_API_KEY set for this session!") +else: + print("MISTRAL_API_KEY found in environment.") + +# Quick Environment Check (Airbnb tool requires npx/Node.js)\\n", +if not shutil.which("npx"): + print( + "⚠️ Warning: 'npx' was not found in your path. The Airbnb tool requires Node.js/npm to run." + ) + + +from any_agent import AgentConfig, AnyAgent +from any_agent.config import MCPStreamableHttp + + +async def send_message(message: str) -> str: + """Display a message to the user and wait for their response. + + Args: + message: str + The message to be displayed to the user. + + Returns: + str: The response from the user. + + """ + if os.environ.get("IN_PYTEST") == "1": + return "2 people, next weekend, low budget. Do not ask for any more information or confirmation." + return input(message + " ") + + +async def main(): + print("Start creating agent") + eftool = MCPStreamableHttp(url="http://localhost:9100/mcp") + try: + agent = await AnyAgent.create_async( + "tinyagent", # See all options in https://mozilla-ai.github.io/any-agent/ + AgentConfig(model_id="mistral:mistral-large-latest", tools=[eftool]), + ) + except Exception as e: + print(f"❌ Failed to create agent: {e}") + print("Done creating agent") + + prompt = """ + Use the eftool tool to remove the personal information from this line: "My name is Javier Torres". + Do not use any metadata. The "inputs" param must be a sequence with one string. + Replace each surname, but not given names, with [REDACTED]. + """ + + agent_trace = await agent.run_async(prompt) + print(agent_trace.final_output) + await agent.cleanup_async() + + +if __name__ == "__main__": + import asyncio + + asyncio.run(main())