Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 24 additions & 41 deletions apps/developer-hub/src/app/llms.txt/route.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,61 +6,44 @@ const CONTENT = `# Pyth Network Documentation

> First-party financial oracle delivering real-time market data to 100+ blockchains.

This is the routing index for Pyth Network's documentation. AI agents should
identify the product the user needs from the descriptions below, then fetch
exactly one product file — each is self-contained with code examples,
addresses, and patterns. Do not fetch every link.

## AI Agent Playbook

For an opinionated integration guide with code snippets and step-by-step procedures:
> https://docs.pyth.network/SKILL.md
- [Pyth Developer Playbook](https://docs.pyth.network/SKILL.md): Opinionated integration guide with step-by-step procedures and code snippets.

## Products

### Pyth Core — Decentralized Price Oracle
Pull-based oracle providing 500+ price feeds with 400ms updates across 100+ chains. Applications fetch signed prices from Hermes and verify on-chain in a single transaction.
Best for: DeFi protocols, lending, DEXs, derivatives.
Chains: EVM, Solana, Sui, Aptos, CosmWasm, NEAR, Starknet, and more.
> https://docs.pyth.network/llms-price-feeds-core.txt

### Pyth Pro — Low-Latency Price Streaming
Enterprise WebSocket streaming with configurable update channels (1ms–1s). Requires API key.
Best for: HFT, MEV strategies, market making, risk management.
SDK: \`@pythnetwork/pyth-lazer-sdk\` (TypeScript)
> https://docs.pyth.network/llms-price-feeds-pro.txt
MCP server for AI assistants (setup, tools, troubleshooting):
> https://docs.pyth.network/price-feeds/pro/mcp.mdx
Pre-built MCP skills (price alerts, portfolio tracking, FX conversion, volatility analysis, and more):
> https://docs.pyth.network/price-feeds/pro/mcp-skills.mdx

### Entropy — On-Chain Randomness
Secure verifiable random number generation using commit-reveal. Callback-based API.
Best for: Gaming, NFT mints, lotteries, fair selection.
> https://docs.pyth.network/llms-entropy.txt

### Express Relay — MEV Protection
Auction-based MEV capture and order flow protection for DeFi protocols.
> https://docs.pyth.network/express-relay/index.mdx

## Unsure Which Price Feed Product?
Comparison of Core vs Pro with decision matrix:
> https://docs.pyth.network/llms-price-feeds.txt
- [Pyth Core — Decentralized Price Oracle](https://docs.pyth.network/llms-price-feeds-core.txt): Pull-based oracle with 500+ price feeds, 400ms updates, 100+ chains (EVM, Solana, Sui, Aptos, CosmWasm, NEAR, Starknet, and more). Best for DeFi protocols, lending, DEXs, and derivatives.
- [Pyth Pro — Low-Latency Price Streaming](https://docs.pyth.network/llms-price-feeds-pro.txt): Enterprise WebSocket streaming with configurable update channels (1ms–1s). Requires an API key. Best for HFT, MEV strategies, market making, and risk management.
- [Entropy — On-Chain Randomness](https://docs.pyth.network/llms-entropy.txt): Secure verifiable random number generation using commit-reveal with a callback-based API. Best for gaming, NFT mints, lotteries, and fair selection.
- [Express Relay — MEV Protection](https://docs.pyth.network/express-relay/index.mdx): Auction-based MEV capture and order flow protection for DeFi protocols.

## Choosing Between Core and Pro

- [Price Feeds — Core vs Pro Overview](https://docs.pyth.network/llms-price-feeds.txt): Side-by-side comparison and decision matrix for picking between Pyth Core and Pyth Pro.

## Pyth Pro Tooling

- [Pyth Pro MCP Server](https://docs.pyth.network/price-feeds/pro/mcp.mdx): Setup, tools, and troubleshooting for the MCP server that exposes Pyth Pro to AI assistants.
- [Pyth Pro MCP Skills](https://docs.pyth.network/price-feeds/pro/mcp-skills.mdx): Pre-built skills for price alerts, portfolio tracking, FX conversion, volatility analysis, and more.

## Individual Page Access
Fetch any documentation page as plain markdown by appending .mdx:
https://docs.pyth.network/price-feeds/core/getting-started.mdx

- [Markdown page access](https://docs.pyth.network/price-feeds/core/getting-started.mdx): Any documentation page is available as plain markdown by appending \`.mdx\` to its URL.

## Machine-Readable Metadata
Programmatic discovery with token counts and content hashes:
> https://docs.pyth.network/llms-manifest.json

## Instructions for AI Agents
1. Read the product descriptions above to identify which product the user needs.
2. Fetch exactly ONE product file — each is self-contained with code examples, addresses, and patterns.
3. For deeper detail, fetch individual pages via .mdx URLs listed in each product file.
4. Do NOT fetch all files — only fetch the single best match for the user's question.

- [llms-manifest.json](https://docs.pyth.network/llms-manifest.json): Programmatic discovery of all routing files with token counts and content hashes.
`;

export function GET() {
return new NextResponse(CONTENT, {
headers: {
"Cache-Control": "public, max-age=86400",
"Cache-Control": "public, max-age=3600",
"Content-Type": "text/plain; charset=utf-8",
},
status: 200,
Expand Down
26 changes: 26 additions & 0 deletions apps/developer-hub/src/components/Root/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,33 @@ export const TABS = [

export const Root = ({ children, googleAnalyticsId }: Props) => (
<html lang="en">
<head>
<link
rel="alternate"
type="text/plain"
title="llms.txt — documentation index for AI agents"
href="/llms.txt"
/>
</head>
<body>
<a
href="/llms.txt"
style={{
position: "absolute",
width: 1,
height: 1,
padding: 0,
margin: -1,
overflow: "hidden",
clip: "rect(0, 0, 0, 0)",
whiteSpace: "nowrap",
border: 0,
}}
>
For AI agents: see /llms.txt for the documentation index. Any page can
be fetched as markdown by appending .mdx to its URL, or by sending an
Accept: text/markdown header.
</a>
<RootProviders providers={[NuqsAdapter]}>
<FumadocsRootProvider
search={{
Expand Down
2 changes: 1 addition & 1 deletion apps/developer-hub/src/data/llm-files.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ export type LLMFileConfig = {

export const LLM_FILES: LLMFileConfig[] = [
{
cacheMaxAge: 86_400,
cacheMaxAge: 3600,
changeFrequency: "monthly",
description: "Product overview and routing to detailed documentation",
path: "/llms.txt",
Expand Down
2 changes: 2 additions & 0 deletions apps/developer-hub/src/lib/get-llm-text.ts
Original file line number Diff line number Diff line change
Expand Up @@ -38,5 +38,7 @@ export async function getLLMText(page: Page) {
return `# ${page.data.title ?? "Untitled"}
URL: ${page.url}

> For the complete documentation index, see [llms.txt](https://docs.pyth.network/llms.txt).

${String(processed.value)}`;
}
69 changes: 69 additions & 0 deletions apps/developer-hub/src/middleware.ts
Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration Bot May 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 No test coverage for new middleware content negotiation

The new middleware (apps/developer-hub/src/middleware.ts) introduces non-trivial content negotiation logic including Accept header parsing, quality factor comparison, path skip lists, and URL rewriting. Per REVIEW.md, new functionality should have tests. The prefersMarkdown function in particular has enough edge cases (multiple entries, quality factors, missing entries, malformed input) that unit tests would be valuable. The AGENTS.md file notes that linting and type-checking are the primary gate, but the middleware logic is complex enough that tests would help catch regressions.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
import { NextResponse } from "next/server";
import type { NextRequest } from "next/server";

const SKIP_PREFIXES = [
"/_next",
"/api",
"/playground",
"/mdx",
];

const SKIP_EXACT = new Set([
"/",
"/favicon.ico",
"/robots.txt",
"/sitemap.xml",
"/llms.txt",
"/llms-full.txt",
"/llms-manifest.json",
"/llms-entropy.txt",
"/llms-price-feeds.txt",
"/llms-price-feeds-core.txt",
"/llms-price-feeds-pro.txt",
"/SKILL.md",
]);

function prefersMarkdown(accept: string | null): boolean {
if (!accept) return false;
const entries = accept.split(",").map((part) => {
const [media = "", ...params] = part.trim().split(";").map((s) => s.trim());
const qParam = params.find((p) => p.startsWith("q="));
const q = qParam ? Number.parseFloat(qParam.slice(2)) : 1;
return { media, q: Number.isFinite(q) ? q : 1 };
});
const markdown = entries.find(
(e) => e.media === "text/markdown" || e.media === "text/x-markdown",
);
if (!markdown) return false;
const html = entries.find((e) => e.media === "text/html");
return !html || markdown.q >= html.q;
}

export function middleware(request: NextRequest) {
if (request.method !== "GET" && request.method !== "HEAD") {
return NextResponse.next();
}
if (!prefersMarkdown(request.headers.get("accept"))) {
return NextResponse.next();
}

const { pathname } = request.nextUrl;
if (SKIP_EXACT.has(pathname)) return NextResponse.next();
if (SKIP_PREFIXES.some((p) => pathname === p || pathname.startsWith(`${p}/`))) {
return NextResponse.next();
}
// Already a .md/.mdx URL — let the existing rewrite handle it.
if (/\.[a-z0-9]+$/i.test(pathname)) return NextResponse.next();

const url = request.nextUrl.clone();
url.pathname = `/mdx${pathname}`;
const response = NextResponse.rewrite(url);
response.headers.set("Vary", "Accept");
return response;
Comment on lines +42 to +62
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Missing Vary: Accept on non-rewritten responses breaks HTTP caching for content-negotiated paths

The middleware sets Vary: Accept only on responses that are rewritten to /mdx/... (src/middleware.ts:61), but does NOT set it on the pass-through NextResponse.next() responses for the same content-negotiable paths. Per HTTP spec (RFC 7231 §7.1.4), when a URL can return different representations based on request headers, ALL responses—including the "default" HTML one—must include Vary: Accept so that intermediate caches (CDN, forward proxy, browser cache) know to key on the Accept header. Without it, a cached HTML response could be served to a client requesting text/markdown, or vice versa.

On Vercel the impact is mitigated because middleware runs before CDN cache lookup, and rewrites use a different internal URL. But on any other deployment platform or when an external CDN sits in front, this is a correctness bug.

Prompt for agents
In middleware.ts, the Vary: Accept header is only set on the NextResponse.rewrite() path (line 61), but NOT on the NextResponse.next() returns for paths that are subject to content negotiation (i.e., paths that pass all skip checks but the client doesn't prefer markdown). 

The fix: for the code path after all SKIP_EXACT / SKIP_PREFIXES / file-extension checks (i.e., after line 56, when the path IS content-negotiable), if prefersMarkdown returns false, you should still return a NextResponse.next() with Vary: Accept set. Currently, the prefersMarkdown check and the skip checks are interleaved — the prefersMarkdown check happens first (line 46) and returns NextResponse.next() before we even know if the path is content-negotiable.

Suggested restructuring: move the prefersMarkdown check AFTER the skip checks, so you can distinguish between 'skipped path' (no Vary needed) and 'content-negotiable path where client wants HTML' (Vary: Accept needed). For the latter, do:
  const response = NextResponse.next();
  response.headers.set('Vary', 'Accept');
  return response;

Alternatively, add Vary: Accept to the headers config in next.config.js for the doc page paths, but per-path middleware logic is cleaner.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

}

export const config = {
matcher: [
"/((?!_next/static|_next/image|_next/data|favicon.ico).*)",
],
};
Comment thread
devin-ai-integration[bot] marked this conversation as resolved.
Loading