Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .npmrc.example
Original file line number Diff line number Diff line change
@@ -1 +1 @@
//registry.npmjs.org/:_authToken=YOUR_NPM_AUTOMATION_TOKEN_HERE
//registry.npmjs.org/:_authToken=YOUR_NPM_AUTOMATION_TOKEN_HERE
105 changes: 75 additions & 30 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,30 +1,32 @@
# @llmstxt — `llms.txt`, `llms-full.txt`, and Markdown for agents
# @llmtxt — `llms.txt`, `llms-full.txt`, and Markdown for agents

Generate `llms.txt` (table of contents) and `llms-full.txt` (full content) from your app’s pages, and optionally serve **any page as Markdown** via content negotiation (`Accept: text/markdown`).

- Standard: [llmstxt.org](https://llmstxt.org)
- Works with: Next.js App Router (`@llmstxt/next`), any Node.js framework (`@llmstxt/core`), and Next.js middleware (`@llmstxt/middleware`)
- Works with: Next.js App Router (`@llmtxt/next`), any Node.js framework (`@llmtxt/core`), and Next.js middleware (`@llmtxt/middleware`)

## Packages

| Package | What it does |
|---|---|
| [`@llmstxt/core`](./packages/core) | Scan your `app/` directory and generate `llms.txt` + `llms-full.txt` |
| [`@llmstxt/next`](./packages/next) | Next.js App Router route handlers for `/llms.txt` and `/llms-full.txt` |
| [`@llmstxt/middleware`](./packages/middleware) | Next.js middleware: any page responds as Markdown on `Accept: text/markdown` |
| [`@llmtxt/core`](./packages/core) | Scan your `app/` directory and generate `llms.txt` + `llms-full.txt` |
| [`@llmtxt/next`](./packages/next) | Next.js App Router route handlers for `/llms.txt` and `/llms-full.txt` |
| [`@llmtxt/middleware`](./packages/middleware) | Next.js middleware: any page responds as Markdown on `Accept: text/markdown` |
| [`@llmtxt/react`](./packages/react) | Build-time helpers for React SPAs: generate `llms.txt` + `llms-full.txt` from a route list |

## Keywords (SEO)

llms.txt, llms-full.txt, llmstxt, markdown for agents, content negotiation, Next.js, middleware, App Router, AI, LLM, SEO, website indexing, documentation, crawlable content, sitemap alternative
llms.txt, llms-full.txt, llmtxt, markdown for agents, content negotiation, Next.js, middleware, App Router, AI, LLM, SEO, website indexing, documentation, crawlable content, sitemap alternative

## Table of Contents

- [Quick Start — Next.js (App Router)](#quick-start--nextjs-app-router)
- [Quick Start — Markdown for Agents (Next.js Middleware)](#quick-start--markdown-for-agents-nextjs-middleware)
- [Quick Start — React (Build-time, no backend)](#quick-start--react-build-time-no-backend)
- [Quick Start — Other Frameworks](#quick-start--other-frameworks-express-hono-bun-etc)
- [How It Works](#how-it-works)
- [API Reference (`@llmstxt/core`)](#api-reference-llmstxtcore)
- [Environment Variables (`@llmstxt/next`)](#environment-variables-llmstxtnext)
- [API Reference (`@llmtxt/core`)](#api-reference-llmtxtcore)
- [Environment Variables (`@llmtxt/next`)](#environment-variables-llmtxtnext)
- [Tips for Better Descriptions](#tips-for-better-descriptions)
- [Development (this repo)](#development-this-repo)
- [License](#license)
Expand All @@ -36,7 +38,7 @@ llms.txt, llms-full.txt, llmstxt, markdown for agents, content negotiation, Next
### Install

```bash
npm install @llmstxt/next
npm install @llmtxt/next
```

### Add route handlers
Expand All @@ -51,20 +53,20 @@ src/app/

**`src/app/llms.txt/route.ts`** (zero-config):
```ts
export { GET } from '@llmstxt/next'
export { GET } from '@llmtxt/next'
```

**`src/app/llms-full.txt/route.ts`** (zero-config):
```ts
import { createLlmsFullTxtHandler } from '@llmstxt/next'
import { createLlmsFullTxtHandler } from '@llmtxt/next'
export const GET = createLlmsFullTxtHandler()
```

Set one of:
- `NEXT_PUBLIC_APP_URL` (recommended)
- `VERCEL_URL` (usually already set on Vercel)

> Tip: Set `NEXT_PUBLIC_APP_URL` explicitly in local development and production. `@llmstxt/next` will only fall back to `http://localhost:3000` when `NODE_ENV=development`.
> Tip: Set `NEXT_PUBLIC_APP_URL` explicitly in local development and production. `@llmtxt/next` will only fall back to `http://localhost:3000` when `NODE_ENV=development`.

Visit:
- `/llms.txt`
Expand All @@ -74,7 +76,7 @@ Visit:

```ts
// src/app/llms.txt/route.ts
import { createLlmsTxtHandler } from '@llmstxt/next'
import { createLlmsTxtHandler } from '@llmtxt/next'

export const GET = createLlmsTxtHandler({
title: 'My SaaS App',
Expand All @@ -85,7 +87,7 @@ export const GET = createLlmsTxtHandler({

```ts
// src/app/llms-full.txt/route.ts
import { createLlmsFullTxtHandler } from '@llmstxt/next'
import { createLlmsFullTxtHandler } from '@llmtxt/next'

export const GET = createLlmsFullTxtHandler({
fetchTimeoutMs: 8000,
Expand All @@ -105,14 +107,14 @@ Serve the Markdown version of **any page** when a client sends `Accept: text/mar
### Install

```bash
npm install @llmstxt/middleware
npm install @llmtxt/middleware
```

### Add middleware

```ts
// middleware.ts (project root)
export { middleware, config } from '@llmstxt/middleware'
export { middleware, config } from '@llmtxt/middleware'
```

This returns:
Expand All @@ -131,14 +133,57 @@ For best Markdown quality in production, plug in `@mozilla/readability` + `turnd

---

## Quick Start — React (Build-time, no backend)

For React SPAs (Vite/CRA/React Router), generate the files at build time and ship them from `public/`.

```bash
npm install -D @llmtxt/react
```

Create a route list:

```ts
// llmtxt.routes.ts
import type { LlmtxtRoute } from '@llmtxt/react'

export const routes: LlmtxtRoute[] = [
{ path: '/', title: 'Home', description: 'What this site is about.' },
{ path: '/docs', title: 'Docs' },
]
```

Generate:

```ts
// scripts/generate-llms.ts
import path from 'path'
import { writeLlmsFiles } from '@llmtxt/react'
import { routes } from '../llmtxt.routes'

await writeLlmsFiles({
routes,
baseUrl: process.env.PUBLIC_SITE_URL!, // e.g. https://example.com
outDir: path.join(process.cwd(), 'public'),
})
```

Run it after deploy (recommended) or against a preview server:

```bash
PUBLIC_SITE_URL=https://example.com node scripts/generate-llms.ts
```

---

## Quick Start — Other Frameworks (Express, Hono, Bun, etc.)

```bash
npm install @llmstxt/core
npm install @llmtxt/core
```

```ts
import { generateLlmsTxt, generateLlmsFullTxt } from '@llmstxt/core'
import { generateLlmsTxt, generateLlmsFullTxt } from '@llmtxt/core'
import path from 'path'

app.get('/llms.txt', async (req, res) => {
Expand Down Expand Up @@ -167,7 +212,7 @@ app.get('/llms-full.txt', async (req, res) => {
## How It Works

### `llms.txt` (Table of contents)
`@llmstxt/core` scans your `app/` directory for `page.tsx`/`page.jsx`/`page.ts`/`page.js` files and produces a structured
`@llmtxt/core` scans your `app/` directory for `page.tsx`/`page.jsx`/`page.ts`/`page.js` files and produces a structured
index of links + descriptions. LLMs use this like a sitemap to understand what your site covers.

Example output:
Expand All @@ -183,14 +228,14 @@ Example output:
- [Advanced Usage](https://example.com/blog/advanced): Deep-dives and recipes

---
*Generated by @llmstxt/core · 2025-01-01T00:00:00.000Z · 12 pages*
*Generated by @llmtxt/core · 2025-01-01T00:00:00.000Z · 12 pages*
```

### `llms-full.txt` (Full content)
Same scan, but also fetches and converts each page to Markdown (or plain text by default). This produces one big file containing your site’s content for ingestion.

### `Accept: text/markdown` (Markdown for agents)
`@llmstxt/middleware` adds content negotiation to your Next.js app: when a client requests any URL with `Accept: text/markdown`, it re-fetches that page as HTML, converts it to Markdown, and returns it with `Content-Type: text/markdown`, `Vary: Accept`, and `x-markdown-tokens`.
`@llmtxt/middleware` adds content negotiation to your Next.js app: when a client requests any URL with `Accept: text/markdown`, it re-fetches that page as HTML, converts it to Markdown, and returns it with `Content-Type: text/markdown`, `Vary: Accept`, and `x-markdown-tokens`.

## Why this helps SEO (and AI discoverability)

Expand All @@ -199,7 +244,7 @@ These files are primarily for **LLM discoverability**: they provide a structured

---

## API Reference (`@llmstxt/core`)
## API Reference (`@llmtxt/core`)

### `generateLlmsTxt(options)`
Generates `llms.txt` as a string.
Expand Down Expand Up @@ -235,31 +280,31 @@ Generates `llms-full.txt` as a string (fetches pages and converts HTML→text/Ma

---

## Environment Variables (`@llmstxt/next`)
## Environment Variables (`@llmtxt/next`)

| Variable | Purpose |
|---|---|
| `NEXT_PUBLIC_APP_URL` | Your public app root URL, e.g. `https://example.com` — recommended for production and local dev |
| `VERCEL_URL` | Automatically set by Vercel; used as a fallback if `NEXT_PUBLIC_APP_URL` is missing |

> Note: When `NODE_ENV=development`, `@llmstxt/next` will default to `http://localhost:3000` if neither variable is set. For predictable results, always set `NEXT_PUBLIC_APP_URL` in `.env.local` or your deployment environment.
> Note: When `NODE_ENV=development`, `@llmtxt/next` will default to `http://localhost:3000` if neither variable is set. For predictable results, always set `NEXT_PUBLIC_APP_URL` in `.env.local` or your deployment environment.

---

## Packages included

This repository publishes three npm packages:

- `@llmstxt/core` — framework-agnostic generator for `llms.txt` and `llms-full.txt`
- `@llmstxt/next` — Next.js App Router route handlers for `llms.txt` and `llms-full.txt`
- `@llmstxt/middleware` — Next.js middleware that serves Markdown on `Accept: text/markdown`
- `@llmtxt/core` — framework-agnostic generator for `llms.txt` and `llms-full.txt`
- `@llmtxt/next` — Next.js App Router route handlers for `llms.txt` and `llms-full.txt`
- `@llmtxt/middleware` — Next.js middleware that serves Markdown on `Accept: text/markdown`

Install only the packages you need:

```bash
npm install @llmstxt/core
npm install @llmstxt/next
npm install @llmstxt/middleware
npm install @llmtxt/core
npm install @llmtxt/next
npm install @llmtxt/middleware
```

See each package README for the full option reference.
Expand Down
4 changes: 2 additions & 2 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@

This folder contains two example projects for local testing of the monorepo packages:

- `basic-test` — a simple TypeScript script that imports `@llmstxt/core` and generates `llms.txt` locally.
- `next-app-test` — a minimal Next.js App Router project that consumes `@llmstxt/next` and exposes `/llms.txt` and `/llms-full.txt`.
- `basic-test` — a simple TypeScript script that imports `@llmtxt/core` and generates `llms.txt` locally.
- `next-app-test` — a minimal Next.js App Router project that consumes `@llmtxt/next` and exposes `/llms.txt` and `/llms-full.txt`.

## Getting started

Expand Down
28 changes: 28 additions & 0 deletions examples/my-react-app/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*

node_modules
dist
dist-ssr
*.local

# Generated llms files (generated by scripts/generate-llms.js)
public/llms.txt
public/llms-full.txt

# Editor directories and files
.vscode/*
!.vscode/extensions.json
.idea
.DS_Store
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?
Loading
Loading