feat: add @llmstxt/middleware package

README.md

@@ -1,24 +1,26 @@
-# @llmstxt — Generate `llms.txt` and `llms-full.txt` for your website
+# @llmstxt — `llms.txt`, `llms-full.txt`, and Markdown for agents
 
-Generate `llms.txt` (a table of contents) and `llms-full.txt` (full content) from your app’s pages so LLMs can quickly understand what your site covers.
+Generate `llms.txt` (table of contents) and `llms-full.txt` (full content) from your app’s pages, and optionally serve **any page as Markdown** via content negotiation (`Accept: text/markdown`).
 
 - Standard: [llmstxt.org](https://llmstxt.org)
-- Works with: Next.js App Router (built-in), plus any Node.js framework via `@llmstxt/core`
+- Works with: Next.js App Router (`@llmstxt/next`), any Node.js framework (`@llmstxt/core`), and Next.js middleware (`@llmstxt/middleware`)
 
 ## Packages
 
 | Package | What it does |
 |---|---|
 | [`@llmstxt/core`](./packages/core) | Scan your `app/` directory and generate `llms.txt` + `llms-full.txt` |
 | [`@llmstxt/next`](./packages/next) | Next.js App Router route handlers for `/llms.txt` and `/llms-full.txt` |
+| [`@llmstxt/middleware`](./packages/middleware) | Next.js middleware: any page responds as Markdown on `Accept: text/markdown` |
 
 ## Keywords (SEO)
 
-llms.txt, llms-full.txt, llmstxt, Next.js, App Router, AI, LLM, SEO, website indexing, documentation, crawlable content, sitemap alternative
+llms.txt, llms-full.txt, llmstxt, markdown for agents, content negotiation, Next.js, middleware, App Router, AI, LLM, SEO, website indexing, documentation, crawlable content, sitemap alternative
 
 ## Table of Contents
 
 - [Quick Start — Next.js (App Router)](#quick-start--nextjs-app-router)
+- [Quick Start — Markdown for Agents (Next.js Middleware)](#quick-start--markdown-for-agents-nextjs-middleware)
 - [Quick Start — Other Frameworks](#quick-start--other-frameworks-express-hono-bun-etc)
 - [How It Works](#how-it-works)
 - [API Reference (`@llmstxt/core`)](#api-reference-llmstxtcore)
@@ -94,6 +96,39 @@ export const GET = createLlmsFullTxtHandler({
 
 ---
 
+## Quick Start — Markdown for Agents (Next.js Middleware)
+
+Serve the Markdown version of **any page** when a client sends `Accept: text/markdown`.
+
+### Install
+
+```bash
+npm install @llmstxt/middleware
+```
+
+### Add middleware
+
+```ts
+// middleware.ts (project root)
+export { middleware, config } from '@llmstxt/middleware'
+```
+
+This returns:
+- `Content-Type: text/markdown; charset=utf-8`
+- `Vary: Accept` (so CDNs cache HTML and Markdown separately)
+- `x-markdown-tokens` (token estimate for context sizing)
+- `Content-Signal` (content usage preferences)
+
+Test it:
+
+```bash
+curl https://yoursite.com/blog/my-post -H "Accept: text/markdown"
+```
+
+For best Markdown quality in production, plug in `@mozilla/readability` + `turndown` (see `packages/middleware/README.md`).
+
+---
+
 ## Quick Start — Other Frameworks (Express, Hono, Bun, etc.)
 
 ```bash
@@ -152,6 +187,9 @@ Example output:
 ### `llms-full.txt` (Full content)
 Same scan, but also fetches and converts each page to Markdown (or plain text by default). This produces one big file containing your site’s content for ingestion.
 
+### `Accept: text/markdown` (Markdown for agents)
+`@llmstxt/middleware` adds content negotiation to your Next.js app: when a client requests any URL with `Accept: text/markdown`, it re-fetches that page as HTML, converts it to Markdown, and returns it with `Content-Type: text/markdown`, `Vary: Accept`, and `x-markdown-tokens`.
+
 ## Why this helps SEO (and AI discoverability)
 
 Search engines don’t directly “rank” `llms.txt`, but having clean, crawlable pages with accurate titles/descriptions helps both humans and models.

package.json

@@ -1,7 +1,7 @@
 {
   "name": "@llmstxt/monorepo",
   "private": true,
-  "description": "Monorepo for @llmstxt packages.",
+  "description": "Monorepo for @llmstxt (llms.txt, llms-full.txt, and Markdown for agents).",
   "license": "MIT",
   "workspaces": [
     "packages/*"

packages/middleware/README.md

@@ -0,0 +1,89 @@
+# `@llmstxt/middleware`
+
+Next.js middleware that serves **any page as Markdown** when the request includes `Accept: text/markdown`.
+
+This mirrors Cloudflare's "Markdown for Agents" content-negotiation pattern, but runs entirely inside your own Next.js app.
+
+---
+
+## Install
+
+```bash
+npm install @llmstxt/middleware
+```
+
+---
+
+## Usage (one line)
+
+```ts
+// middleware.ts (project root)
+export { middleware, config } from '@llmstxt/middleware'
+```
+
+Test it:
+
+```bash
+curl https://yoursite.com/blog/my-post -H "Accept: text/markdown"
+```
+
+---
+
+## Response headers
+
+- `Content-Type: text/markdown; charset=utf-8`
+- `Vary: Accept`
+- `x-markdown-tokens: <estimated token count>`
+- `Content-Signal: <content usage preferences>`
+
+---
+
+## With options
+
+```ts
+// middleware.ts
+import { createMarkdownMiddleware } from '@llmstxt/middleware'
+
+export const middleware = createMarkdownMiddleware({
+  // Opt-out of AI training while allowing agents to read
+  contentSignal: 'ai-train=no, search=yes, ai-input=yes',
+
+  // Better HTML→Markdown output (optional)
+  htmlToMarkdown: async (html) => {
+    const TurndownService = (await import('turndown')).default
+    const td = new TurndownService({ headingStyle: 'atx', codeBlockStyle: 'fenced' })
+    return td.turndown(html)
+  },
+})
+
+export const config = {
+  matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'],
+}
+```
+
+---
+
+## Better Markdown quality (Readability + Turndown)
+
+```bash
+npm install turndown @mozilla/readability jsdom
+```
+
+```ts
+import { createMarkdownMiddleware } from '@llmstxt/middleware'
+import TurndownService from 'turndown'
+import { Readability } from '@mozilla/readability'
+import { JSDOM } from 'jsdom'
+
+export const middleware = createMarkdownMiddleware({
+  htmlToMarkdown: async (html, url) => {
+    const dom = new JSDOM(html, { url })
+    const article = new Readability(dom.window.document).parse()
+    if (!article) return ''
+
+    const td = new TurndownService({ headingStyle: 'atx', codeBlockStyle: 'fenced' })
+    return `# ${article.title}\n\n${td.turndown(article.content)}`
+  },
+})
+```
+

packages/middleware/package.json

@@ -0,0 +1,49 @@
+{
+  "name": "@llmstxt/middleware",
+  "version": "0.1.0",
+  "description": "Next.js middleware that serves Markdown when requests send Accept: text/markdown.",
+  "license": "MIT",
+  "author": "Recordly",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/Muhammad-Hashim/llmstxt.git",
+    "directory": "packages/middleware"
+  },
+  "homepage": "https://github.com/Muhammad-Hashim/llmstxt#readme",
+  "keywords": [
+    "llmstxt",
+    "markdown",
+    "markdown for agents",
+    "content negotiation",
+    "accept header",
+    "nextjs",
+    "middleware",
+    "ai",
+    "llm",
+    "seo"
+  ],
+  "type": "commonjs",
+  "main": "./lib/index.js",
+  "types": "./lib/index.d.ts",
+  "readme": "README.md",
+  "files": [
+    "lib/**/*"
+  ],
+  "exports": {
+    ".": {
+      "types": "./lib/index.d.ts",
+      "default": "./lib/index.js"
+    }
+  },
+  "peerDependencies": {
+    "next": ">=13.0.0"
+  },
+  "scripts": {
+    "build": "tsc -p tsconfig.build.json",
+    "clean": "node ../../scripts/rmrf.js lib"
+  },
+  "engines": {
+    "node": ">=18.0.0"
+  }
+}
+