feat: add @llmstxt/middleware package

[Muhammad-Hashim]

Description of change

Pull-Request Checklist

• Code is up-to-date with the main branch
• npm run lint passes with this change
• npm run test passes with this change
• This pull request links relevant issues as Fixes #0000
• There are new or updated unit tests validating the change
• Documentation has been updated to reflect this change
• The new commits follow conventions outlined in the conventional commit spec

Summary by CodeRabbit

• New Features

  • Added Markdown content negotiation—pages respond with Markdown when requests include Accept: text/markdown header.
  • Introduced @llmstxt/middleware, a Next.js middleware package for automatic HTML-to-Markdown conversion.

• Documentation

  • Updated documentation with setup guides and integration examples for the new middleware.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Free

Run ID: 6e403699-b9d2-43d0-992d-f75d68af8347

📥 Commits

Reviewing files that changed from the base of the PR and between e4047bd and 52c8875.

⛔ Files ignored due to path filters (1)

• package-lock.json is excluded by !**/package-lock.json

📒 Files selected for processing (8)

• README.md
• package.json
• packages/middleware/README.md
• packages/middleware/package.json
• packages/middleware/src/index.ts
• packages/middleware/tsconfig.build.json
• packages/middleware/tsconfig.json
• packages/next/README.md

---

📝 Walkthrough

Walkthrough

This PR introduces @llmstxt/middleware, a Next.js middleware package that enables content negotiation to serve any page as Markdown when clients request Accept: text/markdown. It includes built-in HTML-to-Markdown conversion, configurable URL exclusions, optional YAML front-matter, and response metadata headers. Root and integration documentation are updated accordingly.

Changes

Markdown Middleware Package

Layer / File(s)	Summary
Package Configuration packages/middleware/package.json, packages/middleware/tsconfig.json, packages/middleware/tsconfig.build.json	Package metadata defines @llmstxt/middleware with CommonJS distribution, exports mapping, next>=13.0.0 peer dependency, and TypeScript build configuration.
Core Middleware Implementation packages/middleware/src/index.ts	MarkdownMiddlewareOptions type, builtinHtmlToMarkdown() converter, wantsMarkdown() Accept header inspector, createMarkdownMiddleware() factory that fetches HTML, converts to Markdown, adds YAML front-matter and response headers (Content-Type: text/markdown, x-markdown-tokens, Content-Signal), and falls back to NextResponse.next() on errors or non-eligibility.
Middleware Package Documentation packages/middleware/README.md	Installation, one-line usage export, configuration options with createMarkdownMiddleware, response headers, curl test example, and production-quality Markdown conversion using jsdom, @mozilla/readability, and turndown.
Root Repository & Integration Documentation README.md, package.json, packages/next/README.md	Root description updated to reflect monorepo focus on llms.txt, llms-full.txt, and agent Markdown. Root README adds Markdown-for-Agents Quick Start section with install/usage/headers/curl test. Next.js package README documents @llmstxt/middleware optional integration via middleware.ts export.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 A middleware hops through the web with grace,
Converting HTML to Markdown with speed and space,
Content negotiation brings headers that gleam,
Text-markdown flows like a rabbit's dream! 🌿

---

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}