vite-plugin-robots-ts

A Vite plugin to generate robots.txt. Supports blocking AI training crawlers and works in development mode by proxying middleware to /robots.txt.

Installation

npm install -D vite-plugin-robots-ts
pnpm add -D vite-plugin-robots-ts
yarn add -D vite-plugin-robots-ts
bun add -D vite-plugin-robots-ts
deno add -D npm:vite-plugin-robots-ts

Usage

// vite.config.ts
import { robots } from 'vite-plugin-robots-ts'

export default {
  plugins: [
    robots(), // block all robots by default
  ],
}

Examples

Block all (default):

robots()

// or

import { BLOCK_ALL, robots } from 'vite-plugin-robots-ts'

robots({ content: BLOCK_ALL })

Output:

User-agent: *
Disallow: /

Allow all:

import { ALLOW_ALL, robots } from 'vite-plugin-robots-ts'

robots({ content: ALLOW_ALL })

Output:

User-agent: *
Disallow:

Block only AI training crawlers (allow rest):

import { BLOCK_AI_ALLOW_REST, robots } from 'vite-plugin-robots-ts'

robots({ content: BLOCK_AI_ALLOW_REST })

Output:

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: *
Disallow:

Use built-in presets:

import { BLOCK_AI, BLOCK_ALL, robots } from 'vite-plugin-robots-ts'

robots({
  content:
    process.env.NODE_ENV === 'production'
      ? `${BLOCK_AI}\n<Custom content here>`
      : BLOCK_ALL,
})

Output (production):

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

<Custom content here>

Output (all other environments):

User-agent: *
Disallow: /

Use custom content:

robots({
  content: `User-agent: *
Disallow: /admin
Disallow: /secret`,
})

Output:

User-agent: *
Disallow: /admin
Disallow: /secret

Append sitemap directive:

import { ALLOW_ALL, robots } from 'vite-plugin-robots-ts'

robots({
  content: ALLOW_ALL,
  sitemap: 'https://example.com/sitemap.xml'
})

Output:

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

Multiple sitemaps:

import { ALLOW_ALL, robots } from 'vite-plugin-robots-ts'

robots({
  content: ALLOW_ALL,
  sitemap: [
    'https://example.com/sitemap.xml',
    'https://example.com/sitemap-news.xml',
  ]
})

Output:

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-news.xml

Options

All options are optional.

Option	Type	Default	Description
content	string	BLOCK_ALL	Custom content for robots.txt
sitemap	string | string[]	-	Adds Sitemap directive(s) to robots.txt
outDir	string	-	Custom output directory for robots.txt (resolved relative to project root)

Constants

The plugin exports preset constants that can be used with the content option or composed into custom configurations:

Constant	Description
ALLOW_ALL	Allows all robots (User-agent: * / Disallow:)
BLOCK_ALL	Blocks all robots (User-agent: * / Disallow: /)
BLOCK_AI	Blocks known AI training crawlers only
BLOCK_AI_ALLOW_REST	Blocks known AI training crawlers and allows all other robots

Usage

import {
  ALLOW_ALL,
  BLOCK_ALL,
  BLOCK_AI,
  BLOCK_AI_ALLOW_REST,
} from 'vite-plugin-robots-ts'