Discussion: Benchmarking, Scraping Performance, shared pre-scraped and indexed

[xuanswe]

Hello,

Thank you for creating such an impressive tool.

Since I couldn't find a dedicated discussion space, I'll share my questions here:

Have you benchmarked this tool against others (e.g., Context7, Nia, Ref.Tools)? It might be interesting to create a YouTube video where you set up identical environments, run the same tasks with Claude Code, and compare the results.

Scraping large documentation sites (like Flutter or React) is quite slow when dealing with thousands of pages. Would it be more efficient to reuse something like Context7's llms.txt (e.g., https://context7.com/websites/flutter_dev/llms.txt?tokens=100000) instead of scraping the entire site (e.g., flutter.dev)? Which approach tends to work better in practice?

Given the slowness of scraping, what do you think about maintaining a public GitHub repository with pre-scraped and indexed docs? That way, local docs-mcp-server could simply download and use them immediately. It should also allow working with any private Git repositories, of course.

Sorry for putting multiple questions in a single issue.

[arabold]

A benchmark would be nice but it's hard enough to do that I haven't had any motivation for it yet 😅

The latest version also supports scraping llms.txt indexes directly, which can be very helpful but they point of the server is that documents are indexed completely, not just summaries. It also supports requesting markdown directly from the backend service (Cloudfront supports this), which makes scraping a lot faster. If you want, you could point it to index context7 instead, but then you could use Context7 right away.

A public GitHub repo would indeed be nice. My biggest concern would be to ensure all documentation is under a permissible license though, as otherwise you're copying potentially copyrighted content, even if you keep it free. Hard to police but certainly doable. My "dream" would be to have an cloud MCP that provides preindexed docs, so new users don't have to index themselves. Using GitHub for that is a great idea! Let me think this though...

[xuanswe]

cloud MCP that provides preindexed docs

I think this will introduce costs to run the cloud and trigger expenses for the whole community, similar to Context7. One reason I use docs-mcp-server is to avoid Context7 costs. That's why I suggested using public GitHub repos to replace the unnecessary expense of Context7.

The latest version also supports scraping llms.txt indexes directly

Nice feature.

It also supports requesting markdown directly from the backend service (Cloudfront supports this), which makes scraping a lot faster.

Could you explain the technical details of how this works? I'm not entirely clear on the mechanism.

If you want, you could point it to index context7 instead, but then you could use Context7 right away.

My idea is to avoid paying for Context7 :). That's why I want to manually manage docs for major packages in my technology stack. Honestly, there are only a few docs in each project (even large project) and paying for them makes no sense to me.

My biggest concern would be to ensure all documentation is under a permissible license

I also wonder how Context7 handles this issue. Their indexes are publicly accessible for download. If they could do it without issues, then why do we have issues?