Unclear Error when Attempting to Scrape React Docs

[isugimpy]

This is going to be a pretty lacking issue, but I genuinely don't have much data to offer.

Version: 2.2.1 Docker image
Environment Variables

• OPENAI_API_KEY=foo
• OPENAI_API_BASE=http://192.168.1.105:8000/v1
• DOCS_MCP_EMBEDDING_MODEL=nomic-embed-text-v2-moe-GGUF

When attempting to scrape the docs at https://react.dev/reference/react, a large number of errors similar to the below occur, and only a small number of pages seem to be scraped. No other context on the problem is emitted in the logs, so I'm not sure where to even begin investigating this to give more information.

Failed to process content https://react.dev/reference/react/useCallback: ConnectionError: Failed to add documents to store caused by TypeError: Cannot read properties of undefined (reading '0')

[arabold]

This appears to be an embeddings integration failure rather than a scraper failure. The app is successfully fetching and processing the React pages, but fails while generating/storing embeddings. Because the server is using an OpenAI-compatible backend (OPENAI_API_BASE) with a local GGUF model, the most likely cause is that the embedding backend returns an unexpected/empty data payload for some requests, and our current code logs that only as a generic Failed to add documents to store error. Larger React reference pages likely make this worse because the fallback only retries explicit size-limit errors, not malformed responses.

I'll take a look and see if I can reproduce locally.

[isugimpy]

I did some more digging on this and found a couple problems on my side. I set the value to nomic-embed-text-v2-moe-GGUF instead of openai:nomic-embed-text-v2-moe-GGUF. Once I solved that, I noticed that errors were being logged by llama.cpp indicating the batch size was too small. I've fixed that now. llama.cpp then logged that the context size was too small. Turns out that this model had a built in setting which set it to 512 despite the server setting higher. I've overridden that now and things appear to be working. I would still recommend improving the messaging here if possible, because debugging this was some guesswork.