Currently the server logs to tty every scraping, adding and storing step of the indexing process. This means that the log output becomes very long very quickly. I can disable the logging entirely with the --silent flag, but then I miss everything.
Because the output comes to tty, a simple | grep -v won't help in filtering it. Instead something is needed to capture the tty output and redirect it back to stdout. I personally ended up using script command. However the end result feels unnecessarily complex approach:
script -c 'OPENAI_API_KEY="ollama" OPENAI_API_BASE="http://localhost:11434/v1" DOCS_MCP_EMBEDDING_MODEL="openai:nomic-embed-text" DOCS_MCP_TELEMETRY=false npx @arabold/docs-mcp-server@latest --telemetry false --resume' /dev/null | grep -v "Scraping\|Adding\|Storing"
What I think could improve it
Could there be another built-in logging level between the default and silent?
Currently there are log levels quiet, verbose and perhaps the default is different from verbose. It would be helpful to have a level pretty close to the --silent option but still reporting errors, warnings and when a job is queued, started refreshed or completed.
As a user with thousands of indexed documents, I do not really care how the indexing happens, unless there's an issue I need to troubleshoot, I just need to know whether everything succeeded or if I still need to wait. Currently the web ui does not show if a job completed successfully or not, so I need to read that from the logs.
For the cases when the user needs more control of the logging process
- Could the process log everything to stderr by default, that would be easier to process with standard unix tools? It shouldn't conflict with the mcp protocol, but I'm not an expert on this.
- If not, could there be a cli flag or env variable to enable such?
- Could there be an option to disable the emojis? I spent quite a lot of time trying to figure out whether the emoji characters were the problem in piping to grep - that would make troubleshooting and log regex matching easier.
Currently the server logs to tty every scraping, adding and storing step of the indexing process. This means that the log output becomes very long very quickly. I can disable the logging entirely with the
--silentflag, but then I miss everything.Because the output comes to tty, a simple
| grep -vwon't help in filtering it. Instead something is needed to capture the tty output and redirect it back to stdout. I personally ended up using script command. However the end result feels unnecessarily complex approach:What I think could improve it
Could there be another built-in logging level between the default and silent?
Currently there are log levels quiet, verbose and perhaps the default is different from verbose. It would be helpful to have a level pretty close to the
--silentoption but still reporting errors, warnings and when a job is queued, started refreshed or completed.As a user with thousands of indexed documents, I do not really care how the indexing happens, unless there's an issue I need to troubleshoot, I just need to know whether everything succeeded or if I still need to wait. Currently the web ui does not show if a job completed successfully or not, so I need to read that from the logs.
For the cases when the user needs more control of the logging process