Add open telemetry and local development helpers

[MMolineux]

OpenTelemetry provides code instrumentation and a protocol for emitting logs, traces and metrics to a remote collector endpoint of the server operator's choice (compatible with Grafana stack, SigNoz, OpenObserve, etc...)

This PR adds first, rough implementation.

Changes:

• Add open telemetry instrumentation using auto-instrumentation and custom context logger.
• Add local development quick-start script dev.sh
• Refactor docker-compose for local development's infra dependencies, including Aspire dashboard for OTel

Aspire dashboard for OTel collector and display. chosen because its a single container and simplest for local dev.

The telemetry folder uses it's own configuration structures to decouple from Flask config object. Flask's app global config couples everything together and prevents dependency injection, so i wanted to start moving off this. Ideally we'd use pydantic or similar, but dataclass for now. If you'd prefer this module to use flask app config, i can change.

Let me know if you need this PR split up, includes a few changes.

Logs:

Traces:

Metrics:

[MMolineux]

adapter between flask app global config and telemetry config.

[brian7704]

I'm trying to run this PR to test but I'm getting the following error that's preventing anything from starting.

$ /home/administrator/.opentakserver_venv/bin/opentakserver
Traceback (most recent call last):
  File "/home/administrator/.opentakserver_venv/bin/opentakserver", line 5, in <module>
    from opentakserver.app import start
  File "/home/administrator/.opentakserver_venv/lib/python3.12/site-packages/opentakserver/app.py", line 5, in <module>
    from opentakserver.models.role import Role
  File "/home/administrator/.opentakserver_venv/lib/python3.12/site-packages/opentakserver/models/role.py", line 4, in <module>
    from opentakserver.extensions import db
  File "/home/administrator/.opentakserver_venv/lib/python3.12/site-packages/opentakserver/extensions.py", line 32, in <module>
    logger, meter = setup_telemetry(TelemetryOpts(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/administrator/.opentakserver_venv/lib/python3.12/site-packages/opentakserver/telemetry/__init__.py", line 26, in setup_telemetry
    logger = setup_logging(opts.logging)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/administrator/.opentakserver_venv/lib/python3.12/site-packages/opentakserver/telemetry/logs.py", line 112, in setup_logging
    logLevel = logging.getLevelNamesMapping().get(opts.level.upper(), logging.INFO)
                                                  ^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'upper'

Edit:

I think the issue is in configure_logging in telemetry/ots.py. My existing config.yml didn't have any of the new options, for example OTS_LOG_FILE_LEVEL. So level=cfg.get("OTS_LOG_FILE_LEVEL") returned None. I think the fix is to add default values for all of the calls to cfg.get() in configure_logging()

[brian7704]

After adding a temporary workaround for the logging issue I still can't get OTS to start. I've attached a debug log with several errors.

opentakserver_log.txt

[MMolineux]

fixed the missing settings issue. Thanks for pointing it out.

Re the errors you're seeing.

1. RMQ errors, could be related to gevent thread monkey patching, I think i encountered that error when hooking up otel. But the branch does not current have those problems for me on clean clone and venv (will do some research).
2. The warnings for "exporter" logger, are just OTel SDK complaining that it can't get to an otel sink, it shouldn't prevent starting. If you have docker, do docker compose up -d aspire-dashboard.

How are you running the stack? It looks like you're creating venv and pip installing it?

[brian7704]

The commit you just pushed allows OTS to start now.

I haven't installed OpenTelemetry yet. I didn't change my OTS config but the new OpenTelemetry feature is enabled for me be default. Can you please make it disabled by default?

I also saw a new error. Looks like the Babel extension isn't being loaded.

Traceback (most recent call last):
  File "/home/administrator/.opentakserver_venv/lib/python3.12/site-packages/opentelemetry/instrumentation/flask/__init__.py", line 398, in _wrapped_app
    result = wsgi_app(wrapped_app_environ, _start_response)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/administrator/.opentakserver_venv/lib/python3.12/site-packages/flask/app.py", line 1514, in wsgi_app
    response = self.handle_exception(e)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/administrator/.opentakserver_venv/lib/python3.12/site-packages/flask_cors/extension.py", line 176, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
                                                ^^^^^^^^^^^^^^^^^^^^
  File "/home/administrator/.opentakserver_venv/lib/python3.12/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/administrator/.opentakserver_venv/lib/python3.12/site-packages/flask/app.py", line 919, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/administrator/.opentakserver_venv/lib/python3.12/site-packages/flask_cors/extension.py", line 176, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
                                                ^^^^^^^^^^^^^^^^^^^^
  File "/home/administrator/.opentakserver_venv/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/administrator/.opentakserver_venv/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/administrator/.opentakserver_venv/lib/python3.12/site-packages/flask_security/decorators.py", line 411, in decorated_view
    return current_app.ensure_sync(fn)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/administrator/.opentakserver_venv/lib/python3.12/site-packages/opentakserver/blueprints/ots_api/video_api.py", line 36, in thumbnail
    return jsonify({"success": False, "error": gettext(u"Please specify a valid path")}), 400
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/administrator/.opentakserver_venv/lib/python3.12/site-packages/flask_babel/__init__.py", line 756, in gettext
    return get_domain().gettext(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/administrator/.opentakserver_venv/lib/python3.12/site-packages/flask_babel/__init__.py", line 750, in get_domain
    ctx.babel_domain = get_babel().instance.domain_instance
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/administrator/.opentakserver_venv/lib/python3.12/site-packages/flask_babel/__init__.py", line 46, in get_babel
    return app.extensions['babel']
           ^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'babel'

[MMolineux]

Actually, it looks more like your OTS_RABBITMQ_USERNAME is not set in config.yml.

Can reproduce if i remove OTS_RABBITMQ_USERNAME from config.yml.

My last commit 5427dd9 fixes this.

[MMolineux]

I mistakenly removed babel.init_app from app.py in a merge. Fixed now.