Support running `debugpy` in web

[karthiknadig]

Connect mode might be easier to solve than listen mode. when i tried recently with pyodide, this is what I got:

Connect mode error:

>>> debugpy.connect(("localhost",3))
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/lib/python3.10/site-packages/debugpy/common/compat.py", line 208, in kwonly_f
    return f(*args, **kwargs)
  File "/lib/python3.10/site-packages/debugpy/__init__.py", line 135, in connect
    return api.connect(address, access_token=access_token)
  File "/lib/python3.10/site-packages/debugpy/server/api.py", line 143, in debug
    log.reraise_exception("{0}() failed:", func.__name__, level="info")
  File "/lib/python3.10/site-packages/debugpy/server/api.py", line 141, in debug
    return func(address, settrace_kwargs, **kwargs)
  File "/lib/python3.10/site-packages/debugpy/server/api.py", line 276, in connect
    _settrace(host=host, port=port, client_access_token=access_token, **settrace_kwargs)
  File "/lib/python3.10/site-packages/debugpy/server/api.py", line 47, in _settrace
    return pydevd.settrace(*args, **kwargs)
  File "/lib/python3.10/site-packages/debugpy/_vendored/pydevd/pydevd.py", line 2771, in settrace
    _locked_settrace(
  File "/lib/python3.10/site-packages/debugpy/_vendored/pydevd/pydevd.py", line 2922, in _locked_settrace
    py_db.enable_tracing(py_db.trace_dispatch, apply_to_all_threads=True)
  File "/lib/python3.10/site-packages/debugpy/_vendored/pydevd/pydevd.py", line 1086, in enable_tracing
    pydevd_tracing.SetTrace(thread_trace_func)
  File "/lib/python3.10/site-packages/debugpy/_vendored/pydevd/pydevd_tracing.py", line 77, in SetTrace
    if set_trace_to_threads(tracing_func, thread_idents=[thread.get_ident()], create_dummy_thread=False) == 0:
  File "/lib/python3.10/site-packages/debugpy/_vendored/pydevd/pydevd_tracing.py", line 344, in set_trace_to_threads
    start_new_thread(increase_tracing_count, ())
RuntimeError: can't start new thread

Listen mode error:

>>> debugpy.listen(5678)
E+00000.013: Can't listen for adapter endpoints:
             
             Traceback (most recent call last):
               File "/lib/python3.10/site-packages/debugpy/common/sockets.py", line 24, in create_server
                 server = _new_sock()
               File "/lib/python3.10/site-packages/debugpy/common/sockets.py", line 45, in _new_sock
                 sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
             OSError: [Errno 50] Protocol not available
             
             During handling of the above exception, another exception occurred:
             
             Traceback (most recent call last):
               File "/lib/python3.10/site-packages/debugpy/server/api.py", line 158, in listen
                 endpoints_listener = sockets.create_server("127.0.0.1", 0, timeout=10)
               File "/lib/python3.10/site-packages/debugpy/common/sockets.py", line 30, in create_server
                 server.close()
             UnboundLocalError: local variable 'server' referenced before assignment
             
             Stack where logged:
               File "/lib/python3.10/site-packages/pyodide/webloop.py", line 151, in run_handle
                 h._run()
               File "/lib/python3.10/asyncio/events.py", line 80, in _run
                 self._context.run(self._callback, *self._args)
               File "/lib/python3.10/asyncio/tasks.py", line 315, in __wakeup
                 self.__step()
               File "/lib/python3.10/asyncio/tasks.py", line 232, in __step
                 result = coro.send(None)
               File "/lib/python3.10/site-packages/pyodide/console.py", line 474, in runcode
                 return await super().runcode(source, code)
               File "/lib/python3.10/site-packages/pyodide/console.py", line 362, in runcode
                 return await code.run_async(self.globals)
               File "/lib/python3.10/site-packages/_pyodide/_base.py", line 351, in run_async
                 coroutine = eval(self.code, globals, locals)
               File "<console>", line 1, in <module>
               File "/lib/python3.10/site-packages/debugpy/__init__.py", line 113, in listen
                 return api.listen(address)
               File "/lib/python3.10/site-packages/debugpy/server/api.py", line 141, in debug
                 return func(address, settrace_kwargs, **kwargs)
               File "/lib/python3.10/site-packages/debugpy/server/api.py", line 160, in listen
                 log.swallow_exception("Can't listen for adapter endpoints:")
               File "/lib/python3.10/site-packages/debugpy/common/log.py", line 218, in swallow_exception
                 _exception(format_string, *args, **kwargs)
Traceback (most recent call last):
  File "/lib/python3.10/site-packages/debugpy/common/sockets.py", line 24, in create_server
    server = _new_sock()
  File "/lib/python3.10/site-packages/debugpy/common/sockets.py", line 45, in _new_sock
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
OSError: [Errno 50] Protocol not available
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/lib/python3.10/site-packages/debugpy/server/api.py", line 158, in listen
    endpoints_listener = sockets.create_server("127.0.0.1", 0, timeout=10)
  File "/lib/python3.10/site-packages/debugpy/common/sockets.py", line 30, in create_server
    server.close()
UnboundLocalError: local variable 'server' referenced before assignment
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/lib/python3.10/site-packages/debugpy/__init__.py", line 113, in listen
    return api.listen(address)
  File "/lib/python3.10/site-packages/debugpy/server/api.py", line 143, in debug
    log.reraise_exception("{0}() failed:", func.__name__, level="info")
  File "/lib/python3.10/site-packages/debugpy/server/api.py", line 141, in debug
    return func(address, settrace_kwargs, **kwargs)
  File "/lib/python3.10/site-packages/debugpy/server/api.py", line 161, in listen
    raise RuntimeError("can't listen for adapter endpoints: " + str(exc))
RuntimeError: can't listen for adapter endpoints: local variable 'server' referenced before assignment

[int19h]

For "listen", we'd have to implement websockets as a transport, but then VSCode would also need to be able to talk to that somehow - either as a standard transport along with TCP, or the Python extension can supply its own wrapper pipe.

But the bigger problem here is that Pyodide, apparently, just doesn't support threading yet, because the corresponding wasm feature is still in the experimental / proposal stage. It also doesn't support subprocess (module is importable, but actually creating new processes fails). Both are deal-breakers - both pydevd and debugpy use multiple threads by architectural design, and debugpy uses multiple processes. We'd have to significantly change the design to fit the sandbox.

[fabioz]

I think that we could potentially do things without subprocesses (we actually have a request for that already: #532), but not having threads is really much harder.

We'd probably need to change to be able to use an async loop in the debugger instead of plain threads, but if the user code that's running isn't async either then I don't see how to make some features available at all (for instance, changing breakpoints while the user code is running) as we'd only be able to control things when it starts or when a breakpoint is hit.

I think it should be doable, but given the environment constraints, this is a really big feature...

[int19h]

Py_AddPendingCall could be used as a form of cooperative shared-memory multitasking. In principle, it would even work with blocking native calls - if a request comes in while there is a call in progress, it'd be handled only after that's complete, but it's "good enough" for breakpoints. However, the problem then would be that the DAP client might time out waiting for the request to be acknowledged (or even read from the socket). Some tweaks might also be necessary on CPython side to allow the caller to request "run this as soon as you can", rather than eventually.

[rchiodo]

Three basic ideas:

• Py_AddPendingCall to listen to messages and reimplement pydevd to not use threads (about as good as PDB)
• Follow the same pattern mixed mode debugger in Visual Studio does and build off of the C++ debugger in Chrome (experimental, same bucket as using threads)
• Get WASM threads working and update debugpy/pydevd if necessary

[rchiodo]

Is the C++ extension also looking into this? Does their C++ debugger work in WASM? If so we could wrap their debugger to implement the second option.

[rchiodo]

Looks like https://web.dev/wasm-threads/ is actually on by default.

[rchiodo]

Another question, CPython may have all of the threading stubbed out. Need to double check.

[rchiodo]

Safari has support as well:
https://webkit.org/blog/12140/new-webkit-features-in-safari-15-2/

[judej]

This is blocked by the limitation of CPython available on WASM.

[K0IN]

2 years later is has anything changed?

[rchiodo]

There's this:
https://github.com/microsoft/vscode-python-web-wasm