Add a contrib module to generate type hints for capnp schemas

[elagil]

Since there is no movement in #260, I decided to pick up that pull request and fix remaining CI issues.

Update: This is now a separate application, see #289 (comment)

[lgtm-com]

This pull request introduces 1 alert when merging a8f18ca into e93b045 - view on LGTM.com

new alerts:

• 1 for Unused import

[haata]

I'm still interested in this. Will need to get the CI issues fixed.

[elagil]

@haata I am still working on it, but I decided to rewrite it. Testing the original implementation against test.capnp showed irregularities (incorrect nesting, lists of lists, etc.) and missing type hints (e.g. pycapnp readers and builders, the builder method write(), and something like as_builder() or as_reader()), so lots of fixes were required.

Since my rewrite causes increased complexity and modularization, this can also become a standalone package with its own CLI.

The CLI takes:

• A glob (or multiple) that capture(s) input files
• A similar glob (or multiple) for excluding certain files
• Options for recursive search
• Options for automatic clean up of stub files

Then, it creates type hints for all matching schemas, as well as a *.py file that handles loading the schemas and provides a functional Python module. I will show an example tomorrow.

[elagil]

@haata Here is an example. Please let me know, if you would add some important methods that I might have missed.
The generator can now also handle type imports (also outside of the current directory) from other schemas without issues.

This was generated using the following command line call:
capnp-stub-generator -r -p "ex*.capnp"

Consider this nested schema ex.capnp:

@0x9420d0efc6c9fee8;

using import "ex_imp.capnp".TestImport;

struct TestNestedTypes {
  enum NestedEnum1 {
    foo @0;
    bar @1;
  }

  struct NestedStruct {
    enum NestedEnum2 {
      baz @0;
      qux @1;
      quux @2;
    }

    outerNestedEnum @0 :TestNestedTypes.NestedEnum1 = bar;
    innerNestedEnum @1 :NestedEnum2 = quux;
    listOuterNestedEnum @2 :List(NestedEnum1) = [foo, bar];
    listInnerNestedEnum @3 :List(NestedEnum2) = [quux, qux];
  }

  nestedStruct @0 :NestedStruct;

  outerNestedEnum @1 :NestedEnum1 = bar;
  innerNestedEnum @2 :NestedStruct.NestedEnum2 = quux;
  someListofList @3: List(List(List(NestedEnum1)));
  importedVariable @4: TestImport;
}

alongside this ex_imp.capnp file, from which a type is imported:

@0x9420d0efc6c9fed8;

struct TestImport {
  aVariable @0: Float32;
}

The output of the stub generator is currently this ex_capnp.pyi file:

"""This is an automatically generated stub for `ex.capnp`."""
from __future__ import annotations

from contextlib import contextmanager
from io import BufferedWriter
from typing import Iterator, List, Literal, Union, overload

from .ex_imp_capnp import TestImport

class TestNestedTypes:
    class NestedStruct:
        NestedEnum1 = Literal["foo", "bar"]
        NestedEnum2 = Literal["baz", "qux", "quux"]
        outerNestedEnum: TestNestedTypes.NestedStruct.NestedEnum1
        innerNestedEnum: TestNestedTypes.NestedStruct.NestedEnum2
        listOuterNestedEnum: List[TestNestedTypes.NestedStruct.NestedEnum1]
        listInnerNestedEnum: List[TestNestedTypes.NestedStruct.NestedEnum2]
        @staticmethod
        @contextmanager
        def from_bytes(
            data: bytes, traversal_limit_in_words: Union[int, None] = ..., nesting_limit: Union[int, None] = ...
        ) -> Iterator[TestNestedTypes.NestedStructReader]: ...
        def to_bytes(self) -> bytes: ...
        @staticmethod
        def new_message() -> TestNestedTypes.NestedStructBuilder: ...

    class NestedStructReader(TestNestedTypes.NestedStruct):
        def as_builder(self) -> TestNestedTypes.NestedStructBuilder: ...

    class NestedStructBuilder(TestNestedTypes.NestedStruct):
        def as_reader(self) -> TestNestedTypes.NestedStructReader: ...
        @staticmethod
        def write(file: BufferedWriter) -> None: ...
    nestedStruct: TestNestedTypes.NestedStruct
    outerNestedEnum: TestNestedTypes.NestedStruct.NestedEnum1
    innerNestedEnum: TestNestedTypes.NestedStruct.NestedEnum2
    someListofList: List[List[List[TestNestedTypes.NestedStruct.NestedEnum1]]]
    importedVariable: TestImport
    @overload
    def init(self, name: Literal["nestedStruct"]) -> TestNestedTypes.NestedStruct: ...
    @overload
    def init(self, name: Literal["importedVariable"]) -> TestImport: ...
    @staticmethod
    @contextmanager
    def from_bytes(
        data: bytes, traversal_limit_in_words: Union[int, None] = ..., nesting_limit: Union[int, None] = ...
    ) -> Iterator[TestNestedTypesReader]: ...
    def to_bytes(self) -> bytes: ...
    @staticmethod
    def new_message() -> TestNestedTypesBuilder: ...

class TestNestedTypesReader(TestNestedTypes):
    def as_builder(self) -> TestNestedTypesBuilder: ...

class TestNestedTypesBuilder(TestNestedTypes):
    def as_reader(self) -> TestNestedTypesReader: ...
    @staticmethod
    def write(file: BufferedWriter) -> None: ...

And this ex_capnp.py file, handling the load of the schema by means of pycapnp:

"""This is an automatically generated stub for `ex.capnp`."""
import os

import capnp  # type: ignore

capnp.remove_import_hook()
here = os.path.dirname(os.path.abspath(__file__))
module_file = os.path.abspath(os.path.join(here, "ex.capnp"))
TestNestedTypes = capnp.load(module_file).TestNestedTypes
TestNestedTypesBuilder = TestNestedTypes
TestNestedTypesReader = TestNestedTypes

This is the generated stub ex_imp_capnp.pyi

"""This is an automatically generated stub for `ex_imp.capnp`."""
from __future__ import annotations

from contextlib import contextmanager
from io import BufferedWriter
from typing import Iterator, Union

class TestImport:
    aVariable: float
    @staticmethod
    @contextmanager
    def from_bytes(
        data: bytes, traversal_limit_in_words: Union[int, None] = ..., nesting_limit: Union[int, None] = ...
    ) -> Iterator[TestImportReader]: ...
    def to_bytes(self) -> bytes: ...
    @staticmethod
    def new_message() -> TestImportBuilder: ...

class TestImportReader(TestImport):
    def as_builder(self) -> TestImportBuilder: ...

class TestImportBuilder(TestImport):
    def as_reader(self) -> TestImportReader: ...
    @staticmethod
    def write(file: BufferedWriter) -> None: ...

And the load handler ex_imp_capnp.py

"""This is an automatically generated stub for `ex_imp.capnp`."""
import os

import capnp  # type: ignore

capnp.remove_import_hook()
here = os.path.dirname(os.path.abspath(__file__))
module_file = os.path.abspath(os.path.join(here, "ex_imp.capnp"))
TestImport = capnp.load(module_file).TestImport
TestImportBuilder = TestImport
TestImportReader = TestImport

[brainslush]

Is this PR still maintained? What are the current remaining issues?

[elagil]

@brainslush Yes, it is! I will submit a new version of it soon, as I have been optimizing it in a production environment. A lot of bugs have popped up over time, so the module wasn't really ready for many kinds of schemas.

[b0o]

This would be wonderful to have! It's very tedious working with pycapnp with no editor assistance.

[elagil]

Sorry to keep you waiting, I will try to bring something up as fast as possible

[elagil]

Please check https://gitlab.com/mic_public/tools/python-helpers/capnp-stub-generator

This will probably not be the final place/shape for release, but we have been using it successfully like this.

For now I will close this merge request, because this is an entirely separate tool.

[haata]

Cool! I've added a link in the top-level README. https://github.com/capnproto/pycapnp/tree/ed894304a34ce28254779ea5215afe942a0d5b31#stub-file-generation

Longer term it may make more sense to integrate this directly under https://github.com/orgs/capnproto. But there is no rush for this.

[elagil]

Great, thanks for adding it!

I agree, it would be better to integrate it directly. However, it might need to stay as a standalone package and command-line tool, since it is too complex to be put into just a single script.

[b0o]

This is awesome, thank you for sharing @elagil!

[loloxwg]

[elagil]

@loloxwg Yes, it's actually not published on the public pypi server, only on our company internal server. I will fix that soon.

In the mean time, just clone the repository and install it with pip from the local source.

[vincent-lecrubier-skydio]

You can also let pip clone it:

pip install git+https://gitlab.com/mic_public/tools/python-helpers/capnp-stub-generator.git@main