Native Interface (NI)

[encukou]

Let's add a brand new API layer called the Native Interface (NI) that is:

• performant
• complete (provides all necessary building blocks)
• meant as a base for another layer (C-API, Cython, other language bindings)
  • not ergonomic (for C)
  • can use unwieldy names with version information or argument types
• "ABI" only -- no C macros, inline functions, compiler-dependent types, etc.

Then, let's build a C-API layer on top of that, as one of many possible layers: things like Cython, PyO3, ctypes.pythonapi would act as alternatives to C-API (and reimplement it).

The C-API layer would implement the current C-API (Limited API at first), and would consist of

• thin wrappers around the NI, autogenerated from a description that's easy to reuse for the other layers
  • the description would encode similar info as Argument Clinic: argument/return types of a C wrapper and of the called NI function, and how to convert from one to the other
• optional hand-written wrappers that can reach into the internals for performance
  • these should have the same API as the above, and automatically replace them the above is a stable ABI is not needed

[zooba]

This is the kind of proposal I was thinking my proposal from #4 would merge into well (doesn't replace the whole thing, of course, but that was never my intent).

Incidentally, the building blocks are really quite minimal:

• import
• convert from primitive types
• get attribute
• call

Everything else can be implemented on top of these. We should definitely provide a lot of shortcuts, but these are the most core pieces.

I really think we should avoid our type-specific implementations becoming public API again. If you want to add two values, call the general "add" function and let it figure out that PyLongObject_Add is the right implementation.

[encukou]

performant

To meet this requirement, the NI layer should use concrete types (e.g. PyDictObject* rather than PyObject*).
Run-time type checking belongs in the layer above. (Of course, for languages more strongly typed than C, the compiler-can do many of the type checks instead. And perhaps that includes C99/C++ for some cases.)

See #3 and capi-workgroup/api-evolution#29

[pitrou]

You should first explain why you call it "Native Interface" and what that implies.
If that implies CPython-only, then I don't understand how this would be making any progress.
And if that implies that writing CPython-only code is required for performance, then this is entirely misguided IMHO.