Design a new better C API for Python¶
Subtitle: “Make the stable API usable”.
PyPy and Gilectomy projects are using different approaches to optimize Python, but their performance are limited by the current C API on C extensions. The C API should be “fixed” to unlock raw performances!
To be able to introduce backward incompatible changes to the C API without breaking too many C extensions, this project proposes two things:
- design a helper layer providing removed functions;
- a new Python runtime which is only usable with C extensions compiled with the new stricter and smaller C API (and the new stable ABI) for Python 3.8 and newer, whereas the existing “regular python” becomes the “regular runtime” which provides maximum backward compatibility with Python 3.7 and older.
The current C API has multiple issues:
- The Python lifecycle is shorter than the lifecycle of some operating systems: how to get the latest Python on an “old” but stable operating system?
- Python debug build is currently mostly unusable in practice, making development of C extension harder, especially debugging.
Existing C extensions will still be supported and will not have to be modified. The old C API is not deprecated and there is no plan penalize users of the old C API.
Table of Contents¶
- Statistics on the Python C API
- Status of the new Python C API
- Rationale to change the Python C API
- Bad C API
- Borrowed references
- Duplicated functions
- Only keep abstract functions?
- Functions kept for backward compatibility
- No public C functions if it can’t be done in Python
- Array of pointers to Python objects (
- C structures
- PyType_Ready() and setting directly PyTypeObject fields
- Integer overflow
- Functions stealing references
- Open questions
- For internal use only
- A new C API for Python
- Python runtimes
- Old C API
- Implement a PyTypeObject in C
- Optimization ideas
- Backward compatibility
- Supporting multiple Python versions per operating system release
- C API calling conventions
- Python stable ABI
- Consumers of the Python C API
- PyPy cpyext module
- Remove the C API
- Split Include/ directory
- Deprecate old APIs
- Python Intermediate representation (IR)
- Reorganize Python “runtime”
- Opaque PyObject structure
- Implicit and explicit resource management
- Misc notes