A new C API for Python¶
Design goals¶
Reducing the number of backward incompatible changes to be able to use the new C API on the maximum number of existing C extensions which use directly the C API.
Hide most CPython implementation details. The exact list has to be written. One required change is to replace macros with functions calls. The option question remains if it will be possible to replace
Py_INCREF()
andPy_DECREF()
with function calls without killing performances.Reduce the size of the C API to reduce the maintenance burden of Python implementations other than CPython: remove functions.
Reduce the size of the ABI, especially export less symbols.
The backward compatibility issue is partially solved by keeping the existing old C API available as an opt-in option: see the Regular runtime.
Remove functions and macros¶
Removed functions and macros because they use borrowed references:
Py_TYPE()
PyTuple_GET_ITEM()
PyTuple_GetItem()
PyTuple_SetItem()
PyTuple_SET_ITEM()
New functions¶
XXX the following functions have been added to the current WORK-IN-PROGRESS
implementation of the new C API. Maybe they will go away later. It’s just a
small step to move away from borrowed references. Maybe existing
PyObject_GetItem()
and PyObject_SetItem()
are already good enough.
PyTuple_GetItemRef()
: similar toPyTuple_GetItem()
but returns a strong reference, rather than a borrowed referencePyList_GetItemRef()
: similar toPyList_GetItem()
but returns a strong reference, rather than a borrowed referencePyTuple_SetItemRef()
: similar toPyTuple_SetItem()
but uses a strong reference on the itemPySequence_Fast_GetItemRef()
PyStructSequence_SetItemRef()
XXX private functions:
_Py_SET_TYPE()
: see Implement a PyTypeObject in C_Py_SET_SIZE()
If we decide that Py_TYPE() should go away, 3 more functions/features are needed:
Py_GetType()
: similar toPy_TYPE()
but returns a strong referencePy_TYPE_IS(ob, type)
: equivalent toPy_TYPE(ob) == type
%T
format forPyUnicode_FromFormat()
Non-goal¶
Cython and cffi must be preferred to write new C extensions: there is no intent to replace Cython. Moreover, there is no intent to make Cython development harder. Cython will still be able to access directly the full C API which includes implementation details and low-level “private” APIs.
Hide implementation details¶
See also Bad C API.
What are implementation details?¶
“Implementation details” is not well specified at this point, but maybe hiding implementation can be done incrementally.
The PEP 384 “Defining a Stable ABI” is a very good stable to find the borders between the public C API and implementation details: see Stable ABI.
Replace macros with function calls¶
Replacing macros with functions calls is one part of the practical solution. For example:
#define PyList_GET_ITEM(op, i) ((PyListObject *)op)->ob_item[i]
would become:
#define PyList_GET_ITEM(op, i) PyList_GetItem(op, i)
or maybe even:
PyObject* PyList_GET_ITEM(PyObject *op, PyObject *i) { return PyList_GetItem(op, i); }
Adding a new PyList_GET_ITEM()
function would make the ABI larger,
whereas the ABI should become smaller.
This change remains backward compatible in term of C API. Moreover, using function calls helps to make C extension backward compatible at the ABI level as well.
Problem: it’s no longer possible to use Py_TYPE()
and Py_SIZE()
as l-value:
Py_SIZE(obj) = size;
Py_TYPE(obj) = type;
XXX in the current implementation, _Py_SET_SIZE()
and _Py_SET_TYPE()
macros have been added for such use case. For the type, see also
Implement a PyTypeObject in C.
Py_INCREF()¶
The open question remains if it will be possible to replace Py_INCREF()
and
Py_DECREF()
with function calls without killing performances.
Hide C structures¶
The most backward incompatible change is to hide fields of C structures, up to
PyObject. To final goal will be able to hide PyObject.ob_refcnt
from the
public C API.
C extensions must be modified to use functions to access fields.
In the worst case, there will be no way to access to hidden field from the public C API. For these users, the only option will be to stick at the old C API which remains backward compatible and still expose implementation details like C structure fields.