Bad C API¶
The Python C API is just too big. For performance reasons, CPython calls
internally directly the implementation of a function instead of using the
abstract API. For example,
PyDict_GetItem() is preferred over
PyObject_GetItem(). Inside, CPython, such optimization is fine. But
exposing so many functions is an issue: CPython has to keep backward
compatibility, PyPy has to implement all these functions, etc. Third party
C extensions should call abstract functions like
Problem caused by borrowed references¶
A borrowed reference is a pointer which doesn’t “hold” a reference. If the object is destroyed, the borrowed reference becomes a dangling pointer: point to freed memory which might be reused by a new object. Borrowed references can lead to bugs and crashes when misused. Recent example of CPython bug: bpo-25750: crash in type_getattro().
Borrowed references are a problem whenever there is no reference to borrow: they assume that a referenced object already exists (and thus have a positive refcount), so that it is just borrowed.
Tagged pointers are an example of this: since there is
PyObject* to represent the integer, it cannot easily be
PyPy has a similar problem with list strategies: if there is a list containing only integers, it is stored as a compact C array of longs, and the W_IntObject is only created when an item is accessed (most of the time the W_IntObject is optimized away by the JIT, but this is another story).
But for cpyext, this is a problem:
PyList_GetItem() returns a borrowed
reference, but there is no any concrete
PyObject* to return! The current
cpyext solution is very bad: basically, the first time
is called, the whole list is converted to a list of
PyObject*, just to
have something to return: see cpyext get_list_storage().
See also the Specialized list for small integers optimization: same optimization applied to CPython. This optimization is incompatible with borrowed references, since the runtime cannot guess when the temporary object should be destroyed.
PyList_GetItem() returned a strong reference, the
just be allocated on the fly and destroy it when the user decref it. Basically,
by putting borrowed references in the API, we are fixing in advance the data
structure to use!
C API using borrowed references¶
CPython 3.7 has many functions and macros which return or use borrowed
references. For example,
PyTuple_GetItem() returns a borrowed reference,
PyTuple_SetItem() stores a borrowed reference (store an item into a
tuple without increasing the reference counter).
Doc/data/refcounts.dat (file is edited manually) which
documents how functions handle reference count.
See also functions steal references.
PyEval_GetFuncName(): return the internal
const char*inside a
- borrowed reference to a function
PyWeakref_GetObject(): see https://mail.python.org/pipermail/python-dev/2016-October/146604.html
Py_XSETREF(): the caller has to manually increment the reference counter of the new value
Py_TYPE() corner case¶
Py_TYPE() returns a borrowed reference to a
In practice, for heap types, an instance holds already a strong reference
to the type in
PyObject.ob_type. For static types, instances use a borrowed
reference, but static types are never destroyed.
Hugh Fisher summarized:
It don’t think it is worth forcing every C extension module to be rewritten, and incur a performance hit, to eliminate a rare bug from badly written code.
- [Python-Dev] bpo-34595: How to format a type name? (Sept 2018)
- capi-sig: Open questions about borrowed reference. (Sept 2018).
See also Opaque PyObject structure.
PyEval_CallObjectWithKeywords(): almost duplicate
PyObject_Call(), except that args (tuple of positional arguments) can be
PyObject_CallObject(): almost duplicate
PyObject_Call(), except that args (tuple of positional arguments) can be
Only keep abstract functions?¶
Good: abstract functions. Examples:
Bad? implementations for concrete types. Examples:
Implementations for concrete types don’t have to be part of the C API.
Moreover, using directly them introduce bugs when the caller pass a subtype.
For example, PyDict_GetItem() must not be used on a dict subtype, since
__getitem__() be be overridden for good reasons.
Functions kept for backward compatibility¶
PyEval_CallFunction(): a comment says “PyEval_CallFunction is exact copy of PyObject_CallFunction. This function is kept for backward compatibility.”
PyEval_CallMethod(): a comment says “PyEval_CallMethod is exact copy of PyObject_CallMethod. This function is kept for backward compatibility.”
No public C functions if it can’t be done in Python¶
There should not be C APIs that do something that you can’t do in Python.
Example: the C buffer protocol, the Python
memoryview type only expose a
Array of pointers to Python objects (
PyObject** must not be exposed:
has to go.
PyDict_GetItem() API is one of the most commonly called function but
it has multiple flaws:
- it returns a borrowed reference
- it ignores any kind of error: it calls
The dictionary lookup is surrounded by
PyErr_Restore() to ignore any exception.
If hash(key) raises an exception, it clears the exception and just returns
Enjoy the comment from the C code:
/* Note that, for historical reasons, PyDict_GetItem() suppresses all errors * that may occur (originally dicts supported only string keys, and exceptions * weren't possible). So, while the original intent was that a NULL return * meant the key wasn't present, in reality it can mean that, or that an error * (suppressed) occurred while computing the key's hash, or that some error * (suppressed) occurred when comparing keys in the dict's internal probe * sequence. A nasty example of the latter is when a Python-coded comparison * function hits a stack-depth error, which can cause this to return NULL * even if the key is present. */
Functions implemented with
PyDict_GetItemWithError() which doesn’t ignore all errors: it only
KeyError if the key doesn’t exist. Sadly, the function still
returns a borrowed references.
Don’t leak the structures like
PyTupleObject to not
access directly fields, to not use fixed offset at the ABI level. Replace
macros with functions calls. PyPy already does this in its C API (
Example of macros:
PyCell_GET(): access directly
PyList_GET_ITEM(): access directly
PyMethod_GET_FUNCTION(): access directly
PyMethod_GET_SELF(): access directly
PyTuple_GET_ITEM(): access directly
PyWeakref_GET_OBJECT(): access directly
PyType_Ready() and setting directly PyTypeObject fields¶
PyTypeObjectstructure should become opaque
PyType_Ready()should be removed
See Implement a PyTypeObject in C for the rationale.
PyLong_AsUnsignedLongMask() ignores integer overflow.
k format of
Functions stealing references¶
PyErr_Restore(): type, value, traceback
PySet_Discard(): key, no effect if key not found
Py_XDECREF(): o, if o is not NULL
PyModule_AddObject(): o on success, no change on error!
See also borrowed references.
Should we do something for reference counting, Py_INCREF and Py_DECREF? Replace them with function calls at least?
PyObject_CallFunction() API: bpo-28977. Fix the API or document it?
Deprecate finalizer API: PyTypeObject.tp_finalize of PEP 442. Too specific to the CPython
garbage collector? Destructors (
__del__()) are not deterministic in PyPy
because of their garbage collector: context manager must be used
with file:), or resources must be explicitly released
Compact Unicode API¶
Deprecate Unicode API introduced by the PEP 393, compact strings, like
The family of
PyArg_Parse*() functions like
a wide range of argument formats, but some of them leak implementation details:
O: returns a borrowed reference
s: returns a pointer to internal storage
Is it an issue? Should we do something?
For internal use only¶
Public but not documented and not part of Python.h:
These functions should be made really private and removed from the C API.