Python runtimes

To be able to implement backward compatibility, the plan is to provide multiple Python runtimes, at least two. An “old” runtime with maximum backward compatibility, and one “new” runtime which includes experimental new changes but requires using the newer C API.

Welcome into the bright world of multiple new cool and compatible Python runtimes!

Regular Python: /usr/bin/python3

• Python compiled in release mode

• This runtime still provides Py_INCREF() macro: modify PyObject.ob_refcnt at the ABI level.

• Should be fully compatible with old C API (Python 3.7 C API)

• Should be fully compatible with Python 3.7 stable ABI (it may become incompatible with the Python 3.7 full ABI).

Compared to Python 3.7 regular runtime, this runtime no longer check its arguments for performance reasons. The debug runtime should now be preferred to develop C extensions and to run tests.

Example of Python 3.7 code:

int
PyList_Append(PyObject *op, PyObject *newitem)
{
    if (PyList_Check(op) && (newitem != NULL))
        return app1((PyListObject *)op, newitem);
    PyErr_BadInternalCall();
    return -1;
}

The if (PyList_Check(op) && (newitem != NULL)) belongs to the debug runtime and should be removed from the regular Python.

Debug runtime: /usr/bin/python3-dbg

• Compatible with Python 3.7 C API.

• Compatible with regular runtime 3.8 ABI, but not compatible with regular runtime 3.7 ABI.

• CPython compiled with ./configure --with-pydebug

• Provide sys.gettotalrefcount() which allows to check for reference leaks.

• C function calls check most arguments: check type, value range, etc.

• Runtime compiled with C assertion: crash (kill itself with SIGABRT signal) if a C assertion fails (assert(...);).

• Use the debug hooks on memory allocators by default, as PYTHONDEBUG=debug environment variable: detect memory under- and overflow and misuse of memory allocators.

• Compiled without compiler optimizations (-Og or even -O0) to be usable with a debugger like gdb: python-gdb.py should work perfectly. However, the regular runtime is unusable with gdb since most variables and function arguments are stored in registers, and so gdb fails with the “<optimized out>” message.

For example, the debug runtime can check that the GIL is held by the caller.

See also

• Remove debug checks

• Rejected idea: Check index in PyTuple_GET_ITEM/PyTuple_SET_ITEM in debug mode and PyTuple_SET_ITEM could check bounds in debug mode. Issues:

  • Serhiy Storchaka: “I think we can break this only after adding public API for accessing internal storage of a tuple: PyTuple_ITEMS().”

  • Stefan Krah: “I’m using &PyTuple_GET_ITEM(args, 0), so Serhiy’s concern is not theoretical.”

  • Stefan Behnel: “If this is really just about debugging, then I would suggest to not break existing code at all.”

New experimental runtime: python3-exp

• Loading a C extension compiled with Python 3.7 must fail.

• Loading a C extension compiled with the Python C API 3.8 in the backward compatible mode must fail.

• Only C extensions compiled with the new Python C API 3.8 can be loaded. You have to opt-in for this runtime.

• Not compatible with Python 3.7 API: PyDict_GetItem() is gone, PyDict_GetItemRef() must be used instead.

• Not compatible with Python 3.8 ABI: using Py_INCREF() macro uses PyObject.ob_refcnt at the ABI level, whereas this field must not be access at the ABI level.

• Py_GC header and PyObject structure can be very different from the one used by the regular and debug runtimes.

Technically, this experimental runtime can be a opt-in compilation mode of the upstream CPython code base.

See Optimization ideas.

Other Python implementations

Last 10 years, CPython has been forked multiple times to attempt different CPython enhancements:

• Unladen Swallow: add a JIT compiler based on LLVM

• Pyston: add a JIT compiler based on LLVM (CPython 2.7 fork)

• Pyjion: add a JIT compiler based on Microsoft CLR

• Gilectomy: remove the Global Interpreter Lock nicknamed “GIL”

Sadly, none is this project has been merged back into CPython. Unladen Swallow lost its funding from Google, Pyston lost its funding from Dropbox, Pyjion is developed in the limited spare time of two Microsoft employees.

Other Python implementations written from scratch:

• PyPy

• RustPython

• MicroPython

• Jython

• IronPython

Since the C API will be smaller and the stable ABI will become more usable, you can imagine that Python implementations other than CPython will be able to more easily have a full and up-to-date support of the latest full C API.

Put your CPython fork here!

Since a stable ABI have been designed, if all your C extensions have opt-in for the new C API: you are now allowed to fork CPython and experiment your own flavor CPython. Do whatever you want: C extensions only calls your runtime through function calls.

See Optimization ideas.