Reorganize Python “runtime”

Starter point: PEP 554 – Multiple Interpreters in the Stdlib.

Goal

The goal is to support running multiple Python interpreters in parallel with one lock per interpreter (no more “Global Interpreter Lock”, but one “Interpreter Lock” per interpreter). An interpreter would only be able to run one Python thread holding the interpreter lock at the same time, but multiple Python threads which released the interpreter lock (ex: to call a system call like read()) can be run in parallel.

What do we need?

To maximize performances, shared states between interpreters must be minimized. Each share state must be carefully protected by a lock, which prevent to run code in parallel.

Current state of the code (2019-05-24)

During Python 3.7 and 3.8 dev cycle, Eric Snow moved scattered core global variables into a _PyRuntimeState structure which has a single global and shared instance: _PyRuntime.

Most functions access directly to _PyRuntime, directly or indirectly:

  • PyThreadState *tstate = _PyThreadState_GET(); access implicitly _PyRuntime.
  • PyThreadState *tstate = _PyRuntimeState_GetThreadState(&_PyRuntime); gets access explicitly _PyRuntime. Get runtime->gilstate.tstate_current.

_PyRuntimeState fields:

  • ceval
  • exitfuncs, nexitfuncs
  • finalizing
  • gc
  • gilstate
  • interpreters
  • main_thread
  • open_code_hook, open_code_userdata, audit_hook_head
  • pre_initialized, core_initialized, initialized
  • preconfig
  • xidregistry

TODO

  • Move _PyRuntimeState.gilstate to PyInterpreterState:
    • Remove _PyRuntimeState_GetThreadState()
    • Update _PyThreadState_GET()
  • Move most _PyRuntimeState fields into PyInterpreterState
  • Pass the “context” to private C functions: the context can be _PyRuntime, a field of _PyRuntime, the Python thread state (tstate), etc.

Out of the scope

  • Functions of public C API must not be modified at this stage to add new “context” parameters. Only the internal C API can be modified.

Roots

  • Get the current Python thread: _PyRuntimeState_GetThreadState(&_PyRuntime). WIP: gilstate must move to PyInterpreterState
  • Get the current interpreter: tstate->interp.

Status (2019-05-24)

  • PyInterpreterState moved to the internal C API
  • _PyRuntimeState structure and _PyRuntime variable created