Abstract:

Python has had enormous success in part due to its ease of extensibility and interoperability with other systems. In particular, this has led to huge ecosystems of code that heavily leverage the existing C-API for Python. One such ecosystem is the NumFOCUS ecosystem that includes NumPy, Pandas, SciPy, Scikit-Learn, Numba, Dask, and Jupyter.  This ecosystem has propelled Python to become the de facto language for data-science and machine-learning.

The downside of this success is that the particular dependency on the C-API of these extensions has also made this ecosystem an anchor to the further developing of the C-Python runtime as well as other emerging run-times of Python such as PyPy, MicroPython, IronPython, Jython, and RustPython.

Many of the core Python developers and the steering committee members recognize this problem and are eager to see something done about it. Something can be done, but it will take a concerted technical and social effort lasting several years and budget from multiple stake-holders. We will create an embedded domain specific language (DSL) using the Python language itself along with the typing module and specialized objects as necessary to allow nearly all the existing extensions to be written using this DSL. We will then port several key libraries such as NumPy, SciPy, and Pandas to use this DSL. We will also provide a runtime so these extensions can work on PyPy, C-Python, and RustPython.

Timeline:

The initial MVP will be completed in 6 months and then a version of NumPy will be ported in the next 12 months. An initial run-time for PyPy and Jython will be completed in the next 6 months. The remaining 3.5 years of the funded project will be improving the language by porting existing modules to use the new DSL.