Machine Learning is becoming critical infrastructure for many organizations. At the same time, the current tensor and dataframe infrastructure at the foundation of machine learning is fragmented. This makes it difficult for a robust ecosystem to develop around the state of the art.

Even as Python has become a common language to express workflows, the foundational data-structures used in Python such as n-dimensional arrays (tensors) and dataframes have become more divided. For example, tensors are fragmented between Tensorflow, PyTorch, NumPy, CuPy, MXNet, Xarray, Dask, and others. Dataframes are fragmented between Pandas, PySpark, Arrow, RAPIDS, TuriCreate, Vaex, Modin, Dask, Ray, and more.

The differences between these projects provide marginal benefit but also cause inconsistencies and incompatibilities that require additional effort, limit participation, prevent re-use, and slow progress. Innovation and differentiation are most valuable when created using standard building blocks that ensure consistency and compatibility.

Quansight is forming a consortium of leading companies to establish a standard for both tensors and dataframes which are the critical building blocks for machine-learning workflows. The standard will include APIs and specifications for both tensors and dataframes and will be coordinated with the wider open-source community in its creation and later dissemination. Consortium members will be able to participate in defining the standard including a basic reference implementation. This drafting phase will be semi-public but key parts of the process will be restricted to Consortium members until a draft of the Community Enhancement Proposal is produced. This draft will then be further refined in conjunction with the broader NumPy and Pandas communities.

Quansight is well positioned to establish this Consortium because of its expertise with the technologies involved, understanding of the open-source process, strong relationships with many of the relevant projects, and because Quansight does not have commercial interest in the particulars of the standard.

We are looking for around 10 members to join the Consortium. Members will have the option to privately contribute to writing the standard. Membership will not only allow the ability to significantly influence the standard it will also provide members with pre-publication insights into the details of the standard. The funding obligation to participate is $50,000 per member in order to support the development of the standard and reference implementation which is expected to take about 6 months with a team of 3 engineers.

Definition of the standard has commenced and there is a limited window to participate and influence the standard. Several organizations have already committed to the consortium including Quansight, OmniSci, Anaconda, Intel, and other participants who are not named here. We expect to close membership in the Consortium on April 30th, 2020.