Welcome to the data.table ecosystem project!

An NSF-POSE funded venture.

announcements
grant
Author

Toby Hocking

Published

October 15, 2023

Hi! My name is Toby Dylan Hocking, and I have been using R since 2003, which means 20 years, can you believe it?

Toby D. Hocking

Source: wikimedia.

Toby still using R in 50 years (artist rendering)

I work as an Assistant Professor of Computer Science, and my research expertise is machine learning, the modern branch of artificial intelligence which uses big data. R is an important tool in my machine learning work, and in the work of many people in academia/industry/government, because it provides so many useful functions for handling big data.

The {data.table} package

Since 2015, I have been using the R package data.table to do large parts of data processing before and after running machine learning algorithms - to get the raw data into the right format for the algorithm, and also to get the results in the right format for visualization/interpretation.

The {data.table} hex sticker

data.table is highly valued for its long-term stability and its lightning-fast speed in large data calculations.

Ecosystem expansion

With these use cases in mind, I proposed a project “Expanding the data.table ecosystem for efficient big data manipulation in R,” and I am excited to announce that it has been funded by the National Science Foundation’s “Pathways to Enable Open Source Ecosystems” (POSE) grant, for work between September 2023 and August 2025.

Our project attempts to address three issues with the current state of the data.table project:

  1. Informal governance

The data.table package was originally created by Matt Dowle in 2008. His brilliant use of efficient algorithms and C implementations brought the world a package that has stood the test of time, and is now one of the most-used R packages available.

However, the growth of this incredible package will require more leaders than Dowle himself to help build, review, test, and organize new contributions.

Thus, one goal of this grant is to bring together data.table developers and contributors to propose a new governance structure for the package’s source code.

  1. Limited centralized testing infrastructure

Since the primary draw of data.table is the speed of its algorithmic implementations, adding new functionality to the package is not simple. In particular, new elements must be heavily tested to ensure that they do not interfere with the core computations.

In this grant, we plan to develop software to automate the testing of new package contributions, to smooth the growth process. This part of the project includes a centralized reverse dependency checking system, new benchmarks comparing data.table with other systems such as polars and arrow, and new performance testing software.

  1. Limited documentation and outreach

To encourage more people to learn and adopt data.table, we will be massively expanding the number of tutorials, documentations, and guides for how to use the package effectively. Part of this will be translation projects so that data.table will be more accessible in foreign languages. This grant will also include travel awards, to support selected speakers to travel to conferences and share data.table updates and usage.

Interested in contributing a tutorial/vignette or blog post? Email r.data.table@gmail.com!

How you can get involved

Interested in helping grow the data.table ecosystem? There are so many ways to get involved!

A group of friendly sea lions.  Source: Department of the Interior.

Sea lions say “R!” “R!” “R!”
  1. Follow us for updates on social media:
  1. Subscribe to this blog, The Raft

  2. Take the Community Survey to weigh in on next steps.

  3. Participate in the deep discussions on GitHub:

  1. Email r.data.table@gmail.com to:
  • Be added to the community Slack.

  • Propose a guest blog for The Raft.

  • Ask questions, make suggestions, or volunteer your expertise.

  1. Apply for the upcoming Travel Grants and Translation Projects - watch this blog for more information!

Comparing data.table reshape to duckdb and polars

tips
tutorials
developer
benchmarks

Seal of Approval: mlr3

seal of approval
application package

Seal of Approval: collapse

seal of approval
partner package

Newly awarded translation projects

announcements
grant
translation

Seal of Approval: dtplyr

seal of approval
bridge package

Seal of Approval: nc

seal of approval
extension package

Seal of Approval: tidyfast

seal of approval
bridge package

Announcement: The ‘Seal of Approval’

announcements
grant
community
seal of approval

Announcement: Paola Corrales, data.table Ambassador

announcements
grant
ambassadors
travel

Testing infrastructure for data.table

grant
testing
developer

Results of the 2023 survey

community
guest post
governance

The Benefits of data.table Syntax

tips
tutorials
documentation

Piping data.tables

tips
tutorials
documentation
guest post

Announcement: Jan Gorecki, data.table Ambassador

announcements
grant
ambassadors

Announcement: The data.table Ambassadors Travel Grant

announcements
grant
funding opportunity

Announcement: data.table translation projects

announcements
grant
funding opportunity
No matching items