The role of Data Engineer exists as we know it because of a lack of adequate tooling for Data Scientists — In most companies, Data Engineers support the Data Scientists in various ways. Often this means translating or productionizing the notebooks and scripts that a Data Scientist has written. …

We need to start placing a higher value on data scientists’ time than we do on machine time — While data science tools are being optimized to perform well on microbenchmarks, they are becoming more and more difficult to use. Is the benchmark performance worth the human time cost it takes to get there? …

Comparing Modin with Dask, Ray, Vaex, and RAPIDS — Recently, a blog post was written that compared a variety of tools in a set of head to heads. I wanted to take the opportunity to talk about our vision with Modin and where we’d like to take the field of data science. Modin (https://github.com/modin-project/modin) takes a different view of…

Dataframes are losing their statistical computing and machine learning roots — Dataframes emerged from a specific need, but because so many diverse systems now call themselves dataframes, the term is on the verge of meaning nothing. In an effort to preserve the dataframe, we formalized the definition based on the original data model in our recent preprint[2]. Before we get into…

by Ion Stoica and Devin Petersohn — Serverless computing is rapidly gaining in popularity due to its ease of programmability and management. Many see it as the next general purpose computing platform for the cloud [4]. However, while existing serverless platforms have been successful in supporting several popular applications such as event processing and simple ETL, they…