A Unified System for Data Analytics and In Situ Query Processing (2102.09295v2)

Abstract: In today's world data is being generated at a high rate due to which it has become inevitable to analyze and quickly get results from this data. Most of the relational databases primarily support SQL querying with a limited support for complex data analysis. Due to this reason, data scientists have no other option, but to use a different system for complex data analysis. Due to this, data science frameworks are in huge demand. But to use such a framework, all the data needs to be loaded into it. This requires significant data movement across multiple systems, which can be expensive. We believe that it has become the need of the hour to come up with a single system which can perform both data analysis tasks and SQL querying. This will save the data scientists from the expensive data transfer operation across systems. In our work, we present DaskDB, a system built over the Python's Dask framework, which is a scalable data science system having support for both data analytics and in situ SQL query processing over heterogeneous data sources. DaskDB supports invoking any Python APIs as User-Defined Functions (UDF) over SQL queries. So, it can be easily integrated with most existing Python data science applications, without modifying the existing code. Since joining two relations is a very vital but expensive operation, so a novel distributed learned index is also introduced to improve the join performance. Our experimental evaluation demonstrates that DaskDB significantly outperforms existing systems.