Amogh Margoor, Sourabh Goyal, Vikram Agrawal Originally written by us here: https://www.qubole.com/tech-blog/enabling-spark-sql-merge-via-optimized-acid-data-source-v0-6-0/. Importing my technical blogs to this space. We are pleased to announce the 0.6.0 release of the ACID Data source for Apache Spark. This release should further empower Data lake users in enterprises to enable ACID support for…

As a part of thought experiment to create super fast Data processing engine at Petabyte scale, Programming language was one of the major consideration. I asked around my friends working/researching in the field of PL/compilers. Lot of them were quite excited about Rust and hence I started exploring it. While…

Amogh Margoor, Rajat Venkatesh PS: Originally published on March 7, 2018: https://medium.com/qubole-engineering/sql-join-optimizations-in-qubole-presto-3ced3dc75275. I am moving my blogs to this space. SQL Joins are a common and critical component of interactive SQL workloads. …

Amogh Margoor, Rajat Venkatesh, Dr Prasad M. Deshpande PS: Originally published by me on August15, 2017 here: https://www.qubole.com/blog/insights-recommendations-hive-presto-sparksql-engine-configuration/. Importing my technical blogs to this space. Execution engines like M/R, Tez, Presto, and Spark provide a set of knobs or configuration parameters that control the behavior of the execution engine. In…

Amogh Margoor, Goden Yao Presto, Apache Spark, and Apache Hive can generate more efficient query plans with table statistics. For example, Spark, as of version 2.1.1, will perform broadcast joins only if the table size is available in the table statistics stored in the Hive Metastore (see spark.sql.autoBroadcastJoinThreshold). Broadcast joins…

Introduction — We previously wrote about the optimizations we made to optimize Hadoop and Hive on S3. Since then, we’ve applied those same changes across the rest of our Big Data analytics offerings, including Spark and Presto. …