Skip to main content

> ML_LITERATURE // ARMBRUST-2015-SPARK-SQL-RELATIONAL-DATA-PROCESSING-IN-SPARK_v1.0

Spark SQL: Relational Data Processing in Spark

Michael Armbrust, Reynold S. Xin, Cheng Lian, Yin Huai, Davies Liu, Joseph K. Bradley, Xiangrui Meng, Tomer Kaftan, Michael J. Franklin, Ali Ghodsi, Matei Zaharia · ACM SIGMOD International Conference on Management of Data (2015)

systems2015industry-standardnotAssessed
DOI: 10.1145/2723372.2742797Back to Literature

Principal Contribution

Introduced the Catalyst extensible relational query optimizer and DataFrame API, unifying relational and procedural big data computing.

Operational Relevance

Directly guides deployment choices and architecture selection for task-feature-engineering, task-data-processing.

Assumptions

  • Standard empirical regularity and statistical stability hold across evaluation domains

Limitations

  • Performance characteristics depend on domain distribution and compute allocation parameters

Connected Algorithms, Architectures & Tools

Related Algorithms:
Related Architectures:
Implementing Libraries: