Apache: Big Data 2015

This session will discuss the challenges and opportunities of integrating Apache Spark with enterprise data warehouses, especially the impact of columnar storage on the example of IBM DB2 and IBM dashDB. We will show how columnar storage can help to increase scalability and reduce response time, especially when pushing down processing of projections and aggregates to the database instead of processing them in Spark natively. Key takeaways from the session are: 1.How to benefit from the features of closed-source data warehouses from Spark without access to internal data structures, 2.the role of storage when working with large warehouses from Spark, 3. Opportunities of columnar storage vs. row based storage, 4. How such an integration impacts end-to-end analytics based on Spark MLlib.