Apache: Big Data 2015

Spark API exports intuitive and performant one-liners for data processing, which hide complexity and allow applications to be developed quickly. As an in-memory system, Spark has to be configured properly for performance and stability. This can sometimes be challenging. Based on internal deployments and interaction with customers, we conclude that (i) most Spark woes can be traced back to misconfiguration, and (ii) there is a need for tools that can aid configuration, performance optimization and debugging. In this talk, we will discuss common Spark configuration pitfalls and show how they can be avoided with the help of the auto-configuration and optimization tool being developed at Cloudera.