Apache: Big Data 2015

At Hortonworks, we have been working closely with a group of customers to onboard Data Science and Predictive Analytics applications on the Hortonworks Data Platforms using Spark. In this talk, we discuss the use cases and explore some of the challenges and solutions that arise in building real-world predictive analytics systems on top of Spark. These include useful tricks for memory savings, tradeoffs on the choice of the learning and feature engineering algorithms, tips and tricks to avoid performance bottlenecks, methods for assessing and visualizing accuracy, methods for providing confidence estimates for predictions and methods for managing retraining and deployment of trained models to production.