Apache: Big Data 2015

Your company has developed a system that crunches humongous data from multiple data sources. It involves multiple and varied processing modules. Each of the individual modules has been well tested. But, when you try to deploy these modules and connect them either in an integration or a staging environment, you face issues. To debug these errors and re-test them requires you to set up a mirror environment and is time consuming. So, you write complex integration tests and equally complex setup scripts. Apache Falcon and Falcon Unit to your rescue! While Apache Falcon alleviates some of the problems of pipeline orchestration, Falcon Unit, a feature of Falcon, helps users test their entire pipeline and data lifecycle without even setting up a test environment. This talk will outline the capabilities of Falcon Unit and how it helps users test data pipelines early on in the development phase.