Apache: Big Data 2015

YARN has opened up Hadoop to a variety of high performance purpose-built applications specialized for specific domains. Many of these need a common set of capabilities like scheduling, fault tolerance & scalability while not giving up on important aspects like multi-tenancy & security. We will provide an overview of how Apache Tez provides these capabilities via a dataflow based API to model these applications and an extensible orchestration framework for optimal performance. We will cover broad ecosystem adoption by Apache Hive, Pig, Cascading, Scalding, Flink & commercial vendors and provide some experiment results. We will look at the Tez Web UI for progress monitoring and performance debugging tools. Finally, we will look ahead at upcoming Tez features like hybrid execution which enables new types of integration with existing systems.