Apache: Big Data 2015

This presentation will begin by giving a "State of the Union" of Apache Phoenix, a relational database layer on top of HBase for low latency applications, with a brief overview of new and existing features. Next, the approach for transaction support, a work in-progress will be discussed. Lastly, the current means of integrating with the rest of the Hadoop ecosystem will be examined, including the vision for how this will evolve going forward.

Apache Hive provides a convenient SQL query engine and table abstraction for data stored in Hadoop. Hive uses Hadoop to provide highly scalable bandwidth to the data, but until recently did not support updates, deletes, or transaction isolation. This has prevented many desirable use cases such as updating of dimension tables or doing data cleanup. We have implemented the standard SQL commands insert, update, and delete allowing users to insert new records as they become available, update changing dimension tables, repair incorrect data, and remove individual records. This also allows very low latency ingestion of streaming data from tools like Storm and Flume. Additionally, we have added ACID-compliant snapshot isolation between queries so that queries will see a consistent view of the committed transactions when they are launched.

Apache Drill is a next-generation SQL engine for Hadoop and NoSQL. Its unique schema-free approach enables self-service data exploration with the agility that organizations need in this new era of rapidly growing and evolving data.

In this talk, based on demonstrations, you will understand the key features and architecture of Apache Drill. You will also see how to get started with Drill; and start query, using SQL, various data sources such as HBase, Hive, Parquet, and Avro, but also more complex data structure stored in JSON documents.

Apache Hive has wide use cases for batch-oriented SQL workloads for ETL and data analytics in the Hadoop ecosystem. Up to now, most of the workloads are still executed by a 10 year old technology, MapReduce. On the other hand, Apache Spark as a general, open-source data processing framework is positioned to replace MapReduce with faster data processing and efficient memory utilization.

The Hive on Spark initiative introduced Spark as Hive's new execution engine, providing faster SQL on Hadoop while maintaining Hive's feature richness. With a joint effort from the Hive community and feedback from early adopters and beta users, Hive on Spark is ready for production deployment!

This presentation will share with you the motivation, architecture, deployment practice, and performance tuning. A live demo will be given to conclude the presentation.

HAWQ is a massively parallel processing SQL engine sitting on top of HDFS. As a hybrid of MPP database and Hadoop, it inherits the merits from both parties. It is standard SQL compliant, extremely fast and scalable, and unlike other SQL engines on Hadoop, it is fully
transactional. HAWQ is currently being proposed as an Apache incubating project.

In this talk, Dr. Lei Chang will give an overview on HAWQ architecture and the major exciting areas that are soliciting contributions from open source community. And he will also introduce the easiest way contributors can work with HAWQ developers to bring their innovative ideas to the HAWQ kernels.

Apache Kylin is an open source Distributed Analytics Engine contributed by eBay Inc. that provides SQL interface and multi-dimensional analysis (OLAP) on Hadoop supporting extremely large datasets. Kylin’s pre-built MOLAP cubes, distributed architecture, and high concurrency helps users analyze multidimensional queries using Kylin’s SQL interface as well as via other BI tools like Tableau and Microstrategy. Kylin is successfully deployed and used in eBay for a variety of production use cases, including web traffic analysis and geographical expansion analysis. It was open sourced on Oct 1, 2014 and has 320 stars and 125 forks. Kylin has been accepted as Apache Incubator Project on Nov 25, 2014.

In the space of Big Data, two powerful data processing tools compliment each other. Namely HAWQ and Geode. HAWQ is a scalable OLAP SQL-on-Hadoop system, while Geode is OLTP like, in-memory data grid and event processing system. This presentation will show different integration approaches that allow integration and data exchange between HAWQ and Geode. Presentation will walking you through the implementation of the different Integration strategies demonstrating the power of combining various OSS technologies for processing bit and fast data. Presentation will touch upon OSS technologies like HAWQ, Geode, SpringXD, Hadoop and Spring Boot.

The Pivotal HAWQ[1] project, planned for incubation into an Apache project, is designed to provide a highly performant ANSI SQL compliant query engine supporting a sophisticated resource management model, transactional DML and DDL operations, window functions, grouping sets, complex sub-queries, common table expressions, and strong extensibility capabilities for customized analytics and machine learning.

In this session we will present an overview of the product features, describe the key architectural components, and walk through the overall project structure.

[1] Project incubation and Apache project name pending approval by the Apache Foundation.

Trafodion is a world class Transactional SQL RDBMS running on HBase/Hadoop, currently in Apache incubation.

In this talk we will discuss:

• How operational workloads are different from BI and analytical workloads
• The operational (OLTP & Operational Data Store) use cases Trafodion addresses
• Why Trafodion is the right solution for these use cases.  That is, what is the recipe for a world class database engine, and how Trafodion implements the ingredients that make up that recipe: 

1. Time, money, and talent!
2. World class query optimizer
3. World class parallel data flow execution engine
4. World class distributed transaction management system

• Other important aspects such as performance, scale, availability, and future directions