new icn messageflickr-free-ic3d pan white
Neha Narkhede - ETL is dead; long-live streams | by QConPictures
Back to album

Neha Narkhede - ETL is dead; long-live streams

What happens if you take everything that is happening in your company—every click, every database change, every application log—and make it all available as a real-time stream of well-structured data?

 

I will discuss the experience at LinkedIn and elsewhere moving from batch-oriented ETL to real-time streams using Apache Kafka. I’ll talk about how the design and implementation of Kafka was driven by this goal of acting as a real-time platform for event data. I will cover some of the challenges of scaling Kafka to hundreds of billions of events per day at Linkedin, supporting thousands of engineers, applications, and data systems in a self-service fashion.

164 views
0 faves
0 comments
Taken on November 9, 2016