Event-Stream Processing with Kafka

Tracking and analyzing user behaviour is essential for the development of successful games. At Wooga, we generate more than 100 million tracking events per day. Until recently we have simply batch-imported this data into MySQL every night -- straightforward, but not very flexible. Now Kafka is at the core of our new tracking infrastructure. In this talk, I will explain how we leverage Kafka as an "event bus" to enable near-realtime processing of the event stream. Why did we select Kafka? What are its major strengths and weaknesses? How does it fit into the larger picture? What patterns have emerged in production usage of Kafka over the last six months? I will also show examples of actual Ruby code we use in production at Wooga, and demonstrate how the open source library "Kafkaesque" that we have extracted simplifies filtering and processing events.

Watch the video of Tim Lossen's talk here.

Slides: 

Kafka-tlossen-bbuzz12.pdf