We are excited that Chas Honton will be sharing his experience deploying Elastic pipeline and Kafka in Kubernetes.

Doors open at 6pm and talks start at 6:30pm.

Talk Abstract:

Message Intelligence Service (MIS) collects connection and message metadata from thousands of production email clusters for near real-time monitoring, reporting, and searching. The service processes billions of messages a day and manages hundreds of terabytes of indices with a standard 30 day retention.

MIS consists of conventional data pipeline components from the Kafka and Elastic ecosystems. What might be unusual is the choice of using Kubernetes to deploy these components.

Kubernetes is the current choice for deploying stateless services in the cloud. Unfortunately, many of the simplifying assumptions that makes Kubernetes outstanding for deploying stateless services cause complications for stateful services such as Kafka and Elasticsearch.

This presentation discusses some of these challenges and how we dealt with them.

Speaker: Chas Honton

Chas Honton works with Proofpoint, "protecting the Very Attacked Person". He has previously worked with startups and established players such as Amazon Web Services and Intuit. He is an Apache Commons contributor and actively supports his open source productivity components on GitHub.