Forget about shoveling snow for a couple of hours and join us for our first 2015 meetup. As usual, food and refreshments will arrive around 6pm and we'll have 30 minutes to network and chat before the talk begins. Topic: Building unlimited scalable ELK stack on AWS Speaker Bio: Asaf Yigal is an entrepreneur who used to live in Boston for the past 12 years and started a company called Currensee which developed a social network for foreign exchange traders. He now lives in Israel and has started a company that applies machine learning to log analytics. They use ELK as the base of their product and have gained a lot of knowledge around how to deploy it at scale on AWS and how to properly monitor it.
ELK is a great platform for log analytics. It provides a very flexible infrastructure on each and every level from the log input to the Elasticsearch capabilities and the views in Kibana. The question is how do you put it all together if you want to deploy high scale log analytics which provide security, scalability, multi-tenancy and monitoring using AWS components? We will review the different challenges and how we chose to address them. Our solution includes the use of Kafka as a high performance messaging layer. We front Kibana with Play Framework and nginx to provide security and multi tenancy and we deploy multiple logstash instances along the log flow path in order to provide resiliency.
