Taking Elasticsearch from 0 to 88 mph

Everyone wants their Elasticsearch cluster to index and search faster, but optimizing both and finding the balance between the two can be tricky. At Kenna Security, we use Elasticsearch to store over 3 billion vulnerabilities for our clients. All that data needs to be quickly accessible so clients can assess their cyber security risk. At the same time the data is constantly changing. On average, we update 200+ million documents a day which means indexing speed is also a top priority.

In the early days our cluster could barely keep up. Nodes would fall over constantly, indexing queues would get backed up for days, and searches timed out about 50% of the time. Fixing all of these issues did not happen overnight. However, with a lot of testing, tweaking, and a few “OH crap!” moments we were able to build a stable, 21 node cluster that now meets all of our indexing and searching demands. In this talk I will share the insights we gained and the strategies we used to scale our cluster and hopefully that advice will save others some time and frustration as they grow their own.

Molly Struve is a Sr. Site Reliability Engineer at Kenna Security. She has been working with Elasticsearch at Kenna for the past 3 years. During her time at Kenna, she helped lead the team charged with scaling Kenna’s Elasticsearch cluster. Today the cluster holds over 3 billion documents and updates over 200 million of those a day. Given her degree in Aerospace Engineering from MIT, it is no surprise that Molly thrives on optimizing code performance. When not making code run faster, she can be found fulfilling her need for speed by riding and jumping her show horses.

The Data Game

Given the data in your environment, think of a question. A question you know that can be answered by the data. How hard is it to get the answer? Often times the level-of-effort to get the answer is prohibitive. Many of us face this scenario every day as we're swimming in data, but not answers. As we push to be more data-driven, lowering the level-of-effort to get answers is critical to our success.

In this session we'll examine a data source and then start asking questions of it. Hypothetical questions at first, but we'll bring in Elastic to turn theory into practice as we ingest the data and leverage Elastic's search & analytics capabilities. We'll explore different options and their tradeoffs during the process. You'll leave feeling confident that you can do the same with your data.

Scott Gose is a Solutions Architect for Elastic. He's focused on helping companies unlock answers in their data by using Elastic. Prior to Elastic, Scott did two tours at Endeca, which was a company that helped pioneer the field of faceted search & navigation. He's also worked professionally as a software developer building iPhone applications and writing back-end services in Go. Scott has a Computer Science degree from the University of Illinois.

Make sure to attend our Elastic{ON} Chicago Tour Stop on September 25th. Feel free to use the code Meetup10 for 10% off until
September 25th https://www.elastic.co/elasticon/tour/2018/chicago

If you’re interested in Elastic trainings [there is a fee], our team is coming to Chicago on September 26-27. You can register here: https://training.elastic.co/location/Chicago