Elasticsearch is doing their training Sept 8-10 in SF, please come listen to a talk from a team from the hosts, Ancestry.com!
6:30 - 6:45 mingle, get organized
6:45 - 6:50 hello from out hosts Ancestry.com
6:50 - 7:15 Ancestry presents + Q&A, see info below
7:15 - 7:20 community announcements ("Hi, we're hiring", etc)
7:20+ Elasticsearch guys talk + Q&A
At Ancestry we use an aspect-oriented logging framework to capture website and server events from many applications in the company. We use Kafka for collecting and aggregating the logs and push it to HDFS using some custom MapReduce code and Hive.
To give users access to individual log events and provide them the ability to perform analytics, we set up a 10-node Elasticsearch cluster in August and created custom Kibana dashboards. Using the bulk indexer we’ve averaged 45k events/sec for 1.2TB of data and 2.8 billion documents in our latest test run.
We’ll be presenting:
* A rapid development environment for Elasticsearch that uses Jython to allow direct access to HDFS and Elasticsearch libraries
* How to really push the cluster with a configurable number of threaded clients
* Configuration settings to maximize write throughput
* The monitoring and management tools we use
There is a Giants Home Game this same time so parking will be likely be locked up by ballpark patrons.