Webinar takeaway: ELK – Tips, Tricks, and Lessons Learned

My key takeaways

• ELK can serve multiple purposes
  • Log management
  • SIEM
  • File integrity monitoring
  • Netflow
  • Search solution
• ELK is not hard but not intuitive 😀
• Setup: Feeds -> Logstash Servers -> Kafka -> ELK Stacks
  • collect once, use many
• Over-sharding means having to many to small shards
  • should be between 50-100GB
• Under-sharding means having to many data for a shard
• S3 object storage is cheaper then disk storage and can automate snapshots
• To optimize YML for performance set pipeline.workers to 8 and pipeline.batch.size to 2000
• Logstash knows two parsers
  • dissect : delimiter based
  • grok : regex based
    • flexible but intense in memory
• the main pipeline is probably the weakest link in Logstash as issues are often not catched by monitoring tools
• Logstash as producer and consumer for Kafka, so the glue between Kafka and ELK
• Data.gov has tons of sample data
  • perfect for learning ELK, eg Chicago Crime and NYC Vehivle collisions
• search filter for nothing older than a year when searching for Elastic info due to major chances

Env

• Provided by Wild West Hackin’ Fest

• Presenter:

  • Mark McLauchlin
  • Nicolas Taina

additional links

• A python module to load windows event logs into ElasticSearch
• Elasticsearch’s Windows log shipper "Winlogbeat"
• https://github.com/macatak/presentations/tree/master/bhis_hc
• https://www.elastic.co/training/free
• Dissect Pattern Tester and Matcher for Filebeat, Elasticsearch and Logstash