August 18

Webinar takeaway: ELK – Tips, Tricks, and Lessons Learned


My key takeaways

  • ELK can serve multiple purposes
    • Log management
    • SIEM
    • File integrity monitoring
    • Netflow
    • Search solution
  • ELK is not hard but not intuitive 😀
  • Setup: Feeds -> Logstash Servers -> Kafka -> ELK Stacks
    • collect once, use many
  • Over-sharding means having to many to small shards
    • should be between 50-100GB
  • Under-sharding means having to many data for a shard
  • S3 object storage is cheaper then disk storage and can automate snapshots
  • To optimize YML for performance set pipeline.workers to 8 and pipeline.batch.size to 2000
  • Logstash knows two parsers
    • dissect : delimiter based
    • grok : regex based
      • flexible but intense in memory
  • the main pipeline is probably the weakest link in Logstash as issues are often not catched by monitoring tools
  • Logstash as producer and consumer for Kafka, so the glue between Kafka and ELK
  • has tons of sample data
    • perfect for learning ELK, eg Chicago Crime and NYC Vehivle collisions
  • search filter for nothing older than a year when searching for Elastic info due to major chances


additional links


ELK, Kafka, Logstash, SIEM

You may also like

Webinar takeaway: ELK – Tips, Tricks, and Lessons Learned

Webinar takeaway: ELK – Tips, Tricks, and Lessons Learned
Leave a Reply

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit markiert.

{"email":"Email address invalid","url":"Website address invalid","required":"Required field missing"}