August 18

Webinar takeaway: ELK – Tips, Tricks, and Lessons Learned

Security

0  comments

My key takeaways

  • ELK can serve multiple purposes
    • Log management
    • SIEM
    • File integrity monitoring
    • Netflow
    • Search solution
  • ELK is not hard but not intuitive :-D
  • Setup: Feeds -> Logstash Servers -> Kafka -> ELK Stacks
    • collect once, use many
  • Over-sharding means having to many to small shards
    • should be between 50-100GB
  • Under-sharding means having to many data for a shard
  • S3 object storage is cheaper then disk storage and can automate snapshots
  • To optimize YML for performance set pipeline.workers to 8 and pipeline.batch.size to 2000
  • Logstash knows two parsers
    • dissect : delimiter based
    • grok : regex based
      • flexible but intense in memory
  • the main pipeline is probably the weakest link in Logstash as issues are often not catched by monitoring tools
  • Logstash as producer and consumer for Kafka, so the glue between Kafka and ELK
  • Data.gov has tons of sample data
    • perfect for learning ELK, eg Chicago Crime and NYC Vehivle collisions
  • search filter for nothing older than a year when searching for Elastic info due to major chances

Env

additional links

Tags

ELK, Kafka, Logstash, SIEM

You may also like

Webinar takeaway: ELK – Tips, Tricks, and Lessons Learned

My key takeaways ELK can serve multiple purposes Log management SIEM File integrity monitoring Netflow Search solution ELK is not hard but not intuitive 😀 Setup: Feeds -> Logstash Servers -> Kafka -> ELK Stacks collect once, use many Over-sharding means having to many to small shards should be between 50-100GB Under-sharding means having to

Read More
Leave a Comment