Detection Engineering Overview

Detection engineering is the practice of designing, building, and maintaining security detection capabilities. It transforms raw telemetry into actionable signals that drive incident response.

The Detection Pipeline

A modern detection pipeline has four stages:

Collection — Raw telemetry from multiple sources: operating system logs (syslog, auditd), runtime detection (Falco), network monitoring (Suricata), cloud audit logs (AWS CloudTrail, Azure Activity Log), and application logs.

Normalization — Raw logs come in different formats. Normalization maps them to a common schema (Elastic Common Schema) so detections can be written once and applied across all sources.

Detection — Rules and algorithms that identify malicious or suspicious activity. Multiple detection types work together: signature-based (Sigma rules), behavioral (anomaly thresholds), and correlation (multi-event patterns).

Response — Alerting, ticketing, and automated response actions that turn detections into outcomes.

Signal vs Noise

The fundamental challenge of detection engineering is maximizing signal while minimizing noise. A detection that fires constantly is ignored. A detection that never fires may be broken.

Design each detection with a clear answer to: what specific behavior are we detecting, what's the confidence level this indicates malicious activity, and what action should the analyst take?

The Detection Lifecycle

Detections are not write-once artifacts. They need a lifecycle:

1. Design — Threat modeling identifies what to detect. Determine the data sources, detection logic, and response.

2. Implementation — Write the detection rule, test it against known-good and known-bad traffic.

3. Tuning — Deploy in monitoring-only mode. Adjust thresholds and filters based on observed behavior.

4. Production — Enable alerting when false positive rates are acceptable.

5. Review — Reassess detections quarterly. Is the threat still relevant? Are new data sources available?

Log Ingestion & Normalization

Getting logs from your infrastructure into a searchable, analyzable system is the foundation of security monitoring. Without reliable log ingestion, no detection or response capability exists.

The Ingestion Pipeline

The reference pipeline in this product uses Filebeat → Logstash → Elasticsearch. Filebeat ships logs from each host, Logstash parses and normalizes them, and Elasticsearch stores and indexes them.

Filebeat configuration key points:

• Ship auth logs, auditd logs, Suricata EVE JSON, and Falco alerts
• Use TLS encryption between Filebeat and Logstash
• Configure a disk-backed queue for resilience during Logstash outages
• Drop noise-generating events at the shipper level (health checks, routine cron jobs)

Logstash pipeline:

• Grok patterns parse syslog formats into structured fields
• GeoIP enrichment adds location context to source IP addresses
• Event classification maps raw log messages to standardized event.action values
• Output routes normalized events to dated security-YYYY.MM.DD indices

Elastic Common Schema

ECS is a shared field naming standard that ensures consistency across different log sources. When auth logs and Suricata alerts both use source.ip and user.name, correlation across sources becomes trivial.

Key ECS fields to populate:

• @timestamp — When the event occurred
• event.action — What happened (user_login, process_start, network_connection)
• event.category — High-level classification (authentication, network, process)
• user.name and user.id — Who was involved
• source.ip and destination.ip — Network context
• process.name and process.executable — Process context

Pipeline Health Monitoring

A blind sensor is worse than no sensor — it gives a false sense of security. Monitor pipeline health with:

• Alert if no events received from a host in 5 minutes
• Alert if Logstash queue depth exceeds threshold
• Alert if Elasticsearch indexing lag exceeds 60 seconds