Log Management Strategy Guide

A comprehensive guide to log management best practices for the ELK stack.

Part of: Log Management Toolkit by Datanest Digital

Table of Contents

1. Log Levels & When to Use Them

2. Structured Logging

3. What to Log (and What Not To)

4. Log Retention Strategy

5. Index Design & Naming

6. Alerting Strategy

7. Performance Tuning

8. Security & Compliance

9. Scaling the Stack

10. Troubleshooting Common Issues

1. Log Levels & When to Use Them

Consistent log levels are critical for filtering, alerting, and dashboard accuracy.

Rules of thumb:

• Production environments should log INFO and above
• Enable DEBUG per-service temporarily for troubleshooting
• WARN should always be actionable — if nobody will act on it, it's INFO
• ERROR means something broke. Every ERROR should eventually be investigated

2. Structured Logging

Always use structured (JSON) logging. Human-readable text logs are difficult to parse, search, and aggregate.

Good: Structured JSON

{
  "timestamp": "2026-01-15T14:30:00.123Z",
  "level": "ERROR",
  "service": "api-gateway",
  "message": "Payment processing failed",
  "error": {
    "type": "TimeoutError",
    "message": "Upstream timeout after 30s"
  },
  "request_id": "req_abc123",
  "trace_id": "trace_def456",
  "user_id": "usr_789",
  "duration_ms": 30012,
  "endpoint": "/api/v1/payments",
  "method": "POST"
}

Bad: Unstructured Text

2026-01-15 14:30:00 ERROR Payment processing failed for user usr_789 on /api/v1/payments: TimeoutError: Upstream timeout after 30s

Framework Configuration Examples

Node.js (pino):

const pino = require('pino');
const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  timestamp: pino.stdTimeFunctions.isoTime,
  base: { service: 'api-gateway', environment: process.env.NODE_ENV }
});

Python (structlog):

import structlog
structlog.configure(
    processors=[
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.processors.JSONRenderer()
    ]
)
logger = structlog.get_logger(service="api-gateway")

Go (zerolog):

logger := zerolog.New(os.Stdout).With().
    Timestamp().
    Str("service", "api-gateway").
    Logger()

3. What to Log (and What Not To)

Always Log

• Requests: Method, path, status code, duration, request ID
• Errors: Error type, message, stack trace, context
• Authentication: Login attempts (success and failure), token refresh
• Business events: Order placed, payment processed, user registered
• Lifecycle events: Service start/stop, config changes, deployments
• External calls: API calls to third parties with duration and status

Never Log

• Passwords or authentication tokens
• Credit card numbers or financial account details
• Personal data without explicit need (email, SSN, phone)
• Session tokens or API keys
• Health check requests (unless aggregated — they generate noise)
• Request/response bodies containing user data

Redaction Strategy

If sensitive fields might appear in logs, implement redaction:

// Redact sensitive fields before logging
const REDACT_FIELDS = ['password', 'token', 'authorization', 'cookie', 'ssn', 'credit_card'];

function redactObject(obj) {
  const redacted = { ...obj };
  for (const key of Object.keys(redacted)) {
    if (REDACT_FIELDS.some(f => key.toLowerCase().includes(f))) {
      redacted[key] = '[REDACTED]';
    }
  }
  return redacted;
}

4. Log Retention Strategy

Retention should balance storage costs, compliance requirements, and operational needs.

Recommended Retention Tiers

ILM Policy

The rotate-indices.sh script can create an ILM policy automatically:

bash scripts/rotate-indices.sh --setup-ilm --days 90

Storage Estimation

Elasticsearch typically achieves 1.5-3x compression on log data.

5. Index Design & Naming

Naming Convention

Use a consistent naming pattern: {type}-{source}-{date}

logs-nginx-access-2026.01.15
logs-app-2026.01.15
logs-system-2026.01.15
logs-auth-2026.01.15

Index Templates

Always define templates before ingesting data. Templates control:

• Shard count: 1 shard per index for <50GB/day; more for higher volume
• Replica count: 0 for single-node; 1+ for production clusters
• Field mappings: Explicit types prevent mapping conflicts
• ILM policy: Automatic lifecycle management

Shard Sizing Guidelines

6. Alerting Strategy

Alert Tiers

Kibana Alert Examples

Error rate spike:

WHEN count() of log_level:ERROR
OVER last 5 minutes
GROUPED BY service_name
IS ABOVE 50

Slow responses:

WHEN avg(response_time)
OVER last 15 minutes
IS ABOVE 2.0

Disk space:

WHEN max(elasticsearch.node.stats.fs.total.available_in_bytes)
IS BELOW 10737418240  (10 GB)

Avoiding Alert Fatigue

• Set thresholds based on historical baselines, not arbitrary numbers
• Use anomaly detection (Kibana ML) instead of static thresholds where possible
• Require alerts to be actionable — if nobody acts on it, remove it
• Group related alerts to avoid notification storms
• Implement escalation: warn at 80%, alert at 90%, page at 95%

7. Performance Tuning

Elasticsearch

# JVM heap: 50% of available RAM, max 31GB
ES_JAVA_OPTS: "-Xms4g -Xmx4g"

# Bulk indexing optimization
index.refresh_interval: "30s"    # Reduce for lower latency, increase for throughput
index.translog.durability: "async" # Faster writes, slight data loss risk on crash
index.translog.sync_interval: "5s"

Logstash

# Pipeline tuning
pipeline.workers: 4              # Match CPU cores
pipeline.batch.size: 1000        # Increase for throughput
pipeline.batch.delay: 50         # ms to wait for batch to fill

# JVM heap
LS_JAVA_OPTS: "-Xms1g -Xmx1g"

Filebeat

# Queue and batching
queue.mem.events: 4096
queue.mem.flush.min_events: 2048
queue.mem.flush.timeout: 1s

# Output tuning
output.logstash.bulk_max_size: 2048
output.logstash.worker: 2

Monitoring Performance

Key metrics to watch:

• Indexing rate: Events per second into Elasticsearch
• Search latency: p50, p95, p99 query times
• JVM heap usage: Should stay below 75%
• Disk I/O: High iowait indicates storage bottleneck
• Rejected threads: Bulk queue rejections mean you need to scale

8. Security & Compliance

Authentication

• Change default passwords immediately (the setup script does this)
• Use API keys for Filebeat instead of the elastic superuser
• Enable TLS for all inter-component communication in production

Creating a Filebeat API Key

curl -X POST "localhost:9200/_security/api_key" \
  -u elastic:YOUR_PASSWORD \
  -H "Content-Type: application/json" \
  -d '{
    "name": "filebeat-server-01",
    "role_descriptors": {
      "filebeat_writer": {
        "cluster": ["monitor", "read_ilm", "read_pipeline"],
        "index": [
          {
            "names": ["logs-*", "filebeat-*"],
            "privileges": ["create_index", "create_doc", "auto_configure"]
          }
        ]
      }
    }
  }'

Network Security

• Bind Elasticsearch and Kibana to localhost (default in this toolkit)
• Use a reverse proxy (Nginx) with TLS for external access
• Implement IP allowlisting for the Kibana web interface
• Keep Logstash's Beats port (5044) behind a firewall

Compliance Considerations

9. Scaling the Stack

When to Scale

Multi-Node Elasticsearch

For production workloads beyond a single server:

# 3-node minimum for high availability
discovery.seed_hosts: ["es-01", "es-02", "es-03"]
cluster.initial_master_nodes: ["es-01", "es-02", "es-03"]

# Dedicated roles per node type
node.roles: [master]         # Master-eligible nodes (3 minimum)
node.roles: [data_hot]       # Hot data nodes (SSD, write-heavy)
node.roles: [data_warm]      # Warm data nodes (HDD, read-heavy)
node.roles: [ingest]         # Ingest nodes (pipeline processing)

Logstash Load Balancing

Run multiple Logstash instances with Filebeat load balancing:

# filebeat.yml
output.logstash:
  hosts: ["logstash-01:5044", "logstash-02:5044"]
  loadbalance: true
  worker: 2

10. Troubleshooting Common Issues

Elasticsearch Won't Start

# Check vm.max_map_count
sysctl vm.max_map_count
# Fix: sudo sysctl -w vm.max_map_count=262144

# Check disk space
df -h /var/lib/docker

# Check container logs
docker compose logs elasticsearch

Logstash Not Processing Events

# Check pipeline status
curl -s localhost:9600/_node/stats/pipelines | python3 -m json.tool

# Validate config
docker compose exec logstash logstash --config.test_and_exit

# Check for grok parse failures
# Search Kibana for: tags: "_grokparsefailure"

Filebeat Not Shipping Logs

# Test config
filebeat test config
filebeat test output

# Check registry (tracks file read positions)
ls -la /var/lib/filebeat/registry/

# Check permissions
ls -la /var/log/nginx/access.log
# Filebeat needs read access to log files

High Memory Usage

# Check Elasticsearch heap
curl -s localhost:9200/_nodes/stats/jvm | python3 -m json.tool | grep heap

# Check for large aggregations
curl -s localhost:9200/_nodes/stats/breaker | python3 -m json.tool

# Reduce heap if oversized (>31GB triggers no compressed oops)
# ES_JAVA_OPTS="-Xms16g -Xmx16g"

Missing or Delayed Logs

1. Check Filebeat is running and has permission to read log files

2. Check Logstash pipeline isn't dropping events (look for _grokparsefailure tags)

3. Check Elasticsearch indexing rate and rejection counts

4. Verify time synchronization (NTP) across all servers

5. Check for bulk queue rejections in Logstash logs

Quick Reference

Log Management Toolkit is a product of Datanest Digital

Questions or issues? Email support@datanest.dev

Log Management Toolkit

Production-ready ELK stack deployment with structured logging pipelines and operational dashboards.

Complete Elasticsearch, Logstash, Kibana, and Filebeat configurations for centralized log management. Includes ready-to-use parsing pipelines for Nginx, application JSON logs, and system logs, plus operational scripts for index management, backups, and a comprehensive log strategy guide.