ELK Stack Integration: Synchronizing Logstash, Elasticsearch, and Kibana

When Logstash, Elasticsearch, and Kibana are out of sync, logs disappear, dashboards look empty, or fields show up with the wrong type. ELK Stack integration is less about starting three services and more about making sure the index names, mappings, timestamps, credentials, and data views all agree.

This guide walks through a practical logging pipeline: Logstash receives events, parses them, sends them to Elasticsearch, and Kibana reads the resulting indices or data streams. The examples use classic Logstash pipeline syntax and Elasticsearch APIs you can run from Kibana Dev Tools.

Understanding the Data Flow

Trace one event through the stack:

1. Logstash receives data from Beats, TCP, syslog, files, queues, or another input.
2. Logstash filters parse, enrich, rename, and normalize fields.
3. Elasticsearch indexes the event using templates, mappings, and lifecycle policies.
4. Kibana queries Elasticsearch through a data view and displays the event in Discover, dashboards, Lens, or alerts.

Most integration bugs happen at the boundaries. Logstash cannot connect, Elasticsearch rejects the document, or Kibana is looking at the wrong data view or time range.

Logstash Configuration for Clean Data Flow

Logstash pipelines have three main blocks: input, filter, and output. Keep each block boring and testable.

Input Plugins

Common input plugins include:

• beats: Receives events from Filebeat, Metricbeat, and other Beats.
• tcp / udp: Receives events over network sockets.
• file: Reads local files. This is useful for small deployments and tests, but agents are usually better for distributed production hosts.
• syslog: Receives syslog messages.

Example Beats input with TLS:

input {
  beats {
    port => 5044
    ssl_enabled => true
    ssl_certificate => "/etc/pki/tls/certs/logstash.crt"
    ssl_key => "/etc/pki/tls/private/logstash.key"
  }
}

Make sure the port is open, the certificate matches how clients connect, and the option names match your installed plugin version. Recent Beats input plugin versions use ssl_enabled.

Filter Plugins

Filters turn raw events into useful fields. The order matters because Logstash runs filters sequentially.

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }

  date {
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
  }

  mutate {
    remove_field => [ "message" ]
  }
}

Use grok for unstructured text, date to set @timestamp, mutate for field cleanup, and geoip when you need IP-based location enrichment. Test grok patterns against real log lines before you put them in production. A small parsing mistake can send thousands of events into Elasticsearch with missing fields.

Output Plugin

For the ELK stack, the Elasticsearch output is the usual destination.

output {
  elasticsearch {
    hosts => ["https://elasticsearch-node1:9200", "https://elasticsearch-node2:9200"]
    index => "my-logs-%{+YYYY.MM.dd}"
    user => "logstash_writer"
    password => "${LOGSTASH_ES_PASSWORD}"
    ssl_enabled => true
    cacert => "/etc/logstash/certs/http_ca.crt"
  }
}

The index value is the contract with Elasticsearch templates and Kibana data views. If Logstash writes my-logs-2026.05.23, your template and data view should match my-logs-*.

For larger environments, consider data streams and Index Lifecycle Management instead of hand-managed daily indices. If you use data streams, follow Elastic's current Logstash output guidance for data_stream settings rather than mixing data stream and classic index options.

Elasticsearch Templates and Mappings

Elasticsearch needs consistent mappings before documents arrive. Otherwise, the first document can set a field type that breaks later events. A status code that arrives first as "200" may become text or keyword instead of a number.

Example composable index template:

PUT _index_template/my_log_template
{
  "index_patterns": ["my-logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1
    },
    "mappings": {
      "properties": {
        "@timestamp": {"type": "date"},
        "message": {"type": "text"},
        "host.name": {"type": "keyword"},
        "log.level": {"type": "keyword"},
        "http.response.status_code": {"type": "integer"}
      }
    }
  }
}

Use keyword for exact matching and aggregations, text for full-text search, numeric types for metrics and status codes, and date for time fields. Keep shard counts modest unless you have a measured reason to add more. Too many small shards can hurt cluster performance.

Kibana Data Views

Kibana's current UI uses data views. Older versions called them index patterns. Create a data view that matches the index names or data streams Elasticsearch actually has.

Typical setup:

1. Go to Stack Management -> Kibana -> Data Views.
2. Create a data view such as my-logs-*.
3. Choose @timestamp as the time field.
4. Open Discover and widen the time picker while testing.

If Discover is empty, do not assume Logstash failed. Check the time range, the data view pattern, and whether @timestamp parsed correctly.

Troubleshooting Common Integration Issues

Data Does Not Appear in Kibana

Check each hop:

GET _cat/indices/my-logs-*?v
GET my-logs-*/_search?size=1&sort=@timestamp:desc

Then check:

• Logstash logs for connection, authentication, TLS, or mapping errors.
• Elasticsearch logs for rejected documents and security failures.
• The Kibana data view pattern and selected time range.
• Whether the event timestamp is in the future, in the past, or missing.

Documents Are Rejected by Elasticsearch

Mapping conflicts are common. For example, one event sends http.response.status_code as 200, while another sends "OK". Elasticsearch cannot store both in an integer field.

Fix the Logstash filter so the field is consistently typed, or route bad events to a separate index for review. Do not keep deleting and recreating indices without fixing the pipeline that creates the bad documents.

Logstash Uses Too Much CPU

Expensive grok patterns, high event volume, and large multiline events can push Logstash CPU up quickly. Start by measuring which pipeline is busy, then simplify patterns, anchor regexes, and move simple parsing to Beats or Elasticsearch ingest pipelines when that is easier to operate.

Kibana Queries Are Slow

Slow dashboards often come from wide time ranges, high-cardinality aggregations, too many shards, or fields mapped as text when Kibana needs keyword. Use narrower dashboard defaults, ILM rollover, and field mappings that match your visualizations.

Takeaway

Treat ELK Stack integration as a contract between three layers. Logstash must emit predictable fields, Elasticsearch must map and store them correctly, and Kibana must query the right data view over the right time range. When something breaks, follow one sample event from input to index to dashboard.