Logstash release notes

Review the changes, fixes, and more in each version of Logstash.

To check for security updates, go to Security announcements for the Elastic stack.

We’ve added support for compression to the Persistent Queue (PQ), allowing you to spend some CPU in exchange for reduced disk IO. This can help reduce cost and increase throughput in situations where your hardware is rate-limited or metered.

PQ compression is implemented using the industry-standard highly-efficient ZSTD algorithm, and can be activated at one of three levels:

Speed: spend the least amount of CPU to get minimal compression benefit
Balanced: spend moderate CPU to further reduce size
Size: enable maximum compression, at significantly higher cost

The effects of these settings will depend on the shape and size of each pipeline’s events. To help you tune your configuration to meet your own requirements, we have added queue metrics exposing the effective compression ratio and the amount of CPU that is being spent to achieve it.

PQ Compression has been introduced as an opt-in feature in 9.2 because a PQ that contains one or more compressed events cannot be read by previous versions of Logstash, making the feature a rollback-barrier. We recommend validating your pipelines with Logstash 9.2 (or later) before enabling PQ compression so that you have the freedom to roll back if you encounter any issues with your pipelines.

Persisted Queue: improved serialization to be more compact by default (note: queues containing these compact events can be processed by Logstash v8.10.0 and later) #17849
Support for user defined metrics #18218
PQ: Add support for event-level compression using ZStandard (ZSTD) #18121

Batch size metrics

We've added metrics to help you track the size of batches processed by Logstash pipelines.

The Node API pipelines endpoint now shows includes information displaying the showing the average number of events processed per batch, and the average byte size of those batches for each pipeline. This information can be used to help size Logstash instances, and optimize settings for pipeline.batch.size for Logstash pipelines based on real observations of data.

Implements current batch event count and byte size metrics #18160
Implements average batch event count and byte size metrics. The collection of such metric could be disabled, enabled for each batch or done on a sample of the total batches #18000

Additional features and enhancements

Dropped the persistent queue setting queue.checkpoint.interval #17759
Reimplements BufferedTokenizer to leverage pure Java classes instead of use JRuby runtime's classes #17229
Logging improvement while handling exceptions in the pipeline, ensuring that chained exceptions propagate enough information to be actionable. #17935
Support for using ES|QL queries in the Elasticsearch filter to add improved flexibility when ingesting data from Elasticsearch is now in Technical Preview.
Gauge type metrics, such as current and peak connection counts of Elastic Agent, are now available in the _node/stats API response when the vertices=true parameter is included. These metrics are particularly useful for monitoring Logstash plugin activity on the Logstash Integration dashboards #18090
Improve Logstash release artifacts file metadata: mtime is preserved when building tar archives #18091

Plugins

Elastic_integration Filter - 9.2.0

Logging compatability with Elasticsearch 9.2 #373
Utilizes Elasticsearch interfaces via Elasticsearch logstash-bridge #336

Translate Filter - 3.5.0

Introduce opt-in "yaml_load_strategy => streaming" to stream parse YAML dictionaries. This can hugely reduce the memory footprint when working with large YAML dictionaries. #106

Snmp Integration - 4.1.0

Add support for SNMPv3 context engine ID and context name to the snmptrap input #76

9.1.6

No user-facing changes in Logstash core.

Plugins

Csv Output - 3.0.11

Docs: Correct code snippet #28

9.1.5

No user-facing changes in Logstash core.

Plugins

Elasticsearch Filter - 4.3.1

Added support for encoded and non-encoded api-key formats on plugin configuration #203

Elasticsearch Input - 5.2.1

Added support for encoded and non-encoded api-key formats on plugin configuration #237

Jdbc Integration - 5.6.1

Fixes an issue where the jdbc_static filter's throughput was artificially limited to 4 concurrent queries, causing the plugin to become a bottleneck in pipelines with more than 4 workers. Each instance of the plugin is now limited to 16 concurrent queries, with increased timeouts to eliminate enrichment failures. #187

Elasticsearch Output - 12.0.7

Support both, encoded and non encoded api-key formats on plugin configuration #1223

9.1.4

Features and enhancements

ES|QL Support in Elasticsearch Filter (Technical Preview)

Support for using ES|QL queries in the Elasticsearch filter to add improved flexibility when ingesting data from Elasticsearch is now in Technical Preview.

Fixes

Gauge type metrics, such as current and peak connection counts of Elastic Agent, are now available in the _node/stats API response when the vertices=true parameter is included. These metrics are particularly useful for monitoring Logstash plugin activity on the Logstash Integration dashboards #18090
Improve logstash release artifacts file metadata: mtime is preserved when buiilding tar archives #18091

Plugins

Elasticsearch Filter - 4.3.0

ES|QL support #194

Beats Input - 7.0.3

Upgrade netty 4.1.126 #517

Http Input - 4.1.3

Upgrade netty to 4.1.126 #198

Jms Input - 3.3.1

Fixed a regression introduced in 3.3.0 where add_field is no longer enriching events #59

Tcp Input - 7.0.3

Upgrade netty to 4.1.126 #235

Kafka Integration - 11.6.4

Display exception chain comes from kafka client #200

9.1.3

Features and enhancements

Logging improvement while handling exceptions in the pipeline, ensuring that chained exceptions propagate enough information to be actionable. #17935

Plugins

No change to the plugins in this release.

9.1.2

No user-facing changes in Logstash core.

Plugins

No change to the plugins in this release.

9.1.1

Features and enhancements

No user-facing changes in Logstash core.

Plugins

Elastic_integration Filter - 9.1.1

Add terminate processor support #345

Translate Filter - 3.4.3

Allow YamlFile's Psych::Parser and Visitor instances to be garbage collected #104

Xml Filter - 4.3.2

Update Nokogiri dependency version #89

Azure_event_hubs Input - 1.5.2

Updated JWT dependency #101

Snmp Integration - 4.0.7

FIX: The snmptrap input now correctly enforces the user security level set by security_level config, and drops received events that do not match the configured value #75

Elasticsearch Output - 12.0.6

Add headers reporting uncompressed size and doc count for bulk requests #1217

9.1.0

Features and enhancements

Significantly improves write speeds to the persistent queue (PQ) when a pipeline's workers are caught up with already-written events #17791
Eliminated log warning about unknown gauge metric type when using pipeline-to-pipeline. #17721
Improve plugins remove command to support multiple plugins #17030
Deprecated the persistent queue setting queue.checkpoint.interval#17759, which was found to have no effect. This will be removed in a future Logstash release.
Logstash now ships with JRuby 9.4.13.0 to leveragle latest features and improvements in the 9.4 series #17696
Enhanced keystore validation to prevent the creation of secrets in an invalid format #17351

ES|QL Support in Elasticsearch Input (Technical Preview)

Support for using ES|QL queries in the Elasticsearch input to add improved flexibility when ingesting data from Elasticsearch is now in Technical Preview.
Logstash OSS and Full docker images are now based on Ubuntu 24.04.

Field Tracking Support in Elasticsearch Input (Technical Preview)

The Elasticsearch Input now provides support for field value tracking, persisted to disk on each search_after page. This is useful to track new data being written to an index or series of indices.

Updates to dependencies

Update JDK to 21.0.7+6 #17591

Plugins

Elastic Integration Filter - 9.1.0

Introduces proxy param to support proxy #316
Embeds Ingest Node components from Elasticsearch 9.1

Elasticsearch Filter - 4.2.0

Add target configuration option to store the result into it #196

Elasticsearch Input - 5.2.0

Add "cursor"-like index tracking #205
ES|QL support #233

Elasticsearch Output - 12.0.5

Docs: update Cloud terminology #1212
Change connection log entry from WARN to INFO when connecting during register phase #1211

JDBC Integration - 5.6.0

Support other rufus scheduling options in JDBC Input #183

JMS Input - 3.3.0

Added support for decoding multiple events from text or binary messages when using a codec that produces multiple events #56

Kafka Integration - 11.6.3

Update kafka client to 3.9.1 #193
Docs: fixed setting type reference for sasl_iam_jar_paths #192
Expose the SASL client callback class setting to the Logstash configuration #177
Adds a mechanism to load AWS IAM authentication as SASL client libraries at startup #178

Xml Filter - 4.3.1

Update Nokogiri dependency version #88

Tcp Output - 7.0.1

Call connection check after connect #61

9.0.8

No user-facing changes in Logstash core.

Plugins

Elasticsearch Output - 12.0.7

Support both, encoded and non-encoded api-key formats on plugin configuration #1223

9.0.7

Fixes

Gauge type metrics, such as current and peak connection counts of Elastic Agent, are now available in the _node/stats API response when the vertices=true parameter is included. These metrics are particularly useful for monitoring Logstash plugin activity on the Logstash Integration dashboards. #18089
Improve logstash release artifacts file metadata: mtime is preserved when building tar archives. #18111

Plugins

Beats Input - 7.0.3

Upgrade netty 4.1.126 #517

Http Input - 4.1.3

Upgrade netty to 4.1.126 #198

Tcp Input - 7.0.3

Upgrade netty to 4.1.126 #235

Kafka Integration - 11.6.4

Display exception chain comes from kafka client #200

9.0.6

Features and enhancements

Logging improvement while handling exceptions in the pipeline, ensuring that chained exceptions propagate enough information to be actionable. #17934

Adds support for missing terminate processor #345

Translate Filter - 3.4.3

FIX: Reduces memory consumption when configured with a YAML dictionary file by allowing YamlFile's Psych::Parser and Visitor instances to be garbage collected #104

Xml Filter - 4.3.2

Update Nokogiri dependency version #89

Azure_event_hubs Input - 1.5.2

Updated JWT dependency #101

Snmp Integration - 4.0.7

FIX: The snmptrap input now correctly enforces the user security level set by security_level config, and drops received events that do not match the configured value #75

Elasticsearch Output - 12.0.6

Add headers reporting uncompressed size and doc count for bulk requests #1217

9.0.4

Fixes

Significantly improves write speeds to the persistent queue (PQ) when a pipeline's workers are caught up with already-written events #17791
Eliminated log warning about unknown gauge metric type when using pipeline-to-pipeline. #17721

Plugins

Elastic_integration Filter - 9.0.1

Introduces proxy config to support proxy URI to connect to Elasticsearch. #320

Elasticsearch Output - 12.0.4

Docs: update Cloud terminology #1212

9.0.3

No user-facing changes in Logstash core.

Plugins

Kafka Integration - 11.6.3

Update kafka client to 3.9.1 #193

9.0.2

No user-facing changes in Logstash core.

Plugins

Kafka Integration - 11.6.2

Docs: fixed setting type reference for sasl_iam_jar_paths #192
Expose the SASL client callback class setting to the Logstash configuration #177
Adds a mechanism to load AWS IAM authentication as SASL client libraries at startup #178

9.0.1

Important

The 9.0.1 release contains fixes for potential security vulnerabilities. Check out the security advisory for details.

Features and enhancements

Enhanced keystore validation to prevent the creation of secrets in an invalid format #17351

Updates to dependencies

Update JDK to 21.0.7+6 #17591

Plugins

Xml Filter - 4.3.1

Update Nokogiri dependency version #88

Elasticsearch Output - 12.0.3

Change connection log entry from WARN to INFO when connecting during register phase #1211

Tcp Output - 7.0.1

Call connection check after connect #61