metricstarttimeprocessor

Status
Stability	alpha: metrics
Distributions	contrib
Issues
Code coverage
Code Owners	@dashpole, @ridwanmsharif

Description

The metric start time processor (metricstarttime) is used to set the start time for metric points with a cumulative aggregation temporality. It is commonly used with the prometheus receiver, which usually produces metric points without a start time.

Configuration

Configuration allows configuring the strategy used to set the start time.

processors:
    # processor name: metricstarttime
    metricstarttime:

        # specify the strategy to use for setting the start time
        strategy: true_reset_point

Strategy: True Reset Point

The true_reset_point strategy handles missing start times for cumulative points by producing a stream of points that starts with a True Reset Point. The true reset point has its start time set to its end timestamp. It is meant to indicate the absolute value of the cumulative point when the collector first observed it. Subsequent points re-use the start timestamp of the initial True Reset point.

Pros:

• The absolute value of the cumulative metric is preserved.
• It is possible to calculate the correct rate between any two points since the timestamps and values are not modified.

Cons:

• This strategy is stateful because the initial True Reset point is necessary to properly calculate rates on subsequent points.
• The True Reset point doesn't make sense semantically. It has a zero duration, but non-zero values.
• Many backends reject points with equal start and end timestamps.
  • If the True Reset point is rejected, the next point will appear to have a very large rate.

Strategy: Subtract Initial Point

The subtract_initial_point strategy handles missing start times for cumulative points by dropping the first point in a cumulative series, "subtracting" that point's value from subsequent points and using the initial point's timestamp as the start timestamp for subsequent points.

Pros:

• Cumulative semantics are preserved. This means that for a point with a given [start, end] interval, the cumulative value occurred in that interval.
• Rates over resulting timeseries are correct, even if points are lost. This strategy is not stateful.

Cons:

• The absolute value of counters is modified. This is generally not an issue, since counters are usually used to compute rates.
• The initial point is dropped, which loses information.

Strategy: Start Time Metric

The start_time_metric strategy handles missing start times by looking for the process_start_time metric, which is commonly supported by Prometheus exporters. If found, it uses the value of the process_start_time metric as the start time for all other cumulative points in the batch of metrics.

Use the start_time_metric_regex configuration option to change the name of the metric used for the start time.

If the start time metric is not found, it falls back to the time at which the collector started.

This strategy should only be used in limited circumstances:

• When your application has a metric with the start time in Unix nanoseconds, such as process_start_time.
• The metricstarttime processor is used before any batching, so that the batch of metrics all originate from a single application.
• This strategy can be used when the collector is run as a sidecar to the application, where the collector's start time is a good approximation of the application's start time.

Cons:

• If the collector's start time is used as a fallback and the collector restarts, it can produce rates that are incorrect and higher than expected.
• The process' start time isn't the time at which individual instruments or timeseries are initialized. It may result in lower rates if the first observation is significantly later than the process' start time.