Elasticsearch Anomaly Detection Explainer

Anomaly Detection Score Explainer

Explain anomaly scores, model behavior, and why results look the way they do. Use the ML REST API for job config and the standard _search API against .ml-anomalies-* for results — no ES|QL, fully compatible with Elastic Serverless. For job lifecycle (create, start, stop), use the elasticsearch-anomaly-detection skill.

<!-- begin-partial: preamble -->

Environment Configuration

This skill executes Elasticsearch operations through the elastic CLI. If the elastic CLI is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., GET /, GET /_cat/indices, GET /{index}/_mapping, GET /{index}/_settings/index.mode, POST /_query). The Operations table at the end of this document maps each shorthand to the equivalent elastic CLI command — always use the CLI rather than calling the HTTP API directly.

<!-- end-partial: preamble -->

Prerequisite: ML anomaly detection requires a Platinum-equivalent license on self-managed clusters. Serverless projects include ML. The caller needs monitor_ml to read job config and anomaly results.

Serverless note: The _ml/.../results/* REST endpoints return HTTP 410 in Elastic Serverless. Always use POST /.ml-anomalies-*/_search for result queries instead — fully supported everywhere this skill runs.

Process

1. Decide whether to fetch data or interpret what the user supplied. If the user embeds an anomaly record (or job config) in the prompt, interpret it directly using the domain knowledge below — do not call APIs to re-fetch fields already present. If the job ID, time range, or record is missing, retrieve it from the cluster.

   The decision: proceed with judgment-only explanation when the record contains record_score, initial_record_score, actual, typical, and function; otherwise fetch the missing pieces before explaining.

2. Verify connectivity when calling the cluster. Call GET /. If the call fails, stop and surface the connection error — do not guess endpoints or credentials.

3. Resolve the job ID and load config. When the job ID is unknown, call GET /_ml/anomaly_detectors to list candidates. Call GET /_ml/anomaly_detectors/{job_id} for full analysis_config (bucket_span, detectors, custom_rules, use_null, model_plot_config) and GET /_ml/anomaly_detectors/{job_id}/_stats for state, model_size_stats.memory_status, and data counts.

   The decision: confirm detector function and direction match the user's question before interpreting scores. A low_count job legitimately fires on drops; a high_count job does not.

4. Retrieve anomaly records for the time range. Call POST /.ml-anomalies-*/_search with result_type: record, the job ID, a timestamp range, and optional record_score filter. Read initial_record_score, record_score, actual, typical, function, multi_bucket_impact, and anomaly_score_explanation.

   Always show both initial_record_score and record_score. The gap is the renormalization story.

5. Classify the score pattern before speculating on causes.

   • initial_record_score >> record_score — Renormalization. A later, more extreme anomaly rescale this record downward. This is expected, healthy model behavior — not a broken model or reason to distrust the detection. Use initial_record_score for alerting severity; show both scores and explain the gap explicitly.

   • initial_record_score == record_score — No renormalization has occurred since detection.

   • actual << typical with low_count, count, or low_mean — Absence / drop anomaly. A high score is legitimate — the job detected an outage, pipeline stall, or service failure. This is not a false positive. Recommend incident investigation, not score tuning.

   • actual >> typical with high_count or high_mean — Spike anomaly; confirm with single_bucket_impact.

   Only cite anomaly_score_explanation factors present in the record. If high_variance_penalty is false, do not blame variance. If a factor is absent, note that it was not returned — do not invent it.

6. Quantify renormalization across the job (optional). Re-query POST /.ml-anomalies-*/_search for records in the time range sorted by timestamp ascending. Compute score_drift = initial_record_score − record_score per record and filter to |score_drift| ≥ 20. Large negative drift (initial >> record) confirms renormalization after a more extreme anomaly appeared later.

7. Add context when the user asks "what caused this?" or "why so low/high?"

   • Model bounds — If model_plot_config.enabled is true, call POST /.ml-anomalies-*/_search with result_type: model_plot for the same job and time range. Compare actual to model_lower / model_upper.

   • Influencers — Call POST /.ml-anomalies-*/_search with result_type: influencer for the bucket time range; sort by influencer_score descending.

   • Categorization jobs — Call POST /.ml-anomalies-*/_search with result_type: category_definition to list learned log patterns (terms, regex, examples per category_id).

   For aggregations, cross-job queries, bucket-level results, or custom filters beyond score and time, see references/explainer-reference.md.

Common multi-step workflows

Critical principles

• Retrieve the record first (or use the one the user supplied). Never explain scores without initial_record_score, record_score, actual, typical, and function.

• Renormalization is healthy. When initial_record_score >> record_score, a more extreme anomaly appeared later and lowered this score — expected behavior, not a model failure.

• Direction matters. low_count fires when values drop; high_count fires on spikes. A high score on a traffic stop with low_count is correct detection, not a false positive.

• Explain factors before speculating. Read anomaly_score_explanation from the record. Only address factors that are present and relevant.

• Job config is essential. bucket_span, detector function, custom_rules, use_null, and memory status all affect scores. Inspect job config when a score is surprising.

• Model plot is the most visual explanation. When enabled, show model bounds to illustrate where the actual value falls relative to the expected range.

• For job health ("missing documents", "memory limit", "datafeed not running") use the elasticsearch-anomaly-detection skill.

Domain knowledge

Score types

anomaly_score_explanation factors

The anomaly_score_explanation field on each record breaks the score into components:

Why a score might be unexpectedly low

• high_variance_penalty: The metric is historically noisy — wide confidence bounds absorb the spike.

• Renormalization: A more extreme anomaly appeared later and pushed this score down (initial_record_score >> record_score).

• Insufficient training history: Need ≥3 weeks for weekly seasonality, ≥2 full cycles for any detected period.

• bucket_span too large: Short-duration spikes get smoothed. Use a smaller bucket_span for high-frequency events.

• Detector function mismatch: mean vs high_mean, count vs high_count — only one direction fires.

• incomplete_bucket_penalty: Bucket received less data than expected (ingest latency or gaps).

• custom_rules: A detector filter may be suppressing the anomaly.

Why a score might be unexpectedly high

• Insufficient history: Model hasn't learned the normal pattern yet — early anomalies are unreliable.

• Model split thin: High-cardinality partition_field or by_field → very few points per entity → unreliable probabilities.

• use_null: If use_null: true, missing entities produce "null" anomalies that may not be meaningful.

• Absence / drop detection: With low_count or low_mean, actual << typical produces a legitimately high score — treat as a real incident, not a false positive.

Model behavior concepts

Model plot and categories

• Model plot: Shows the model's learned upper and lower bounds at each time point. If actual is within bounds, no anomaly; if outside, the score depends on the distance from bounds. Only available when model_plot_config is enabled on the job. Query via POST /.ml-anomalies-*/_search with result_type: model_plot.

• Categories: For jobs with a categorization_field_name, query result_type: category_definition to show log message patterns (terms, regex, examples per category_id). Anomaly records use by_field_value = <category_id>.

Score troubleshooting protocol

1. List jobs — Call GET /_ml/anomaly_detectors when the job ID is unknown.

2. Get job config and stats — Call GET /_ml/anomaly_detectors/{job_id} and GET /_ml/anomaly_detectors/{job_id}/_stats. Verify bucket_span, detector function, custom_rules, use_null, job state (opened/closed/failed), and model_size_stats.memory_status.

3. Retrieve the record — Call POST /.ml-anomalies-*/_search with result_type: record, the job ID, time range, and optional minimum record_score. Inspect initial_record_score, record_score, actual, typical, function, multi_bucket_impact, and anomaly_score_explanation.

4. Check renormalization — Compare initial_record_score vs record_score. If initial >> record, re-query records sorted by timestamp and compute score_drift to quantify renormalization across the job.

5. Visualize model bounds — If model_plot_config is enabled, query result_type: model_plot and show where the actual value fell relative to model_lower and model_upper.

6. Influencers — Query result_type: influencer for the anomaly bucket time range; sort by influencer_score.

7. Explain factors — From the record's anomaly_score_explanation, address each present relevant factor: high_variance_penalty, incomplete_bucket_penalty, anomaly_length, single_bucket_impact, multi_bucket_impact. Do not cite factors absent from the record.

Examples

• "Why is my anomaly score only 15 when the spike looks huge?" → Check renormalization: initial_record_score (~92) >> record_score (~15). The spike was real; the current score was rescaled down. Use the initial score for alerting.

• "Traffic stopped and I got a HIGH score — false positive?" → No. low_count with actual far below typical is legitimate absence detection. Investigate the outage.

• "Which entities contributed most to the anomalies in job X last night?" → Query influencers for the time range.

• "Show me the model bounds for this job." → Query model plot when model_plot_config is enabled.

• "List records where the score was renormalized down a lot." → Records sorted by timestamp; filter large initial_record_score − record_score.

Guidelines

• Report only what the API or the user-supplied record contains; do not invent scores, timestamps, entity values, or explanation factors.

• Always show both initial_record_score and record_score when explaining a record; state explicitly whether renormalization occurred.

• When a score factor is missing from the record, do not assert it; note that the field was not returned.

• Do not attribute low scores to high_variance_penalty or incomplete_bucket_penalty when those flags are false or absent in the record.

• For investigation ("what caused this?", "which service is responsible?") query influencers or construct cross-job searches via references/explainer-reference.md.

• For job health use the elasticsearch-anomaly-detection skill.

Operations

Query body shapes for each result_type (record, influencer, model_plot, category_definition) are documented in references/explainer-reference.md.