Kibana Context Engine

Query Context Engine AI Indices

<!-- begin-partial: preamble -->

Environment Configuration

This skill executes Elasticsearch operations through the elastic CLI. If the elastic CLI is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., GET /, GET /_cat/indices, GET /{index}/_mapping, GET /{index}/_settings/index.mode, POST /_query). The Operations table at the end of this document maps each shorthand to the equivalent elastic CLI command — always use the CLI rather than calling the HTTP API directly.

<!-- end-partial: preamble -->

Concepts

AI Index — an Elasticsearch index named ai-index-idx-*, or a data stream named ai-index-ds-*, registered with the Context Engine under a logical id. The registration lives in Kibana; the knowledge lives in the backing index. Only the list operation knows which ai-index-* indices are registered and what to put in FROM; the index name alone does not tell you.

Knowledge Indicator (KI) — one document in an AI Index: context prepared for agents, such as a description of some data, a summary, an access pattern, a runnable query, or a record of a Kibana resource. A KI may answer a question outright, or help you locate and use the source that does.

A KI is retired by a tombstone (governance.lifecycle.status: deleted) or by an expires_at in the past. On a data stream, one KI is several revisions under the same id, the newest of which is its current state. The query API returns only current, active, unexpired KIs unless the query asks otherwise (see Lifecycle).

Search the AI Indices before broader retrieval whenever their KIs may cover the question. This skill reads KIs; generating them belongs to the Context Engine's automations and feedback loop.

Three read-only Context Engine HTTP APIs back this skill. The Operations table maps each to its elastic CLI command:

Use them in this order: list → describe → query. Never guess an index id, a FROM target, or a field name; describe supplies all three. Nothing else can read an AI Index correctly: a generic ES|QL query (POST /_query) or index listing neither knows which indices are registered nor applies the space scoping.

Every call is scoped to one space, and the space comes from the request rather than from anything written into the query. Point the request at another space to read that space's KIs.

The canonical KI schema

Elastic's canonical KI indices share these fields. Other AI Indices differ, so treat this as a starting shape and let describe settle the fields for any given index.

Semantic sub-fields use dot notation: title.semantic, description.semantic, content.semantic. They exist only where the mapping defines them; the describe block's Semantic fields section lists the real ones.

attributes.esql — a KI's runnable ES|QL

When a KI carries runnable ES|QL, it lives in attributes.esql. That is the canonical location; do not look elsewhere in attributes. The field is optional, so a query must tolerate its absence. The same access pattern is also presented as prose inside content — templates, params, and returns — so reach for content when a KI has no attributes.esql, or when the description of the query matters more than the query itself.

attributes is flattened, so extract the value with FIELD_EXTRACT before filtering or keeping it:

FROM <dest.value>
| WHERE governance.lifecycle.status IS NULL OR governance.lifecycle.status == "active"
| WHERE expires_at IS NULL OR expires_at > NOW()
| WHERE type == "detection"
| EVAL esql = FIELD_EXTRACT(attributes, "esql")
| WHERE esql IS NOT NULL
| KEEP title, description, esql
| LIMIT 5

A KI may store an array of queries, making the extracted value multivalued. IS NOT NULL still filters correctly, but scalar comparisons like == skip multivalued rows. Add | MV_EXPAND esql after the IS NOT NULL to get one row per query.

Flow

Step 1 — List the AI indices

Call GET kbn:/api/context_engine/ai_index. Each entry gives:

• id, to pass to describe.

• dest.value, the exact string to put after FROM. Use it verbatim: it differs from the id (sales-knowledge → ai-index-idx-sales-knowledge). dest.type says whether it is an index or a data_stream.

• description and managed, to choose between entries.

The list is scoped to the caller's space and to what the caller can read. An entry whose backing index is empty or does not exist yet is still listed; an index the caller lacks read on is silently left out.

Skip this step when your instructions already name the index. Step 2 still applies: describe supplies the FROM target and the field names either way.

Step 2 — Describe the one you picked

Call GET kbn:/api/context_engine/ai_index/{aiIndexId}/_describe with the id from Step 1. The response is a text block meant to be read, not parsed: no regexing it into structured data. It contains:

• the Query with ES|QL against: line (the FROM target),

• Fields, every mapped field as path: type plus searchable / aggregatable (capped at 500; a path mapped inconsistently across matched indices shows as conflict),

• Semantic fields, the searchable semantic_text fields, when any,

• Knowledge item types and Tags, the top 20 KI type / tags values by document count in this space, active and unexpired only,

• Example queries, three ready ES|QL shapes, each opening with the lifecycle pipeline for the index's destination type.

The block is capped at ~16k tokens. Read it and use it; do not echo it back to the user.

Step 3 — Query

Call POST kbn:/api/context_engine/ai_index/_query with { query, params?, limit? }. Build the ES|QL from the describe block's field list, starting from its example queries, and pass user input as named parameters (?query, ?type, ?tag) rather than interpolating it into the query string. The response is { columns, values }; summarize it for the user rather than dumping raw rows.

What the server owns and you cannot override:

There is no time_range and no filter parameter. Express time windows in the ES|QL itself (WHERE @timestamp >= NOW() - 24 hours) or through params.

Lifecycle

Open every query with the two lifecycle filters, unless reading retired KIs, and end row-returning queries in KEEP (aggregations end in STATS): the query then reads as what it returns without pulling governance.* into context.

| WHERE governance.lifecycle.status IS NULL OR governance.lifecycle.status == "active"
| WHERE expires_at IS NULL OR expires_at > NOW()

The query API adds these two filters and DROP governance.* to any query that omits them, and on a data stream always keeps only the newest revision per id (adding METADATA _id, _index when missing). A top-level WHERE on a lifecycle field switches off that field's filter only; showing the field in KEEP does not. Any governance.* column named anywhere in the query, the opening status filter included, switches off the DROP, so a query that opens with the two filters returns governance.* columns unless it ends in KEEP or STATS. An index that is not registered as an AI Index is read as-is.

Examples

These run as-is against canonical KI indices. For any other AI Index, adapt the field names from the describe block: title.semantic does not exist unless the mapping has it.

Full-text search, lexical and semantic fused

FROM ai-index-idx-sales-knowledge METADATA _id, _index, _score
| WHERE governance.lifecycle.status IS NULL OR governance.lifecycle.status == "active"
| WHERE expires_at IS NULL OR expires_at > NOW()
| FORK
    ( WHERE MATCH(title, ?query) OR MATCH(description, ?query) OR MATCH(content, ?query) | SORT _score DESC | LIMIT 20 )
    ( WHERE MATCH(title.semantic, ?query) OR MATCH(description.semantic, ?query) OR MATCH(content.semantic, ?query) | SORT _score DESC | LIMIT 20 )
| FUSE
| SORT _score DESC, _id ASC
| KEEP title, description, content, type, tags
| LIMIT 5

Called with the search phrase as a parameter:

{ "query": "<the esql above>", "params": { "query": "refund policy for annual plans" }, "limit": 5 }

Filter by KI type and tag

FROM ai-index-idx-sales-knowledge
| WHERE governance.lifecycle.status IS NULL OR governance.lifecycle.status == "active"
| WHERE expires_at IS NULL OR expires_at > NOW()
| WHERE type == ?type AND MATCH(tags, ?tag)
| KEEP title, description, content, type, tags
| LIMIT 5

Count KIs by type

A first probe to confirm the index holds KIs visible from this space:

FROM ai-index-idx-sales-knowledge
| WHERE governance.lifecycle.status IS NULL OR governance.lifecycle.status == "active"
| WHERE expires_at IS NULL OR expires_at > NOW()
| STATS count = COUNT(*) BY type
| SORT count DESC
| LIMIT 20

Read retired KIs, only when asked

Only when the user asks about deleted or expired knowledge. Tombstoned and expired KIs stay readable, on an index or a data stream:

• Tombstoned: a WHERE on governance.lifecycle.status turns off the status filter and, being a governance.* column, the DROP too; the expiry filter still applies.

• Expired: a WHERE on expires_at turns off the expiry filter; the status filter still applies.

Nothing else is readable: a KI whose document was removed is gone, and older revisions on a data stream are never returned.

FROM ai-index-idx-sales-knowledge
| WHERE governance.lifecycle.status == "deleted"
| KEEP title, governance.provenance.created_by.uri
| LIMIT 20

On a data stream, the same query returns the KIs whose newest revision is a tombstone:

FROM ai-index-ds-sales-knowledge
| WHERE governance.lifecycle.status == "deleted"
| KEEP title, governance.provenance.created_by.uri
| LIMIT 20

Diagnosing

These failure modes look alike from the outside and have different fixes. Work down the table.

Guidelines

• Never skip describe. A query written from memory could fail, or silently returns nothing.

• Read AI Indices only through POST kbn:/api/context_engine/ai_index/_query. A direct Elasticsearch query or a cluster-wide ES|QL query (POST /_query) bypasses the space scoping, the lifecycle filter, and the row cap: it returns every revision, deleted and expired ones included. Consult the elasticsearch-esql skill for ES|QL syntax.

• Start at limit 5 and narrow the columns with KEEP. The whole result enters your context.

• If the top 5 don't answer the question, check how they fall short: on topic but incomplete → raise the limit; off topic → the query is wrong, so reword the search terms or loosen the WHERE rather than asking for more rows.

• Answer from the rows and cite the KI titles; do not dump the raw result.

• An empty result means no current KI. Do not read tombstones or expired KIs unless the user asks about deleted or expired knowledge.

Out of scope

• Creating, updating, or deleting AI indices, and managing their sources / automations / feedback_analysis config.

• Generating Knowledge Indicators, or deciding what an AI Index should contain. That is the Context Engine's automations and its analyze-and-improve feedback loop, not query time.

Operations