Mongodb Search And AI

Lexical Search - Indexing

This guide covers how to configure MongoDB Atlas Search indexes. Use this reference to build index definitions with proper field types, analyzers, mappings, and optimization settings.

Table of Contents

• Atlas Search Index Definition

• Analyzer Selection

• Field Types

• Dynamic vs Explicit Mappings

• Stored Source

• Synonyms

Atlas Search Index Definition

Syntax

{
  "analyzer": "<analyzer-for-index>",
  "searchAnalyzer": "<analyzer-for-query>",
  "mappings": {
    "dynamic": <boolean> | {
      "typeSet": "<typeSet-name>"
    },
    "fields": {
      <field-definition>
    }
  },
  "numPartitions": <integer>,
  "analyzers": [ <custom-analyzer> ],
  "storedSource": <boolean> | {
    <stored-source-definition>
  },
  "synonyms": [
    {
      <synonym-mapping-definition>
    }
  ],
  "typeSets": [
    {
      "types": [
        {<field-types-definition>}
      ]
    }
  ]
}

Options

Basic Definition

Most indexes only need the mappings configuration:

{
  "mappings": {
    "dynamic": <boolean> | { <typeSet-definition> },
    "fields": { <field-definition> }
  }
}

Analyzer Selection

The analyzer determines how text is processed for indexing and searching.

Default behavior: Most queries don't specify an analyzer and use MongoDB's default standard analyzer, which:

• Divides text into terms based on word boundaries (language-neutral)

• Converts terms to lowercase and removes punctuation

• Recognizes email addresses, acronyms, CJK characters, alphanumerics, and more

Common built-in analyzers:

Index Analyzer (Applied at Index Time)

Specify per-field or top-level for all fields:

// Field-level (add to mappings.fields):
{
  "title": {
    "type": "string",
    "analyzer": "lucene.standard"  // Or omit to use default
  },
  "category": {
    "type": "token"   // Exact matching — use token, not string with lucene.keyword
  }
}

// Top-level (applies to all fields unless overridden):
{
  "analyzer": "lucene.standard",
  "mappings": {
    "fields": {
      "title": { "type": "string" }  // Uses top-level analyzer
    }
  }
}

Search Analyzer (Applied at Query Time)

Apply different analysis to queries than to indexed content:

// Top-level searchAnalyzer:
{
  "searchAnalyzer": "lucene.simple",  // Query-time analyzer
  "mappings": {
    "fields": {
      "description": {
        "type": "string",
        "analyzer": "lucene.standard"  // Index-time analyzer
      }
    }
  }
}

Use case: Index with standard analysis, but search with simpler/synonym-aware analyzer.

If omitted, uses the index analyzer. If both omitted, defaults to lucene.standard.

Multi Analyzer (Alternate Analyzers for Same Field)

Index the same field with multiple analyzers:

// Field configuration (add to mappings.fields):
{
  "title": {
    "type": "string",
    "analyzer": "lucene.standard",  // Default analyzer
    "multi": {
      "keywordAnalyzer": {
        "type": "string",
        "analyzer": "lucene.keyword"  // Alternate analyzer
      }
    }
  }
}

Use case: Support both fuzzy matching and exact matching on the same field.

To query using the alternate analyzer, specify the path as fieldName.alternateAnalyzerName (e.g., title.keywordAnalyzer).

Custom Analyzers

Define custom tokenization and filtering:

// Index definition with custom analyzer:
{
  "analyzers": [
    {
      "name": "customAnalyzer",
      "tokenizer": {
        "type": "standard"
      },
      "charFilters": [],
      "tokenFilters": [
        { "type": "lowercase" },
        { "type": "stop", "tokens": ["the", "a", "an"] }
      ]
    }
  ],
  "mappings": {
    "fields": {
      "content": {
        "type": "string",
        "analyzer": "customAnalyzer"  // Reference custom analyzer
      }
    }
  }
}

Use case: Need specific tokenization or filtering not provided by built-in analyzers.

Normalizers (Token Type Only)

Normalizers produce a single token (used with token field type):

// Field configuration (add to mappings.fields):
{
  "username": {
    "type": "token",
    "normalizer": "lowercase"  // Options: "lowercase", "none"
  }
}

Normalizers:

• lowercase: Transforms to lowercase, creates single token

• none: No transformation, creates single token

Use case: Exact matching with case normalization for token fields.

Decision Guide

• lucene.standard (or omit): Default for most text fields

• lucene.keyword: Categories, tags

• Language-specific: When you know the content language

• searchAnalyzer: Different analysis for queries vs indexed content (e.g., synonyms)

• multi: Support multiple search patterns on same field

• Custom: Need specific tokenization/filtering logic

• Normalizers: Token fields requiring case normalization

Field Types

Dynamic mapping includes: boolean, date, number, objectId, string, uuid Must configure explicitly: autocomplete, token, geo, embeddedDocuments, vector

Quick Reference

Field definition structure:

{
  "mappings": {
    "fields": {
      "<field-name>": {
        "type": "<field-type>",
        // type-specific options here
      }
    }
  }
}

Multiple types on same field:

"<field-name>": [
  { "type": "string" },
  { "type": "token", "normalizer": "lowercase" }
]

Arrays: MongoDB Search automatically flattens arrays during indexing. Specify only the element type, not that it's an array.

Dynamic vs Explicit Mappings

Choose based on user's needs:

Use dynamic (true) when:

• User is prototyping or exploring data with unknown schema

• Need to get started quickly without defining all fields

• All or most fields need to be searchable

• Accept larger index size and slower performance for convenience

Use explicit (dynamic: false) (recommended for production) when:

• User has completed early stages of prototyping and knows exactly which fields to search

• Performance and index size are priorities

• Schema is stable and well-defined

• Only a subset of fields need to be searchable

Use typeSets (recommended for production) when:

• User wants automatic indexing but with control over which types

• Document schema is dynamic and new fields need to be indexed automatically without an index rebuild

• Want different indexing strategies for different nested documents

• Balance between convenience and performance is important

Dynamic mappings automatically index all fields:

{
  "mappings": {
    "dynamic": true  // Index everything
  }
}

• Pros: Quick setup, works immediately

• Cons: Larger index, slower queries, wastes resources on unused fields

Explicit mappings define exactly what to index:

{
  "mappings": {
    "dynamic": false,  // Only index specified fields
    "fields": {
      "title": { "type": "string" },
      "genre": { "type": "token" }
    }
  }
}

• Pros: Smaller index, faster queries, precise control

• Cons: Requires knowing your schema

Configurable dynamic with typeSets (recommended middle ground):

{
  "mappings": {
    "dynamic": {
      "typeSet": "customTypes"
    },
    "fields": {
      "metadata": {
        "type": "document",
        "dynamic": {
          "typeSet": "metadataTypes"
        }
      }
    }
  },
  "typeSets": [
    {
      "name": "customTypes",
      "types": [
        { "type": "string" },
        { "type": "number" }
      ]
    },
    {
      "name": "metadataTypes",
      "types": [
        {
          "type": "string",
          "analyzer": "lucene.standard"
        }
      ]
    }
  ]
}

• Pros: Automatically indexes specified field types, more control than full dynamic, can configure different typeSets for sub-documents

• Cons: Still indexes all fields of specified types

Recommendation: Use static mappings or a dynamic typeSet (within a specific path, not at the root document level) in production for optimized index size and performance.

Stored Source

Store frequently accessed fields directly in the search index (mongot) to avoid full document lookups from the database. This dramatically improves query performance, especially when filtering or sorting.

Requirements:

• Available on clusters running MongoDB 7.0+

• Stored fields must still be indexed separately to query them

• Retrieve stored fields at query time using returnStoredSource: true (see lexical-search-querying.md)

Syntax:

{
  "storedSource": true | false | {
    "include" | "exclude": ["<field-name>", ...]
  }
}

Options:

true - Store all fields in documents. Not supported if index contains vector type field. Can significantly impact performance.

false - Don't store any fields (default behavior).

{ "include": [...] } - Store only specified fields. MongoDB Search also stores _id by default. List field names or dot-separated paths.

{ "exclude": [...] } - Store all fields except specified ones.

Examples:

Store specific fields:

{
  "mappings": {
    "dynamic": false,
    "fields": {
      "title": { "type": "string" },
      "genre": { "type": "token" },
      "year": { "type": "number" },
      "rating": { "type": "number" }
    }
  },
  "storedSource": {
    "include": ["title", "genre", "year", "rating"]
  }
}

Exclude specific fields:

{
  "storedSource": {
    "exclude": ["largeTextField", "unusedField"]
  }
}

Store all fields:

{
  "storedSource": true
}

When to use:

• Fields used for filtering, sorting, or projection after $search

• Frequently accessed fields in search results

• When avoiding database lookups is critical for performance

When NOT to use:

• Very large text fields (increases index size significantly)

• Fields rarely used in queries

• When index size is a concern

Note: For using stored source at query time, see lexical-search-querying.md. For vector field storage considerations, see vector-search.md.

Synonyms

Use when user wants query expansion with equivalent terms (e.g., "car" also finds "automobile", "vehicle").

Agent Workflow:

1. Ask user to create synonym collection in the same database as their indexed collection.

2. Provide synonym document format based on user's needs:

For bidirectional synonyms (all terms interchangeable):

db.synonyms.insertMany([
  {
    "mappingType": "equivalent",
    "synonyms": ["car", "vehicle", "automobile"]
  },
  {
    "mappingType": "equivalent",
    "synonyms": ["happy", "joyful", "glad"]
  }
])

For one-way synonyms (input maps to synonyms only):

db.synonyms.insertMany([
  {
    "mappingType": "explicit",
    "input": ["pants"],
    "synonyms": ["trousers", "slacks"]
  }
])

3. Add synonym mapping to index definition:

{
  "mappings": {
    "fields": {
      "<field-name>": {
        "type": "string",
        "analyzer": "lucene.standard"  // Note the analyzer
      }
    }
  },
  "synonyms": [
    {
      "name": "<synonym-mapping-name>",
      "analyzer": "lucene.standard",  // Must match field analyzer
      "source": {
        "collection": "<synonym-collection-name>"
      }
    }
  ]
}

Critical Rules:

• Synonym mapping analyzer MUST match the field analyzer being queried

• Only one synonym mapping allowed per index

• Changes to synonym collection auto-update (no reindex needed)

• Works only with text and phrase operators

Example: If user wants "car" to also find "vehicle" and "automobile":

1. Tell user to create collection: db.synonyms.insertOne({ "mappingType": "equivalent", "synonyms": ["car", "vehicle", "automobile"] })

2. Add to index with analyzer matching the field being searched

3. Queries automatically expand (user searches "car", MongoDB Search searches: car OR vehicle OR automobile)

Searching on Views

Requires MongoDB 8.0+. Create Atlas Search indexes on Views to partially index a collection, transform documents, or support incompatible data types.

Note: Programmatic index creation via mongosh/driver methods requires 8.1+. On 8.0, also note that queries must run against the source collection referencing the view's index name. On 8.1+, you can query the view directly.

Supported view stages: $addFields, $set, $match with $expr only.

Key limitations:

• Index names must be unique across source collection and all its views

• No operators producing dynamic results (e.g., $USER_ROLES, $rand)

• Queries return original source documents. Use storedSource to retrieve transformed fields

Example: partial index (filter documents)

db.createView("movies_After2000", "movies", [
  { $match: { $expr: { $gt: ["$released", ISODate("2000-01-01")] } } }
])

db.movies_After2000.createSearchIndex(
  "after2000Index",
  { "mappings": { "dynamic": true } }
)

// 8.1+: query view directly; 8.0: query source collection using index name
db.movies_After2000.aggregate([
  { $search: { index: "after2000Index", text: { path: "title", query: "<query>" } } }
])

Editing a view: Use collMod. MongoDB Search auto-reindexes on view definition changes with no downtime.

Performance: Complex transformations slow performance. For heavy transformations consider a materialized view, or query the source collection directly.

Troubleshooting:

• Index goes FAILED: view is incompatible with Search, or source collection was removed/changed

• Index goes STALE: view's pipeline fails on a document. Index remains queryable while STALE; returns to READY after fixing the document or view definition

• $search is only valid as first stage error: you're on MongoDB 8.0 querying the view directly. Query the source collection instead, or upgrade to 8.1+