Generated banner for Mlkit Agent API
Generated logo for Mlkit Agent API

Mlkit Agent API

Android (Google)OfficialCreated by Google LLCDistributed by Android (Google)

  • Skill
  • Tool integration
  • Data and AI
  • Apache-2.0
  • No issues found

From the author

Builds autonomous on-device agentic workflows in Android apps using the ML Kit GenAI Agent API powered by Gemini Nano and AICore. Use this skill when implementing conversational agents, multi-turn chat with AgentSession in a ViewModel, autonomous tool calling using Android AppFunctions (@AppFunctionDeclaration and AppFunctionAgentTools.discover), ephemeral single-turn execution with runner.runSingleTurnAsync, or handling GenAiException error resilience. It solves the problem of orchestrating local reasoning loops, typed tool dispatch, and session persistence on-device with zero cloud latency and privacy preservation. Don't use for cloud-based Gemini models or Vertex AI, single-turn text generation without agent loops (suggest Prompt API), or non-Android platforms.
— the description by Google LLC

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The ML Kit GenAI Agent API enables developers to create autonomous agentic workflows directly on Android devices using Gemini Nano and AICore. This skill provides patterns for implementing conversational agents, managing multi-turn chat sessions within ViewModels, and integrating local tools via Android AppFunctions. It focuses on orchestrating reasoning loops, handling typed tool dispatch, and managing session persistence while ensuring proper error handling for on-device AI operations.

What it does

  • Implements multi-turn conversational agents using AgentSession.
  • Integrates local tools through Android AppFunction declarations.
  • Supports ephemeral single-turn execution for quick tasks.
  • Provides patterns for session lifecycle management in Android ViewModels.
  • Handles GenAiException for robust error resilience.
  • Enables advanced tool discovery and decorator patterns for custom logic.

When to use it

  • When building conversational agents that require on-device reasoning.
  • When you need to integrate local app functions as tools for an AI agent.
  • When managing complex multi-turn dialogues that must persist across configuration changes.
  • When performing one-off tasks like summarization or data formatting without session history.

Advantages

  • Enables on-device agentic workflows without requiring cloud connectivity.
  • Leverages standard Android AppFunctions for tool definitions.
  • Provides structured lifecycle management for conversational sessions.
  • Supports deterministic tool selection via greedy decoding.

Usage examples

“How do I initialize an agent for a chat interface in an Android ViewModel?”
The agent is initialized using Agent.builder, wrapped in an AgentRunner, and the session is managed within the ViewModel to ensure persistence and proper cleanup.
“How can I register a local function as a tool for my agent?”
Annotate the function with @AppFunctionDeclaration and enable AppFunctions in the Agent.Builder to allow the agent to discover and call the tool.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/android/mlkit-agent-api/download
  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip mlkit-agent-api.zip -d .agents/skills

Documentation by Google LLC

Targeting AICore Developer Preview. This skill provides authoritative execution patterns and failure-prevention rules for on-device agentic workflows in Android apps.


1. Prerequisites and setup

Gradle dependencies
// app/build.gradle.kts
dependencies {
    // Core ML Kit GenAI Agent API
    implementation("com.google.mlkit:genai-agent:1.0.0-alpha1")

    // Android AppFunctions dependency for on-device tool declarations
    implementation("androidx.appfunctions:appfunctions:1.0.0-alpha13")
}

[!IMPORTANT] Always use official SDK interfaces from com.google.mlkit.genai.agent.*, androidx.appfunctions.*, com.google.mlkit.genai.prompt.*, and com.google.mlkit.genai.common.*. Never declare local placeholder interfaces or fallback mocks.


2. Minimal agent and runner initialization

Configure an Agent with Agent.builder and wrap execution in AgentRunner:

import com.google.mlkit.genai.agent.Agent
import com.google.mlkit.genai.agent.AgentRunner

val agent = Agent.builder()
    .setName("OnDeviceAssistant")
    .setInstructions("You are a helpful on-device assistant.")
    .build()

val runner = AgentRunner(agent)
  • Agent.builder defaults to maxTurns = 10 automatically to prevent infinite tool loops.

  • AgentRunner defaults to greedy decoding (temperature = 0.0f) for deterministic tool selection and schema compliance.

  • AgentRunner(agent) automatically resolves application context using MlKitContext for AppFunction execution in Android apps. An explicit AgentRunner(context, agent) overload is available for headless tests or custom context scopes.


3. Multi-turn chat and lifecycle (ViewModel)

AgentSession preserves conversational history in memory and is not Parcelable . To preserve dialogue history across Android configuration changes (such as screen rotation), retain session inside an AndroidX ViewModel.

When executing single-turn ephemeral queries, managing concurrent multi-sessions, or requiring explicit RAM cleanup, you must follow the guidelines in Session lifecycle.

package com.example.app

import androidx.lifecycle.ViewModel
import androidx.lifecycle.viewModelScope
import com.google.mlkit.genai.agent.Agent
import com.google.mlkit.genai.agent.AgentRunner
import com.google.mlkit.genai.agent.AgentSession
import com.google.mlkit.genai.agent.textTokens
import com.google.mlkit.genai.common.GenAiException
import com.google.mlkit.genai.prompt.Content
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.asStateFlow
import kotlinx.coroutines.launch

class ChatViewModel : ViewModel() {
    private val agent = Agent.builder()
        .setInstructions("You are a helpful on-device assistant.")
        .build()
    private val runner = AgentRunner(agent)
    private val session: AgentSession =
        runner.getOrCreateSession("chat_session")

    private val _uiState = MutableStateFlow<ChatUiState>(ChatUiState.Idle)
    val uiState: StateFlow<ChatUiState> = _uiState.asStateFlow()

    fun sendMessage(prompt: String) {
        viewModelScope.launch {
            _uiState.value = ChatUiState.Loading("")
            try {
                // Content.builder() required; runAsync takes Content
                val content = Content.builder().text(prompt).build()
                var fullResponse = ""
                session.runAsync(content)
                    // Extension function requiring textTokens import
                    .textTokens()
                    .collect { token ->
                        fullResponse += token
                        _uiState.value = ChatUiState.Loading(fullResponse)
                    }
                _uiState.value = ChatUiState.Success(fullResponse)
            } catch (e: GenAiException) {
                // Catch GenAiException to prevent app crashes on AI errors
                val err = e.localizedMessage ?: "AI error (${e.errorCode})"
                _uiState.value = ChatUiState.Error(err)
            }
        }
    }

    override fun onCleared() {
        super.onCleared()
        // Evict conversation session from RAM when ViewModel is destroyed
        session.close()
    }
}
Essential lifecycle and coroutine rules
  1. Content object : session.runAsync and runner.runSingleTurnAsync require a Content object. Always build it using Content.builder().text(userPrompt).build().

  2. textTokens extension : textTokens is a Kotlin extension function on Flow<AgentEvent>. Always include import com.google.mlkit.genai.agent.textTokens.

  3. Session cleanup : Always invoke session.close in ViewModel.onCleared to evict conversational RAM and cancel pending tasks.

  4. AgentEvent stream : When collecting Flow<AgentEvent> directly instead of textTokens, always include import com.google.mlkit.genai.agent.AgentEvent. Match against AgentEvent.TextMessage(text, isPartial), AgentEvent.FunctionCall(name, args), and AgentEvent.FunctionResponse(name, response), and always include an else -> {} branch in when (event) expressions (AgentEvent is non-exhaustive for forward compatibility):

    import com.google.mlkit.genai.agent.AgentEvent

    session.runAsync(content).collect { event -> when (event) { is AgentEvent.FunctionCall -> { _uiState.update { it.copy(activeToolStatus = "Tool started: ${event.name}") } } is AgentEvent.FunctionResponse -> { _uiState.update { it.copy(activeToolStatus = "Tool completed: ${event.name}") } } is AgentEvent.TextMessage -> { _uiState.update { it.copy(responseText = it.responseText + event.text) } } else -> {} // Required: AgentEvent is non-exhaustive for forward compat } }


4. Tool calling recipe

All tools used with Agent originate from Android AppFunctions (@AppFunctionDeclaration), registered using the declarative path (enableAppFunctions(true) or addAppFunctionIds(...)) or advanced discovery (AppFunctionAgentTools.discover(...)).

When you need to restrict caller access levels for private in-app tools or wrap discovered AppFunctions with human-in-the-loop (HITL) confirmation decorators, you must consult Tools and functions.

Option A: Declarative AppFunctions path

Define tools declaratively in your app using AndroidX AppFunctions:

package com.example.app.tools

import androidx.appfunctions.AppFunctionDeclaration
import androidx.appfunctions.AppFunctionSerializable

@AppFunctionSerializable
data class CalendarEvents(val date: String, val events: List<String>)

class CalendarTools {
    /**
     * Fetches scheduled calendar appointments for a specific date in
     * YYYY-MM-DD format.
     *
     * @param date Date string in YYYY-MM-DD format
     */
    @AppFunctionDeclaration(isDescribedByKDoc = true)
    fun fetch_calendar(
        date: String
    ): CalendarEvents {
        return CalendarEvents(
            date = date,
            events = listOf(
                "Team Standup at 10:00 AM",
                "Product Review at 2:00 PM"
            )
        )
    }
}

Enable AppFunctions on Agent.Builder and initialize AgentRunner:

import com.google.mlkit.genai.agent.Agent
import com.google.mlkit.genai.agent.AgentRunner

val agent = Agent.builder()
    .setName("CalendarAssistant")
    .setInstructions("You assist with calendar queries.")
    // Automatically discovers AppFunctions in your app package
    .enableAppFunctions(true)
    // Optional: scope to specific function IDs or method names
    .addAppFunctionIds("fetch_calendar")
    .build()

val runner = AgentRunner(agent)
  • AppFunction Contract : Tool methods take domain parameters directly (no framework context parameter required). Returns must be primitives or @AppFunctionSerializable data classes (Map<String, Any?> isn't supported for AppFunctions).

  • Restricting Callers with ACCESS_LEVEL_SELF : AppFunctions don't have to be public to external callers. To restrict execution strictly to your app's internal agent (blocking external system assistants and IPC callers), annotate with @OptIn(ExperimentalAppFunctionsApi::class) and @AppFunctionAccessLevel(level = AppFunctionMetadata.ACCESS_LEVEL_SELF, isCompatEnforcementEnabled = true).

Option B: Advanced discovery and decorator pattern

For inspecting, filtering, or decorating discovered AppFunctions (such as Human-in-the-Loop confirmation wrappers using AgentTool(delegate)) before registering with the agent:

import com.google.mlkit.genai.agent.Agent
import com.google.mlkit.genai.agent.AgentRunner
import com.google.mlkit.genai.agent.AgentTool
import com.google.mlkit.genai.agent.AppFunctionAgentTools

// Discover tools (optionally scoped to function IDs)
val discoveredTools = AppFunctionAgentTools.discover(
    enabledAppFunctionIds = listOf("fetch_calendar")
)

// Optional: Wrap sensitive tools using AgentTool(delegate) decorator
class HitlConfirmationTool(
    delegate: AgentTool,
    private val onConfirm: suspend (
        toolName: String,
        args: Map<String, Any>
    ) -> Boolean
) : AgentTool(delegate) {
    override suspend fun execute(args: Map<String, Any>): Map<String, Any?> {
        if (!onConfirm(name, args)) {
            return mapOf("status" to "CANCELLED_BY_USER")
        }
        return checkNotNull(delegate).execute(args)
    }
}

val agent = Agent.builder()
    .setName("CalendarAssistant")
    .setInstructions("You assist with calendar queries.")
    .addTools(discoveredTools)
    .build()

val runner = AgentRunner(agent)

[!WARNING] Mutual exclusivity : Configure AppFunctions using either the declarative path (enableAppFunctions(true) on Agent.Builder) or the advanced path (AppFunctionAgentTools.discover passed to addTools), but not both. Using both simultaneously triggers a warning and registers duplicate tool schemas.

Deprecated annotations (@Tool and @Param)

The legacy @Tool and @Param annotations in com.google.mlkit.genai.schema.annotations.*, along with AgentTool.fromAnnotatedObject, Agent.Builder.addTools(annotatedObject: Any), and standalone FunctionTool, have been deprecated or removed. All tools added to Agent must originate from @androidx.appfunctions.AppFunctionDeclaration using Agent.Builder.enableAppFunctions(true) or AppFunctionAgentTools.discover.


5. Ephemeral single-turn execution

For one-off tasks where conversation history isn't needed (such as summarizing text, formatting data, or answering standalone questions), invoke runner.runSingleTurnAsync(content) directly:

import com.google.mlkit.genai.agent.AgentRunner
import com.google.mlkit.genai.agent.textTokens
import com.google.mlkit.genai.prompt.Content
import kotlinx.coroutines.flow.Flow

// Executes a single-turn query ephemerally without session persistence:
fun askQuickQuestion(runner: AgentRunner, question: String): Flow<String> {
    val content = Content.builder().text(question).build()
    return runner.runSingleTurnAsync(content).textTokens()
}
  • runner.runSingleTurnAsync(content) automatically creates and evicts an ephemeral session upon Flow completion.

  • Cleanup reference :

    • session.close: Releases session resources and cancels coroutines (call in ViewModel.onCleared).

    • session.delete or runner.deleteSession(sessionId): Permanently deletes the session from the runner.


6. Error handling and GenAiException reference

Runtime failures during on-device execution (such as model unavailability or exceeding maximum turn limits) are emitted or thrown as GenAiException:

import com.google.mlkit.genai.common.GenAiException
Key error codes
  • GenAiException.ErrorCode.MAX_TURNS_EXCEEDED (-112): The agent exceeded maxTurns (default 10) without completing the reasoning loop.

  • GenAiException.ErrorCode.NOT_AVAILABLE (8): The Gemini Nano model or AICore service isn't available on this device.

  • GenAiException.ErrorCode.BUSY (9): System or AICore is busy, or battery or thermal quota has been reached.

Files

LICENSE


                                 Apache License
                           Version 2.0, January 2004
                        http://www.apache.org/licenses/

   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

   1. Definitions.

      "License" shall mean the terms and conditions for use, reproduction,
      and distribution as defined by Sections 1 through 9 of this document.

      "Licensor" shall mean the copyright owner or entity authorized by
      the copyright owner that is granting the License.

      "Legal Entity" shall mean the union of the acting entity and all
      other entities that control, are controlled by, or are under common
      control with that entity. For the purposes of this definition,
      "control" means (i) the power, direct or indirect, to cause the
      direction or management of such entity, whether by contract or
      otherwise, or (ii) ownership of fifty percent (50%) or more of the
      outstanding shares, or (iii) beneficial ownership of such entity.

      "You" (or "Your") shall mean an individual or Legal Entity
      exercising permissions granted by this License.

      "Source" form shall mean the preferred form for making modifications,
      including but not limited to software source code, documentation
      source, and configuration files.

      "Object" form shall mean any form resulting from mechanical
      transformation or translation of a Source form, including but
      not limited to compiled object code, generated documentation,
      and conversions to other media types.

      "Work" shall mean the work of authorship, whether in Source or
      Object form, made available under the License, as indicated by a
      copyright notice that is included in or attached to the work
      (an example is provided in the Appendix below).

      "Derivative Works" shall mean any work, whether in Source or Object
      form, that is based on (or derived from) the Work and for which the
      editorial revisions, annotations, elaborations, or other modifications
      represent, as a whole, an original work of authorship. For the purposes
      of this License, Derivative Works shall not include works that remain
      separable from, or merely link (or bind by name) to the interfaces of,
      the Work and Derivative Works thereof.

      "Contribution" shall mean any work of authorship, including
      the original version of the Work and any modifications or additions
      to that Work or Derivative Works thereof, that is intentionally
      submitted to Licensor for inclusion in the Work by the copyright owner
      or by an individual or Legal Entity authorized to submit on behalf of
      the copyright owner. For the purposes of this definition, "submitted"
      means any form of electronic, verbal, or written communication sent
      to the Licensor or its representatives, including but not limited to
      communication on electronic mailing lists, source code control systems,
      and issue tracking systems that are managed by, or on behalf of, the
      Licensor for the purpose of discussing and improving the Work, but
      excluding communication that is conspicuously marked or otherwise
      designated in writing by the copyright owner as "Not a Contribution."

      "Contributor" shall mean Licensor and any individual or Legal Entity
      on behalf of whom a Contribution has been received by Licensor and
      subsequently incorporated within the Work.

   2. Grant of Copyright License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      copyright license to reproduce, prepare Derivative Works of,
      publicly display, publicly perform, sublicense, and distribute the
      Work and such Derivative Works in Source or Object form.

   3. Grant of Patent License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      (except as stated in this section) patent license to make, have made,
      use, offer to sell, sell, import, and otherwise transfer the Work,
      where such license applies only to those patent claims licensable
      by such Contributor that are necessarily infringed by their
      Contribution(s) alone or by combination of their Contribution(s)
      with the Work to which such Contribution(s) was submitted. If You
      institute patent litigation against any entity (including a
      cross-claim or counterclaim in a lawsuit) alleging that the Work
      or a Contribution incorporated within the Work constitutes direct
      or contributory patent infringement, then any patent licenses
      granted to You under this License for that Work shall terminate
      as of the date such litigation is filed.

   4. Redistribution. You may reproduce and distribute copies of the
      Work or Derivative Works thereof in any medium, with or without
      modifications, and in Source or Object form, provided that You
      meet the following conditions:

      (a) You must give any other recipients of the Work or
          Derivative Works a copy of this License; and

      (b) You must cause any modified files to carry prominent notices
          stating that You changed the files; and

      (c) You must retain, in the Source form of any Derivative Works
          that You distribute, all copyright, patent, trademark, and
          attribution notices from the Source form of the Work,
          excluding those notices that do not pertain to any part of
          the Derivative Works; and

      (d) If the Work includes a "NOTICE" text file as part of its
          distribution, then any Derivative Works that You distribute must
          include a readable copy of the attribution notices contained
          within such NOTICE file, excluding those notices that do not
          pertain to any part of the Derivative Works, in at least one
          of the following places: within a NOTICE text file distributed
          as part of the Derivative Works; within the Source form or
          documentation, if provided along with the Derivative Works; or,
          within a display generated by the Derivative Works, if and
          wherever such third-party notices normally appear. The contents
          of the NOTICE file are for informational purposes only and
          do not modify the License. You may add Your own attribution
          notices within Derivative Works that You distribute, alongside
          or as an addendum to the NOTICE text from the Work, provided
          that such additional attribution notices cannot be construed
          as modifying the License.

      You may add Your own copyright statement to Your modifications and
      may provide additional or different license terms and conditions
      for use, reproduction, or distribution of Your modifications, or
      for any such Derivative Works as a whole, provided Your use,
      reproduction, and distribution of the Work otherwise complies with
      the conditions stated in this License.

   5. Submission of Contributions. Unless You explicitly state otherwise,
      any Contribution intentionally submitted for inclusion in the Work
      by You to the Licensor shall be under the terms and conditions of
      this License, without any additional terms or conditions.
      Notwithstanding the above, nothing herein shall supersede or modify
      the terms of any separate license agreement you may have executed
      with Licensor regarding such Contributions.

   6. Trademarks. This License does not grant permission to use the trade
      names, trademarks, service marks, or product names of the Licensor,
      except as required for reasonable and customary use in describing the
      origin of the Work and reproducing the content of the NOTICE file.

   7. Disclaimer of Warranty. Unless required by applicable law or
      agreed to in writing, Licensor provides the Work (and each
      Contributor provides its Contributions) on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
      implied, including, without limitation, any warranties or conditions
      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
      PARTICULAR PURPOSE. You are solely responsible for determining the
      appropriateness of using or redistributing the Work and assume any
      risks associated with Your exercise of permissions under this License.

   8. Limitation of Liability. In no event and under no legal theory,
      whether in tort (including negligence), contract, or otherwise,
      unless required by applicable law (such as deliberate and grossly
      negligent acts) or agreed to in writing, shall any Contributor be
      liable to You for damages, including any direct, indirect, special,
      incidental, or consequential damages of any character arising as a
      result of this License or out of the use or inability to use the
      Work (including but not limited to damages for loss of goodwill,
      work stoppage, computer failure or malfunction, or any and all
      other commercial damages or losses), even if such Contributor
      has been advised of the possibility of such damages.

   9. Accepting Warranty or Additional Liability. While redistributing
      the Work or Derivative Works thereof, You may choose to offer,
      and charge a fee for, acceptance of support, warranty, indemnity,
      or other liability obligations and/or rights consistent with this
      License. However, in accepting such obligations, You may act only
      on Your own behalf and on Your sole responsibility, not on behalf
      of any other Contributor, and only if You agree to indemnify,
      defend, and hold each Contributor harmless for any liability
      incurred by, or claims asserted against, such Contributor by reason
      of your accepting any such warranty or additional liability.

   END OF TERMS AND CONDITIONS

   APPENDIX: How to apply the Apache License to your work.

      To apply the Apache License to your work, attach the following
      boilerplate notice, with the fields enclosed by brackets "[]"
      replaced with your own identifying information. (Don't include
      the brackets!)  The text should be enclosed in the appropriate
      comment syntax for the file format. We also recommend that a
      file or class name and description of purpose be included on the
      same "printed page" as the copyright notice for easier
      identification within third-party archives.

   Copyright [yyyy] [name of copyright owner]

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
Google LLC
Publisher
Android (Google) (Official)
Distributed by
Android (Google)
Source
android/skills · device-ai/mlkit-agent-api · commit bf68e2d
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE.txt)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by Google LLC or Android (Google).

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download mlkit-agent-api.zip