Azure AI Projects Py

MicrosoftOfficialCreated by Microsoft

Skill
Tool integration
Software development
MIT
Low risk
Has scripts

From the author

Build AI applications using the Azure AI Projects Python SDK (azure-ai-projects). Use when working with Foundry project clients, creating versioned agents with PromptAgentDefinition, running evaluations, managing connections/deployments/datasets/indexes, or using OpenAI-compatible clients. This is the high-level Foundry SDK - for low-level agent operations, use azure-ai-agents-python skill.

— the description by Microsoft

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The azure-ai-projects-py skill provides an interface for interacting with the Azure AI Projects Python SDK. It enables developers to manage the lifecycle of AI agents, deployments, datasets, and indexes within the Foundry environment. The SDK supports both synchronous and asynchronous operations, integrates with Azure identity management, and provides an OpenAI-compatible client for standard API interactions.

What it does

Create and manage versioned AI agents using PromptAgentDefinition
List and manage model deployments, datasets, and indexes
Execute and manage agent threads, messages, and runs
Access an OpenAI-compatible client for standard chat completions
Configure and run evaluations on AI models
Manage project connections to external services

When to use it

When building AI applications that require integration with Microsoft Foundry
When you need to manage agent versions and definitions programmatically
When performing batch evaluations or quality checks on AI outputs
When you need to interface with Azure-managed AI resources and deployments

Advantages

Provides a unified client for managing diverse AI project resources
Supports both sync and async programming patterns
Enables versioning for agent definitions
Integrates with standard Azure authentication via DefaultAzureCredential

Usage examples

“How do I initialize the AIProjectClient?”: Use the AIProjectClient class with your project endpoint and a DefaultAzureCredential instance, wrapped in a context manager.
“How can I create a versioned agent?”: Use the client.agents.create_version method, passing a PromptAgentDefinition object with your model, instructions, and version label.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/microsoft/azure-ai-projects-py/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip azure-ai-projects-py.zip -d .agents/skills

Documentation by Microsoft

Azure AI Projects Python SDK (Foundry SDK)

Build AI applications on Microsoft Foundry using the azure-ai-projects SDK.

Installation

pip install azure-ai-projects azure-identity

Environment Variables

AZURE_AI_PROJECT_ENDPOINT="https://<resource>.services.ai.azure.com/api/projects/<project>"  # Required for all auth methods
AZURE_AI_MODEL_DEPLOYMENT_NAME="gpt-4o-mini"  # Required for all auth methods
AZURE_TOKEN_CREDENTIALS=prod # Required only if DefaultAzureCredential is used in production

Authentication & Lifecycle

🔑 Two rules apply to every code sample below:
Prefer DefaultAzureCredential. It works locally (Azure CLI / VS Code / Developer CLI) and in Azure (managed identity, workload identity) with no code change. Avoid connection strings, account/API keys — they bypass Entra audit and rotation.
Local dev: DefaultAzureCredential works as-is.
Production: set AZURE_TOKEN_CREDENTIALS=prod (or AZURE_TOKEN_CREDENTIALS=<specific_credential>) to constrain the credential chain to production-safe credentials.
Wrap every client in a context manager so HTTP transports, sockets, and token caches are released deterministically:
Sync: with <Client>(...) as client:
Async: async with <Client>(...) as client: and async with DefaultAzureCredential() as credential: (from azure.identity.aio)
Snippets may abbreviate this setup, but production code should always follow both rules.

import os
from azure.identity import DefaultAzureCredential, ManagedIdentityCredential
from azure.ai.projects import AIProjectClient

# Local dev: DefaultAzureCredential. Production: set AZURE_TOKEN_CREDENTIALS=prod or AZURE_TOKEN_CREDENTIALS=<specific_credential>
credential = DefaultAzureCredential()
# Or use a specific credential directly in production:
# See https://learn.microsoft.com/python/api/overview/azure/identity-readme?view=azure-python#credential-classes
# credential = ManagedIdentityCredential()
with AIProjectClient(
    endpoint=os.environ["AZURE_AI_PROJECT_ENDPOINT"],
    credential=credential,
) as client:
    deployments = list(client.deployments.list())

Client Operations Overview

Operation	Access	Purpose
`client.agents`	`.agents.*`	Agent CRUD, versions, threads, runs
`client.connections`	`.connections.*`	List/get project connections
`client.deployments`	`.deployments.*`	List model deployments
`client.datasets`	`.datasets.*`	Dataset management
`client.indexes`	`.indexes.*`	Index management
`client.evaluations`	`.evaluations.*`	Run evaluations
`client.red_teams`	`.red_teams.*`	Red team operations

Two Client Approaches

1. AIProjectClient (Native Foundry)

from azure.ai.projects import AIProjectClient

with AIProjectClient(
    endpoint=os.environ["AZURE_AI_PROJECT_ENDPOINT"],
    credential=DefaultAzureCredential(),
) as client:
    # Use Foundry-native operations
    agent = client.agents.create_agent(
        model=os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
        name="my-agent",
        instructions="You are helpful.",
    )

2. OpenAI-Compatible Client

# Get OpenAI-compatible client from project
openai_client = client.get_openai_client()

# Use standard OpenAI API
response = openai_client.chat.completions.create(
    model=os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
    messages=[{"role": "user", "content": "Hello!"}],
)

Agent Operations

Create Agent (Basic)

agent = client.agents.create_agent(
    model=os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
    name="my-agent",
    instructions="You are a helpful assistant.",
)

Create Agent with Tools

from azure.ai.agents.models import CodeInterpreterTool, FileSearchTool

agent = client.agents.create_agent(
    model=os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
    name="tool-agent",
    instructions="You can execute code and search files.",
    tools=[CodeInterpreterTool(), FileSearchTool()],
)

Versioned Agents with PromptAgentDefinition

from azure.ai.projects.models import PromptAgentDefinition

# Create a versioned agent
agent_version = client.agents.create_version(
    agent_name="customer-support-agent",
    definition=PromptAgentDefinition(
        model=os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
        instructions="You are a customer support specialist.",
        tools=[],  # Add tools as needed
    ),
    version_label="v1.0",
)

See references/agents.md for detailed agent patterns.

Tools Overview

Tool	Class	Use Case
Code Interpreter	`CodeInterpreterTool`	Execute Python, generate files
File Search	`FileSearchTool`	RAG over uploaded documents
Bing Grounding	`BingGroundingTool`	Web search (requires connection)
Azure AI Search	`AzureAISearchTool`	Search your indexes
Function Calling	`FunctionTool`	Call your Python functions
OpenAPI	`OpenApiTool`	Call REST APIs
MCP	`McpTool`	Model Context Protocol servers
Memory Search	`MemorySearchTool`	Search agent memory stores
SharePoint	`SharepointGroundingTool`	Search SharePoint content

See references/tools.md for all tool patterns.

Thread and Message Flow

# 1. Create thread
thread = client.agents.threads.create()

# 2. Add message
client.agents.messages.create(
    thread_id=thread.id,
    role="user",
    content="What's the weather like?",
)

# 3. Create and process run
run = client.agents.runs.create_and_process(
    thread_id=thread.id,
    agent_id=agent.id,
)

# 4. Get response
if run.status == "completed":
    messages = client.agents.messages.list(thread_id=thread.id)
    for msg in messages:
        if msg.role == "assistant":
            print(msg.content[0].text.value)

Connections

# List all connections
connections = client.connections.list()
for conn in connections:
    print(f"{conn.name}: {conn.connection_type}")

# Get specific connection
connection = client.connections.get(connection_name="my-search-connection")

See references/connections.md for connection patterns.

Deployments

# List available model deployments
deployments = client.deployments.list()
for deployment in deployments:
    print(f"{deployment.name}: {deployment.model}")

See references/deployments.md for deployment patterns.

Datasets and Indexes

# List datasets
datasets = client.datasets.list()

# List indexes
indexes = client.indexes.list()

See references/datasets-indexes.md for data operations.

Evaluation

# Using OpenAI client for evals
openai_client = client.get_openai_client()

# Create evaluation with built-in evaluators
eval_run = openai_client.evals.runs.create(
    eval_id="my-eval",
    name="quality-check",
    data_source={
        "type": "custom",
        "item_references": [{"item_id": "test-1"}],
    },
    testing_criteria=[
        {"type": "fluency"},
        {"type": "task_adherence"},
    ],
)

See references/evaluation.md for evaluation patterns.

Async Client

from azure.ai.projects.aio import AIProjectClient

async with AIProjectClient(
    endpoint=os.environ["AZURE_AI_PROJECT_ENDPOINT"],
    credential=DefaultAzureCredential(),
) as client:
    agent = await client.agents.create_agent(...)
    # ... async operations

See references/async-patterns.md for async patterns.

Memory Stores

# Create memory store for agent
memory_store = client.agents.create_memory_store(
    name="conversation-memory",
)

# Attach to agent for persistent memory
agent = client.agents.create_agent(
    model=os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
    name="memory-agent",
    tools=[MemorySearchTool()],
    tool_resources={"memory": {"store_ids": [memory_store.id]}},
)

Best Practices

Pick sync OR async and stay consistent. Do not mix azure.ai.projects sync clients with azure.ai.projects.aio async clients in the same call path. Choose one mode per module.
Always use context managers for clients and async credentials. Wrap every client in with AIProjectClient(...) as client: (sync) or async with AIProjectClient(...) as client: (async). For async DefaultAzureCredential from azure.identity.aio, also use async with credential: so tokens and transports are cleaned up.
Clean up agents when done: client.agents.delete_agent(agent.id)
Use create_and_process for simple runs, streaming for real-time UX
Use versioned agents for production deployments
Prefer connections for external service integration (AI Search, Bing, etc.)

SDK Comparison

Feature	`azure-ai-projects`	`azure-ai-agents`
Level	High-level (Foundry)	Low-level (Agents)
Client	`AIProjectClient`	`AgentsClient`
Versioning	`create_version()`	Not available
Connections	Yes	No
Deployments	Yes	No
Datasets/Indexes	Yes	No
Evaluation	Via OpenAI client	No
When to use	Full Foundry integration	Standalone agent apps

Reference Files

references/agents.md: Agent operations with PromptAgentDefinition
references/tools.md: All agent tools with examples
references/evaluation.md: Evaluation operations overview
references/built-in-evaluators.md: Complete built-in evaluator reference
references/custom-evaluators.md: Code and prompt-based evaluator patterns
references/connections.md: Connection operations
references/deployments.md: Deployment enumeration
references/datasets-indexes.md: Dataset and index operations
references/async-patterns.md: Async client usage
references/api-reference.md: Complete API reference for all 373 SDK exports (v2.0.0b4)
scripts/run_batch_evaluation.py: CLI tool for batch evaluations

Files

ATTRIBUTION.mdAdded by AutoGineer971 B
LICENSELicence file added by AutoGineer1.1 KiB
SKILL.md10.8 KiB
references/agents.md6.7 KiB
references/api-reference.md27.3 KiB
references/async-patterns.md6.8 KiB
references/built-in-evaluators.md11.5 KiB
references/connections.md4.7 KiB
references/custom-evaluators.md14.1 KiB
references/datasets-indexes.md4.2 KiB
references/deployments.md3.5 KiB
references/evaluation.md10.7 KiB
references/tools.md11.7 KiB
scripts/run_batch_evaluation.pyScript12.2 KiB

references/built-in-evaluators.md

Built-in Evaluators Reference

Complete reference for Microsoft Foundry's built-in evaluators using the azure-ai-projects SDK.

Discovering Evaluators

List All Built-in Evaluators

from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient

endpoint = os.environ["AZURE_AI_PROJECT_ENDPOINT"]

with (
    DefaultAzureCredential() as credential,
    AIProjectClient(endpoint=endpoint, credential=credential) as project_client,
):
    evaluators = project_client.evaluators.list_latest_versions(type="builtin")
    for e in evaluators:
        print(f"{e.name}: {e.description}")
        print(f"  Categories: {e.categories}")

Get Evaluator Schema

Before using an evaluator, query its schema to discover required inputs:

evaluator = project_client.evaluators.get_version(
    name="builtin.task_adherence",
    version="latest"
)
print(f"Init Parameters: {evaluator.definition.init_parameters}")
print(f"Data Schema: {evaluator.definition.data_schema}")
print(f"Metrics: {evaluator.definition.metrics}")

Using Built-in Evaluators

All built-in evaluators use the azure_ai_evaluator type with builtin. prefix:

testing_criteria = [
    {
        "type": "azure_ai_evaluator",
        "name": "my_coherence_check",          # Your custom name for results
        "evaluator_name": "builtin.coherence", # The actual evaluator
        "data_mapping": {
            "query": "{{item.query}}",
            "response": "{{item.response}}"
        },
        "initialization_parameters": {
            "deployment_name": "gpt-4o-mini"   # Required for LLM-based evaluators
        }
    }
]

Quality Evaluators

builtin.coherence

Measures logical flow and consistency of the response.

{
    "type": "azure_ai_evaluator",
    "name": "coherence",
    "evaluator_name": "builtin.coherence",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

Inputs: query, response
Output: Score 1-5 (5 = highly coherent)

builtin.fluency

Measures grammatical correctness and natural language quality.

{
    "type": "azure_ai_evaluator",
    "name": "fluency",
    "evaluator_name": "builtin.fluency",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

Inputs: query, response
Output: Score 1-5 (5 = perfectly fluent)

builtin.relevance

Measures how well the response addresses the query given context.

{
    "type": "azure_ai_evaluator",
    "name": "relevance",
    "evaluator_name": "builtin.relevance",
    "data_mapping": {
        "query": "{{item.query}}",
        "response": "{{item.response}}",
        "context": "{{item.context}}"
    },
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

Inputs: query, response, context
Output: Score 1-5 (5 = highly relevant)

builtin.groundedness

Measures whether the response is factually grounded in the provided context.

{
    "type": "azure_ai_evaluator",
    "name": "groundedness",
    "evaluator_name": "builtin.groundedness",
    "data_mapping": {
        "query": "{{item.query}}",
        "response": "{{item.response}}",
        "context": "{{item.context}}"
    },
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

Inputs: query, response, context
Output: Score 1-5 (5 = fully grounded)

builtin.response_completeness

Measures whether the response fully addresses all aspects of the query.

{
    "type": "azure_ai_evaluator",
    "name": "response_completeness",
    "evaluator_name": "builtin.response_completeness",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

Inputs: query, response
Output: Score 1-5

Safety Evaluators

Safety evaluators detect harmful content. They don't require deployment_name.

builtin.violence

Detects violent content.

{
    "type": "azure_ai_evaluator",
    "name": "violence",
    "evaluator_name": "builtin.violence",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"}
}

Inputs: query, response
Output: pass/fail with severity score

builtin.sexual

Detects inappropriate sexual content.

{
    "type": "azure_ai_evaluator",
    "name": "sexual",
    "evaluator_name": "builtin.sexual",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"}
}

builtin.self_harm

Detects content promoting or describing self-harm.

{
    "type": "azure_ai_evaluator",
    "name": "self_harm",
    "evaluator_name": "builtin.self_harm",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"}
}

builtin.hate_unfairness

Detects biased or hateful content.

{
    "type": "azure_ai_evaluator",
    "name": "hate_unfairness",
    "evaluator_name": "builtin.hate_unfairness",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"}
}

Agent Evaluators

Agent evaluators assess AI agent behavior and tool usage.

builtin.task_adherence

Evaluates whether the agent follows its system instructions.

{
    "type": "azure_ai_evaluator",
    "name": "task_adherence",
    "evaluator_name": "builtin.task_adherence",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_items}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

Note: Use {{sample.output_items}} for agent responses to include tool call information.

builtin.intent_resolution

Evaluates whether the agent correctly understood user intent.

{
    "type": "azure_ai_evaluator",
    "name": "intent_resolution",
    "evaluator_name": "builtin.intent_resolution",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_text}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

builtin.task_completion

Evaluates whether the agent completed the task end-to-end.

{
    "type": "azure_ai_evaluator",
    "name": "task_completion",
    "evaluator_name": "builtin.task_completion",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_items}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

builtin.tool_call_accuracy

Evaluates whether tool calls are correct (selection + parameters).

{
    "type": "azure_ai_evaluator",
    "name": "tool_call_accuracy",
    "evaluator_name": "builtin.tool_call_accuracy",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_items}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

builtin.tool_call_success

Evaluates whether tool calls executed without failures.

{
    "type": "azure_ai_evaluator",
    "name": "tool_call_success",
    "evaluator_name": "builtin.tool_call_success",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_items}}"}
}

builtin.tool_selection

Evaluates whether the correct tools were selected.

{
    "type": "azure_ai_evaluator",
    "name": "tool_selection",
    "evaluator_name": "builtin.tool_selection",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_items}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

NLP Evaluators

NLP evaluators compare responses to ground truth without requiring an LLM.

builtin.f1_score

Token-level F1 score between response and ground truth.

{
    "type": "azure_ai_evaluator",
    "name": "f1",
    "evaluator_name": "builtin.f1_score",
    "data_mapping": {"response": "{{item.response}}", "ground_truth": "{{item.ground_truth}}"}
}

Output: Score 0-1

builtin.bleu_score

BLEU score for generation quality.

{
    "type": "azure_ai_evaluator",
    "name": "bleu",
    "evaluator_name": "builtin.bleu_score",
    "data_mapping": {"response": "{{item.response}}", "ground_truth": "{{item.ground_truth}}"}
}

builtin.rouge_score

ROUGE score for summarization quality.

{
    "type": "azure_ai_evaluator",
    "name": "rouge",
    "evaluator_name": "builtin.rouge_score",
    "data_mapping": {"response": "{{item.response}}", "ground_truth": "{{item.ground_truth}}"}
}

builtin.similarity

Semantic similarity between response and ground truth.

{
    "type": "azure_ai_evaluator",
    "name": "similarity",
    "evaluator_name": "builtin.similarity",
    "data_mapping": {
        "query": "{{item.query}}",
        "response": "{{item.response}}",
        "ground_truth": "{{item.ground_truth}}"
    },
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

Evaluator Sets by Use Case

Quick Health Check

testing_criteria = [
    {"type": "azure_ai_evaluator", "name": "coherence", "evaluator_name": "builtin.coherence", ...},
    {"type": "azure_ai_evaluator", "name": "fluency", "evaluator_name": "builtin.fluency", ...},
    {"type": "azure_ai_evaluator", "name": "violence", "evaluator_name": "builtin.violence", ...},
]

Safety Audit

testing_criteria = [
    {"type": "azure_ai_evaluator", "name": "violence", "evaluator_name": "builtin.violence", ...},
    {"type": "azure_ai_evaluator", "name": "sexual", "evaluator_name": "builtin.sexual", ...},
    {"type": "azure_ai_evaluator", "name": "self_harm", "evaluator_name": "builtin.self_harm", ...},
    {"type": "azure_ai_evaluator", "name": "hate_unfairness", "evaluator_name": "builtin.hate_unfairness", ...},
]

Agent Evaluation

testing_criteria = [
    {"type": "azure_ai_evaluator", "name": "task_adherence", "evaluator_name": "builtin.task_adherence", ...},
    {"type": "azure_ai_evaluator", "name": "intent_resolution", "evaluator_name": "builtin.intent_resolution", ...},
    {"type": "azure_ai_evaluator", "name": "tool_call_accuracy", "evaluator_name": "builtin.tool_call_accuracy", ...},
]

RAG Evaluation

testing_criteria = [
    {"type": "azure_ai_evaluator", "name": "groundedness", "evaluator_name": "builtin.groundedness", ...},
    {"type": "azure_ai_evaluator", "name": "relevance", "evaluator_name": "builtin.relevance", ...},
    {"type": "azure_ai_evaluator", "name": "response_completeness", "evaluator_name": "builtin.response_completeness", ...},
]

Data Mapping Reference

Data Source	Response Mapping	Use Case
JSONL dataset	`{{item.response}}`	Pre-recorded query/response pairs
Agent target	`{{sample.output_text}}`	Plain text response
Agent target	`{{sample.output_items}}`	Structured JSON with tool calls

When to use sample.output_items:

Tool-related evaluators (tool_call_accuracy, tool_selection, etc.)
Task adherence evaluator
Any evaluator needing tool call context

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an API key, Needs Python

Security

What our scanner noticed· Low risk

Instructions that use the network
Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

Microsoft

Publisher

Microsoft (Official)

Source

microsoft/skills · .github/plugins/azure-sdk-python/skills/azure-ai-projects-py · commit 0696600

Licence

MIT License (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

AutoGineer is not affiliated with or endorsed by Microsoft or Microsoft.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download azure-ai-projects-py.zip