Huggingface Trackio

Hugging FaceOfficialPublished by Hugging Face

Skill
Tool integration
Data and AI
Apache-2.0
Medium risk

From the author

Track and visualize ML training experiments with Trackio. Use when logging metrics during training (Python API), firing alerts for training diagnostics, or retrieving/analyzing logged metrics (CLI). Supports real-time dashboard visualization, alerts with webhooks, HF Space syncing, and JSON output for automation.

— the description by Hugging Face

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

Trackio provides a Python API and CLI for tracking machine learning experiments. It supports local metric storage, real-time dashboard visualization, and synchronization with Hugging Face Spaces. Designed with a focus on autonomous workflows, it allows developers and LLM agents to log metrics, fire diagnostic alerts, and retrieve training data programmatically via JSON output.

What it does

Logs training metrics and hyperparameters using a Python API.
Provides real-time experiment visualization via local or remote Hugging Face Space dashboards.
Enables diagnostic alerting with severity levels (INFO, WARN, ERROR) and webhook support.
Allows programmatic retrieval of metrics and alerts via CLI with JSON output.
Supports automatic integration with TRL trainers.
Syncs local experiment data to persistent remote storage.

When to use it

When you need to track and visualize metrics during ML model training.
When building autonomous agents that need to monitor training health and iterate on hyperparameters.
When you require persistent dashboards for cloud-based training runs.
When you need to integrate training diagnostics with external notification systems like Slack or Discord.

Advantages

Lightweight and local-first design.
API-compatible with existing experiment tracking tools.
Enables autonomous experiment iteration through structured alerts.
Supports both local and cloud-based monitoring workflows.
Provides structured JSON output for easy integration with automation scripts.

Usage examples

“How do I log training metrics with Trackio?”: Initialize the run with trackio.init(), use trackio.log() to record metrics during your loop, and call trackio.finish() to save the data.
“How can an agent monitor for training failures?”: The agent can use trackio.alert() in the training script to flag issues, then poll for these alerts using the CLI command 'trackio list alerts --json'.
“How do I sync my local training metrics to a Hugging Face Space?”: Pass the space_id parameter to trackio.init() during initialization, or use trackio.sync() to upload an existing local project.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/huggingface/huggingface-trackio/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip huggingface-trackio.zip -d .agents/skills

Documentation by Hugging Face

Trackio - Experiment Tracking for ML Training

Trackio is an experiment tracking library for logging and visualizing ML training metrics. It syncs to Hugging Face Spaces for real-time monitoring dashboards.

Three Interfaces

Task	Interface	Reference
Logging metrics during training	Python API	references/logging_metrics.md
Firing alerts for training diagnostics	Python API	references/alerts.md
Retrieving metrics & alerts after/during training	CLI	references/retrieving_metrics.md

When to Use Each

Python API → Logging

Use import trackio in your training scripts to log metrics:

Initialize tracking with trackio.init()
Log metrics with trackio.log() or use TRL's report_to="trackio"
Finalize with trackio.finish()

Key concept: For remote/cloud training, pass space_id — metrics sync to a Space dashboard so they persist after the instance terminates. Auto-created Spaces are public by default — pass private=True if the metrics should not be public.

→ See references/logging_metrics.md for setup, TRL integration, and configuration options.

Python API → Alerts

Insert trackio.alert() calls in training code to flag important events — like inserting print statements for debugging, but structured and queryable:

trackio.alert(title="...", level=trackio.AlertLevel.WARN) — fire an alert
Three severity levels: INFO, WARN, ERROR
Alerts are printed to terminal, stored in the database, shown in the dashboard, and optionally sent to webhooks (Slack/Discord)

Key concept for LLM agents: Alerts are the primary mechanism for autonomous experiment iteration. An agent should insert alerts into training code for diagnostic conditions (loss spikes, NaN gradients, low accuracy, training stalls). Since alerts are printed to the terminal, an agent that is watching the training script's output will see them automatically. For background or detached runs, the agent can poll via CLI instead.

→ See references/alerts.md for the full alerts API, webhook setup, and autonomous agent workflows.

CLI → Retrieving

Use the trackio command to query logged metrics and alerts:

trackio list projects/runs/metrics — discover what's available
trackio get project/run/metric — retrieve summaries and values
trackio list alerts --project <name> --json — retrieve alerts
trackio show — launch the dashboard
trackio sync — sync to HF Space

Key concept: Add --json for programmatic output suitable for automation and LLM agents.

→ See references/retrieving_metrics.md for all commands, workflows, and JSON output formats.

Minimal Logging Setup

import trackio

# Spaces are PUBLIC by default (good for shareable dashboards);
# pass private=True if the metrics should not be public
trackio.init(project="my-project", space_id="username/trackio", private=True)
trackio.log({"loss": 0.1, "accuracy": 0.9})
trackio.log({"loss": 0.09, "accuracy": 0.91})
trackio.finish()

Minimal Retrieval

trackio list projects --json
trackio get metric --project my-project --run my-run --metric loss --json

Autonomous ML Experiment Workflow

When running experiments autonomously as an LLM agent, the recommended workflow is:

Set up training with alerts — insert trackio.alert() calls for diagnostic conditions
Launch training — run the script in the background
Poll for alerts — use trackio list alerts --project <name> --json --since <timestamp> to check for new alerts
Read metrics — use trackio get metric ... to inspect specific values
Iterate — based on alerts and metrics, stop the run, adjust hyperparameters, and launch a new run

import trackio

trackio.init(project="my-project", config={"lr": 1e-4})

for step in range(num_steps):
    loss = train_step()
    trackio.log({"loss": loss, "step": step})

    if step > 100 and loss > 5.0:
        trackio.alert(
            title="Loss divergence",
            text=f"Loss {loss:.4f} still high after {step} steps",
            level=trackio.AlertLevel.ERROR,
        )
    if step > 0 and abs(loss) < 1e-8:
        trackio.alert(
            title="Vanishing loss",
            text="Loss near zero — possible gradient collapse",
            level=trackio.AlertLevel.WARN,
        )

trackio.finish()

Then poll from a separate terminal/process:

trackio list alerts --project my-project --json --since "2025-01-01T00:00:00"

Files

ATTRIBUTION.mdAdded by AutoGineer882 B
LICENSELicence file added by AutoGineer11.1 KiB
SKILL.md5.0 KiB
references/alerts.md5.8 KiB
references/logging_metrics.md5.4 KiB
references/retrieving_metrics.md8.5 KiB

references/alerts.md

Trackio Alerts

Alerts let you flag important training events directly from code. They are the primary mechanism for LLM agents to diagnose runs and iterate autonomously on ML experiments.

Alerts are printed to the terminal, stored in the database, displayed in the dashboard, and optionally sent to webhooks (Slack/Discord).

Core API

trackio.alert()

trackio.alert(
    title="Loss divergence",                    # Short title (required)
    text="Loss 5.2 still high after 200 steps", # Detailed description (optional)
    level=trackio.AlertLevel.WARN,               # INFO, WARN, or ERROR (default: WARN)
    webhook_url="https://hooks.slack.com/...",   # Per-alert webhook override (optional)
)

Alert Levels

Level	Usage
`trackio.AlertLevel.INFO`	Informational milestones (checkpoints saved, eval completed)
`trackio.AlertLevel.WARN`	Potential issues (loss plateau, low accuracy, high gradient norm)
`trackio.AlertLevel.ERROR`	Critical failures (NaN loss, divergence, OOM)

Webhook Support

Set a global webhook URL via trackio.init() or the TRACKIO_WEBHOOK_URL environment variable. Alerts are auto-formatted for Slack and Discord URLs.

trackio.init(
    project="my-project",
    webhook_url="https://hooks.slack.com/services/...",
    webhook_min_level=trackio.AlertLevel.WARN,  # Only send WARN+ to webhook
)

Per-alert override:

trackio.alert(
    title="Critical failure",
    level=trackio.AlertLevel.ERROR,
    webhook_url="https://hooks.slack.com/services/...",  # Overrides global URL
)

Environment variables:

TRACKIO_WEBHOOK_URL — global webhook URL
TRACKIO_WEBHOOK_MIN_LEVEL — minimum level for webhook delivery (info, warn, error)

Retrieving Alerts (CLI)

# List all alerts for a project
trackio list alerts --project my-project --json

# Filter by run or level
trackio list alerts --project my-project --run my-run --level error --json

# Poll for new alerts since a timestamp (efficient for agents)
trackio list alerts --project my-project --json --since "2025-06-01T12:00:00"

JSON Output Structure

{
  "project": "my-project",
  "run": null,
  "level": null,
  "since": "2025-06-01T12:00:00",
  "alerts": [
    {
      "run": "run-name",
      "title": "Loss divergence",
      "text": "Loss 5.2 still high after 200 steps",
      "level": "warn",
      "step": 200,
      "timestamp": "2025-06-01T12:05:30"
    }
  ]
}

Autonomous Agent Workflow

The recommended pattern for an LLM agent running ML experiments:

1. Insert Alerts Into Training Code

Add diagnostic trackio.alert() calls for conditions the agent should react to:

import trackio

trackio.init(project="hyperparam-sweep", config={"lr": lr, "batch_size": bs})

for step in range(num_steps):
    loss = train_step()
    trackio.log({"loss": loss, "step": step})

    if step > 200 and loss > 5.0:
        trackio.alert(
            title="Loss divergence",
            text=f"Loss {loss:.4f} still above 5.0 after {step} steps — learning rate may be too high",
            level=trackio.AlertLevel.ERROR,
        )

    if step > 500 and loss_delta < 0.001:
        trackio.alert(
            title="Training stall",
            text=f"Loss barely changed over last 100 steps (delta={loss_delta:.6f})",
            level=trackio.AlertLevel.WARN,
        )

    if math.isnan(loss):
        trackio.alert(
            title="NaN loss",
            text="Loss became NaN — training is broken",
            level=trackio.AlertLevel.ERROR,
        )
        break

trackio.finish()

2. Monitor Alerts

Alerts are automatically printed to the terminal when fired. If the agent is watching the training script's output (e.g. running in the foreground or tailing logs), it will see alerts immediately — no polling needed.

For background or detached runs, poll for alerts via CLI:

# Poll for alerts (run periodically)
trackio list alerts --project hyperparam-sweep --json --since "2025-06-01T00:00:00"

3. Inspect Metrics Around the Alert

When an alert fires, use trackio get snapshot to see all metrics at that point:

# Alert fired at step 200 — get all metrics in a ±5 step window
trackio get snapshot --project hyperparam-sweep --run run-1 --around 200 --window 5 --json

# Or inspect a single metric around the alert's timestamp
trackio get metric --project hyperparam-sweep --run run-1 --metric loss --around 200 --window 10 --json

4. React and Iterate

Based on alerts:

ERROR alerts → stop the run, adjust hyperparameters, relaunch
WARN alerts → inspect metrics with trackio get snapshot ..., decide whether to intervene
INFO alerts → note progress, continue monitoring

5. Compare Across Runs

# Check metrics from previous runs
trackio get run --project hyperparam-sweep --run run-1 --json
trackio get metric --project hyperparam-sweep --run run-1 --metric loss --json

# Launch new run with adjusted config
python train.py --lr 5e-5

Using Alerts with Transformers / TRL

When using report_to="trackio", you don't control the training loop directly. Use a TrainerCallback to fire alerts:

from transformers import TrainerCallback

class AlertCallback(TrainerCallback):
    def on_log(self, args, state, control, logs=None, **kwargs):
        if "trackio" not in args.report_to:
            return
        if logs and "loss" in logs:
            if logs["loss"] > 5.0 and state.global_step > 100:
                trackio.alert(
                    title="High loss",
                    text=f"Loss {logs['loss']:.4f} at step {state.global_step}",
                    level=trackio.AlertLevel.ERROR,
                )

trainer = SFTTrainer(
    model=model,
    args=SFTConfig(output_dir="./out", report_to="trackio"),
    callbacks=[AlertCallback()],
    ...
)

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Medium risk

Possible data-exfiltration endpoints
Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

Hugging Face

Publisher

Hugging Face (Official)

Source

huggingface/skills · skills/huggingface-trackio · commit 25910b3

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

AutoGineer is not affiliated with or endorsed by Hugging Face or Hugging Face.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download huggingface-trackio.zip