Generated banner for Nvflare Convert Huggingface
Generated logo for Nvflare Convert Huggingface

Nvflare Convert Huggingface

NVIDIAOfficialCreated by NVIDIA FLARE Team

  • Skill
  • Tool integration
  • Automation and integrations
  • Apache-2.0
  • Medium risk
  • Has scripts

From the author

Convert existing Hugging Face Transformers Trainer or TRL SFTTrainer training code into an NVFLARE federated job using flare.patch(trainer), local validation, and job export; use when the user names Hugging Face or preliminary source inspection identifies one Hugging Face owner, and not for manual PyTorch loops, Lightning, inference-only pipelines, deployment, or experiment workflows.
— the description by NVIDIA FLARE Team

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured procedure for migrating existing Hugging Face Transformers or TRL SFTTrainer training scripts into an NVFLARE-compatible federated learning job. It leverages the NVFLARE Client API to patch the Trainer, enabling federated training while preserving the original model architecture, dataset handling, and evaluation logic. The process involves static inspection of the source code, recipe selection, job construction using maintained assets, and validation through simulation.

What it does

  • Inspects source code to confirm Hugging Face Trainer ownership.
  • Patches the Trainer using flare.patch(trainer) for federated model exchange.
  • Constructs an NVFLARE job using standard FedAvg recipes.
  • Preserves original Trainer arguments, metrics, and PEFT configurations.
  • Validates the converted job through simulation and evidence reporting.

When to use it

  • When you have existing training code built around transformers.Trainer or TRL SFTTrainer.
  • When you need to transition a centralized Hugging Face training workflow to a federated environment.
  • When you require federated fine-tuning of models using PEFT/LoRA.
  • When you need to maintain existing Trainer callbacks and evaluation metrics in a federated setup.

Advantages

  • Maintains existing Hugging Face training logic and Trainer semantics.
  • Automates the integration of federated learning capabilities via patching.
  • Ensures consistent model exchange using standard NVFLARE recipes.
  • Provides a clear validation ladder to verify the converted job.

Usage examples

“Convert my existing Hugging Face SFTTrainer script to run on NVFLARE.”
The agent will inspect your script, confirm Trainer ownership, apply the required patches, and generate an NVFLARE job structure ready for simulation.
“How do I make my Transformers Trainer-based model work in a federated setting?”
The agent will guide you through the conversion workflow, including patching your Trainer and configuring the server-client job assets.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/nvidia/nvflare-convert-huggingface/download
  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip nvflare-convert-huggingface.zip -d .agents/skills

Documentation by NVIDIA FLARE Team

NVFLARE Convert Hugging Face

Use When

Use when converting training code built around transformers.Trainer, Seq2SeqTrainer, TRL SFTTrainer, or another Trainer subclass. Support full-model and PEFT/LoRA fine-tuning, datasets/tokenizers, Trainer callbacks and metrics, checkpoint continuity, and replicated torch.distributed training.

Do Not Use When

Do not use for an AutoModel driven by a manual PyTorch loop without a Hugging Face Trainer (route to nvflare-convert-pytorch), PyTorch Lightning (route to nvflare-convert-lightning, including Lightning modules that contain Transformers models), inference-only pipelines, model serving, failed jobs (route to nvflare-diagnose-job), or federated statistics without training (route to nvflare-fed-stats). Route a project with active Lightning and Hugging Face Trainer entrypoints to nvflare-orient to select one training-loop owner or separate jobs. Route unresolved Trainer ownership, such as a Trainer factory without a bound owner call, to nvflare-orient; do not patch either Trainer. Out of scope: DeepSpeed, FSDP, production/POC deployment, controller rewrites, experiment search, and privacy-protection requests such as HE, encrypted aggregation, differential privacy, or privacy filters; never substitute an unprotected recipe or present a disclaimer as implementation. If a request combines federated statistics and model-training conversion, treat it as two independent jobs and workflows: do not merge or automatically chain them, do not route the combination to nvflare-orient, and ask which workflow to run first before generating or running either job. Recommend nvflare-fed-stats first only when the user's purpose is to understand data distribution; handle conversion later as a separate request.

Available Scripts

ScriptPurposeArguments
scripts/resolve_model_snapshot.pyResolve a local or Hub model/dataset snapshot and emit JSON evidence.Identifier; required --source; optional source-root, download, revision, cache, and repository-type flags.

Use run_script() only during the validation phase after loading references/huggingface-validation.md. For example:

  • run_script("scripts/resolve_model_snapshot.py", ["--source", "local", "--source-root", "<absolute-source-root>", "<configured-path>"])

  • run_script("scripts/resolve_model_snapshot.py", ["--source", "hub", "<org/model>"])

Workflow

  1. Load ../nvflare-shared/references/conversion-common.md and apply it for the whole conversion; this SKILL.md states only the framework-specific deltas. Load ../nvflare-shared/references/conversion-workflow.md only for a non-standard rerun, authorization, or missing-semantics case; it no longer holds the data-location or partitioning contracts, whose invariants conversion-common.md owns. Load ../nvflare-shared/references/site-data-and-paths.md for generated partitions, relative paths, or per-site data locations.

  2. Inspect before editing with nvflare agent inspect source <path> --format json plus direct source reading. Load references/huggingface-detection.md during this phase. If inspect recommends nvflare-orient for unresolved Trainer ownership or active Lightning/Hugging Face owners, stop before editing. Extract the entrypoint, Trainer subclass, model constructor, tokenizer or processor, datasets and collator, Trainer arguments, compute_metrics, callbacks, checkpoint and PEFT settings, precision, local budget, distributed launcher, site/round counts, data location, and aggregation intent. Do not import or execute user training modules to discover them.

  3. Apply the dependency-install ordering rule in ../nvflare-shared/references/conversion-common.md before any Python command imports user, framework, NVFLARE, or declared dependencies. Keep dependency inventory single-purpose: check NVFLARE separately, inventory non-product packages separately, and do not append Hugging Face cache or filesystem discovery. Defer model and dataset availability checks to the maintained resolver during validation. If an optional path must be inspected, run it separately and report a missing directory with exit code zero.

  4. Select the recipe from FL intent. For explicit FedAvg, run nvflare recipe show fedavg-pt --format json, then immediately load ../nvflare-shared/references/pytorch-family-recipe-construction.md and use the returned module, class, and parameters with the required construction and execution shape in assets/job.py. Import FedAvgRecipe from nvflare.app_opt.pt.recipes.fedavg, never from nvflare.recipe. Treat class_path as the public recipe key and path as its normalized exported representation; do not inspect Recipe source or signatures to reconcile them. Do not guess adjacent symbols or add per-site recipe config unless sites genuinely differ. Load ../nvflare-shared/references/pytorch-family-recipe-selection.md only for ambiguous, evaluation-only, or non-FedAvg requests.

  5. Convert with references/huggingface-conversion.md and adapt assets/client_with_eval.py rather than drafting a new round loop. Preserve model, tokenizer/processor, datasets, collator, Trainer arguments, callbacks, and metrics. Apply the step-1 data-location rules to the client's data argument. Import the Client API as import nvflare.client.hf as flare, so flare.init(), flare.patch(), and flare.is_running() resolve to nvflare.client.hf. Keep flare.patch(trainer) simple with inferred params_scope="auto" and encode one per-round budget in Trainer arguments: requested steps use max_steps, requested epochs use num_train_epochs, and a silent prompt uses the reported default max_steps=10 unless source-budget preservation was requested. Do not duplicate the budget in patch local_steps/local_epochs. When the client uses HfArgumentParser, construct it with allow_abbrev=False.

  6. Adapt assets/server_model.py and assets/job.py instead of inventing server-model, packaging, export, or SimEnv wiring. Keep generated and packaged project-local modules in the same writable source directory. Never use .. in train_script, add_server_file(), or add_client_file(); use an existing resolved absolute path when co-location is impossible. Keep the server and Trainer model factory and exchange keyspace identical. Use the recipe's documented class_path or path key plus complete args for required or overridden values; a direct zero-argument instance must not use from_pretrained(), downloads, or checkpoint loading during job construction. Apply only options confirmed by the construction reference. Preserve the job asset's recipe-before-parser ordering, ArgumentParser(allow_abbrev=False), and strict parse_args(); do not use parse_known_args().

  7. Only after generated files exist, load ../nvflare-shared/references/validation-evidence.md and references/huggingface-validation.md. Follow the shared compile, construction, simulation, and terminal-evidence ladder. Inspect export/package evidence only for an exported final target; inspect a local target's materialized evidence after its run. Apply only the standard HF Trainer checks and stop at the first failed rung. Review and exercise the maintained assets directly; do not inspect NVFLARE implementation source, improvise Recipe API probes, or write one-off AST programs to re-prove them. Use references/huggingface-state-and-distributed.md only when inspection found PEFT, DDP, checkpoint/restore overrides, auxiliary trainable models, or another non-default patch setting.

  8. Report the recipe, source facts, parameter scope, data partition, changed files, validation status, and exact artifact paths. When validation produces metrics, load ../nvflare-shared/references/metrics-and-artifact-reporting.md before the final response and report each observed primary scalar with its metric name, numeric value, and artifact or bounded-log source.

Requirements

  • Must use flare.patch(trainer) as the sole model-exchange owner. receive() inside a patched loop may inspect task metadata only; it must not load a second copy of the global model.

  • Must follow the Client API initialization and conditional rank contract in ../nvflare-shared/references/conversion-common.md. Keep the generated client rankless; nvflare.client.hf.init() owns distributed-rank resolution and rejection. Load references/huggingface-state-and-distributed.md only for a distributed multi-process source path.

  • Must preserve source evaluation. When per-round global-model evaluation is required, call trainer.evaluate() before trainer.train() on every rank. Do not invent compute_metrics, label mappings, averaging denominators, or metric direction.

  • Must follow the Best-Model Metric policy in ../nvflare-shared/references/pytorch-family-recipe-construction.md; the Hugging Face delta is only how the delivered key is named and produced. Must preserve source metric names when practical: if the generated trainer.evaluate() emits accuracy, set key_metric="accuracy"; if Trainer emits a prefixed key such as eval_accuracy, set the server to that exact key and report the source-to-server mapping. When best-model selection is requested, every lower-is-better metric, including Trainer-generated eval_loss, is delivered as an explicitly negated companion and selected by that key — never as raw loss. When selection is not requested, use key_metric=""; do not omit it and accidentally activate the recipe default.

  • Must preserve PEFT configuration exactly and verify adapter key compatibility between the server model and patched Trainer. Do not infer LoRA target modules, silently switch adapter/full-model scope, or solve key mismatches with non-strict loading.

  • Must verify that trainer.model owns all federated trainable state for Trainer subclasses with reference, reward, value-head, or other auxiliary models. Ask or fail closed when params_scope="auto" would omit trainable state required by the algorithm.

  • Must preserve model constructor values needed on both server and clients per ../nvflare-shared/references/pytorch-model-exchange.md (State-Dict Compatibility). Ask one semantic question or fail closed when required values are not statically available.

  • Must patch only one Trainer per Python process. Preserve a single Trainer lifecycle across rounds when restore_state=True.

  • Must use a positive TrainingArguments.max_steps budget for a length-less iterable training dataset and let flare.patch(trainer) infer it.

  • Must reject or report DeepSpeed, FSDP, save_only_model=True with restore_state=True, load_best_model_at_end=True, prebuilt optimizer/scheduler instances with restore_state=False, and checkpoint paths not visible to every distributed rank. Do not rewrite these settings silently. launch_once is a framework-neutral recipe parameter owned by ../nvflare-shared/references/pytorch-family-recipe-construction.md; the Hugging Face delta is only that the product rejects explicit launch_once=False together with restore_state=True.

  • Must initialize torch.distributed before patching when rank environment variables declare multiple ranks. All ranks must call patched Trainer methods in identical order.

  • Must use the maintained HF validation resolver with an explicit local/Hub source. For authorized downloads it obtains or validates a full commit-SHA revision before downloading. Must not copy it into generated job code, set trust_remote_code=True, download model/data artifacts unless requested, or recover from a cache-only miss by going online. Cache misses, remote identifiers, and validation requests do not authorize online retries; see ../nvflare-shared/references/conversion-common.md.

  • Site partitioning, custom aggregation, the Source Of Truth Boundary, and user input/authorization follow ../nvflare-shared/references/conversion-common.md.

Always read this converter SKILL.md together with ../nvflare-shared/references/conversion-common.md. Complete each workflow phase before loading the next phase's reference. Do not preload validation, state/DDP, broad workflow, dependency, or reporting references. The standard FedAvg path loads, in order: ../nvflare-shared/references/conversion-common.md, references/huggingface-detection.md, and ../nvflare-shared/references/site-data-and-paths.md only for its stated triggers; ../nvflare-shared/references/pytorch-family-recipe-construction.md, references/huggingface-conversion.md, and ../nvflare-shared/references/pytorch-model-exchange.md; then ../nvflare-shared/references/validation-evidence.md and references/huggingface-validation.md; load references/huggingface-state-and-distributed.md and other shared references only under the triggers above. Do not depend on repository examples.

Files

references/huggingface-validation.md

Hugging Face Conversion Validation

Use ../../nvflare-shared/references/validation-evidence.md for compilation, Recipe construction, export, package inspection, simulation, terminal evidence, and reporting. Run these Trainer-specific checks only after the shared ladder reaches the applicable phase, and stop at the first failure.

Generated Trainer Contract

  • Adapt ../assets/client_with_eval.py, ../assets/server_model.py, and ../assets/job.py rather than drafting replacements. Review the generated differences and exercise the asset functions directly; do not write one-off AST, class-loader, persistor, Recipe-source, or simulator-source probes.

  • Confirm the generated client keeps one persistent Trainer, one flare.patch(trainer) call, no manual model receive/send path, and the source-prescribed evaluate/train sequence.

  • Keep model, tokenizer/processor, datasets, collator, Trainer arguments, and callbacks outside the FL round loop. The generated FL client must always reach flare.init() and flare.patch(trainer); preserve standalone behavior only behind an explicit non-client entry parameter.

  • Pass the final generated train_args through the client entry's actual argument mechanism and actual dataclass types in parse-only mode; reject every unused argument. When the generated client uses HfArgumentParser, construct it with allow_abbrev=False and parse with the same project and framework dataclass types. When it preserves argparse or another parser, use that parser instead. Test the exact arguments produced by the generated job helper; do not append an argument only in preflight that the recipe will not deliver.

  • Run intentional typo and abbreviation rejection cases through the shared assertion-wrapper rule. HfArgumentParser.parse_args_into_dataclasses() may raise ValueError for unused arguments rather than SystemExit; the wrapper must catch that exact ValueError, confirm its diagnostic names the rejected argument, and treat the rejection as success so the wrapper itself exits zero. A wrapper that catches only SystemExit is invalid.

  • Version-check only fields claimed to belong to a framework TrainingArguments or SFTConfig base class. Preserve a project-defined subclass field when its source definition is verified and the actual parser accepts it.

  • For the default in-process executor, require unquoted whitespace-free train_args values. Do not assume shell parsing or call internal command splitters. If a required value contains whitespace and no documented structured argument surface exists, fail closed.

  • Only when source inspection found a source-project-relative local file or directory argument, change to a fresh working directory outside the project, construct the Recipe with that relative source value, and inspect the final train_args. Require the source argument name to be unchanged and its transmitted value to be absolute and equal to the expected location under the explicit inspected source-project root before running the full simulation. SOURCE_DIR is that root only when generated job.py is colocated with the source; a read-only-source flow must validate against the separate original root. Skip this path-specific check when the source has no local path argument. Do not pass absolute local paths, per-site paths, Hub identifiers, or URLs through the relative-path test; validate their classification using ../../nvflare-shared/references/site-data-and-paths.md instead.

Hugging Face Artifacts And Compatibility

  • Before constructing a model or tokenizer/processor, run the maintained ../scripts/resolve_model_snapshot.py with --source local for a configured path or --source hub for a Hub repository ID. Never infer the source type from slash syntax: models/checkpoint and org/model are intentionally distinguished by that required option. The cache-only Hub path catches LocalEntryNotFoundError, emits a structured missing result, and exits zero; report that result as a blocker.

  • Use the identifier as the positional argument; do not invent a --model option. These are the canonical invocations:

    python <skill-dir>/scripts/resolve_model_snapshot.py --source local --source-root <absolute-source-root> <configured-path>
    python <skill-dir>/scripts/resolve_model_snapshot.py --source hub <org/model>
    python <skill-dir>/scripts/resolve_model_snapshot.py --source hub --repo-type dataset <org/dataset>

    A relative local identifier requires the absolute original source-project root so resolution never depends on the validation command's working directory. An absolute local identifier must omit --source-root; the option resolves relative identifiers and is not a sandbox boundary.

  • Only when the user authorizes downloading the specific public Hub artifact if uncached, rerun its resolver once with the matching canonical invocation:

    python <skill-dir>/scripts/resolve_model_snapshot.py --source hub --allow-download <org/model>
    python <skill-dir>/scripts/resolve_model_snapshot.py --source hub --repo-type dataset --allow-download <org/dataset>

    In that authorized invocation, the resolver uses the matching public HfApi().model_info(...).sha or HfApi().dataset_info(...).sha result, validates the full immutable 40-character commit SHA, and passes it to snapshot_download. An already audited full SHA may instead be supplied with --revision. For a model repository, use the returned immutable revision and resolved_path for the server and every client. For a dataset repository, pass resolved_path only to a source-compatible local dataset loader; if the source cannot load that snapshot layout, ask or fail closed rather than inventing a loader. Do not run a preceding snapshot_download(..., local_files_only=True) probe, copy resolver logic into generated job.py, or download without authorization. For a configured Hub dataset ID, use the same resolver with --repo-type dataset; for a source-prescribed local dataset path, use the existing local path without a Hub lookup.

  • Never recover from an offline/cache-only miss by removing HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE, dropping local_files_only=True, or rerunning online unless the user explicitly requested the download. Do not substitute another local checkpoint. Keep the job as a draft and report full-model validation blocked when the required artifact is unavailable. A cache-only Hugging Face error can mention huggingface.co while reporting a local cache miss; that URL alone is not evidence that a network request was attempted. Judge the configured offline flags, local_files_only setting, and exception/result type instead.

  • Pass the same resolved local model path and cache configuration from the resolver to the server model and every client. Do not validate with a cached Hub identifier and export a job that depends on an unverified online lookup.

  • Do not call flare.init(), flare.patch(), flare.is_running(), or patched Trainer methods in a standalone preflight; they require the Client API context created by the recipe or simulator launcher. Construct the Trainer without patching, inspect the generated patch call and supported public signature statically, and let the first bounded simulation validate runtime patch acceptance. Reject or report DeepSpeed, FSDP, best-model-at-end, save-only-model, prebuilt optimizer/scheduler, and checkpoint/restore combinations identified in huggingface-state-and-distributed.md.

Parameter Checks

  • Instantiate the adapted server_model.ServerModel and the source Trainer model through the same importable factory path and with the same local constructor arguments. Always compare state-dict key sets, shapes, and dtypes without training. Do not inspect NVFLARE persistors or class loaders; the exported job and final simulation validate product-side construction.

  • Before comparing values, classify checkpoint-loaded versus missing or newly initialized parameters from source evidence and the available model load report/loading information. Loading the same fixed checkpoint proves value determinism only for parameters actually loaded from it; a checkpoint that omits a task head or other exchanged parameters does not prove deterministic construction of the complete model.

  • For checkpoint-loaded parameters, or when the factory explicitly resets all initialization seeds/state on every call, require exact per-tensor equality or an equivalent stable state hash between independently constructed source and server models. A single coincidentally equal run is not proof. For keys reported as missing/newly initialized without a deterministic reset, require matching keys, shapes, and dtypes, record their names and provenance, and do not require their independently initialized values to match. If per-key provenance is unavailable, treat the factory as nondeterministic rather than assuming the entire fixed-checkpoint construction is deterministic.

  • Determine this comparison policy before asserting values. Do not first run an unconditional full-state equality assertion and then recover by excluding newly initialized keys after it fails.

  • Require exact full-model key agreement after any documented prefix transformation. For PEFT or auxiliary trainable models, load huggingface-state-and-distributed.md and run its adapter/ownership checks.

  • Confirm tensor-native exchange only when selected by the capability profile.

Standard Execution Checks

  • Follow the shared compile, construction, export, package-inspection, and one final foreground simulation path. Do not add separate Recipe-method, argument-transport, simulator-process, or export-dispatch probes when the maintained job asset compiles and the public validation rungs pass.

  • Require terminal completion evidence, positive per-round training step counts, and finite source-backed evaluation metrics for the claimed stage.

  • Confirm the requested budget appears once, metrics advance across requested rounds, and server artifacts contain the selected metric.

Conditional Execution Checks

  • When the real model may exceed host capacity, estimate server and concurrent client memory before simulation. Capability-check optional diagnostics such as free or nvidia-smi, run them separately from correctness checks, and make absence an exit-zero result; missing diagnostics mean capacity evidence is unavailable, not conversion failure. Use a reduced one-round topology smoke only when the real model cannot fit, keep the generated default unchanged, and report full-model validation blocked.

  • After partial aggregation, unexplained disconnect, or exit -9, inspect logs and either reduce workload with a changed causal factor or report a resource blocker. Do not retry the same payload with guessed concurrency settings, and make at most one expensive real-model retry.

  • For non-default checkpoint/restore behavior, PEFT, auxiliary trainable models, or DDP, load huggingface-state-and-distributed.md and run only its applicable checks.

  • Do not report DDP validated from a single-process or rank-zero-only simulation. A DDP validation claim requires a two-process torchrun case in which each process records its launcher rank and product-resolved Client API rank, and the observed ranks are exactly {0, 1}.

Report

Report commands, exit codes, terminal status, metric keys/values, parameter scope, checkpoint mode, process/rank count, exact workspace/export/result paths, and unresolved validation blockers with concrete reasons.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Medium risk

  • Large base64-encoded blobs
  • Instructions that use the network
  • Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
NVIDIA FLARE Team
Publisher
NVIDIA (Official)
Source
NVIDIA/skills · skills/nvflare-convert-huggingface · commit fce3cca
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE-APACHE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by NVIDIA FLARE Team or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download nvflare-convert-huggingface.zip