Doca Rdma

DOCA RDMA workflows

Where to start: The verbs run configure → build → modify → run → test → debug. Skip ahead only when the user is already past a verb. The ## test verb is an iterative loop (capability check → permission check → bidi smoke → completion drain → loop back if the permission set or task set changed), not a one-shot pass — see the eval-loop overlay in ## test below.

Read this file when the loader sent you here from SKILL.md. For the underlying capability matrix, version compatibility, error taxonomy, observability surface, and safety policy that these workflows assume, see CAPABILITIES.md. For where to find docs, the installed DOCA layout, or release notes, route through doca-public-knowledge-map/SKILL.md.

Each verb below describes the shape of the workflow, not a copy-paste recipe. The agent’s job is to walk the user through the steps in order, verifying preconditions before recommending the next call.

configure

Goal: bring up a DOCA RDMA context on a host or BlueField and confirm both sides are in a state where data-movement tasks are meaningful.

Core order, named in full every time: create → configure → start → connect → use → stop → destroy. Task configuration and permission setting belong in configure and MUST finish before doca_ctx_start(). Connect is its own step after start; do not fold it into use.

Steps the agent should walk the user through:

1. Confirm the installed DOCA version. Use the procedure in doca-setup CAPABILITIES.md ## Version compatibility (doca-setup/CAPABILITIES.md#version-compatibility). Quote the version observed (pkg-config --modversion doca, then doca_caps --version); do not assume "latest".

2. Discover the device capability surface for RDMA. Run doca_caps --list-devs (doca-caps/SKILL.md) to see which devices have RDMA capability, then run the per-doca_devinfo doca_rdma_cap_* queries against the candidate device. Record the link layer the active port already exposes (IB or Ethernet/RoCE) separately from the DOCA RDMA transport type (RC or DC, with DC alpha and restricted as described in the capabilities matrix). IB / RoCE is not an argument to doca_rdma_set_transport_type(). Also record which task types are supported for the selected transport and what doca_rdma_cap_get_max_* returns for the queue / buf-list sizes. The capability matrix to compare against lives in CAPABILITIES.md ## Capabilities and modes.

3. Decide the connection method. RDMA CM (callback-driven) vs bridge / OOB vs gRPC. The pick decides the next-step code shape; the trade-off matrix lives in CAPABILITIES.md ## Capabilities and modes connection-methods table. Do not pick for the user when their intent is ambiguous — ask.

4. Configure the RDMA instance. Mandatory before doca_ctx_start(): enable at least one task type (doca_rdma_task_*_set_conf); set permissions (doca_rdma_set_permissions); set the mmap permissions (doca_mmap_set_permissions) to match the task type per the matrix in CAPABILITIES.md ## Safety policy. Optional but commonly needed: doca_rdma_set_transport_type(), doca_rdma_set_max_num_connections(), doca_rdma_set_* queue sizes. Then doca_ctx_start(), then connect with the method from step 3, then use. Never connect or submit before start; tear-down is stop then destroy.

5. Sanity check before any task submission. Confirm with the user: which side is requesting work (Read/Write originator), which side is exporting memory (doca_mmap_export_rdma()), which transport, which connection method. If any of those are unclear, stop and ask — do not invent.

If any step fails with a DOCA_ERROR_*, route through the error taxonomy in CAPABILITIES.md ## Error taxonomy before retrying. Configure retries use the bounded identity defined in ## test: the same error family and configure step on the same device, link layer, transport, task set, permission set, and connection method, with unchanged logs, capability results, and connection-state evidence. Stop after two unchanged iterations and escalate instead of retrying indefinitely.

build

Goal: produce a binary that links DOCA RDMA against the user's installed DOCA, using the canonical cross-library build pattern.

The build pattern for any DOCA C/C++ consumer is identical across libraries — pkg-config for include + link flags, meson or CMake as the build system — and is fully documented in doca-programming-guide TASKS.md ## build (doca-programming-guide/TASKS.md#build).

Step 0 — resolve the pkg-config module (do this first; do not assume). DOCA RDMA is normally delivered inside the umbrella doca pkg-config module; split installs may expose a per-library module. Resolve the actual module on the target before writing any build file:

1. Make sure pkg-config can see DOCA. If pkg-config --exists doca fails, discover the install's pkgconfig directory first with find /opt -name 'doca-*.pc' -path '*/pkgconfig/*', then prepend that discovered directory to PKG_CONFIG_PATH.

2. List what actually exists: pkg-config --list-all | grep -i doca. Use the most specific RDMA module that exists if a split install exposes one; otherwise use the umbrella doca (the normal case). The public RDMA header is doca_rdma.h and the shared object is libdoca_rdma.so regardless of which .pc resolves them.

Guardrail — DOCA is the deliverable, not "any RDMA". If pkg-config doca-rdma (or any --libs/--cflags probe) fails, the cause is almost always a wrong module name or an unset PKG_CONFIG_PATH — fix that. Do NOT silently fall back to raw libibverbs / librdmacm / RDMA-CM and hand-roll a verbs program. That technically moves bytes but abandons DOCA entirely, which defeats the purpose of this work. A build that does not link libdoca_rdma is a failed DOCA-RDMA build — troubleshoot the module resolution above (and escalate via the error taxonomy) instead of bypassing DOCA.

Concrete path — build the shipped DOCA RDMA sample pair first. Before writing any new code, start from the samples the DOCA RDMA guide ships under /opt/mellanox/doca/samples/doca_rdma/. Each sample directory holds <name>_sample.c, <name>_main.c, and meson.build. The pairs are:

1. ls /opt/mellanox/doca/samples/doca_rdma/ — quote the pairs this install actually ships before naming one.

2. pkg-config --modversion doca-rdma (or the module Step 0 resolved) — record the version next to the build.

3. Build each side out of tree with meson + ninja, as the guide documents: cd /opt/mellanox/doca/samples/doca_rdma/rdma_receive, meson /tmp/build-rdma-receive, ninja -C /tmp/build-rdma-receive; repeat for rdma_send into its own build directory. The binary is doca_<sample_name> in that build directory. Never edit files under /opt/mellanox/doca/samples/ or leave build directories there — when the user's intent needs a code change, copy the whole samples tree to a workspace first so the shared helper files still resolve, and edit and build the copy.

4. Prove the routing on the exact binary you will run, as its own unfiltered command: ldd /tmp/build-rdma-send/doca_rdma_send, and read the libdoca_rdma.so => … line in the full output. No such line means this is not a DOCA RDMA build — fix the build; do not switch to another tool.

This skill carries only the RDMA-specific overlay:

For non-C consumers (Rust, Go, Python), the link surface is the same *.so files (cgo #cgo pkg-config: doca, Rust pkg-config crate on doca, etc.); the FFI wrapper layer is the language-specific binding and is out of scope for this skill — but the slots above (and the guardrail) are still the load-bearing inputs the wrapper needs.

modify

Goal: take a shipped DOCA RDMA sample as the verified starting point and apply a minimum-diff modification to express the user's intent.

Non-C languages (Go / Rust / Python): this is still the right verb. Do not reimplement RDMA in raw libibverbs to avoid C. Start from the shipped sample under $(pkg-config --variable=prefix doca)/samples/doca_rdma/<name>/ (substitute the module resolved in ## build Step 0), then expose its entry points (e.g. the sample's send/receive or write/read driver functions) through a thin FFI shim — for Go, one small *.c/*.h pair compiled via #cgo pkg-config: doca calling the sample functions, with main.go orchestrating. This keeps the verified DOCA datapath intact and links libdoca_rdma; it is a single small wrapper, not a full re-binding of the DOCA API.

The universal modify-a-shipped-sample workflow lives in doca-programming-guide TASKS.md ## modify (doca-programming-guide/TASKS.md#modify). Use it as-is. The RDMA-specific overlay is the modify-from-sample schema fill — the five slots the agent must elicit from the user before recommending any code-level edit:

The agent emits an intent description + the five filled slots; the actual unified diff against the sample source is produced by the user (this bundle has no diff renderer). The agent must walk the user through the diff line-by-line against the sample source they read on disk, and have the user paste back the result for validation. Modify is complete only when (1) all five slots match the edited sample, (2) the resulting binary links DOCA RDMA (libdoca_rdma is present in the resolved link/runtime dependency surface), and (3) the edited sample builds successfully with the install-resolved pkg-config flags.

run

Goal: actually execute the built binary against the user's installed DOCA on a host or BlueField, including a peer to connect to.

Steps the agent should walk the user through:

1. Confirm the peer is reachable. RDMA needs a peer; running the binary on one side alone produces a misleading hang. Either both sides are on networks that route IB or RoCE to each other, or both run as local processes on a single host over the same doca_dev (see Single-host loopback below) — the latter is the right first-run smoke test when only one BlueField / NIC is on hand.

2. Run the side that listens first. Server (RDMA CM) or accept-side (bridge / OOB) must be running and at the connection-listening state before the client / connect-side starts. Confirm with the user which side is which.

3. Capture the structured log. Set DOCA_LOG_LEVEL=trace for the first run (see doca-debug CAPABILITIES.md ## Observability (doca-debug/CAPABILITIES.md#observability)). This is the cheapest way to make the lifecycle and connection state transitions visible on first failure.

4. Capture the completion events on the PE. A run that produces no completion events but doesn't error is almost always a missed doca_pe_progress() call. Confirm the progress engine is being driven on both sides.

Two peers on one box (only one BlueField / NIC available). A peer does not have to be a second host. Run the responder / receiver and the requester / sender as two separate processes on the same active port (-d <ibdev>), or across two ports, or host ↔ BlueField. Two processes on one port still move the bytes through the NIC. What does not count is a same-process, same-buffer loopback, or a client / server pointed at 127.0.0.1 / localhost. Exchange connection details with the samples' own arguments (DOCA 3.5 RDMA guide, common sample arguments):

• Out-of-band descriptor files (default). -ld / --local-descriptor-path is where a side writes its own descriptor; -re / --remote-descriptor-path is where it reads the peer's; the one-sided read/write samples also take -m / --mmap-descriptor-path for the peer's exported mmap. On one box the receiver's local path is the sender's remote path and vice versa. Start the responder / receiver first, then the requester / sender.

• RDMA CM. -cm / --use-rdma-cm on both sides; the listener takes -lp / --listen-port, the connector takes -sa / --server-addr with -sat / --server-addr-type (IPv4, IPv6, or GID). Use the port's own IP or GID, never a loopback address.

Two caveats that bite agents: (a) the descriptor handshake waits for Enter on each side once both descriptor files exist, and the write/atomic samples expect a second Enter on the responder after the requester reports its task done — a single Enter looks like a hang; drive these prompts from the same foreground command and wait for both processes, do not leave them in the background; (b) prove the transfer with the independent evidence in ## test (receiver-side data check plus the receiving port's RDMA byte counters sampled before and after), not from the sample's log alone.

test

Goal: prove the configured RDMA instance can actually move data correctly between the two sides on the user's hardware, and that the permission and capability set was sized right.

This is a loop, not a one-shot pass. Each iteration narrows either the capability set, the permission set, the task set, or the sizing. The loop terminates when either (a) the user's intended data-movement pattern flows end-to-end with the expected completions, or (b) the agent has narrowed the failure cause to a layer outside RDMA itself (driver / firmware / network) and escalated to the matching skill.

What proves a DOCA RDMA transfer — and what does not. Declare the data path working only when one run captures all four:

1. Routing. Plain ldd on the executed requester and responder binaries (built from the shipped sample per ## build) shows libdoca_rdma.so.

2. Two peers. Responder / receiver and requester / sender are separate processes with different PIDs (two hosts, host ↔ BlueField, two ports, or one active port). The responder starts first, the command waits for both, and both exit 0.

3. Data check. The receiver validates the payload it actually received against what the sender sent — for example the SHA-256 of the sent and received buffers. Use synthetic test bytes, never user data.

4. NIC evidence. Sample the receiving port's RDMA byte counters before and after the run — map the device to its netdev with ibdev2netdev, then read ethtool -S <netdev> (the mlx5 rx_vport_rdma_unicast_bytes / tx_vport_rdma_unicast_bytes counters). The receive-side delta should be at least the payload size.

ib_write_bw, ib_send_bw, rping, ibv_rc_pingpong, and other rdma-core / perftest tools exercise only the verbs layer and the fabric: they can be a fabric pre-check, but they do not validate the DOCA RDMA path and are never the deliverable. Neither is a loopback shortcut (same process / same buffer, or 127.0.0.1).

Iteration shape:

1. Capability re-check. Re-run doca_rdma_cap_task_*_is_supported for every task the user intends to submit, against the active doca_devinfo. If false → that's the answer; the user's device or DOCA version does not support the task. Update the user's intent or update the install.

2. Permission cross-check. Compare the configured mmap + RDMA-permission pair against the matrix in CAPABILITIES.md ## Safety policy. Mismatches surface as DOCA_ERROR_NOT_PERMITTED on the first task submission, not at configure time.

3. Bidirectional smoke. Always start with a Send + Receive pair, even if the user's intent is one-sided (Read / Write). If the bidirectional control path works, the connection + two-sided permissions are correct, and any subsequent one-sided failure narrows cleanly to the one-sided permission set. Skipping this step is the most common reason "Write fails and we don't know why".

4. Completion drain. Confirm completion events arrive on the PE for every submitted task. Submitted but no completion is the most expensive class of bug to discover late; confirm it on the smoke pair before adding bulk submissions.

5. Negative test. Once the positive path works, intentionally submit one task type the device should NOT support (per step 1) and confirm the failure is the expected DOCA_ERROR_NOT_SUPPORTED. This validates the agent's capability-discovery is itself correct. Run this only in a non-production test environment, on a dedicated test connection and mmap that contain no production data.

Eval-loop overlay — why this is a loop, not a one-shot pass:

Loop identity is the same trigger or DOCA_ERROR_* family for the same device, link layer, transport, task set, permission set, connection method, and configured timeout, with unchanged program log, capability results, completion events, and connection-state callbacks. Stop after two consecutive iterations with that identity produce no evidence change — that means the cause is below RDMA. Escalate to doca-debug ## debug (doca-debug/TASKS.md#debug) with the captured layer-1-through-5 evidence. The configure workflow reuses this exact identity and bound.

debug

Goal: when a DOCA RDMA call returns a DOCA_ERROR_* (or the program doesn't make forward progress), narrow the cause to a specific layer and act on it.

The cross-library debug ladder lives in doca-debug TASKS.md ## debug (doca-debug/TASKS.md#debug). Walk through it in order — install → version → build → link → runtime → program → driver — before recommending RDMA-specific fixes. This skill's overlay names the RDMA-specific manifestation at layers 5 (runtime) and 6 (program):

Layer 5 (runtime) — RDMA overlay.

• Walk the connection state machine: was the context started? Was the connection request received? Was the connection established? Was a task submitted before the connection reached established? The wrong order returns DOCA_ERROR_BAD_STATE, not a clear symptom.

• Confirm the PE is being progressed. No completion events is almost always a missing doca_pe_progress() in the user's main loop.

• Confirm both sides agreed on transport type. A mixed-transport pair returns DOCA_ERROR_DRIVER from the layer below; the fix is at configure time, not in the RDMA call.

Layer 6 (program) — RDMA overlay.

• Permission matrix: the most common RDMA program-layer bug is an mmap-permission / RDMA-permission mismatch surfaced as DOCA_ERROR_NOT_PERMITTED. Walk the matrix in CAPABILITIES.md ## Safety policy against the user's configured permissions.

• Lifecycle order: create → configure → start → connect → use → stop → destroy. Out-of-order returns DOCA_ERROR_BAD_STATE. The most common case is destroying an mmap before doca_ctx_destroy().

• Bulk-submit sizing: DOCA_ERROR_FULL on bulk submit is a sizing-vs-drain mismatch, not a hardware failure.

Once the layer is identified, route to the matching debug verb on the matching skill: install / build / link / driver to doca-setup ## debug (doca-setup/TASKS.md#debug); cross-cutting runtime to doca-debug ## debug (doca-debug/TASKS.md#debug); program-layer Core-context patterns to doca-programming-guide TASKS.md ## debug (doca-programming-guide/TASKS.md#debug).

Deferred task verbs

The following verbs are out of scope for this skill but are commonly asked in the same conversations. Route them as follows so the agent does not invent guidance:

• install. Installing DOCA, choosing packages, post-install verification, pkg-config wiring — defer to doca-setup/SKILL.md and to the install-tree layout in doca-public-knowledge-map ## Layout of an installed DOCA package (doca-public-knowledge-map/SKILL.md#layout-of-an-installed-doca-package). This skill assumes DOCA is already installed.

• deploy. Deploying RDMA-using applications at scale across many hosts / DPUs, Kubernetes operator workflows for RDMA workloads, multi-tenant RDMA isolation — out of scope for this skill. For single-host first-run testing, the right verb in this skill is ## run; do not invent a "deploy" workflow.

• rollback. Coordinated rollback of RDMA-using applications across multiple hosts / DPUs — out of scope for this skill. For a single in-session RDMA configuration rollback, the right verb in this skill is destroying the context (doca_ctx_stop → doca_ctx_destroy) and re-running ## configure with the corrected parameters; do not invent a "rollback" workflow.

• kernel-level driver install / firmware burn. Installing the mlx5_core driver, burning new ConnectX firmware, or modifying mlxconfig parameters that need a reset — out of scope for this skill. Route to doca-setup ## debug (doca-setup/TASKS.md#debug) layer 5 (driver), then to the upstream MLNX OFED / firmware documentation reachable through doca-public-knowledge-map/SKILL.md.

Command appendix

Every command below is cross-cutting on DOCA RDMA — it answers a recurring class of question that comes up in the verbs above. The agent should treat the class as load-bearing; the worked example is a single instance. Run-as user is the unprivileged user unless noted. Rows that need elevated privileges call that out explicitly.

Infra-aware preamble (every row below). Per the bundle's detect → prefer → fall back → report contract documented in doca-structured-tools-contract ## The agent behavior contract (doca-structured-tools-contract/SKILL.md#the-agent-behavior-contract), the agent should:

1. Probe for the matching structured helper FIRST (doca-env --json for version + devices + libraries + drivers + hugepages in one shot; doca-capability-snapshot for per-device capability flags; version-matrix.json for “available since” lookups).

2. If the probe succeeds, the structured tool's output is the authoritative answer and the agent SHOULD NOT also run the manual command in the row below. Report “using structured <tool>”.

3. If the probe fails, fall back to the manual command in the row. Report “falling back to manual chain”.

4. The schemas the structured tools emit are defined in doca-structured-tools-contract ## Schemas (doca-structured-tools-contract/SKILL.md#schemas); the version-handling semantics (four-way match, NGC, headers-win) are owned by doca-version/SKILL.md.

For commands shared across libraries (pkg-config --modversion, doca_caps, the install-prefix applications/VERSION, DOCA_LOG_LEVEL) the cross-library overlay is in doca-debug TASKS.md ## Command appendix (doca-debug/TASKS.md#command-appendix); this table adds the RDMA-specific rows on top.