Golang Performance

Runtime Tuning

Runtime settings control garbage collection frequency, memory limits, CPU scheduling, and compiler optimizations. Tune them after profiling — the defaults are well-chosen for most workloads.

Table of Contents

• Garbage Collector Tuning

  • GOGC (default: 100)

  • GOMEMLIMIT (Go 1.19+)

  • Programmatic control

  • Ballast pattern (pre-Go 1.19)

• GC Profiling and Diagnostics

  • GODEBUG=gctrace=1

  • runtime.ReadMemStats

  • GC pacing

• Allocation Rate Reduction

• GOMAXPROCS in Containers

• Profile-Guided Optimization (PGO)

• Logging Overhead in Hot Paths

• Panic/Recover Cost

Garbage Collector Tuning

Diagnose: 1- GODEBUG=gctrace=1 — print one line per GC cycle; look for high GC frequency (cycles/s), high CPU% (>5% means GC is competing for CPU), or heap growing faster than expected 2- runtime.ReadMemStats — inspect Alloc, TotalAlloc, NumGC, PauseNs; compare Alloc vs Sys to see how much memory the GC is reclaiming vs how much the OS allocated 3- go tool trace — visualize GC stop-the-world pauses and GC assist stealing CPU from application goroutines; look for long STW bars or frequent assist marks 4- debug.ReadGCStats — get pause time percentiles (p50, p95, p99); high p99 pauses indicate large heap scans or too many pointers 5- runtime/metrics — programmatic access to GC stats for dashboards; monitor /gc/cycles/total, /gc/heap/allocs, /gc/pauses 6- GODEBUG=gcpacertrace=1 — trace the GC pacer's decisions; useful to understand why GC triggers earlier or later than expected 7- Prometheus rate(go_gc_duration_seconds_count[5m]) — monitor GC frequency in production; >2 cycles/s sustained suggests excessive allocation rate

GOGC (default: 100)

Controls the heap growth ratio that triggers the next GC cycle. GOGC=100 means GC runs when the heap doubles since the last collection. Higher values reduce GC frequency but use more memory:

GOGC=50  ./myapp  # latency-sensitive: more frequent, shorter GC pauses
GOGC=200 ./myapp  # throughput-oriented: less frequent GC, more memory used
GOGC=off ./myapp  # disable GC entirely (testing only!)

GOMEMLIMIT (Go 1.19+)

Soft memory limit — the runtime increases GC frequency to stay under this limit. Essential for containerized applications where exceeding the container limit triggers an OOM kill:

# Container with 512MB limit: leave headroom for non-heap memory (goroutine stacks, OS buffers)
GOMEMLIMIT=450MiB ./myapp

# Container with 1GB limit
GOMEMLIMIT=900MiB ./myapp

The GC pacer adjusts collection timing based on both GOGC and GOMEMLIMIT. When the heap approaches the limit, the GC runs more aggressively regardless of GOGC.

Programmatic control

import "runtime/debug"

debug.SetGCPercent(200)                    // equivalent to GOGC=200
debug.SetMemoryLimit(450 * 1024 * 1024)   // 450 MiB soft limit

Use programmatic control for dynamic tuning based on observed workload, or when environment variables cannot be set.

Ballast pattern (pre-Go 1.19)

Before GOMEMLIMIT, teams allocated a large byte array at startup to inflate the live heap size, reducing GC frequency:

var ballast [1 << 30]byte // 1 GB — obsolete pattern

GOMEMLIMIT is strictly better — it provides the same benefit (fewer GC cycles) without wasting physical memory. Use GOMEMLIMIT instead.

GC Profiling and Diagnostics

GODEBUG=gctrace=1

Prints a line per GC cycle to stderr:

GODEBUG=gctrace=1 ./myapp 2>&1 | head -20

Sample output:

gc 5 @1.234s 2%: 0.012+12+0.9 ms clock, 0.25+8.9/20+18 ms cpu, 45->92->50 MB, 200 MB goal, 8 P

Key fields:

• gc 5 — 5th GC cycle

• @1.234s — time since program start

• 2% — total CPU time spent in GC

• 45->92->50 MB — heap before → peak during collection → after

• 200 MB goal — target heap size (based on GOGC and GOMEMLIMIT)

• 8 P — number of processors

Watch for: GC frequency (too often = too many allocations), pause times (high = large heap or many pointers), CPU% (high = tune GOGC or reduce allocations).

runtime.ReadMemStats

Programmatic monitoring for dashboards and alerting:

var m runtime.MemStats
runtime.ReadMemStats(&m)

fmt.Printf("Alloc: %d MB\n", m.Alloc/1024/1024)       // currently allocated
fmt.Printf("TotalAlloc: %d MB\n", m.TotalAlloc/1024/1024) // cumulative
fmt.Printf("Sys: %d MB\n", m.Sys/1024/1024)            // requested from OS
fmt.Printf("NumGC: %d\n", m.NumGC)                      // completed collections
fmt.Printf("LastPause: %d ms\n", m.PauseNs[(m.NumGC+255)%256]/1_000_000)

GC pacing

The GC pacer predicts when to start the next collection based on:

1. Live heap size after the last collection

2. GOGC percentage — how much growth to allow

3. GOMEMLIMIT — soft ceiling (if set)

4. Current allocation rate — how fast the heap is growing

The pacer starts collection early enough to finish before hitting the target. Fast allocation rates cause earlier starts.

Allocation Rate Reduction

Diagnose: 1- go tool pprof -alloc_objects — rank functions by allocation count; the top allocators are where allocation reduction will have the biggest GC impact 2- GODEBUG=gctrace=1 — monitor GC frequency before and after reducing allocations; expect fewer GC cycles per second as allocation rate drops 3- Prometheus rate(go_memstats_alloc_bytes_total[5m]) — track allocation rate trend in production; compare before/after deploy to detect regressions

Reducing allocations helps more than tuning GOGC — it addresses the root cause instead of managing the symptom:

• Value types over pointer types where possible — values stay on the stack (no GC), pointers escape to the heap

• Pool frequently allocated objects with sync.Pool (see memory.md)

• Preallocate slices and maps — → See samber/cc-skills-golang@golang-data-structures skill

• Avoid interface boxing in hot paths — use typed parameters or generics

GOMAXPROCS in Containers

Diagnose: 1- go tool pprof (CPU profile) — look for high runtime.schedule or runtime.findRunnable overhead; this indicates too many P's competing for work or too few P's starving goroutines 2- go tool trace — check if goroutines are evenly distributed across P's; uneven distribution suggests GOMAXPROCS is misconfigured for the container 3- GODEBUG=schedtrace=1000 — print scheduler state every second; look for runqueue imbalances or idle P's when work is available 4- runtime.GOMAXPROCS(0) — query the current value; if it returns the host CPU count (e.g., 64) instead of the container limit (e.g., 2), the runtime is over-scheduling 5- Prometheus rate(process_cpu_seconds_total[5m]) — monitor CPU cores consumed in production; if consistently near GOMAXPROCS value, the app is CPU-saturated

Go 1.25+ improves container CPU detection, particularly for cgroup v2. The runtime sets GOMAXPROCS based on:

• Logical CPUs on the machine

• Process CPU affinity mask

• cgroup CPU quota limits (on Linux)

In a container with 2 CPU cores on a 64-core host running Go 1.25+ with cgroup v2, GOMAXPROCS is correctly set to 2 by default. For cgroup v1 environments, validate the detected value at startup and consider using go.uber.org/automaxprocs to ensure correctness.

For Go 1.24 and earlier, use the go.uber.org/automaxprocs library to handle container CPU detection:

// Pre-Go 1.25: explicit container-aware detection
import _ "go.uber.org/automaxprocs"

func main() {
    // GOMAXPROCS is now correctly set to container CPU limit
    startServer()
}

Manual override (if needed):

GOMAXPROCS=2 ./myapp
GODEBUG=updatemaxprocs=0 ./myapp  # disable dynamic updates (Go 1.25+)

Known limitations (Go 1.25): cgroup v1 on certain systems (Oracle OCPUs) may not properly detect Kubernetes CPU limits. Manually set GOMAXPROCS as a workaround in these cases.

Profile-Guided Optimization (PGO)

Diagnose: 1- go tool pprof (CPU profile) — collect a representative production profile (30+ seconds); look for hot interface method calls and deep call chains that PGO can optimize via devirtualization and inlining 2- go test -bench — benchmark before and after placing default.pgo; expect 2-7% improvement on interface-heavy code, less on already-optimized paths

Go 1.21+ supports PGO — the compiler uses a production CPU profile to make better inlining and devirtualization decisions. Expected improvement: 2-7% for minimal effort.

Workflow:

1. Collect a production CPU profile (30+ seconds of representative load):

   Copy

   curl http://localhost:6060/debug/pprof/profile?seconds=60 > cpu.pprof

2. Place as default.pgo in the main package directory:

   Copy

   cp cpu.pprof ./cmd/myapp/default.pgo

3. Build — go build auto-detects default.pgo:

   Copy

   go build ./cmd/myapp

What the compiler optimizes:

• Inlining — hot function calls are inlined more aggressively

• Devirtualization — interface method calls with high probability of targeting specific types become direct calls

When it helps most: code with many interface calls, hot inlining opportunities, deep call stacks. When it helps least: already-optimized code, memory-bound workloads.

Rebuild profiles after significant code changes — stale profiles can mislead the compiler.

Logging Overhead in Hot Paths

Diagnose: 1- go tool pprof (CPU profile) — look for fmt.Sprintf, log.Printf, or slog.(*Logger).log appearing in hot paths; these indicate log formatting consuming CPU even when the log level filters the message 2- go build -gcflags="-m" — check if log arguments escape to the heap; expect "moved to heap" for arguments boxed into any interface by logging functions 3- go test -bench -benchmem — benchmark with logging enabled vs disabled; if allocs/op doesn't change, the logger is allocating even when the level is off

Log formatting allocates memory and consumes CPU even when the message is discarded because it's below the configured level:

// Bad — fmt.Sprintf runs BEFORE the logger checks the level
logger.Debug(fmt.Sprintf("processing item %d with data %v", item.ID, item.Data))

// Good — slog defers formatting until level check passes (Go 1.21+)
slog.Debug("processing item", slog.Int("id", item.ID), slog.Any("data", item.Data))

// Best — LogAttrs: zero allocations when level is disabled
slog.LogAttrs(ctx, slog.LevelDebug, "processing item",
    slog.Int("id", item.ID))

In hot paths, even slog.Any can allocate. Prefer typed attributes: slog.Int, slog.String, slog.Bool.

Panic/Recover Cost

Diagnose: 1- go tool pprof (CPU profile) — look for runtime.gopanic or runtime.gorecover in the profile; their presence in hot paths means panic/recover is being used for control flow 2- go test -bench — benchmark panic/recover vs error-return versions; expect 10-100x overhead from stack unwinding and defer execution

panic triggers stack unwinding, running all deferred functions up the call stack. recover catches the panic but the unwinding itself is expensive. Never use panic/recover for control flow:

// Bad — panic overhead for a normal condition
defer func() { recover() }()
v, _ := strconv.Atoi(s) // relies on panic for invalid input

// Good — explicit error check, no panic overhead
v, err := strconv.Atoi(s)
if err != nil { continue }

Panic is appropriate only for truly unrecoverable situations (programmer errors, corrupted state). Always convert panics to errors at package boundaries.