Redshift Guide
AWSOfficialPublished by AWS
- Skill
- Tool integration
- Cloud and DevOps
- Apache-2.0
- No issues found
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
The Redshift Guide is a technical reference designed to prevent common errors when working with Amazon Redshift. Because Redshift shares a wire protocol with PostgreSQL, AI models often incorrectly apply PostgreSQL syntax and behaviors to Redshift. This guide provides the necessary corrections, covering DDL, system metadata, query performance, storage management, and authentication. It distinguishes between provisioned clusters and Serverless workgroups, ensuring users apply the correct system views and API calls for their specific environment.
What it does
- Corrects common misconceptions where PostgreSQL syntax is incorrectly applied to Redshift.
- Provides guidance on Redshift-specific DDL, including distribution and sort keys.
- Outlines the correct system views (SYS_ vs STL_) for monitoring and metadata discovery.
- Details secure authentication practices for JDBC/ODBC and the Data API.
- Offers troubleshooting steps for storage issues, query performance, and COPY/UNLOAD operations.
- Enforces the use of Redshift-specific commands over PostgreSQL-native alternatives.
When to use it
- When you are unsure if a PostgreSQL command or function is supported in Redshift.
- When troubleshooting performance issues, disk space, or query errors in Redshift.
- When setting up new tables, loading data via COPY, or exporting data via UNLOAD.
- When configuring IAM roles, permissions, or connectivity for Redshift clusters or workgroups.
- When you need to identify the correct system views for monitoring a specific Redshift deployment.
Advantages
- Prevents errors caused by assuming PostgreSQL compatibility.
- Provides clear distinctions between provisioned and Serverless environments.
- Focuses on high-impact performance tuning like distribution and sort keys.
- Promotes secure, least-privilege access patterns for data operations.
- Reduces time spent debugging by providing a structured routing table for common issues.
Usage examples
- “How do I create a table in Redshift with a sort key?”
- The agent will provide a CREATE TABLE statement using DISTSTYLE and SORTKEY, explaining that Redshift uses these instead of standard PostgreSQL indexes.
- “Why is my query failing with a 'relation does not exist' error?”
- The agent will guide you to use SHOW commands or SYS_ views to inspect the schema and verify table existence, rather than querying information_schema.
- “How do I debug a failed COPY command?”
- The agent will instruct you to query SYS_LOAD_ERROR_DETAIL to identify the specific row or column causing the load failure.
- “Is my Redshift cluster running out of disk space?”
- The agent will help you distinguish between actual data growth and temporary query spill, and guide you to use the appropriate storage utilization references.
Install
Download the skill as a ZIP:
https://autogineer.com/agent-skills/aws/redshift-guide/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip redshift-guide.zip -d .agents/skills
Documentation by AWS
Amazon Redshift Guide
Redshift is NOT PostgreSQL (read first)
Redshift speaks PostgreSQL's wire protocol and shares much of its surface syntax, so
LLMs assume PostgreSQL behavior carries over — it frequently does not. Divergences span
system tables (pg_catalog is incomplete), DDL (no indexes, no sequences), functions
(string_agg, SUBSTR on tables, leader-node-only functions), types (a text column
becomes VARCHAR(256)), and comparison semantics (trailing blanks, unenforced constraints). Assume
divergence and verify against the reference below — do not answer from PostgreSQL habit.
Common PostgreSQL→Redshift divergences are in references/redshift-sql-syntax.md.
Works best with the AWS MCP server — it runs the AWS CLI and Redshift Data API calls below in a sandboxed, audit-logged environment. All guidance here is plain AWS CLI and SQL and works without it.
STEP 0: Serverless or Provisioned?
Establish this before answering — APIs, system tables, and capabilities differ. Take it
from the question when it says which one; ask when it does not. SELECT version()
does not identify it.
Serverless — identified by a workgroup (and namespace). Data API calls take
--workgroup-name; the user says "workgroup"/"Serverless".Provisioned — identified by a cluster. Data API calls take
--cluster-identifier; the user says "cluster".
| Target | System Views | Credentials API |
|---|---|---|
| Provisioned | SYS_, all SVV_ + STL_, STV_, SVL_, SVCS_ (single-AZ only — disabled on Multi-AZ) | redshift:GetClusterCredentials |
| Serverless | SYS_ + a subset of SVV_ ONLY (no STL/STV/SVL/SVCS) | redshift-serverless:GetCredentials |
Critical Facts
SHOW commands are the primary metadata interface — SHOW DATABASES, SHOW SCHEMAS, SHOW TABLES, SHOW COLUMNS, SHOW TABLE, SHOW VIEW. Do NOT default to pg_catalog or information_schema. → Load
references/redshift-sql-metadata.mdfor metadata/discovery questions and any "relation does not exist" report — it has the diagnostic flow.SYS_views are the preferred system views — they work everywhere.STL_,STV_,SVL_, andSVCS_are provisioned single-AZ only, and someSVV_views are unsupported on Serverless. → Loadreferences/redshift-sql-metadata.mdfor any system-view or monitoring question.sys_load_error_detailfor COPY debugging (notstl_load_errors, which is provisioned single-AZ only).DATEADD/DATEDIFF — unit-first argument order:
DATEADD(day, -30, GETDATE()),DATEDIFF(day, start, end).APPROXIMATE COUNT(DISTINCT col) — Redshift-specific, ~2% error, much faster than exact COUNT(DISTINCT) on large datasets.
MERGE ... REMOVE DUPLICATES — simplified dedup when source and target have identical schemas.
COPY should use IAM_ROLE (the namespace role, not the caller role) + supports MANIFEST for explicit file lists + MAXERROR for error tolerance.
SUBSTR()is leader-node-only — works on literals but errors on table columns (SUBSTR() function is not supported (Hint: use SUBSTRING instead)). UseSUBSTRING()on columns.UNIQUE / PRIMARY KEY / FOREIGN KEY are informational only — NOT enforced (duplicate rows are accepted with no error). Optimizer hints; enforce integrity in the application or via MERGE.
NOT NULLIS enforced.SHOW VIEW <schema.name>returns the definition of a regular view, materialized view, or late-binding view. MV freshness:SVV_MV_INFO(is_stale).No
FILTER (WHERE ...)on aggregates — PostgreSQL-only, Redshift rejects it. WriteCOUNT(CASE WHEN cond THEN 1 END)/SUM(CASE WHEN cond THEN x END)instead. This applies to every query you emit, including ones you derive from a reference example.TOP NandLIMIT Nboth work (TOP N PERCENTdoes not). Atextcolumn becomesVARCHAR(256)— useVARCHAR(max)or explicit length.Iceberg tables use
CREATE TABLE ... USING ICEBERG(notSTORED AS ICEBERG, notTABLE_FORMAT=ICEBERG).Datashares support read and write operations — consumers can write once the producer grants write privileges. Treat "permission denied" on a datashare write as a missing grant, not an unsupported operation. → Load
references/redshift-sql-metadata.mdfor requirements and limits.Storage: attribute before you remediate, and never reflex-VACUUM. Disk-full is usually NOT data growth — rule out transient query spill (temp blocks) and per-node/slice skew first, because the cluster average hides a single full node. On RA3/Serverless a high local reading is cache, not out-of-space, and free managed storage does not prevent a node-local Disk Full.
SVV_TABLE_INFO.unsortedalone is misleading — gate sorting onvacuum_sort_benefit. "Storage is growing / what is driving it" is a different question from "storage is high": it needs a trend over months (CloudWatch, not a point-in-time view), per-table attribution is always partial, and the per-table sum is never the cluster total. → Loadreferences/redshift-storage-utilization.mdto attribute space, thenreferences/redshift-storage-remediation.mdto act.
Safety Guardrails
BLOCK: DROP DATABASE, DELETE without WHERE, publicly-accessible=true, GRANT ALL ON ALL
WARN then confirm: RESIZE, RESTORE, VACUUM on large tables, ALTER PASSWORD, WLM config change, TRUNCATE / DROP TABLE — TRUNCATE forces an implicit COMMIT and cannot be rolled back (it also commits other pending work in the same transaction), so name the target table and say the rows are unrecoverable before proceeding; treat it as routine only on temp/staging tables the user has already called disposable
Confirm: CREATE, GRANT specific, COPY, UNLOAD
Security Considerations
Apply these defaults when generating anything that connects, loads, or exports. Details are in the reference files noted.
In transit: the Data API is HTTPS-only. For JDBC/ODBC set the
require_sslparameter and connect withsslmode=verify-fullso the server certificate is checked.At rest: keep cluster/namespace encryption enabled, and add
ENCRYPTED KMS_KEY_ID '<arn>'toUNLOAD— it writes query results to S3, outside Redshift's own encryption. →references/redshift-sql-ddl-copy.mdCredentials: prefer
SecretArn(Secrets Manager) or IAM Identity Center;DbUseris acceptable because it issues temporary credentials. Never place database passwords in code, environment variables, or SQL text. →references/redshift-sql-recipes-load-api.mdLeast privilege: scope the namespace
IAM_ROLEto the specific bucket and prefix (s3:GetObjectonarn:aws:s3:::<bucket>/<prefix>/*), nots3:*or a managed full-access policy, and condition its trust policy on bothaws:SourceArn(the cluster/namespace ARN) andaws:SourceAccount—SourceArnalone still allows another resource in the account to assume it. Grant per-object privileges rather thanGRANT ALL ON ALL.Audit: CloudTrail records
redshift-data:*API calls but not the SQL executed; enable Redshift audit logging (useractivitylog,connectionlog,userlog) for that. Both capture query text and user activity, so encrypt every destination in use: the CloudWatch Logs group (aws logs associate-kms-key), the CloudTrail trail (SSE-KMS), and the audit-log S3 bucket (SSE-S3 — audit logging to S3 supports only S3-managed keys, not KMS). Serverless only supports sending audit logs to CloudWatch.Network: keep
PubliclyAccessible=falseand connect over a VPC endpoint. Do not open port 5439 to0.0.0.0/0or::/0— scope inbound rules to specific CIDRs or to a referencing security group.Sensitive data: Data API results persist for 24h and
sys_load_error_detailcan echo fragments of rejected rows, so treat statement IDs and load-error output as sensitive.Further reading: Security in Amazon Redshift for the full guidance behind these defaults.
Routing Table
MANDATORY: When a question matches a row below, you MUST load and read the referenced file BEFORE answering.
Ask whether the target is provisioned or Serverless before giving troubleshooting steps — unless the question already says which one, in which case use that and do not re-confirm.
| User Intent | Route To |
|---|---|
| "CREATE TABLE", "DISTKEY/SORTKEY", "ENCODE", "IDENTITY", "COPY", "UNLOAD", "IAM_ROLE", "Iceberg table" | references/redshift-sql-ddl-copy.md |
| "LISTAGG", "DATEADD/DATEDIFF", "NVL/DECODE", "type mapping", "text type", "VARBYTE", "recursive CTE" | references/redshift-sql-functions-types.md |
| "QUALIFY", "PIVOT/UNPIVOT", "MERGE", "TOP N", "SUBSTR error", "UNIQUE/PK not enforced", "trailing blanks", "leader-node function", "JSON", "SUPER", "PartiQL", "nested/semi-structured data" | references/redshift-sql-extensions-semantics.md |
| "system view", "SVV_/SYS_", "SHOW commands", "STL vs SYS", "list tables", "distkey/sortkey lookup", "datashare discovery", "2-part vs 3-part", "permission denied", "GRANT", "privileges", "relation/table does not exist" | references/redshift-sql-metadata.md |
| "how do I write SQL", "PostgreSQL vs Redshift", "which SQL reference", general dialect question | references/redshift-sql-syntax.md (index of the 6 SQL references + PostgreSQL-vs-Redshift failure table) |
| "COPY failed", "load error", "Data API poll", "async query", "Data API throttle" | references/redshift-sql-recipes-load-api.md |
| "materialized view", "MV refresh", "AUTO REFRESH", "stale view" | references/redshift-sql-materialized-views.md |
"disk full" / error 1016 / 34507, "PercentageDiskSpaceUsed", "storage grew", "what's consuming space", "ghost rows", "tombstone", "spill to disk", "managed storage vs local SSD", "datashare cache", "one node full / storage skew", "Serverless storage inflation", "what is driving the storage increase" / "storage keeps climbing" / "grew N% per month" / "which tables are growing" / "is vacuum missing" — a growth-over-time question, answered by the "Is it growing, and what is driving it?" section: split it into four (is it really that full, is it vacuum debt, how much would VACUUM return, what is driving the trend), trend from CloudWatch over months, per-table attribution from SYS_VACUUM_HISTORY, and state plainly what cannot be attributed from customer-side views, any snapshot / backup / recovery-point billing or cost question — including "AWS Backup", "cross-region snapshot copy", "backup billed at full size", "why is snapshot storage this expensive", "incremental so it should be cheaper" (answer from the per-type × per-platform matrix in S16, never from AWS Backup's own incremental-copy documentation, which describes AWS Backup's behavior and not Redshift's manual-snapshot metering), "cluster unavailable / stuck after hitting 100% storage" (confirm the fill first — PercentageDiskSpaceUsed in CloudWatch, since the SQL views are unreachable while the cluster is down, and rule out the non-storage causes before assuming disk) — attribute the space first | references/redshift-storage-utilization.md |
| "VACUUM won't finish / times out", "which VACUUM mode — DELETE ONLY / SORT ONLY / FULL / RECLUSTER", "vacuum_sort_benefit", "unsorted", "reclaim space", "deep copy", "auto-vacuum CPU", "distribution key skew", "recovering a cluster stuck or unavailable at ~100% storage — headroom plus escalation", "resize failed and stuck in canceling-resize" — take action | references/redshift-storage-remediation.md |
| "slow Redshift query", "query was fast now slow", "why is my query slow", "query performance", "time decomposition", "queue vs execution time", "planning/compile time", "result cache hit / false fast", "EXPLAIN plan", "DS_BCAST_INNER / broadcast / nested loop / merge join", "stale statistics / ANALYZE / bad row estimate", "function on join key / plan flip", "distribution skew / skew_rows / one slice", "did the query change / query hash", "row multiplication / join fan-out / DISTINCT slow", "sys_query_history / sys_query_detail" | references/redshift-sql-query-performance.md |
| "ready-to-run performance query", "which SQL to run", "diagnostic queries", "execution step / skew / spill", "plan diff between runs", "svv_table_info table health", "auto-analyze / auto-vacuum history", "lock wait / transaction history" | references/redshift-sql-query-performance-queries.md |
| "connection timed out/refused", "SSO/federated login fails", "Response signature invalid", "assumeRoleWithSAML", "RedshiftDbUser principal tag", "sts:TagSession AccessDenied", "IAM Identity Center region", "connection limit exceeded", "ODBC/JDBC driver", "ODBC 1.x EOL/CVE", "SSL error/certificate verify failed", "permission denied to query database created from datashare", "SYSLOG ACCESS/system-table row visibility", "GetClusterCredentials AccessDenied" | references/redshift-auth-authorization.md |
| General Redshift question not matching above | Answer directly from general knowledge |
| Aurora, RDS, DynamoDB, Athena (non-Redshift) | REFUSE. State this skill is for Amazon Redshift only. Do not provide guidance for other database services. |
Data API Quick Reference
→ Load references/redshift-sql-recipes-load-api.md before answering ANY Data API, COPY-error, or async-query question. It carries the bounded poll loop, the HasResultSet and ResourceNotFoundException handling, the per-target parameters, and the auth options.
Data API calls are async by default — use long polling (--wait-time-seconds, 1–30)
rather than blind sleeps, and keep a bounded loop for work that can exceed 30s.
Serverless takes --workgroup-name, provisioned takes --cluster-identifier.
Files
- ATTRIBUTION.mdAdded by AutoGineer908 B
- LICENSELicence file added by AutoGineer9.9 KiB
- NOTICELicence file added by AutoGineer67 B
- SKILL.md14.7 KiB
- references/redshift-auth-authorization.md24.1 KiB
- references/redshift-sql-ddl-copy.md4.9 KiB
- references/redshift-sql-extensions-semantics.md4.0 KiB
- references/redshift-sql-functions-types.md4.2 KiB
- references/redshift-sql-materialized-views.md2.9 KiB
- references/redshift-sql-metadata.md10.3 KiB
- references/redshift-sql-query-performance-queries.md22.8 KiB
- references/redshift-sql-query-performance.md34.6 KiB
- references/redshift-sql-recipes-load-api.md7.7 KiB
- references/redshift-sql-syntax.md4.0 KiB
- references/redshift-storage-remediation.md37.8 KiB
- references/redshift-storage-utilization.md34.0 KiB
references/redshift-sql-query-performance.md
Redshift Query Performance — Time Decomposition First
Workflow for "a query that used to be fast is now slow" and similar "why is this query slow?" questions. Establish Serverless vs provisioned first (see STEP 0 in SKILL.md for the deployment split and which views are provisioned-only) — the remediation differs by deployment.
Decision tree (start here)
Decompose
elapsed_timeinSYS_QUERY_HISTORY(next section) and find the bucket that grew. First check that the "fast" run wasn't served from the result cache (result_cache_hit = true): then Redshift returned a stored result without running the query, so its time isn't a real baseline. Also confirm your fast/slow runs are the same statement (apples-to-apples) before comparing — an unchanged query can still degrade from data, stats, plan, or contention shifts.Branch on the dominant bucket:
queue_time→ WLM / concurrency contention; find concurrent long-runners (same queue under manual WLM, cluster-wide otherwise) → Whenqueue_timedominates — remediation.compile_time→ first-run / post-patch recompile — expected and transient → Segment compilation.planning_time→ plan/query complexity, catalog, or datasharing elements; no deeper public telemetry, so if it dominates and you need more detail → escalate to AWS.lock_wait_time→ blocking transaction / concurrent DDL.execution_time→ real work; drill in by symptom: broadcast/redistribution or bad join → Reading EXPLAIN;SELECT DISTINCT/ many LEFT JOINs → Row multiplication (join fan-out); estimate ≠ actual → Estimate divergence (stale/missing statistics); slow after a function on a join/dist key → Functions on join/distribution keys; one slice/node hot → Distribution skew; grew on an unchanged plan → Stalled VACUUM.
Same query, slower, plan unchanged? Check data-volume growth, then compare
redshift_versionacross runs for a patch boundary → Did the query change… / escalate.
Escalating to AWS = open an AWS Support case (Support Center → Create case → Technical, service Amazon Redshift) with your evidence: the
SYS_QUERY_HISTORYbucket breakdown, thequery_id(s), andredshift_versionbefore/after. See Creating support cases.
Decompose the time before proposing any fix
When a query "got slow," do not jump to adding a DISTKEY/SORTKEY, rewriting the
query, or running VACUUM. First find which time bucket regressed. elapsed_time is
the total wall-clock; the other columns attribute time to individual phases (all in
microseconds), all available per query in SYS_QUERY_HISTORY (works on Serverless
and provisioned):
| Bucket | Column | What it means | If this bucket dominates |
|---|---|---|---|
| Queue wait | queue_time | Waiting for a WLM / concurrency slot | WLM config, concurrency scaling |
| Planning | planning_time | Time spent planning the query (parse, rewrite, plan) | Plan/query complexity (large join search space) or catalog/datasharing work. No public view decomposes planning_time further |
| Compile | compile_time | Segment compilation (see compilation below) | First-run only — warm cache expected |
| Lock wait | lock_wait_time | Waiting on a relation lock | Blocking transactions / concurrent DDL |
| Execution | execution_time | Actual work on compute nodes | Data volume, distribution skew, spill, or a stalled VACUUM (see below) |
These are separate metrics and need not add up exactly to elapsed_time — read them as
attribution hints to see which phase grew, not as a strict disjoint sum.
If planning_time is the dominant bucket, the public surface exposes only the single
planning_time value — there is no further user-facing telemetry to break it down. It
reflects plan/query complexity (large join search space, catalog work, possible datasharing
elements); if it dominates and more detail is needed, escalate to AWS.
-- Phase triage: where did the time go? Branch on whichever pct_* bucket dominates.
-- note: planning_time and lock_wait_time are independent metrics — lock wait is its own bucket.
SELECT query_id, user_id, database_name, query_type, status, start_time,
service_class_id, -- WLM queue: needed for the queue_time branch
compute_type, -- provisioned only: primary (main) / primary-scale (concurrency scaling) / secondary; not populated on Serverless
elapsed_time, queue_time, planning_time, lock_wait_time, compile_time, execution_time,
100.0 * queue_time / NULLIF(elapsed_time,0) AS pct_queue,
100.0 * planning_time / NULLIF(elapsed_time,0) AS pct_planning,
100.0 * lock_wait_time / NULLIF(elapsed_time,0) AS pct_lockwait,
100.0 * compile_time / NULLIF(elapsed_time,0) AS pct_compile,
100.0 * execution_time / NULLIF(elapsed_time,0) AS pct_execution,
result_cache_hit, query_text
FROM sys_query_history
WHERE user_id > 1 -- drop system/internal noise
AND ( query_id = <query_id, integer, no quotes>
OR query_text LIKE '%<tag_or_snippet, string, no quotes>%' )
ORDER BY elapsed_time DESC
LIMIT 50;The pct_* columns turn the raw buckets into shares of elapsed_time — read whichever is
largest to decide which phase to investigate.
Compare the buckets across a fast run and a slow run of the same query — the bucket that grew is the one to investigate. Only then pick a fix, targeted at that bucket.
In particular, separate queue wait from execution: if the extra time is queue_time,
the query didn't get slower to execute — it waited behind other work. So investigate
what it queued behind (next paragraph), not the queued query itself: other concurrently
running queries were using the cluster's concurrency and memory, and tuning them often relieves
it. A large execution_time with small
queue_time instead points at the query's own work — data volume, distribution skew, or spill.
Confirm queue pressure by comparing queue_time to elapsed_time per query in
SYS_QUERY_HISTORY (a high queue-time share = WLM contention); on provisioned clusters, query_priority and
compute_type (primary-scale = the query ran on a concurrency-scaling cluster) show how
queuing and scaling are being applied.
To find what it queued behind, look in SYS_QUERY_HISTORY for concurrent long-running
queries whose execution overlapped the slow query's queue window, ordered by
execution_time desc. Scope depends on the WLM mode: under Manual WLM each queue
(service_class_id) has fixed slots and a fixed memory share, so filter to the same
service_class_id to find the slot-hogs. Under Automatic WLM, concurrency and
memory are assigned dynamically per query across the whole cluster — not partitioned
per service class — so don't restrict to one service_class_id; look at the heavy concurrent
queries cluster-wide in the overlap window. Either way, those long-runners are the
contention source.
Then treat the long-runner as the new subject: decompose its time and drill into its
dominant bucket (usually execution_time → spill / skew / plan / stats) and tune it —
fixing the hog at the source relieves the queue for everything behind it. Reprioritizing
(WLM slots/priority, QMR, separate queue) or scaling only redistributes the contention if
the query itself isn't addressed.
When queue_time dominates — remediation. The query queued because other concurrently
running queries were using the cluster's concurrency and memory. Find them (see the paragraph
above; the ready-to-run query is in the queries reference); tuning them is often what relieves the
queuing.
The levers below are configuration changes, so recommend them to the user rather than making
them. Automatic WLM, concurrency scaling, and SQA are provisioned levers; on Serverless, raise
capacity instead. WLM query queues and query monitoring rules (QMR) work on both
deployments, configured through the wlm_json_configuration parameter: in the cluster's
parameter group on provisioned, and in the workgroup's configuration parameters on Serverless.
Confirm Automatic WLM is on — the slow query's
service_class_idinSYS_QUERY_HISTORYtells you: 100–107 is automatic WLM, 6–13 is a manual WLM queue. If it's manual, recommend switching to automatic WLM so Redshift manages concurrency and memory from the workload rather than fixed manual slots.Recommend enabling concurrency scaling — adds transient clusters for spiky/bursty demand. Free-tier credits accrue per running day; confirm the current allowance on the Redshift concurrency-scaling pricing page rather than quoting a figure, and cap spend with a concurrency-scaling usage limit.
Confirm Short Query Acceleration (SQA) is on (on by default; off only if
wlm_json_configurationhas"short_query_queue": false) — routes short queries past the queue, cutting their queue wait.Separate WLM queues by workload (BI / ETL / ad-hoc) and, on provisioned clusters, set queue priorities so critical queries aren't stuck behind heavy ones.
Add query monitoring rules (QMR) to catch runaway queries (nested loops, long runtime, disk spill) before they clog a queue.
Serverless — scale capacity (no WLM mode or SQA to enable). Two alternative approaches, not simultaneous knobs:
AI-driven scaling (on by default for new workgroups). The Price-performance target slider (Optimized for Cost ↔ Optimized for Performance; default Balanced) lets Redshift manage RPUs for you toward a cost/perf goal — you do not hand-set a base capacity in this mode. Bound the spend with Max capacity / Max RPU-hours. Best when you don't know the right base. Nudging the slider toward performance is the lever here.
Manual base capacity. Set a fixed Base capacity (RPUs) floor (plus a Max capacity ceiling) when you want a predictable, hand-tuned size instead of AI-driven scaling.
Stalled VACUUM as an execution regression: if execution_time (and blocks_read in
SYS_QUERY_DETAIL) grew while the plan is unchanged, suspect table maintenance. When
auto-vacuum stops, deleted rows linger as ghost rows (tbl_rows − estimated_visible_rows
in SVV_TABLE_INFO) and unsorted% climbs — both defeat sort-key zone-map pruning, so
the scan reads more blocks for the same post-filter result. Check every table the slow query
reads, one by one: take each table_name from SYS_QUERY_DETAIL for the slow query_id (it
covers tables behind views; table_name is db.schema.table), and look up each in
SVV_TABLE_INFO. Look first at the table whose scan sits in the segment whose blocks_read grew (blocks_read is
recorded on the segment row, not the step).
Confirm with SYS_VACUUM_HISTORY
(did auto-vacuum stop?) and SVV_TABLE_INFO, then pick the phase the evidence points to:
ghost rows → VACUUM DELETE ONLY; unsorted rows → VACUUM SORT ONLY, but only when
vacuum_sort_benefit > 0 (high unsorted% alone is not a reason to sort).
For the per-step (operator-level) breakdown of the execution bucket, query SYS_QUERY_DETAIL
for the query_id. Columns are recorded at different levels:
Segment rows only (0 on step rows):
blocks_read,spilled_block_local_diskandspilled_block_remote_diskSegment rows, copied onto each of their steps:
durationandtime_skewness, so they can't tell a segment's steps apartStep rows only:
input_rowsandoutput_rows,data_skewness(how unevenly the step's rows were spread), andalert(missing stats, nested loop, large broadcast)
Find the slow segment first, then the expensive step inside it.
A spill is a symptom, not a root cause. When a segment spilled (the spilled_block_local_disk /
spilled_block_remote_disk columns on its segment row), its working set exceeded the memory it was granted, so
Redshift wrote intermediate data to disk. The goal is not zero spill: some steps spill even
when the query is well written. Check the step against these best practices, and once they're
met, treat any remaining spill as acceptable:
Reduce the data the step handles: filter earlier, select only the columns needed (no
SELECT *), pre-aggregate, and remove any join fan-out.For a skewed spill (one slice spills much more than the others), co-locate the join: align the
DISTKEYs, or join on the distribution key, so rows aren't concentrated on one slice.Refresh statistics: a low row estimate gives the step too small a memory grant, so run
ANALYZE.
Adding nodes or disk does not fix a skewed spill.
Declared VARCHAR width matters only when the planner has no statistics. Rows are stored at
their actual length, whatever the declared width. With statistics, the planner uses the column's
real average width and distinct-value count. Without them, it estimates width from the declared
width (larger for wider declarations, up to a cap). That oversizes memory grants and can
misestimate join row counts. Run ANALYZE to give the planner statistics — except on columns
declared very wide (e.g. VARCHAR(65535)), which ANALYZE skips; right-size those instead.
Segment compilation (first run)
Redshift compiles optimized code for each query plan and keeps the compiled segments in a
compiled-code cache, locally and in a remote cache that persists across reboots, so later
runs skip compilation. For a query whose segments aren't in that cache yet, composition (on by
default) starts it immediately with pre-existing logic while the optimized code compiles in the
background, so first-run compile_time is rarely on the critical path
(Compiled code).
To confirm compilation is the cause, check whether compile_time is the bucket that grew in
SYS_QUERY_HISTORY (SVL_COMPILE gives per-segment detail, provisioned only). If it is, a rerun
should be fast.
Patch caveat. A version upgrade / patch clears the compiled-code cache (a plain reboot does
not), so first runs after a patch can recompile — but since composition keeps that off the
critical path, don't assume post-patch slowness is "just recompile." Confirm it's
compile_time that grew and that it vanishes on rerun; if it's in execution_time / the
plan or persists after reruns, treat it as a real regression (compare redshift_version,
buckets, plan).
Rule out result-cache hits before comparing runs
If result_cache_hit = true, Redshift returned a cached result — execution_time
is near-zero and tells you nothing about the query's real cost. A benchmark where the
"fast" run was a cache hit is not a valid measurement, and a fast→slow comparison
against a cached baseline is comparing a cache return to real compute.
Confirm with
result_cache_hitinSYS_QUERY_HISTORY.To measure real (uncached) cost, disable the result cache for the session:
SET enable_result_cache_for_session TO off;or compare only cache-miss executions (
result_cache_hit = false).
Reading EXPLAIN: distribution and join strategy
Two things in an EXPLAIN plan explain most "execution bucket" slowness — data
movement and join strategy.
Data distribution operators (the DS_* labels):
| Label | Meaning | Good/bad |
|---|---|---|
DS_DIST_NONE | No movement — both sides already co-located on the join key | Ideal |
DS_DIST_ALL_NONE | No movement — the inner is DISTSTYLE ALL, already on every node | Ideal |
DS_DIST_INNER | Inner table redistributed on the join key | Costly for a large inner |
DS_DIST_OUTER | Outer table redistributed on the join key | Costly for a large outer |
DS_DIST_BOTH | Both tables redistributed on the join key | Costly — the most data movement |
DS_DIST_ALL_INNER | Entire inner redistributed to a single slice (outer is DISTSTYLE ALL) | Usually a distribution mismatch |
DS_BCAST_INNER | The whole inner table is broadcast to every node | Not inherently bad — a cheap fan-out the planner picks on purpose when the inner is small; costly only when the inner is large |
Join strategy:
Merge Join — fastest; both tables distributed and sorted on the join key, and less than 20% of each table unsorted (
SVV_TABLE_INFO.unsorted).Hash Join — the common case; builds a hash table on the inner side.
Nested Loop — the correct (and only) strategy for non-equi joins (inequality / range conditions like
a.x < b.y) and intentional cross joins. It is a red flag only when it shows up for what should be an equi-join — usually a missing or mistyped join predicate producing an accidental near-Cartesian product (a huge estimated cost is the tell). Check the join condition.
Fix broadcast/redistribution by co-locating the join: set a matching DISTKEY on
the join column of both tables, or DISTSTYLE ALL for a small, slow-moving dimension table.
ALL copies the whole table to every node, which multiplies its storage and slows every load
and update. There's no fixed size cutoff; when unsure, use DISTSTYLE AUTO, which starts a
small table as ALL and changes it as the table grows. To see the executed plan (not just the estimate), read SYS_QUERY_DETAIL
(step_name values like hashjoin, nestloop, broadcast, distribute, scan) or
SYS_QUERY_EXPLAIN.
Row multiplication (join fan-out)
A SELECT DISTINCT or aggregate over many LEFT JOINs can be slow because the joins
fan out — one row matches many in a dimension/lookup table, so an intermediate join
step processes billions of rows that DISTINCT then collapses back to a few million.
Spot it in
SYS_QUERY_DETAIL: a join step'soutput_rowsis far larger than the query's final returned rows (huge join row count vs small result). Broadcast (DS_BCAST_INNER) on those joins compounds it.Find the offending join by testing each
LEFT JOINincrementally withCOUNT(*).Fix (only when the query's semantics permit it — i.e. the multiplied rows aren't actually needed): pre-aggregate or deduplicate the lookup tables in a
WITHclause / subquery so each returns one row per key (restore the one-to-one relationship) before joining; and co-locate the joins on the distribution key.
Diff the plan across the fast and slow run
Reading one plan shows the current shape; diffing the fast run's plan against the slow run's shows what changed. Do this when the buckets point at execution/plan and you have both a fast and a slow run of the same statement.
Pair the two runs. Get one fast and one slow
query_idfor the same query — group runs by the built-ingeneric_query_hash/user_query_hash, or by a normalized SQL text fingerprint (mask literal values, keep identifiers). Use whichever is handy.Fingerprint each plan to decide whether the shape changed — before eyeballing nodes. Pull
SYS_QUERY_EXPLAIN(populated per child query:query_id+child_query_sequence; a rewritten statement has several child plans). Raw plan text is noisy:plan_infoembeds the WHERE literals (Filter: (dt >= '2026-01-01'::date)) andplan_nodeembeds volatilecost=…/rows=…/width=…— all vary run-to-run even when the shape is identical. So build a normalized plan hash perchild_query_sequence:Order rows by
plan_node_id; concatenateplan_node+plan_info.Strip the volatile parts: replace literal constants in
plan_info('…',…::type, bare numbers) with a placeholder, and drop thecost=/rows=/width=numbers fromplan_node.Keep the structure: operator,
DS_*label, join type, and the predicate columns.Hash the result. Equal fast-vs-slow hash → the plan shape did not change — stop diffing the plan and go to the data/stats check (next section) or the other buckets. Different → the plan flipped; drop to the tree diff.
Reading the plan as a tree.
SYS_QUERY_EXPLAINreturns nodes:plan_node_ididentifies each node andplan_parent_idpoints to its parent, so together they reconstruct the operator tree.plan_node_idis within-plan — it also joins toSYS_QUERY_DETAIL(query_id+child_query_sequence+plan_node_id) to show which execution segment/step ran a node — but it is not a stable identity across two runs, so don't assume node N in the fast run is node N in the slow run. Localize the change by walking parent→child and finding the subtree whereplan_node(operator /DS_*/ join type) orplan_info(predicate columns) differs.Tree diff (only when the hashes differ), per
child_query_sequence:plan_node— aDS_DIST_NONE/DS_DIST_ALL_NONEthat becameDS_BCAST_INNER/DS_DIST_INNER/DS_DIST_BOTH, or a Hash/Merge Join that became a Nested Loop, is the regression.plan_info— join conditions and WHERE restrictions; a predicate that moved (e.g. a filter no longer applied at theSeq Scanbut late in a join) shows predicate placement changed — often the cause of a redistribution flip.SYS_QUERY_EXPLAINalso emits a----- Tables missing statistics -----node — a direct stale/missing-stats signal.Tie the flip back to the execution evidence — map node → segment. A plan change that raised
execution_timeshould be traceable to a specific segment, but the changed node isn't always the slowest one: a flip in an earlier segment (e.g. a redistribution that lands rows badly) can make a later segment the bottleneck. So map the changedplan_nodeto its segment, pull the slowest segments for the slow run fromSYS_QUERY_DETAIL(order byduration), and check whether the slowest segment is, or is fed by, the changed node — comparing the same segments against a successful (fast) run. If the slowest segment can't be traced back to the plan change, the flip may not be the cause — re-check the other buckets.
Estimate drift on the base-table scans (no separate hash — just read
rows=). For theSeq Scannodes on base tables, compare the scan'srows=estimate between the fast and slow run. Arows=estimate that changed run-to-run — or a----- Tables missing statistics -----node — means the stats/data moved under the query, so the plan didn't have to change shape to get slower. Read the estimate off the scan node. To compare it with the actual rows, use the queries reference's plan-diff Step 4, which joins each step to its own plan node, so a table scanned several times is still matched correctly. This is the hand-off to the next section.
If the two plans are identical (same hash) but the timing still differs, the plan shape is not
the cause. Four common culprits: (a) the WHERE literals select more rows — same plan, but a
wider predicate reads/returns more data (see the data/stats section below); (b) the base tables
grew — same plan, but more input rows to scan/join means linearly more execution_time; confirm
by comparing the scan steps' input_rows (and output_rows) in SYS_QUERY_DETAIL between the
comparable runs, not SVV_TABLE_INFO.tbl_rows (current snapshot only, no history); or
(c) concurrency contention — more queries running concurrently during the slow run's window,
especially long-runners or heavy spillers competing for compute / memory / disk (this is not
queue-slot waiting, which shows up as queue_time; see the concurrent long-runner query in the
queue section); or (d) duplicate data / join fan-out — new duplicate keys (or an accidental
near-cartesian join) make an intermediate join step explode to far more rows even when the
final returned rows look close to the fast run, so the extra work is invisible in the result
count; confirm by comparing a join step's intermediate output_rows in SYS_QUERY_DETAIL
between the runs (see Row multiplication (join fan-out) above). If none fits, re-decompose the
time buckets (top of this doc) to see which one actually grew.
An identical plan doesn't guarantee identical row counts per step. On hash joins, Redshift can
build a runtime (Bloom) filter from the build side's join keys and apply it inside the
probe-side scan, dropping rows that can't match before they reach the join. EXPLAIN doesn't show
it, and SYS_QUERY_DETAIL gives it no step of its own. It shows only as that scan's
output_rows, which also reflects any filter in the plan, so the two can't be separated. Whether
the filter is used, and how much it removes, can vary between runs of the same plan. So when two
same-plan runs show different output_rows on a hash join's probe-side scan, a runtime filter
behaving differently is a candidate cause, not only a change in the data.
Across environments (e.g. a dc2→ra3 migration or a rebuilt table), also confirm the
underlying tables share the same physical design — compare diststyle / distkey /
sortkey via SVV_TABLE_INFO (or SHOW TABLE) on each side. DISTSTYLE AUTO /
SORTKEY AUTO can pick different keys per environment, which by itself changes the plan.
Data changes flip the plan — even when no ANALYZE has run
The optimizer builds each plan from the row counts and statistics it holds at plan time.
Those inputs change without any change to the query: data loads and deletes change row counts,
and ANALYZE (manual or automatic) refreshes the statistics. So the same SQL can get a new
plan (or the same plan, run slower) with no query-text change. When the plan hash changed, or the base-table scan's rows=
estimate differed between the runs (above), work this checklist to find what moved underneath
the query:
Statistics were refreshed between the runs. An
ANALYZE— manual or auto — updates the statistics, which alone can flip the plan. Confirm one ran between the two runs inSYS_ANALYZE_HISTORY(is_automatictells you which; check for either).Row counts grew (feeds the cost model). A larger table raises estimated scan/join cost, so a co-located
DS_DIST_NONEjoin can flip toDS_BCAST_INNER/ redistribution past a threshold. The signal is the base-table scan'srows=estimate differing between the comparable runs (above). Don't try to prove growth fromSVV_TABLE_INFO.tbl_rows— it is only the current row count (a snapshot, no history), and aWHEREpredicate would filter the scan estimate below the full-table count anyway, so the two aren't comparable.Stats went stale. As data changes, the planner's whole-table estimate drifts from reality. Read staleness from
SVV_TABLE_INFO.stats_off(0= current,100= fully stale), and check whether DML has landed since the lastANALYZE(last-analyze time is inSYS_ANALYZE_HISTORY). UseSVV_TABLE_INFO(stats_off,tbl_rows) for this — notpg_class.reltuples, which is only a rough estimate. A stale estimate flips the plan before you would think to runANALYZE. Check every referenced table (see the next section).The physical design changed — auto or manual. A dist/sort-key change alters the plan: Redshift can auto-apply dist/sort recommendations, and a manual
ALTER TABLE … ALTER DISTKEY/SORTKEY(or a table rebuild) does the same. Check both: recent DDL history for manual changes, andSYS_AUTO_TABLE_OPTIMIZATIONfor automatic ones.Sort order / pruning degraded. Appends raise
unsorted%, weakening zone-map pruning so scans read more blocks for the same rows. This slows the query on an unchanged plan. The signal is the scan's segmentblocks_readrising inSYS_QUERY_DETAILwhile itsinput_rowsstays comparable across the runs. Confirm withSVV_TABLE_INFO.unsortedandvacuum_sort_benefitfor that table, and withSYS_VACUUM_HISTORY(did auto-vacuum stop or fall behind?). Fix:VACUUM SORT ONLY <schema_table, identifier, no quotes>;, but only whenvacuum_sort_benefit > 0. See Stalled VACUUM above.
The common thread: a plan or timing change on unchanged SQL usually traces to statistics, row-count, physical-design, or sort-order movement, not the query — verify these before rewriting the query or forcing DDL.
Estimate divergence: stale or missing statistics
When EXPLAIN's estimated rows are wildly off from the actual rows, the optimizer is
planning on stale or missing statistics, and a bad estimate produces a bad plan. Not every gap
means bad statistics. For example, on a hash join's probe-side scan a runtime filter can make the
actual rows much lower than the estimate even when the statistics are current (see the
runtime-filter note under Diff the plan across the fast and slow run).
Check staleness with
SVV_TABLE_INFO.stats_off(0= current,100= fully stale).Fix by running
ANALYZE <schema_table, identifier, no quotes>;. It refreshes both the row counts and the column statistics (distinct values, most common values) that filter and join estimates depend on.ANALYZE, notVACUUM, is what updates the planner statistics.Confirm the divergence by comparing the estimate (
EXPLAINrows=) to the actualinput_rows/output_rowsper step inSYS_QUERY_DETAIL(the queries reference's plan-diff Step 4 shows both side by side).Before recommending a manual
ANALYZE, checkSYS_ANALYZE_HISTORY— auto-analyze may already have refreshed the statistics.Check
stats_offfor every table the query references (all joined tables / CTE sources), not just the largest or most obvious one — a single stale table's bad estimate can flip the whole plan.ANALYZEeach stale table.
Functions on join / distribution keys slow the join
Wrapping a join or distribution key in a function or expression — ON LOWER(a.id) = LOWER(b.id), a.id || '' = b.id || '', CAST, SUBSTRING, etc. — makes the join slower,
for two reasons:
It defeats
DISTKEYco-location. The expression's value no longer matches the column's distribution, so Redshift must redistribute or broadcast instead of doing a co-located join.It defeats column statistics. Statistics are collected on columns, not on expressions, so the optimizer can't estimate the expression's cardinality and falls back to a poor default — an underestimate that can flip a co-located join to broadcast or redistribution.
The expression isn't always in the SQL you're looking at. If the query reads from a view,
the function can come from the view's definition, and redefining the view changes the join
without any change to the query text. Check the join condition in the plan (Hash Cond in
SYS_QUERY_EXPLAIN.plan_info), not only the SQL.
Fix: join on the raw column. If the transformation is genuinely needed, persist it as a materialized/precomputed column (and distribute/sort on that) so both the distkey and the statistics stay usable.
Distribution skew: one slice doing all the work
If one slice/node is doing far more work than the others while the rest sit idle, the query has skew. Two distinct causes:
Table distribution skew — a
DISTKEYon a low-cardinality or heavily-repeated column piles disproportionate rows onto one slice; a fixed property of the table. Detect withSVV_TABLE_INFO.skew_rows— the ratio of the most-populated to least-populated slice (near1.0is even; large values are skewed).Runtime (redistribution) skew — a hash join or broadcast redistributes rows so one slice receives most of them for a step, even when the base tables are evenly distributed; per-query, not a table property.
Detect runtime skew with SYS_QUERY_DETAIL's data_skewness and time_skewness (0–100%,
higher = more unbalanced), which report how unevenly work fell across slices. data_skewness
is per step (how unevenly the step's rows were spread); time_skewness is the segment's value,
repeated on each of its steps. They flag that a step or segment is skewed, not where the skew
sits; you don't need that to fix it. Tell table skew from runtime skew (SVV_TABLE_INFO.skew_rows
vs SYS_QUERY_DETAIL.data_skewness), then apply the remediation below.
Remediation depends on the case: table skew is typically addressed by a better
distribution (a higher-cardinality DISTKEY, or DISTSTYLE EVEN); runtime redistribution
skew by co-locating the join (aligning DISTKEYs / joining on the distribution key). Whether a
distribution change is worth it depends on the query mix, table size, and how other queries join
the table — so prefer DISTSTYLE AUTO, which chooses the distribution from the observed
workload, and check SVV_ALTER_TABLE_RECOMMENDATIONS for a pending recommendation before
choosing a key by hand. Resizing the cluster or adding nodes does not fix skew — it just
spreads the same skewed data across more slices, so one slice still carries the
disproportionate load.
Did the query change, or something around it?
Before assuming "the query got slower," confirm whether the SQL text even changed:
SYS_QUERY_HISTORYexposesuser_query_hash(same text including literals) andgeneric_query_hash(same text ignoring literals). If the hash is unchanged across the fast and slow runs, the query did not change — look elsewhere.If the text is unchanged, investigate what changed around it: data-volume growth (compare the base-table scan steps'
input_rowsinSYS_QUERY_DETAILbetween the fast and slow runs), stale stats, or a plan change. Compare the fast vs slow executions withSYS_QUERY_HISTORY(time buckets) andSYS_QUERY_DETAIL(per-step rows and thealertcolumn for plan-quality warnings).Patch/version regression: only after ruling out buckets, data volume, stats, plan, and spill/skew, compare
redshift_versionacross the before/after runs inSYS_QUERY_HISTORY— a version change lining up with the regression points at a patch. A patch is far more credible when the regression affects many unrelated queries across the cluster at the same date boundary — plans flipping across multiple queries at comparable row counts and stats — not just this one; a single query regressing usually has a local cause, so confirm the symptom spans multiple queries across the cluster first. Patch internals and restart reasons are not public, so conclude "a patch caused it" only once the above are eliminated, and escalate with the bucket + version + multi-query evidence rather than blaming the patch from timing alone.
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Needs an MCP server, Needs Node.js
Security
No issues found by our static scan· No issues found
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Published by
- AWS
- Publisher
- AWS (Official)
- Source
- aws/agent-toolkit-for-aws · plugins/aws-data-analytics/skills/redshift-guide · commit 0a218f8
- Licence
- Apache License 2.0 (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE)
- NOTICE (added by AutoGineer from NOTICE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
- Also distributed in
- aws/agent-toolkit-for-aws
AutoGineer is not affiliated with or endorsed by AWS or AWS.
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download redshift-guide.zip