Redshift Guide
AWSOfficialPublished by AWS
- Skill
- Tool integration
- Cloud and DevOps
- Apache-2.0
- No issues found
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
The Redshift Guide is a technical reference designed to prevent common errors when working with Amazon Redshift. Because Redshift shares a wire protocol with PostgreSQL, AI models often incorrectly apply PostgreSQL syntax and behaviors to Redshift. This guide provides the necessary corrections, covering DDL, system metadata, query performance, storage management, and authentication. It distinguishes between provisioned clusters and Serverless workgroups, ensuring users apply the correct system views and API calls for their specific environment.
What it does
- Corrects common misconceptions where PostgreSQL syntax is incorrectly applied to Redshift.
- Provides guidance on Redshift-specific DDL, including distribution and sort keys.
- Outlines the correct system views (SYS_ vs STL_) for monitoring and metadata discovery.
- Details secure authentication practices for JDBC/ODBC and the Data API.
- Offers troubleshooting steps for storage issues, query performance, and COPY/UNLOAD operations.
- Enforces the use of Redshift-specific commands over PostgreSQL-native alternatives.
When to use it
- When you are unsure if a PostgreSQL command or function is supported in Redshift.
- When troubleshooting performance issues, disk space, or query errors in Redshift.
- When setting up new tables, loading data via COPY, or exporting data via UNLOAD.
- When configuring IAM roles, permissions, or connectivity for Redshift clusters or workgroups.
- When you need to identify the correct system views for monitoring a specific Redshift deployment.
Advantages
- Prevents errors caused by assuming PostgreSQL compatibility.
- Provides clear distinctions between provisioned and Serverless environments.
- Focuses on high-impact performance tuning like distribution and sort keys.
- Promotes secure, least-privilege access patterns for data operations.
- Reduces time spent debugging by providing a structured routing table for common issues.
Usage examples
- “How do I create a table in Redshift with a sort key?”
- The agent will provide a CREATE TABLE statement using DISTSTYLE and SORTKEY, explaining that Redshift uses these instead of standard PostgreSQL indexes.
- “Why is my query failing with a 'relation does not exist' error?”
- The agent will guide you to use SHOW commands or SYS_ views to inspect the schema and verify table existence, rather than querying information_schema.
- “How do I debug a failed COPY command?”
- The agent will instruct you to query SYS_LOAD_ERROR_DETAIL to identify the specific row or column causing the load failure.
- “Is my Redshift cluster running out of disk space?”
- The agent will help you distinguish between actual data growth and temporary query spill, and guide you to use the appropriate storage utilization references.
Install
Download the skill as a ZIP:
https://autogineer.com/agent-skills/aws/redshift-guide/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip redshift-guide.zip -d .agents/skills
Documentation by AWS
Amazon Redshift Guide
Redshift is NOT PostgreSQL (read first)
Redshift speaks PostgreSQL's wire protocol and shares much of its surface syntax, so
LLMs assume PostgreSQL behavior carries over — it frequently does not. Divergences span
system tables (pg_catalog is incomplete), DDL (no indexes, no sequences), functions
(string_agg, SUBSTR on tables, leader-node-only functions), types (a text column
becomes VARCHAR(256)), and comparison semantics (trailing blanks, unenforced constraints). Assume
divergence and verify against the reference below — do not answer from PostgreSQL habit.
Common PostgreSQL→Redshift divergences are in references/redshift-sql-syntax.md.
Works best with the AWS MCP server — it runs the AWS CLI and Redshift Data API calls below in a sandboxed, audit-logged environment. All guidance here is plain AWS CLI and SQL and works without it.
STEP 0: Serverless or Provisioned?
Establish this before answering — APIs, system tables, and capabilities differ. Take it
from the question when it says which one; ask when it does not. SELECT version()
does not identify it.
Serverless — identified by a workgroup (and namespace). Data API calls take
--workgroup-name; the user says "workgroup"/"Serverless".Provisioned — identified by a cluster. Data API calls take
--cluster-identifier; the user says "cluster".
| Target | System Views | Credentials API |
|---|---|---|
| Provisioned | SYS_, all SVV_ + STL_, STV_, SVL_, SVCS_ (single-AZ only — disabled on Multi-AZ) | redshift:GetClusterCredentials |
| Serverless | SYS_ + a subset of SVV_ ONLY (no STL/STV/SVL/SVCS) | redshift-serverless:GetCredentials |
Critical Facts
SHOW commands are the primary metadata interface — SHOW DATABASES, SHOW SCHEMAS, SHOW TABLES, SHOW COLUMNS, SHOW TABLE, SHOW VIEW. Do NOT default to pg_catalog or information_schema. → Load
references/redshift-sql-metadata.mdfor metadata/discovery questions and any "relation does not exist" report — it has the diagnostic flow.SYS_views are the preferred system views — they work everywhere.STL_,STV_,SVL_, andSVCS_are provisioned single-AZ only, and someSVV_views are unsupported on Serverless. → Loadreferences/redshift-sql-metadata.mdfor any system-view or monitoring question.sys_load_error_detailfor COPY debugging (notstl_load_errors, which is provisioned single-AZ only).DATEADD/DATEDIFF — unit-first argument order:
DATEADD(day, -30, GETDATE()),DATEDIFF(day, start, end).APPROXIMATE COUNT(DISTINCT col) — Redshift-specific, ~2% error, much faster than exact COUNT(DISTINCT) on large datasets.
MERGE ... REMOVE DUPLICATES — simplified dedup when source and target have identical schemas.
COPY should use IAM_ROLE (the namespace role, not the caller role) + supports MANIFEST for explicit file lists + MAXERROR for error tolerance.
SUBSTR()is leader-node-only — works on literals but errors on table columns (SUBSTR() function is not supported (Hint: use SUBSTRING instead)). UseSUBSTRING()on columns.UNIQUE / PRIMARY KEY / FOREIGN KEY are informational only — NOT enforced (duplicate rows are accepted with no error). Optimizer hints; enforce integrity in the application or via MERGE.
NOT NULLIS enforced.SHOW VIEW <schema.name>returns the definition of a regular view, materialized view, or late-binding view. MV freshness:SVV_MV_INFO(is_stale).No
FILTER (WHERE ...)on aggregates — PostgreSQL-only, Redshift rejects it. WriteCOUNT(CASE WHEN cond THEN 1 END)/SUM(CASE WHEN cond THEN x END)instead. This applies to every query you emit, including ones you derive from a reference example.TOP NandLIMIT Nboth work (TOP N PERCENTdoes not). Atextcolumn becomesVARCHAR(256)— useVARCHAR(max)or explicit length.Iceberg tables use
CREATE TABLE ... USING ICEBERG(notSTORED AS ICEBERG, notTABLE_FORMAT=ICEBERG).Datashares support read and write operations — consumers can write once the producer grants write privileges. Treat "permission denied" on a datashare write as a missing grant, not an unsupported operation. → Load
references/redshift-sql-metadata.mdfor requirements and limits.Storage: attribute before you remediate, and never reflex-VACUUM. Disk-full is usually NOT data growth — rule out transient query spill (temp blocks) and per-node/slice skew first, because the cluster average hides a single full node. On RA3/Serverless a high local reading is cache, not out-of-space, and free managed storage does not prevent a node-local Disk Full.
SVV_TABLE_INFO.unsortedalone is misleading — gate sorting onvacuum_sort_benefit. "Storage is growing / what is driving it" is a different question from "storage is high": it needs a trend over months (CloudWatch, not a point-in-time view), per-table attribution is always partial, and the per-table sum is never the cluster total. → Loadreferences/redshift-storage-utilization.mdto attribute space, thenreferences/redshift-storage-remediation.mdto act.
Safety Guardrails
BLOCK: DROP DATABASE, DELETE without WHERE, publicly-accessible=true, GRANT ALL ON ALL
WARN then confirm: RESIZE, RESTORE, VACUUM on large tables, ALTER PASSWORD, WLM config change, TRUNCATE / DROP TABLE — TRUNCATE forces an implicit COMMIT and cannot be rolled back (it also commits other pending work in the same transaction), so name the target table and say the rows are unrecoverable before proceeding; treat it as routine only on temp/staging tables the user has already called disposable
Confirm: CREATE, GRANT specific, COPY, UNLOAD
Security Considerations
Apply these defaults when generating anything that connects, loads, or exports. Details are in the reference files noted.
In transit: the Data API is HTTPS-only. For JDBC/ODBC set the
require_sslparameter and connect withsslmode=verify-fullso the server certificate is checked.At rest: keep cluster/namespace encryption enabled, and add
ENCRYPTED KMS_KEY_ID '<arn>'toUNLOAD— it writes query results to S3, outside Redshift's own encryption. →references/redshift-sql-ddl-copy.mdCredentials: prefer
SecretArn(Secrets Manager) or IAM Identity Center;DbUseris acceptable because it issues temporary credentials. Never place database passwords in code, environment variables, or SQL text. →references/redshift-sql-recipes-load-api.mdLeast privilege: scope the namespace
IAM_ROLEto the specific bucket and prefix (s3:GetObjectonarn:aws:s3:::<bucket>/<prefix>/*), nots3:*or a managed full-access policy, and condition its trust policy on bothaws:SourceArn(the cluster/namespace ARN) andaws:SourceAccount—SourceArnalone still allows another resource in the account to assume it. Grant per-object privileges rather thanGRANT ALL ON ALL.Audit: CloudTrail records
redshift-data:*API calls but not the SQL executed; enable Redshift audit logging (useractivitylog,connectionlog,userlog) for that. Both capture query text and user activity, so encrypt every destination in use: the CloudWatch Logs group (aws logs associate-kms-key), the CloudTrail trail (SSE-KMS), and the audit-log S3 bucket (SSE-S3 — audit logging to S3 supports only S3-managed keys, not KMS). Serverless only supports sending audit logs to CloudWatch.Network: keep
PubliclyAccessible=falseand connect over a VPC endpoint. Do not open port 5439 to0.0.0.0/0or::/0— scope inbound rules to specific CIDRs or to a referencing security group.Sensitive data: Data API results persist for 24h and
sys_load_error_detailcan echo fragments of rejected rows, so treat statement IDs and load-error output as sensitive.Further reading: Security in Amazon Redshift for the full guidance behind these defaults.
Routing Table
MANDATORY: When a question matches a row below, you MUST load and read the referenced file BEFORE answering.
Ask whether the target is provisioned or Serverless before giving troubleshooting steps — unless the question already says which one, in which case use that and do not re-confirm.
| User Intent | Route To |
|---|---|
| "CREATE TABLE", "DISTKEY/SORTKEY", "ENCODE", "IDENTITY", "COPY", "UNLOAD", "IAM_ROLE", "Iceberg table" | references/redshift-sql-ddl-copy.md |
| "LISTAGG", "DATEADD/DATEDIFF", "NVL/DECODE", "type mapping", "text type", "VARBYTE", "recursive CTE" | references/redshift-sql-functions-types.md |
| "QUALIFY", "PIVOT/UNPIVOT", "MERGE", "TOP N", "SUBSTR error", "UNIQUE/PK not enforced", "trailing blanks", "leader-node function", "JSON", "SUPER", "PartiQL", "nested/semi-structured data" | references/redshift-sql-extensions-semantics.md |
| "system view", "SVV_/SYS_", "SHOW commands", "STL vs SYS", "list tables", "distkey/sortkey lookup", "datashare discovery", "2-part vs 3-part", "permission denied", "GRANT", "privileges", "relation/table does not exist" | references/redshift-sql-metadata.md |
| "how do I write SQL", "PostgreSQL vs Redshift", "which SQL reference", general dialect question | references/redshift-sql-syntax.md (index of the 6 SQL references + PostgreSQL-vs-Redshift failure table) |
| "COPY failed", "load error", "Data API poll", "async query", "Data API throttle" | references/redshift-sql-recipes-load-api.md |
| "materialized view", "MV refresh", "AUTO REFRESH", "stale view" | references/redshift-sql-materialized-views.md |
"disk full" / error 1016 / 34507, "PercentageDiskSpaceUsed", "storage grew", "what's consuming space", "ghost rows", "tombstone", "spill to disk", "managed storage vs local SSD", "datashare cache", "one node full / storage skew", "Serverless storage inflation", "what is driving the storage increase" / "storage keeps climbing" / "grew N% per month" / "which tables are growing" / "is vacuum missing" — a growth-over-time question, answered by the "Is it growing, and what is driving it?" section: split it into four (is it really that full, is it vacuum debt, how much would VACUUM return, what is driving the trend), trend from CloudWatch over months, per-table attribution from SYS_VACUUM_HISTORY, and state plainly what cannot be attributed from customer-side views, any snapshot / backup / recovery-point billing or cost question — including "AWS Backup", "cross-region snapshot copy", "backup billed at full size", "why is snapshot storage this expensive", "incremental so it should be cheaper" (answer from the per-type × per-platform matrix in S16, never from AWS Backup's own incremental-copy documentation, which describes AWS Backup's behavior and not Redshift's manual-snapshot metering), "cluster unavailable / stuck after hitting 100% storage" (confirm the fill first — PercentageDiskSpaceUsed in CloudWatch, since the SQL views are unreachable while the cluster is down, and rule out the non-storage causes before assuming disk) — attribute the space first | references/redshift-storage-utilization.md |
| "VACUUM won't finish / times out", "which VACUUM mode — DELETE ONLY / SORT ONLY / FULL / RECLUSTER", "vacuum_sort_benefit", "unsorted", "reclaim space", "deep copy", "auto-vacuum CPU", "distribution key skew", "recovering a cluster stuck or unavailable at ~100% storage — headroom plus escalation", "resize failed and stuck in canceling-resize" — take action | references/redshift-storage-remediation.md |
| "slow Redshift query", "query was fast now slow", "why is my query slow", "query performance", "time decomposition", "queue vs execution time", "planning/compile time", "result cache hit / false fast", "EXPLAIN plan", "DS_BCAST_INNER / broadcast / nested loop / merge join", "stale statistics / ANALYZE / bad row estimate", "function on join key / plan flip", "distribution skew / skew_rows / one slice", "did the query change / query hash", "row multiplication / join fan-out / DISTINCT slow", "sys_query_history / sys_query_detail" | references/redshift-sql-query-performance.md |
| "ready-to-run performance query", "which SQL to run", "diagnostic queries", "execution step / skew / spill", "plan diff between runs", "svv_table_info table health", "auto-analyze / auto-vacuum history", "lock wait / transaction history" | references/redshift-sql-query-performance-queries.md |
| "connection timed out/refused", "SSO/federated login fails", "Response signature invalid", "assumeRoleWithSAML", "RedshiftDbUser principal tag", "sts:TagSession AccessDenied", "IAM Identity Center region", "connection limit exceeded", "ODBC/JDBC driver", "ODBC 1.x EOL/CVE", "SSL error/certificate verify failed", "permission denied to query database created from datashare", "SYSLOG ACCESS/system-table row visibility", "GetClusterCredentials AccessDenied" | references/redshift-auth-authorization.md |
| General Redshift question not matching above | Answer directly from general knowledge |
| Aurora, RDS, DynamoDB, Athena (non-Redshift) | REFUSE. State this skill is for Amazon Redshift only. Do not provide guidance for other database services. |
Data API Quick Reference
→ Load references/redshift-sql-recipes-load-api.md before answering ANY Data API, COPY-error, or async-query question. It carries the bounded poll loop, the HasResultSet and ResourceNotFoundException handling, the per-target parameters, and the auth options.
Data API calls are async by default — use long polling (--wait-time-seconds, 1–30)
rather than blind sleeps, and keep a bounded loop for work that can exceed 30s.
Serverless takes --workgroup-name, provisioned takes --cluster-identifier.
Files
- ATTRIBUTION.mdAdded by AutoGineer908 B
- LICENSELicence file added by AutoGineer9.9 KiB
- NOTICELicence file added by AutoGineer67 B
- SKILL.md14.7 KiB
- references/redshift-auth-authorization.md24.1 KiB
- references/redshift-sql-ddl-copy.md4.9 KiB
- references/redshift-sql-extensions-semantics.md4.0 KiB
- references/redshift-sql-functions-types.md4.2 KiB
- references/redshift-sql-materialized-views.md2.9 KiB
- references/redshift-sql-metadata.md10.3 KiB
- references/redshift-sql-query-performance-queries.md22.8 KiB
- references/redshift-sql-query-performance.md34.6 KiB
- references/redshift-sql-recipes-load-api.md7.7 KiB
- references/redshift-sql-syntax.md4.0 KiB
- references/redshift-storage-remediation.md37.8 KiB
- references/redshift-storage-utilization.md34.0 KiB
references/redshift-storage-remediation.md
Redshift Storage Remediation — VACUUM, Deep Copy, Skew & Disk-Full Recovery
The action side of references/redshift-storage-utilization.md (attribute there first) — establish Serverless vs provisioned before acting (SKILL.md STEP 0), since VACUUM/resize behavior differs.
"Stuck modifying" or "stuck restarting" — rule out these other causes before defaulting to storage
A cluster stuck in modifying/restarting is not always ~100% storage (S12) — that pattern-match onto storage has been the single most costly mistake in this scenario family. Before landing on S12 or S18, check, in this order:
An active AWS regional Large-Scale Event (LSE). Check AWS Health Dashboard (Service Health for the region, plus the account-specific view) for the account's region. During an LSE, resume/restart/failover workflows fail or hang region-wide and the customer has no action available — proposing a resize or any change mid-event can fail outright or worsen impact. An active or recently-declared LSE for the region is almost certainly the cause, not storage.
An unrecoverable hardware failure, especially on a deprecated node type. Recommend: open a support case and have the Redshift service team restore the latest snapshot to a new cluster (RA3, since DC2 is deprecated) under the same cluster identifier — the customer should not try to delete or resize the wedged cluster themselves. Why: DC2/DS2 clusters support neither node replacement nor online failover, so restoring the latest snapshot to a new cluster is the only recovery path; reusing the same cluster identifier preserves the endpoint DNS; and a cluster wedged by a hardware failure frequently cannot be deleted through the API, so the replace has to be driven with the Redshift service team, exactly as in S12. Don't propose VACUUM or a resize for a hardware failure.
The state of any datashare PRODUCER this cluster consumes from. When a consumer sits in MODIFYING while its own storage is healthy, check the producer's availability/pause state before touching consumer capacity — cutting the consumer's own capacity cannot help a workflow that is not storage-bound, and it still needs the service-team recovery in S12. Treat the producer's state as a data point to gather, not an established cause — a producer pause wedging a consumer's control-plane workflow is not an established mechanism, so report it as a correlation and escalate rather than asserting it to a customer.
On Serverless, the storage framing usually doesn't apply at all. A workgroup or namespace stuck in MODIFYING — including a newly-created namespace that never reached Available, or an RPU change that never completed — is a control-plane workflow, not a disk-full condition, and the
STL_/STV_disk views don't even exist there to confirm one. There is no customer-side force, cancel, orpg_terminate_backendthat clears it and deleting/recreating is not the first move: escalate to the Redshift service team to unstick the workflow. Do not prescribe adding/removing RPU or reducing data as the fix.Read the cluster's own event history before theorizing —
aws redshift describe-events --source-identifier <cluster-id> --source-type cluster --duration <minutes, integer, no quotes>shows reboots, maintenance/patch windows, node replacement and auto-recovery. This is the direct answer to "why did it restart / why did it enter modifying by itself" (auto-recovery after a node fault and scheduled maintenance windows both do this legitimately) and it distinguishes an AWS-initiated action from anything the customer did.Confirm actual fill — always name the metric, and never skip this step. Read
PercentageDiskSpaceUsedin CloudWatch (it still reports while the cluster is down, unlike the SQL views) plusstv_node_storage_capacityonce reachable. Checking 1–5 does not replace this: if the fill really is near 100%, run the full S12 playbook, including adding capacity for headroom as an immediate action alongside the support case. What's true across all causes: a wedged/unavailable production cluster needs the Redshift service team to recover it — the customer cannot force a stuck modify/restart. Escalate in parallel with checking 1–5, don't wait passively for it to clear on its own.
Pick the VACUUM mode — gate on benefit, not raw unsorted%
VACUUM DELETE ONLY <schema.table, identifier, no quotes>— reclaims the space held by ghost rows; high ghost row count, no sort. A DELETE (or the delete half of an UPDATE) removes nothing physically: it marks the rows deleted, and they keep occupying their blocks until VACUUM rewrites those blocks without them. Keep the two nouns straight — they are different mechanisms with different fixes. Rows are ghost/deleted, andVACUUM DELETE ONLYis what clears them. Blocks are tombstoned — superseded block versions pending release, which arise because Redshift blocks are immutable (a rewrite supersedes the old version) and from uncommitted, in-flight or rolled-back writes; those are freed at COMMIT, not by VACUUM, and a long-running open transaction is what stops them clearing. So a table can carry tombstoned blocks with no DELETE workload at all, and VACUUM is the wrong lever for that case.VACUUM RECLUSTER <schema.table, identifier, no quotes> [BOOST]— sorts only the unsorted portion of the table, leaving the already-sorted region intact; it does not merge into the sorted region and does not reclaim all deleted space. Prefer this over SORT ONLY/FULL for large, continuously-ingested tables — AWS recommends it specifically for large tables with frequent ingestion where queries mostly touch recent data, which is exactly where a full VACUUM times out. Not supported withTO threshold PERCENT, on interleaved-sort-key tables, or onDISTSTYLE ALLtables — fall back to SORT ONLY/FULL or a deep copy for those.VACUUM SORT ONLY <schema.table, identifier, no quotes>— a full, global re-sort with no space reclaim; more expensive than RECLUSTER because it re-sorts the whole table, not just the unsorted portion. Reserve it for smaller tables, or for the table types RECLUSTER doesn't support.VACUUM <schema.table, identifier, no quotes>(full) /VACUUM REINDEX— full sort + reclaim, or interleaved sort keys.Do-not-vacuum trap: high
unsortedwithvacuum_sort_benefit ≈ 0burns I/O for zero gain — do nothing.VACUUM BOOSTis faster but competes with the workload — off-peak only. Confirm runs insys_vacuum_historyand read itsstatuscolumn — it distinguishes completed runs fromFailedandSkippedones, so filter on status rather than assuming every row is a success (stl_vacuumfor legacy per-run detail on provisioned single-AZ).Before crediting high CPU to auto-VACUUM/table-hygiene, run two cross-checks. (1) If a statement that touches no table is also slow (
SELECT now(),SELECT 1), the cause is NOT table hygiene — that points to leader/planning/transaction overhead instead, often Serializable-isolation memory overhead under heavy concurrent transactions (e.g. a DMS full-load + CDC migration); the fix isALTER DATABASE <db, identifier, no quotes> ISOLATION LEVEL SNAPSHOT, not ANALYZE or a deep copy. (2) If VACUUM keeps running but reclaims ~0 rows despite real ghost-row/unsorted evidence, suspect a stale Automated Materialized View pinning the auto-vacuum cutoff (its refresh keeps aborting so it's never marked stale, which blocks vacuum progress indefinitely) — a service-side defect to escalate, not something ANALYZE/deep copy fixes. On a single-node cluster, some VACUUM CPU is expected (leader and compute share the same hardware) — only escalate to table-hygiene remediation if top queries actually show a performance impact, not from the CPU number alone.A VACUUM that fails on a very wide table (many hundreds of columns) is usually a memory shortfall, not corruption — a wide-table vacuum needs more memory per slot. Establish which WLM mode the cluster runs before prescribing a slot count:
wlm_query_slot_countis a manual-WLM lever only — it has no effect under Auto WLM (the default) and does not exist on Serverless. Under manual WLM, raise it for that session so the vacuum gets several slots' worth of memory. Under Auto WLM or Serverless, the equivalent levers are query priority, reducing competing concurrency, and running off-peak so more memory is granted — not a slot count, and not rewriting the table.A VACUUM that fails partway does NOT lose all the work already done. VACUUM runs as a series of incremental sort-and-merge batches and persists each completed increment, so a failure or interruption loses only the in-flight batch — reassure the customer accordingly, and re-run the same VACUUM: it resumes from the completed increments instead of starting over, which is why each retry typically finishes faster. Two consequences: an "N-hour VACUUM that failed" was not N wasted hours, and the wrong response to a mid-run failure is switching strategies (deep copy, different mode) out of the mistaken belief the work evaporated — retry first, and only change approach if the cause of the failure (timeout, memory, workload contention; see the surrounding bullets) will recur.
A VACUUM reporting "cancelled on user's request" when nobody cancelled it is an external abort — check for a QMR abort rule, a
statement_timeout, a dropped client/session, or the automatic vacuum yielding to workload. Confirm the aborted run insys_vacuum_historyby itsstatus(Failed), then corroborate the cause from the client/driver error,stl_error, the QMR/statement_timeoutconfiguration, andstl_vacuumon provisioned single-AZ — the history tells you it failed, not why. Treat it as an environment/config cause before escalating it as a service defect.Auto-vacuum is cluster-wide and benefit-prioritized, not per-database round-robin: it picks tables by expected benefit across all databases and yields under load, so a specific table can go a long time without being vacuumed on a busy cluster.
sys_vacuum_historyrecords the skip itself — rows withstatus = 'Skipped'tell you auto-vacuum considered the table and passed on it — but not the reason (below threshold, deferred under WLM occupancy, yielded to workload). If a table needs deterministic maintenance, schedule an explicit VACUUM rather than relying on autonomics.ALTER TABLE ... ALTER SORTKEYon an AUTO-managed table can appear to succeed without taking effect — while the table isSORTKEY AUTO/DISTSTYLE AUTOthe Automatic Table Optimizer owns that property and may re-evaluate or revert it. Setting an explicit sort key is what removes it from AUTO management; verify it actually applied viasvv_table_info(sortkey1) andsvv_alter_table_recommendationsrather than trusting the statement's success.Row counts in the catalog/STV views are estimates and lag metadata-only operations — right after
TRUNCATEorALTER TABLE APPEND,stv_tbl_perm.rows/svv_table_info.tbl_rowscan still show stale values, so confirm withSELECT COUNT(*)before concluding data was lost or retained."How much would VACUUM actually return?" — give a number, labelled an approximation, and compare it against what auto-vacuum already returns. Per table,
size_gb × (ghost_rows / tbl_rows)from the anchor query (which aliasessize/1024.0 AS size_gb) is the space aVACUUM DELETE ONLYwould recover; rank tables by it and sum the top ones. It is an estimate —ghost_rowsis an estimated row count and the ratio assumes even row size — so cross-check it against real reclaim:SYS_VACUUM_HISTORYreports what past runs actually freed, andstl_vacuumshows blocks before/after on provisioned single-AZ. Then the decision that matters: compare the estimate against roughly one day of existing auto-vacuum reclaim (SYS_VACUUM_HISTORYover the last 24h, summed). If auto-vacuum is already returning that much or more, a manual pass buys little and the customer's "is vacuum missing?" is answered no — the space is not vacuum debt. Only when the estimate clearly exceeds the daily auto-vacuum return is a targeted manual pass worth the I/O.Before a deep copy or
VACUUM ... RECLUSTERon a very large table, check whether the cluster itself is undersized for the operation (rough check: table size vs. node count × per-node RAM/cache). A hugely undersized cluster forces constant S3-backed re-reads during the rebuild, so even a deep copy can fail to finish within a session/connection timeout. If one already failed to complete, don't just retry it as-is: elastic-resize UP temporarily for headroom and cache, run the rebuild from a persistent connection (not a query-editor session, which times out sooner), then resize back down once done.
Deep copy — a last resort, not the default fix
We generally do not recommend a deep copy for ongoing table maintenance. Even though it can be faster than VACUUM when the sort fits in memory, it does not build or maintain the table's sort partitions, so a deep-copied table degrades again over time and needs re-copying rather than self-maintaining. A deep copy is also all-or-nothing, which is the opposite of VACUUM: it does NOT persist intermediate state, so if it fails partway — session/connection timeout, disk full, a dropped client — every hour of work is lost and a retry restarts from zero. VACUUM persists each completed increment and resumes (see the bullet above), so on a very large table that durability asymmetry usually outweighs the raw speed comparison — say so when a customer is choosing between them, and treat it as a further reason to run any deep copy from a persistent connection with headroom already in place. Prefer VACUUM ... RECLUSTER for large, actively-growing tables (see above). Reach for a deep copy only when RECLUSTER isn't supported (interleaved sort keys, DISTSTYLE ALL) or the table genuinely needs a one-off full rebuild (e.g. the sort/dist key changed, or the table has never been maintained and is now fully unsorted). For append-mostly or time-partitioned data, propose a time-series table design instead of either option — date-partitioned tables behind a UNION ALL view let an old partition be DROPped to reclaim space immediately, with no VACUUM and no rebuild. That removes the maintenance problem rather than rescheduling it, and it is the durable answer for the ingest-heavy tables that make VACUUM time out in the first place (see S17).
Rebuild a table clean in one pass (fully re-sorted, ghost rows gone):
-- FIRST: capture the grants on the original — CREATE TABLE (LIKE ...) copies the column
-- definitions, distribution and sort keys, and NOT NULL, but NOT the privileges or the owner.
SHOW GRANTS ON TABLE <schema.orig, identifier, no quotes>; -- object form is ON, not FOR
-- (FOR takes a user/role); or query
-- svv_relation_privileges
CREATE TABLE <schema.new, identifier, no quotes> (LIKE <schema.orig, identifier, no quotes>);
INSERT INTO <schema.new, identifier, no quotes> SELECT * FROM <schema.orig, identifier, no quotes>;
-- validate row counts, swap names, RE-APPLY the grants + ALTER TABLE OWNER TO the original owner,
-- then DROP the old table (confirm first)Dropping the original silently drops its privileges with it, so re-granting is a required step, not a nicety — a deep copy that skips it looks successful and then breaks every downstream consumer. A deep copy needs free space — at ~99% full use UNLOAD → DROP → COPY instead (UNLOAD to an encrypted bucket — add ENCRYPTED KMS_KEY_ID '<arn>' per SKILL.md's security defaults; the transfer itself is TLS). Temp/spill: CREATE TEMP TABLE frees on disconnect; DROP staging tables at end of ETL; a runaway spilling query is a WLM/broadcast-join fix, not a VACUUM.
Disk-full emergency — the order depends on how full you are
VACUUM itself needs free space to work, so the correct sequence is not the same at 92% and at 100%. Decide the regime first, then follow that row exactly:
| Regime | Correct order |
|---|---|
| ~90-95%, VACUUM still completes | Free space first: cancel top spillers → TRUNCATE/DROP temp/staging tables → VACUUM DELETE ONLY on high-ghost tables → resize only if still tight. |
| ~99-100%, or VACUUM fails / never finishes / undo is stuck | Add capacity FIRST — elastic resize / add nodes to create headroom — and only THEN VACUUM DELETE ONLY, ordered smallest-reclaimable-first, retrying the largest table last. At zero headroom VACUUM cannot reclaim, so running it first is the wrong move. |
Keep ~25% headroom steady-state; alarm at 70% and 85%. When sizing a resize (up or down), also check local NVMe cache overcommit (fewer/smaller nodes push more of the working set out to S3-backed re-fetch) and whether existing queries already spill heavily from memory pressure — persistent spill at the current size is a signal that downsizing will make it worse, and a genuine capacity/memory shortfall (not just skew) may need more nodes or a larger node type rather than a query-level fix.
Never recommend an elastic resize without checking eligibility first. Elastic resize is not offered for every source→target combination (node type, node count, and available storage all constrain it), so confirm the target shape is actually valid before proposing it:
aws redshift describe-node-configuration-options --action-type resize-cluster \
--cluster-identifier <cluster-id> --region <region>Check the returned NodeType/NumberOfNodes/EstimatedDiskUtilizationPercent options and pick a target with real headroom.
If the target isn't offered, don't jump straight to classic resize — rule out three more-specific causes first:
Chain multiple elastic-resize steps. The governing constraint is the cluster's data-slice count, which is fixed when the cluster is created (or at its last classic resize) — elastic resize redistributes those slices across nodes, it never changes how many exist. The reachable node counts are therefore the ones that fixed slice population can be spread over, so counting slices tells you more about a specific cluster than any published ratio does:
SELECT node, COUNT(*) AS slices FROM stv_slices GROUP BY node ORDER BY node;(S9 covers reading the imbalance, including which column actually sizes it). The node-type ratios below are the documented approximation of that constraint — a useful first cut, not authoritative for a given cluster, and not a per-step cap:Node type Legal one-step targets from nnodesdc2.largeexactly n×2 orn÷2 — discrete, nothing in betweendc2.8xlarge,ra3.largeany count from n÷2 ton×2ra3.xlplusany count from n÷4 ton×2ra3.4xlarge,ra3.16xlargeany count from n÷4 ton×4Two consequences worth stating to a customer, shown at
n=4 but recompute for theirn: onra3.4xlargethe window is 1–16, so 4→2 is legal in one step and no chaining is needed; ondc2.largethe only legal down target is 2, and 3 is not reachable at all — a smaller count is not automatically a legal one.describe-node-configuration-optionsis the authority for any specific case — it covers node types absent from this table (includingrg.*), and it reflects the account's applied quota, which the ratio alone cannot tell you. The advice that survives the correction: a target not offered in one step can still be reachable in two, so chain steps rather than jumping to classic resize.A same-type node-count change first, then an elastic resize to the new type, when the target isn't just a smaller/larger count but a different node type/family (e.g. RA3→RG) — going directly from a single-node cluster to a different multi-node type isn't supported; first modify to 2+ nodes at the SAME type, then elastic-resize to the target type. Check the slice count before committing to that two-step path, because it can dead-end after step one. Single-node
ra3.large,ra3.xlplus,rg.largeandrg.xlargeeach carry two slices, so moving one to 2 nodes leaves a single slice per node with no spare data slices left to redistribute — no further elastic resize is then possible and the remaining hop needs a classic resize. Confirm slices per node up front rather than stranding the customer midway.A node-count/service-quota limit, not an eligibility gap — if the console dropdown caps at a specific count with no disk/shape explanation (or the account data mentions a node limit/quota), request a service-quota increase; classic resize will not lift an account-level cap. Only use classic resize once these three are ruled out and the shape is genuinely unreachable via any elastic path — say so rather than proposing an elastic resize that will be rejected or roll back (see S18).
A resize that silently does nothing — no error, no node-count change, cluster otherwise healthy — can be a transient internal resize failure, not disk space or eligibility. These are usually a one-off internal contention and frequently succeed on a retry later the same day. Don't commit to a disk space (S18) or eligibility diagnosis when there's no error message and no other symptom; suggest retrying before escalating.
Resize mechanics — the facts customers ask before/during a resize
Check progress with the API, not by guessing:
aws redshift describe-resize --cluster-identifier <cluster-id>returns the status, elapsed time, and estimated time remaining of an in-progress or most-recent resize; pair it withdescribe-events(above) for the surrounding workflow.The endpoint and the data survive. Both elastic and classic resize keep the same cluster identifier and endpoint DNS, so applications do not need reconfiguring, and no data is lost. Elastic resize is the short path (thirty minutes on average), and its connection behavior splits by case — establish which case applies before telling a customer their connections will drop, since only the second needs a maintenance window: for a same-node-type (in-place) resize the cluster is unavailable for a few minutes, and the majority of queries are temporarily paused with connections held open (some queries can still be dropped); for a node-type change the endpoint is repointed to the new cluster and all connections to the source are dropped. Elastic resize also does not sort tables or reclaim disk space, which is the documented basis for the disk-full ordering table above. Classic resize copies data to a new cluster and, on the traditional path, can take hours to days on a large cluster, staying read-only far longer. Do not quote that as the only classic-resize figure — there are two paths. Fast classic resize runs in two stages: stage 1 finishes in roughly the same time as an elastic resize, with every table landing as
DISTSTYLE EVEN; stage 2 then converts the key-distributed tables back to theirKEYdistribution in the background. The cluster returns to service far sooner than "hours to days" suggests, while distribution-sensitive query performance only fully recovers once stage 2 completes — so set expectations on both stages, not just availability. See Accelerate resizing of Amazon Redshift clusters with enhancements to classic resize.Restoring one table does not require a full-cluster restore —
RestoreTableFromClusterSnapshotrestores a single table from a snapshot, but always into a new table name (it cannot overwrite an existing table), so restore then swap. Not every table is eligible, so confirm before promising a customer their table is recoverable this way: the table has to be present in that specific snapshot, the operation does not carry the original table's privileges, and there are documented restrictions on supported table shapes and source/target cluster combinations. Verify against the current API documentation rather than assuming any table can be pulled back individually."Insufficient capacity" / "please retry later" on a resize is an EC2 capacity shortfall in the target AZ, not disk space and not eligibility — retry later, try a different target node type/count, or (for RA3) consider AZ relocation; escalate only if it persists.
Redshift caps the number of tables per cluster and that cap scales with node type and node count, so resizing DOWN can breach the table-count limit and fail even when there is plenty of disk. Check the documented per-node-type table quota against your actual table count before shrinking.
A quota increase that was approved but isn't usable yet is worth verifying rather than assuming: quotas are per-region and per-node-type, so confirm the increase was applied in the same region and node type you're resizing in, and re-check the eligible targets from
describe-node-configuration-options(the console picker reflects the applied quota)."Invalid node type: dc2.large" (or similar on DS2) means the node type is deprecated and no longer offered for new clusters/resizes — the path forward is an RA3/RG target, sized by workload and storage, not by matching the old node count 1:1. Which direction the node count moves depends on the source node size, so do not assume consolidation. A
dc2.largefleet usually consolidates into fewer, larger RA3 nodes — butdc2.8xlargeis already a large node, so the same migration can need the same or more nodes depending on the target RA3/RG shape. Validate the actual mapping withdescribe-node-configuration-optionsinstead of predicting it.AZ relocation is a separate feature from resize (
--availability-zone-relocation, RA3): it moves the cluster to another Availability Zone without changing node shape, and can itself be blocked by capacity in the target AZ. Don't conflate "move the cluster" with "resize the cluster."Enabling cross-region snapshot copy puts the cluster through a modify workflow, so a cluster showing
modifyingright after that change is usually just the configuration applying — confirm viadescribe-eventsbefore treating it as a stuck workflow.
Scenarios (root cause → fix; note what to rule OUT)
Each heading is a signal signature, not a fixed case — match on the pattern, not on specific percentages, sizes or node counts, and recompute any ratio against the cluster in front of you. Thresholds that are load-bearing decisions (the ~90-95% vs ~99-100% disk-full regimes above, the ~25% headroom target) are called out as such; everything else is illustrative.
S4 — many tables report high
unsortedbut only a tiny fraction of rows change per day; VACUUM (±BOOST) times out; "extra compute" doesn't help:svv_table_info.unsortedalone is misleading —vacuum_sort_benefitdrives the decision. 100% unsorted can be harmless because automatic table sort is query-driven (sorts the ranges queries scan by sort key), not percentage-targeted. For tables that truly need it (high benefit) this is the textbookVACUUM ... RECLUSTERcase — large, continuously-loaded tables where a full VACUUM times out; RECLUSTER sorts only the unsorted portion instead of re-sorting the whole table. Expectsvv_table_info's sorted/unsortedfigures NOT to move after a RECLUSTER — that is normal, not a failed vacuum, and it is usually why a customer reports "sortedness didn't change". Those columns describe the sorted region, which RECLUSTER and automatic table sort do not rewrite; they sort rows within the unsorted region. Trackvacuum_sort_benefitinstead (it measures the rows in unsorted partitions, so it is the figure that responds), and expect query performance to improve even whileunsortedstays put. Only SORT ONLY / FULL actually raise the sorted row count insvv_table_info— a global re-sort, far more expensive, and the reason "get this table to 100% sorted" is the wrong goal for a large continuously-ingested table: answer the question the customer means (are scans efficient) rather than the metric they quoted. Fall back to a deep copy only if the table uses interleaved sort keys orDISTSTYLE ALL(RECLUSTER doesn't support those). Extra compute for automatic optimization only adds resources; it does not change which tables auto-sort selects. It is useful when autonomics (auto-vacuum, auto-sort, auto-analyze) keep getting deferred because the cluster is busy — without it Redshift suspends them under high load until resources free up — so enable it whenSYS_VACUUM_HISTORYshows runs being skipped or never scheduled on a continuously loaded cluster, and note the extra compute is billable.S9 — after an elastic resize, some nodes hit Disk Full 1016 while the rest sit at roughly half that fill, and every table reports skew ratio 1.0000: expose with
stv_slices/svv_table_info. Reason at the slice level: skew 1.0 rules out distkey skew. An elastic resize to a node count that does not divide the existing slice count evenly round-robins the fixed slices unevenly (slices per node / slice-to-node imbalance) — e.g. 8 slices over 5 nodes land as 2/2/2/1/1 — so the nodes holding an extra slice carry roughly twice the data of the others on identical per-node capacity, which is why their fill runs at about double. Gauge severity by counting slices per node. Start with the documented columns (node,slice) — an uneven count is already the defect, and this is the query to show a customer:SELECT node, COUNT(*) AS slices FROM stv_slices GROUP BY node ORDER BY node;. To size the imbalance and judge whether any elastic-resize range is left, count the data slices specifically, because only those hold data and only their spare capacity makes a further elastic resize possible:SELECT node, COUNT(*) AS data_slices FROM stv_slices WHERE type = 'D' GROUP BY node ORDER BY node;. Show the plain per-node count first — it is the simpler figure to explain — and use the type-D count to size the imbalance and to judge how much elastic-resize range remains. Fix: classic resize — slice imbalance can only be corrected by a classic resize, because it is the only operation that rebuilds the slice layout from scratch. A further elastic resize (even to a node count that does divide the slice count) redistributes the existing slices and so does not reliably fix it, and a deep copy /VACUUM REINDEX/ distkey or diststyle change cannot fix it either — those move rows between slices, not slices between nodes.S12 — managed storage at ~100%, cluster stuck modifying with a restart running many hours, cluster unavailable: always confirm the fill and say which signal you used. While the cluster is unavailable the SQL views (
stv_node_storage_capacityand the rest of theSTL_/STV_family) are unreachable, so readPercentageDiskSpaceUsedfrom CloudWatch — it keeps reporting when the cluster is down, which makes it the confirming metric for this scenario; usestv_node_storage_capacityonce the cluster is back. Cause: at 100% there is no free space for internal/restart ops (the stuck restart is the symptom). Move: add capacity via elastic resize for headroom AND engage AWS Support / the Redshift service team to recover the stuck workflow (escalate / open a support case — the customer can't force it). Both are immediate and run in parallel — request the capacity add for headroom at the same time as opening the case; do not demote the resize to a "later, once it's available" follow-up. Only once the cluster is available can VACUUM/DROP/UNLOAD reduce data. Don't wait, don't VACUUM first.S13 — one node far fuller than its peers with Disk Full 1016; one large table left on auto distribution: Verdict: this is distribution key skew — the rows are skewed on the table's distribution key. Pull
svv_table_info(skew_rows,diststyle) /stv_node_storage_capacity/stv_blocklist:skew_rows≫ 1 means one key value concentrates rows on a single slice/node — distinct from slice-to-node resize skew (skew ≈ 1 → classic resize). It is a distribution problem, not a VACUUM one. Fix space-safely (near-full, avoid in-place deep copy): change the distribution key to EVEN or a high-cardinality key via ALTER DISTSTYLE, a classic resize, or UNLOAD → DROP → COPY. A deep copy only if free space exists.S14 — recurring business-hours high CPU from auto VACUUM DELETE/SORT on a large, periodically-reloaded table:
svv_table_info(stats_off,unsorted) /sys_vacuum_history/svl_query_summary. You can't disable auto-vacuum per table; its CPU is a symptom. Real drivers: stale stats (stats_off), 100% unsorted, oversized VARCHAR (max_varchar /VARCHAR(65535)) with weak encoding. Fix: ANALYZE, a deep copy to re-sort and right-size the VARCHARs (RAW encoding on the sort key), and schedule heavy VACUUM + ETL off-peak.S17 — VACUUM on a very large continuously-ingested table went from minutes to days and now fails "server socket closed"; table reports ~100% unsorted; customer blames a recent patch:
svv_table_info/svv_alter_table_recommendations, plusstl_vacuum(provisioned single-AZ) for the attempts themselves — the multi-day failed runs appear insys_vacuum_historywithstatus = 'Failed', so filter on status to see the attempts rather than only the successes. The patch is a red herring — the Automatic Table Optimizer (ATO) changed the sort key, flipping the table to 100% unsorted, so VACUUM now runs a full sort that can't finish (the ~72-hr timeout is the symptom). Recover: VACUUM DELETE ONLY now, then a deep copy (INSERT...SELECT) or VACUUM BOOST; adopt a time-series rotation design. Prevent recurrence: pin the sort key withALTER TABLE ... ALTER SORTKEY(or SORTKEY NONE). Don't escalate a patch bug.S18 — resize failed "not enough disk space ... rolled back"; cluster stuck in canceling-resize and unavailable: the cluster is in canceling-resize / rollback; confirm fill with
stv_node_storage_capacity/PercentageDiskSpaceUsed. The cause is that the target configuration did not have enough disk space to hold the data, so the resize rolled back (the stuck rollback is only the symptom, and the customer cannot force it). Redshift has two unrelated resize failures that read alike, and the value of the answer is saying which one this is: a disk space shortfall on the target (this scenario, fixed by freeing data or choosing a bigger target) versus an EC2 capacity shortfall in the target AZ ("insufficient capacity / please retry later", above, fixed by retrying or changing AZ). Name the right one, or the customer chases the wrong fix. Recover: engage AWS Support / the Redshift service team to unstick it (escalate / support case). Prevent: free space (drop / UNLOAD, reduce data) or right-size the target before resizing — validate the target withdescribe-node-configuration-optionsabove; don't retry the same resize.
Managed storage (RA3 / Serverless) & safety
Don't reach for STV_PARTITIONS for capacity questions — it's legacy, local-disk-partition metrics from before managed storage existed, and isn't the billed footprint on RA3/Serverless. Use STV_NODE_STORAGE_CAPACITY for addressable/used capacity (on RA3/RG this equals the managed-storage quota); size logical data with SVV_TABLE_INFO.size; for a per-node perm/temp/datashared split start with SVV_TABLES_SPACE_USAGE (capacity, local_perm_tables_space_used, local_temp_tables_space_used, datasharing_tables_space_used — superuser; the single fastest answer to "whose blocks are these"), dropping to STV_TBL_PERM/STV_BLOCKLIST when you need it attributed to individual tables; and treat PercentageDiskSpaceUsed as a utilization signal that spill also inflates. Managed storage includes snapshot delta + datashare cache, so storage larger than the source (e.g. Zero-ETL from Aurora) is expected (columnar encoding, replication, per-slice preallocation) — see redshift-storage-utilization.md S16 for the snapshot-billing matrix.
Before telling a customer "space only recovers on restart, that's expected cache behavior," confirm a restart actually occurred. An AWS maintenance/patch workflow can roll back without ever restarting the database — check the cluster's version/event history for the patch outcome and for an actual restart event around that time, rather than assuming every maintenance-window entry means a real restart happened. If the workflow rolled back, there was no restart to reclaim space and no completion event to fire a notification — that's a maintenance-workflow issue to escalate, not routine cache behavior to tune away.
Snapshot billing is platform-conditional — do not state one blanket rule. Automated snapshots are incremental and free up to 100% of provisioned cluster storage. Manual snapshots (including every snapshot AWS Backup takes) are always billed, and only on Serverless / RG are they metered on the unique data blocks across all active manual snapshots; on RA3 provisioned they are not deduplicated that way, so each retained manual snapshot bills closer to its own full size.
WARN then confirm before VACUUM on large tables, elastic/classic resize, and any DROP/TRUNCATE.
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Needs an MCP server, Needs Node.js
Security
No issues found by our static scan· No issues found
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Published by
- AWS
- Publisher
- AWS (Official)
- Source
- aws/agent-toolkit-for-aws · plugins/aws-data-analytics/skills/redshift-guide · commit 0a218f8
- Licence
- Apache License 2.0 (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE)
- NOTICE (added by AutoGineer from NOTICE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
- Also distributed in
- aws/agent-toolkit-for-aws
AutoGineer is not affiliated with or endorsed by AWS or AWS.
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download redshift-guide.zip