Ingesting Into Data Lake

AWSOfficialPublished by AWS

Skill
Workflow
Data and AI
Apache-2.0
Low risk

From the author

Import data into the AWS data lake from S3 files, local uploads, JDBC databases (Oracle, SQL Server, PostgreSQL, MySQL, RDS, Aurora), Amazon Redshift, Snowflake, BigQuery, DynamoDB, or existing Glue catalog tables (migration). Default target is S3 Tables; standard Iceberg on a general purpose bucket is supported where S3 Tables is not adopted. Handles one-time loads, recurring pipelines, migrations. Triggers on: import data, load data, ingest, sync database, migrate table, move data to AWS, set up pipeline, ETL, pull from Snowflake, query BigQuery into S3, export DynamoDB, CTAS, convert to Iceberg. Do NOT use for setting up or troubleshooting Glue connections (use connecting-to-data-source), creating empty tables (use creating-data-lake-table), running queries (use querying-data-lake), finding tables by fuzzy name (use finding-data-lake-assets), catalog audit (use exploring-data-catalog), or SaaS platforms like Salesforce, ServiceNow, SAP, MongoDB, Kafka.

— the description by AWS

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured workflow for importing data from various sources—including JDBC databases, cloud warehouses like Snowflake and BigQuery, DynamoDB, and S3 files—into an AWS data lake. It prioritizes modern S3 Tables or standard Iceberg formats, ensuring data is queryable and well-structured. The skill guides users through connection verification, target configuration, and validation, while delegating complex setup tasks to specialized sub-skills.

What it does

Imports data from JDBC databases, cloud warehouses, and S3 into a data lake
Supports one-time loads, recurring ETL pipelines, and catalog migrations
Configures target tables using S3 Tables or standard Iceberg formats
Validates data integrity via row counts, null checks, and sample verification
Automates scheduling for recurring ingestion tasks using Glue triggers
Provides clear delegation paths for connection setup and table creation

When to use it

When you need to move data from an external database or cloud service into your AWS data lake
When setting up a new recurring ETL pipeline for data synchronization
When migrating existing Hive-style tables to modern Iceberg formats
When you need to perform a one-time bulk load of files from S3 or local storage

Advantages

Standardizes ingestion patterns across multiple source types
Ensures data quality through built-in validation steps
Promotes modern data lake practices by defaulting to S3 Tables and Iceberg
Reduces configuration errors by enforcing connection verification before execution

Usage examples

“Import the customers table from our Oracle database into the data lake.”: The agent verifies the existing JDBC connection, confirms the target S3 Tables configuration, and initiates a Glue ETL job to migrate the data.
“Set up a recurring pipeline to pull data from BigQuery into S3.”: The agent configures a Glue job with the necessary BigQuery connection, sets up the target Iceberg table, and creates a Glue trigger for scheduled execution.
“Convert our existing Hive tables to Iceberg.”: The agent performs a catalog migration, updating the table metadata to Iceberg format while maintaining data accessibility.
“Load these local CSV files into the data lake.”: The agent guides the user through the local upload process and maps the data into a queryable S3 Tables format.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/aws/ingesting-into-data-lake/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip ingesting-into-data-lake.zip -d .agents/skills

Documentation by AWS

Ingest into Data Lake

Move data from a source into a queryable table in the data lake. This skill assumes the source connection (if one is needed) already exists. For Glue connection setup or troubleshooting, delegate to connecting-to-data-source.

Philosophy

Default to S3 Tables unless the environment says otherwise. S3 Tables is the recommended target for new data lake work. If the user's catalog inventory shows they haven't adopted S3 Tables, recommend standard Iceberg on their existing general-purpose bucket instead of forcing them to change posture.

Common Tasks

You MUST execute commands using AWS MCP server tools when connected -- they provide validation, sandboxed execution, and audit logging. Fall back to AWS CLI only if MCP is unavailable. You MUST explain each step before executing.

Workflow

1. Verify Dependencies and Context

You MUST check whether AWS MCP tools or AWS CLI are available and inform the user if missing
You MUST confirm target AWS region and verify credentials with aws sts get-caller-identity
For SageMaker Unified Studio project roles, note that target tables and connections may be scoped to the project. See the caller ARN detection pattern in querying-data-lake.

2. Classify the Source

User says...	Source type	Reference
"upload my file", "local CSV", "move to S3"	Local file	local-upload.md
"load from S3", "import CSV/JSON/Parquet from s3://"	S3 files	s3-files.md
"import from Oracle/Postgres/MySQL/SQL Server/Redshift/RDS/Aurora"	JDBC	jdbc-ingest.md
"pull from Snowflake", "Snowflake table to S3"	Snowflake	snowflake-ingest.md
"import from BigQuery", "GCP analytics to S3"	BigQuery	bigquery-ingest.md
"export DynamoDB", "DynamoDB to data lake"	DynamoDB	dynamodb-ingest.md
"migrate Glue table", "convert Hive to Iceberg"	Catalog migration	catalog-migration.md

If the user names Salesforce, ServiceNow, SAP, MongoDB, Kafka, or another SaaS/streaming source, decline -- these are not supported in this release.

If the source table is referenced by a fuzzy or business name ("migrate our orders table", "pull from the sales warehouse"), delegate to finding-data-lake-assets to resolve before proceeding.

3. Confirm Connection Exists (if applicable)

For JDBC, Snowflake, and BigQuery sources, a Glue connection is required. Check:

aws glue get-connection --name <CONNECTION_NAME> --region <REGION>

If the connection does not exist, stop and delegate to connecting-to-data-source to create and test it. Do not proceed with ingest until the connection is verified.

Local files, S3 files, DynamoDB, and catalog migration do not need a Glue connection.

4. Clarify the Target

You MUST ask the user (or suggest based on catalog inventory) before creating or writing to any table:

Database/namespace: Does a specific target database exist? Or should one be created?
Table: Existing table (append/merge) or new table (delegate to creating-data-lake-table)?
Format: S3 Tables (default), standard Iceberg, or raw Parquet?

Inventory-aware defaults:

If you have already run exploring-data-catalog or can quickly check, use what exists:

Account has an s3tablescatalog federated catalog and active table buckets: recommend S3 Tables
Account has general-purpose buckets with Iceberg tables and no S3 Tables usage: recommend standard Iceberg on their existing bucket
Account uses Parquet/ORC on S3 without Iceberg metadata: ask whether to adopt Iceberg now (recommend yes) or continue with raw files

Do not force S3 Tables on customers who haven't adopted it. See iceberg-catalog-config-and-usage.md.

Delegations from this step:

Target table doesn't exist -> creating-data-lake-table
Target database named by fuzzy term -> finding-data-lake-assets
User doesn't know what exists -> exploring-data-catalog

5. Execute Source Workflow

Read the source-specific reference and follow its phases. Each is self-contained with job templates, gotchas, and troubleshooting:

Local / S3 / JDBC / Snowflake / BigQuery / DynamoDB / catalog migration -- one reference per source

Common Glue 5.1 or higher job configuration and PySpark templates are shared in glue-job-config.md and glue-job-scripts.md.

6. Validate

Run all three, do not skip:

Row count matches expected (source vs target)
Null check on critical columns
Spot-check 3-5 sample rows

See data-quality-validation.md.

7. Schedule (if recurring)

For recurring pipelines, create a Glue Trigger with a cron schedule. See testing-and-scheduling.md. Simple single-step pipelines use Glue Triggers; multi-step with branching uses MWAA.

Argument Routing

S3 path only: Infer one-time load, start Step 2 with S3 files
Connection name: Start Step 3 with the named connection
Table name: Start Step 4, ask whether this is source or target
--target flag: Pre-fill the target format in Step 4
No args: Walk through interactively

Gotchas

S3 Tables requires Glue 5.1 or higher and --datalake-formats iceberg job argument
All spark.sql.catalog.* config MUST go in --conf job arguments, never in spark.conf.set(). Glue 5.x throws AnalysisException: Cannot modify the value of a static config otherwise. See iceberg-catalog-config-and-usage.md for correct catalog configs.
The warehouse parameter is required in S3 Tables catalog config. Without it Spark fails with "Cannot derive default warehouse location".
Table and column names in S3 Tables MUST be all lowercase
overwritePartitions() only replaces partitions present in the DataFrame -- for full refresh with deletes, use createOrReplace()
Standard Iceberg targets MUST include a LOCATION clause; S3 Tables MUST NOT
DynamoDB does not need a Glue connection -- do not attempt to create one
Connection failures during ingest delegate back to connecting-to-data-source; do not debug network/credentials in this skill
For target tables in SageMaker Unified Studio projects, ensure the project role has write access to the target namespace before the Glue job runs

Troubleshooting

Error	Likely cause	Action
Access Denied on S3	Missing IAM permissions	Check Glue role has s3:GetObject, s3:PutObject
Access Denied on S3 Tables	Missing s3tables:* permissions	Add S3 Tables inline policy to Glue role
CTAS timeout	Dataset too large for Athena	Switch to Glue ETL or batch with WHERE filters
JDBC connection timeout/auth failure	Connection-level issue	Delegate to `connecting-to-data-source`
Throughput exceeded (DynamoDB)	Read percent too high	Lower `read.percent` or use native export

See error-handling.md for the full catalog.

References

Source-specific

local-upload.md -- Local files
s3-files.md -- S3 files (CSV, JSON, Parquet, Avro, ORC)
jdbc-ingest.md -- Oracle, SQL Server, PostgreSQL, MySQL, RDS, Aurora, Redshift
snowflake-ingest.md -- Snowflake
bigquery-ingest.md -- BigQuery
dynamodb-ingest.md -- DynamoDB (export and Glue direct read)
catalog-migration.md -- Existing Glue catalog tables (Hive, self-managed Iceberg)

Cross-cutting

iceberg-catalog-config-and-usage.md -- S3 Tables, standard Iceberg, raw files: catalog config, engine access patterns
glue-job-config.md -- Job sizing, monitoring, retry
glue-job-scripts.md -- PySpark templates (append, upsert, custom SQL, full refresh)
incremental-loading.md -- Watermark strategies
testing-and-scheduling.md -- Glue Triggers, MWAA
data-quality-validation.md -- Row counts, null checks, Glue Data Quality
schema-evolution.md -- ALTER TABLE ADD COLUMNS, nested JSON
type-transformations.md -- Type conflict resolution
format-specific-loading.md -- CSV/JSON/Parquet/Avro/ORC specifics
athena-loading.md -- Athena INSERT INTO as simple-load fallback
error-handling.md -- Ingest errors (connection errors delegate to connecting-to-data-source)
upload-options.md -- aws s3 cp vs sync, multipart

Migration-specific

ctas-patterns.md -- Athena CTAS syntax and partition transforms
glue-etl-migration.md -- Large-table migration via Glue 5.1 or higher PySpark
migration-validation.md -- Full validation checklist
migration-troubleshooting.md -- CTAS failures, visibility, partitions

JDBC-specific

jdbc-schema-discovery.md -- Crawler, direct inspection, custom SQL
jdbc-performance.md -- Parallel reads, partitioning

Files

ATTRIBUTION.mdAdded by AutoGineer928 B
LICENSELicence file added by AutoGineer9.9 KiB
NOTICELicence file added by AutoGineer67 B
SKILL.md10.8 KiB
references/athena-loading.md2.8 KiB
references/bigquery-ingest.md3.7 KiB
references/catalog-migration.md6.9 KiB
references/ctas-patterns.md3.0 KiB
references/data-quality-validation.md12.2 KiB
references/dynamodb-ingest.md7.2 KiB
references/error-handling.md12.3 KiB
references/format-specific-loading.md12.8 KiB
references/glue-etl-migration.md4.5 KiB
references/glue-job-config.md9.4 KiB
references/glue-job-scripts.md9.6 KiB
references/iceberg-catalog-config-and-usage.md7.9 KiB
references/incremental-loading.md13.1 KiB
references/jdbc-ingest.md5.7 KiB
references/jdbc-performance.md10.3 KiB
references/jdbc-schema-discovery.md12.7 KiB
references/local-upload.md4.5 KiB
references/migration-troubleshooting.md2.6 KiB
references/migration-validation.md2.7 KiB
references/s3-files.md5.7 KiB
references/schema-evolution.md9.3 KiB
references/snowflake-ingest.md3.6 KiB
references/testing-and-scheduling.md13.6 KiB
references/type-transformations.md7.8 KiB
references/upload-options.md1.0 KiB

references/jdbc-schema-discovery.md

Schema Discovery for External Databases

Complete guide for discovering schemas in external database systems and inferring target S3 Table schemas.

Overview

Schema discovery identifies what data is available in the source system and maps it to Iceberg-compatible types for the target S3 Table.

Identifying Source Data

Ask the User

Gather information about what to import:

Which table(s) or view(s) to import:

Single table: CUSTOMERS, SALES_ORDERS, INVENTORY
Multiple tables: List of tables to import
Views: Database views are treated like tables

Schema/database name (if system supports multiple databases):

Oracle: Schema name (e.g., SALES_SCHEMA)
SQL Server: Database and schema (e.g., SalesDB.dbo)
PostgreSQL: Schema within database (e.g., public, analytics)
MySQL: Database name (e.g., sales_db)

Custom SQL query (optional):

If user wants to filter or transform at source
Useful for reducing data volume before transfer
Example: SELECT * FROM CUSTOMERS WHERE status = 'ACTIVE' AND created_date >= CURRENT_DATE - 90

Auto-Discovering Available Tables

Use Glue crawlers to discover available tables in the source database.

Create Temporary Crawler

# Create crawler
aws glue create-crawler \
  --name "temp-discovery-crawler" \
  --role "<glue-service-role-arn>" \
  --database-name "temp_discovery_db" \
  --targets '{
    "JdbcTargets": [{
      "ConnectionName": "<connection-name>",
      "Path": "<database>/%"
    }]
  }' \
  --region <region>

Path Patterns for Different Databases

Database	Path Pattern	Example
Oracle	`<schema>/%`	`SALES_SCHEMA/%`
SQL Server	`<database>/<schema>/%`	`SalesDB/dbo/%`
PostgreSQL	`<database>/<schema>/%`	`analytics/public/%`
MySQL	`<database>/%`	`sales_db/%`

Run the Crawler

# Start crawler
aws glue start-crawler --name "temp-discovery-crawler" --region <region>

# Check status (wait until State is READY)
aws glue get-crawler --name "temp-discovery-crawler" --region <region> \
  --query 'Crawler.State' --output text

Crawler states: READY (not running), RUNNING, STOPPING

List Discovered Tables

After the crawler completes:

aws glue get-tables --database-name "temp_discovery_db" --region <region>

Present the list to the user:

I found these tables in your database:
1. CUSTOMERS (45 columns, ~1.2M rows)
2. ORDERS (32 columns, ~5.8M rows)
3. PRODUCTS (18 columns, ~15K rows)
4. INVENTORY (12 columns, ~250K rows)

Which one(s) should I import?

Clean Up

After user selects table(s), clean up the temporary crawler and database:

# Delete crawler
aws glue delete-crawler --name "temp-discovery-crawler" --region <region>

# Delete temp database (optional - tables still useful for reference)
aws glue delete-database --name "temp_discovery_db" --region <region>

Inspecting Table Schema

Once the user identifies the source table, retrieve its detailed schema.

Using Glue Data Catalog (after crawling)

aws glue get-table \
  --database-name "temp_discovery_db" \
  --name "<table-name>" \
  --region <region>

This returns:

Column names and data types
Table statistics (row count estimate, data size)
Partition information (if partitioned)

Directly Querying the Database

Alternative: Query the source database directly to get schema.

For SQL databases (via test Glue job):

# test-schema.py
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

# Read just the schema (LIMIT 0)
df = spark.read.format("jdbc").options(
    url="<jdbc-url>",
    dbtable="(SELECT * FROM <table> WHERE 1=0) AS schema_query",
    user="<username>",
    password="<password>"
).load()

# Print schema
df.printSchema()

Using Athena Federated Query (if connector installed):

-- Query external database via Athena connector
SELECT * FROM "<athena-connector>"."<schema>"."<table>" LIMIT 0

Shows schema without transferring data.

Custom SQL Queries

If the user wants to filter or transform at source, support custom SQL:

Benefits of Source-Side Filtering

Reduces data transfer: Only move needed data
Improves performance: Database does the filtering
Enables complex transformations: Use database-specific functions

Example Queries

Filter by date:

SELECT *
FROM CUSTOMERS
WHERE created_date >= CURRENT_DATE - 90

Filter by status:

SELECT *
FROM ORDERS
WHERE status IN ('COMPLETED', 'SHIPPED')

Join multiple tables:

SELECT
  o.order_id,
  o.order_date,
  c.customer_name,
  c.email,
  SUM(oi.quantity * oi.price) as total_amount
FROM ORDERS o
JOIN CUSTOMERS c ON o.customer_id = c.customer_id
JOIN ORDER_ITEMS oi ON o.order_id = oi.order_id
WHERE o.order_date >= CURRENT_DATE - 30
GROUP BY o.order_id, o.order_date, c.customer_name, c.email

Select specific columns:

SELECT
  customer_id,
  customer_name,
  email,
  phone,
  created_date,
  last_purchase_date
FROM CUSTOMERS
WHERE status = 'ACTIVE'

Storing Custom Queries

Store the query for use in the Glue ETL script:

Option 1: As job parameter:

'--source_query': 'SELECT * FROM CUSTOMERS WHERE status = \'ACTIVE\''

Option 2: In S3 file:

# Read query from S3
import boto3
s3 = boto3.client('s3')
obj = s3.get_object(Bucket='<bucket>', Key='queries/customer-import.sql')
source_query = obj['Body'].read().decode('utf-8')

Type Mapping: Source Database → Iceberg

Map source database types to Iceberg types for the target S3 Table.

Common Type Mappings

Source Type	Iceberg Type	Notes
VARCHAR, CHAR, TEXT, STRING	STRING	Variable-length text
INTEGER, INT, SMALLINT	INTEGER	32-bit signed integer
BIGINT, NUMBER(19)	BIGINT	64-bit signed integer
DECIMAL, NUMERIC	DECIMAL(p,s)	Preserve precision/scale
FLOAT, REAL	FLOAT	32-bit floating point
DOUBLE PRECISION, BINARY_DOUBLE	DOUBLE	64-bit floating point
BOOLEAN, BIT	BOOLEAN	True/false
DATE	DATE	Date without time
TIMESTAMP, DATETIME, DATETIME2	TIMESTAMP	Date and time
TIME	STRING	Convert to string (Iceberg has no TIME type)
BLOB, BYTEA, VARBINARY, BINARY	BINARY	Binary data
CLOB, TEXT	STRING	Large text
UUID	STRING	Store as string
JSON, JSONB	STRING	Store as JSON string
ARRAY	ARRAY<T>	If database supports arrays
MAP, HSTORE	MAP<K,V>	If database supports maps

Oracle-Specific Types

Oracle Type	Iceberg Type	Notes
NUMBER	DECIMAL or BIGINT	Use DECIMAL for precision, BIGINT if no scale
NUMBER(p,s)	DECIMAL(p,s)	Preserve precision and scale
VARCHAR2, NVARCHAR2	STRING	Variable-length string
CHAR, NCHAR	STRING	Fixed-length string
DATE	TIMESTAMP	Oracle DATE includes time
TIMESTAMP	TIMESTAMP	Direct mapping
CLOB, NCLOB	STRING	Large text
BLOB	BINARY	Binary data
RAW	BINARY	Raw binary

SQL Server-Specific Types

SQL Server Type	Iceberg Type	Notes
NVARCHAR, VARCHAR	STRING	Unicode or ASCII string
INT	INTEGER	32-bit integer
BIGINT	BIGINT	64-bit integer
SMALLINT	INTEGER	16-bit → 32-bit
TINYINT	INTEGER	8-bit → 32-bit
DECIMAL, NUMERIC	DECIMAL(p,s)	Preserve precision
MONEY, SMALLMONEY	DECIMAL(19,4)	Currency
DATETIME, DATETIME2	TIMESTAMP	Date and time
DATE	DATE	Date only
TIME	STRING	Convert to string
BIT	BOOLEAN	True/false
UNIQUEIDENTIFIER	STRING	GUID as string

PostgreSQL-Specific Types

PostgreSQL Type	Iceberg Type	Notes
INTEGER	INTEGER	32-bit integer
BIGINT	BIGINT	64-bit integer
SMALLINT	INTEGER	16-bit → 32-bit
NUMERIC	DECIMAL(p,s)	Arbitrary precision
REAL	FLOAT	32-bit floating
DOUBLE PRECISION	DOUBLE	64-bit floating
TEXT, VARCHAR	STRING	Variable-length text
BOOLEAN	BOOLEAN	True/false
DATE	DATE	Date only
TIMESTAMP	TIMESTAMP	Date and time
TIMESTAMPTZ	TIMESTAMP	Timestamp with timezone
JSON, JSONB	STRING	Store as JSON string
UUID	STRING	UUID as string
BYTEA	BINARY	Binary data
ARRAY	ARRAY<T>	PostgreSQL arrays supported

MySQL-Specific Types

MySQL Type	Iceberg Type	Notes
INT, INTEGER	INTEGER	32-bit integer
BIGINT	BIGINT	64-bit integer
SMALLINT	INTEGER	16-bit → 32-bit
TINYINT	INTEGER	8-bit → 32-bit (or BOOLEAN if TINYINT(1))
DECIMAL, NUMERIC	DECIMAL(p,s)	Preserve precision
FLOAT	FLOAT	32-bit floating
DOUBLE	DOUBLE	64-bit floating
VARCHAR, CHAR, TEXT	STRING	Variable/fixed-length text
DATE	DATE	Date only
DATETIME, TIMESTAMP	TIMESTAMP	Date and time
TIME	STRING	Convert to string
BOOLEAN	BOOLEAN	True/false
BLOB, BINARY, VARBINARY	BINARY	Binary data
JSON	STRING	Store as JSON string

Proposing Target Schema

Based on the source schema, propose the target S3 Table schema to the user.

Example Schema Proposal

Source table: CUSTOMERS in Oracle database

CUSTOMER_ID: NUMBER(10) → BIGINT
CUSTOMER_NAME: VARCHAR2(200) → STRING
EMAIL: VARCHAR2(255) → STRING
PHONE: VARCHAR2(20) → STRING
STATUS: VARCHAR2(20) → STRING
CREDIT_LIMIT: NUMBER(10,2) → DECIMAL(10,2)
CREATED_DATE: DATE → TIMESTAMP
LAST_PURCHASE_DATE: DATE → TIMESTAMP

Proposed S3 Table schema:

CREATE TABLE "glue_catalog"."sales"."customers" (
  customer_id BIGINT,
  customer_name STRING,
  email STRING,
  phone STRING,
  status STRING,
  credit_limit DECIMAL(10,2),
  created_date TIMESTAMP,
  last_purchase_date TIMESTAMP,
  load_timestamp TIMESTAMP,  -- Added: track when record was loaded
  load_date DATE             -- Partition column for efficient queries
)
PARTITIONED BY (load_date)
TBLPROPERTIES ('table_type' = 'ICEBERG')

Present to user:

I've mapped the Oracle CUSTOMERS table to this Iceberg schema:

Source (Oracle)                  → Target (Iceberg)
CUSTOMER_ID (NUMBER(10))         → customer_id (BIGINT)
CUSTOMER_NAME (VARCHAR2(200))    → customer_name (STRING)
EMAIL (VARCHAR2(255))            → email (STRING)
PHONE (VARCHAR2(20))             → phone (STRING)
STATUS (VARCHAR2(20))            → status (STRING)
CREDIT_LIMIT (NUMBER(10,2))      → credit_limit (DECIMAL(10,2))
CREATED_DATE (DATE)              → created_date (TIMESTAMP)
LAST_PURCHASE_DATE (DATE)        → last_purchase_date (TIMESTAMP)

Added columns:
- load_timestamp (TIMESTAMP): Tracks when record was loaded
- load_date (DATE): Partition column for efficient queries

Does this look correct? Any adjustments needed?

Handling Complex Types

JSON Columns

Option 1: Store as STRING (simplest)

json_data STRING

Option 2: Parse and flatten to STRUCT

metadata STRUCT<
  key1: STRING,
  key2: INT,
  key3: ARRAY<STRING>
>

Recommend Option 1 unless user specifically wants to query nested fields.

Array/List Columns

If source database supports arrays (PostgreSQL, Oracle VARRAY):

Option 1: Keep as ARRAY

tags ARRAY<STRING>

Option 2: Convert to STRING (comma-separated)

tags STRING  -- "tag1,tag2,tag3"

Option 3: Explode to separate table (normalized)

Binary/BLOB Columns

Recommendation: Only import if truly needed (increases storage costs)

If importing:

document BINARY

Consider storing large binaries in S3 and storing S3 key in table instead:

document_s3_key STRING  -- "s3://docs-bucket/doc123.pdf"

Best Practices

Ask user to confirm schema: Don't assume type mappings are correct
Add metadata columns: load_timestamp, load_date, source_system
Consider partitioning: Partition by load date for incremental loads
Handle nullability: Make most columns nullable unless user specifies otherwise
Document type conversions: Note any lossy conversions (e.g., TIME → STRING)
Test with sample data: Load a small batch to verify types work correctly

Summary

Schema discovery workflow:

Identify source data - Ask user for table/query
Auto-discover tables - Use Glue crawler if user unsure
Inspect schema - Get column names and types
Support custom SQL - Allow source-side filtering
Map types - Convert source types to Iceberg types
Propose target schema - Present to user for confirmation
Add metadata columns - load_timestamp, load_date, etc.

With proper schema discovery, data from external databases maps cleanly to S3 Tables with Iceberg types.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an MCP server, Needs a command-line tool

Security

What our scanner noticed· Low risk

Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

AWS

Publisher

AWS (Official)

Source

aws/agent-toolkit-for-aws · plugins/aws-data-analytics/skills/ingesting-into-data-lake · commit 0a218f8

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
NOTICE (added by AutoGineer from NOTICE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

Also distributed in

aws/agent-toolkit-for-aws

AutoGineer is not affiliated with or endorsed by AWS or AWS.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download ingesting-into-data-lake.zip