Connecting To Data Source

AWSOfficialPublished by AWS

Skill
Tool integration
Cloud and DevOps
Apache-2.0
No issues found

From the author

Create and troubleshoot AWS Glue connections to JDBC databases (Oracle, SQL Server, PostgreSQL, MySQL, RDS), Redshift, Snowflake, and BigQuery. Gathers connection hints from user, discovers existing connections and RDS/Redshift candidates, registers credentials in Secrets Manager or IAM DB auth, configures VPC, and tests. Triggers on: connect to database, set up Glue connection, register data source, connect to Snowflake/BigQuery/RDS, connection timeout, test connection, troubleshoot connection. Do NOT use for moving data (use ingesting-into-data-lake), creating tables (use creating-data-lake-table), queries (use querying-data-lake), catalog exploration (use exploring-data-catalog), or SaaS (Salesforce, ServiceNow, SAP, MongoDB, Kafka).

— the description by AWS

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured workflow for registering external data sources with AWS Glue. It handles the creation of Glue connections, which act as reusable network and credential configurations for ETL jobs. The process includes discovering existing resources, setting up secure credential storage via AWS Secrets Manager or IAM database authentication, configuring VPC network requirements, and performing end-to-end connectivity testing to ensure the connection is ready for production use.

What it does

Discovers existing RDS, Redshift, and Glue connection configurations.
Registers database credentials securely using AWS Secrets Manager or IAM DB authentication.
Configures VPC, subnet, and security group requirements for private data sources.
Creates Glue connection objects for JDBC databases, Redshift, Snowflake, and BigQuery.
Performs multi-phase connectivity testing including API checks and engine-level smoke tests.
Provides diagnostic steps for common network, credential, and driver-related connection failures.

When to use it

When you need to establish a new connection between AWS Glue and an external database or warehouse.
When you need to troubleshoot an existing Glue connection that is failing during ETL jobs.
When you are setting up a new data source for use with Glue ETL or Athena federated queries.
When you need to migrate from plaintext credentials to secure AWS Secrets Manager or IAM-based auth.

Advantages

Ensures secure credential handling by prioritizing Secrets Manager and IAM over plaintext.
Validates network and authentication settings before they are used in production jobs.
Reduces configuration errors by providing source-specific setup patterns.
Provides a clear, repeatable process for managing data source connectivity across environments.

Usage examples

“Set up a new Glue connection for my PostgreSQL RDS instance.”: The agent will guide you through identifying the RDS instance, configuring the VPC settings, setting up IAM DB authentication or a secret, and running a test connection.
“I need to connect Glue to Snowflake.”: The agent will prompt for your Snowflake account details, warehouse, and role, then create a native SNOWFLAKE-type Glue connection and verify it.
“Troubleshoot my Glue connection 'oracle-prod-sales' which is timing out.”: The agent will check your VPC routing, security group rules, and S3 VPC endpoints to diagnose the connectivity issue.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/aws/connecting-to-data-source/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip connecting-to-data-source.zip -d .agents/skills

Documentation by AWS

Connect to Data Source

Register an external data source with AWS Glue so downstream skills (ingesting-into-data-lake) can move data from it. A Glue connection stores the network config, driver, and credential reference for one source. Create once per source, reuse across jobs.

Philosophy

A connection is a named pipe, not a pipeline. This skill produces a tested, reusable Glue connection. It does not move data.

Common Tasks

You MUST execute commands using AWS MCP server tools when connected -- they provide validation, sandboxed execution, and audit logging. Fall back to AWS CLI only if MCP is unavailable. You MUST explain each step before executing.

Workflow

1. Verify Dependencies and Context

You MUST check whether AWS MCP tools or AWS CLI are available and inform the user if missing
You MUST confirm target AWS region and verify credentials with aws sts get-caller-identity

2. Classify the Source

Ask the user which source type they want to connect to, or infer from hints:

User says...	Source type	Connection type	Reference
"Oracle", "SQL Server", "Postgres", "MySQL", "RDS <engine>"	JDBC database	`JDBC`	jdbc-setup.md
"Redshift", "my cluster", "my data warehouse on AWS"	Redshift	`JDBC`	jdbc-setup.md (Redshift section)
"Snowflake"	Snowflake	`SNOWFLAKE`	snowflake-setup.md
"BigQuery", "Google analytics warehouse"	BigQuery	`BIGQUERY`	bigquery-setup.md

If the user names DynamoDB or a local file, stop and tell them: DynamoDB is read directly by Glue without a connection, and local files belong in the ingesting-into-data-lake skill's local-upload workflow.

3. Gather Connection Hints from the User

You MUST ask for hints the user can provide -- do not guess.

For all sources:

Desired connection name (lowercase, hyphens: oracle-prod-sales, snowflake-analytics)
Existing Secrets Manager secret, or create one
Is source reachable from a Glue VPC (same, peered, VPN, Direct Connect)

JDBC: hostname/endpoint, port, database, whether RDS/Aurora/self-managed, IAM DB auth enabled (Aurora/RDS MySQL/Postgres), SSL required.

Snowflake: account identifier, warehouse, role, default database, auth (password, key-pair, OAuth).

BigQuery: GCP project ID, location, whether service account JSON is provisioned.

4. Discover Existing Connections and Candidate Sources

Check what exists before creating.

Existing Glue connections:

aws glue get-connections --filter ConnectionType=<TYPE> --region <REGION>

If a suitable one exists, confirm and skip to Step 7.

Candidate sources in account (JDBC/Redshift only):

RDS: aws rds describe-db-instances
Aurora: aws rds describe-db-clusters
Redshift: aws redshift describe-clusters

Present candidates to user; let them pick. See discovery.md.

5. Register Credentials

You MUST encourage AWS Secrets Manager over plaintext passwords. You SHOULD prefer IAM database authentication where supported (Aurora/RDS MySQL and PostgreSQL, Redshift). See credential-security.md.

You MUST confirm with user before creating a new Secrets Manager secret
You MUST NOT write plaintext credentials into chat or logs
For IAM DB auth, no secret is needed

6. Create the Glue Connection

Follow the source-specific reference for connection properties:

aws glue create-connection --connection-input '<JSON>' --region <REGION>

Private sources require PhysicalConnectionRequirements (SubnetId, SecurityGroupIdList, AvailabilityZone). See network-setup.md.

7. Test the Connection

You MUST test before handing off. Testing is two-phase: a quick API check, then an engine-level verification.

Phase A: Glue TestConnection (network and credential sanity check)

aws glue test-connection --connection-name <NAME> --region <REGION>

This validates that Glue can reach the source and authenticate. It does NOT prove the connection works end-to-end with the query engine the user plans to use.

Phase B: Engine-level verification

After TestConnection passes, verify the connection works with the user's intended engine by running a minimal query through it:

Glue ETL (default): Run a smoke-test Glue job that reads one row via the connection. See troubleshooting.md.
Athena: If the user plans to query via Athena with a federated connector, run a SELECT 1 through the Athena connection to confirm the Lambda-based connector can reach the source.
Glue Crawler: If the user plans to crawl the source, run a test crawl on a single table.

Phase B catches issues that TestConnection misses: driver compatibility at job runtime, catalog configuration, Spark-level serialization, and engine-specific auth flows (e.g., Snowflake SNOWFLAKE type works in ETL but not via JDBC crawlers).

On success in both phases, tell user the connection name is ready for ingesting-into-data-lake. On failure in either phase, Step 8.

8. Troubleshoot (only if test failed)

Diagnose in order: network, credentials, driver. See troubleshooting.md.

Constraints:

You MUST check VPC routing, security groups, and S3 VPC endpoint before blaming credentials
You MUST verify Glue role can read the Secrets Manager secret
You MUST NOT rotate credentials without user confirmation

Argument Routing

No args: Walk through Steps 1-7 interactively
Source type keyword (e.g., snowflake, oracle): Skip to Step 2 with the type prefilled
Existing connection name: Skip to Step 7 (test) then Step 8 if failing
Hostname or RDS endpoint: Skip to Step 4 with the candidate prefilled

Gotchas

Glue's SNOWFLAKE connection type is distinct from JDBC configured for Snowflake. You MUST use SNOWFLAKE for Spark ETL jobs; do not use JDBC.
Connection names are immutable. Choose carefully.
PhysicalConnectionRequirements.AvailabilityZone MUST match the subnet's AZ or the connection fails at job runtime, not creation time.
IAM database authentication tokens expire in 15 minutes. The Glue job generates a fresh token on each connection; do not cache.
An S3 VPC gateway endpoint MUST exist in the VPC used by private-source connections. Without it, Glue jobs cannot read their scripts or write results to S3.

Troubleshooting

Error	Likely cause	Fix
`Connect timed out`	VPC routing, SG rule, or NAT gateway missing	See troubleshooting.md
`Access denied for user` / `ORA-01017`	Credentials wrong, Secrets Manager access missing, or IAM DB auth misconfigured	See troubleshooting.md
`No suitable driver found`	Custom driver JAR not set or wrong class name	See troubleshooting.md
`SSL handshake failed`	`JDBC_ENFORCE_SSL` mismatch between Glue and source	See troubleshooting.md
`UnableToFindVpcEndpoint`	S3 VPC endpoint missing	Create S3 gateway endpoint in the connection's VPC

References

jdbc-setup.md -- Oracle, SQL Server, PostgreSQL, MySQL, RDS, Redshift
snowflake-setup.md -- Glue SNOWFLAKE type, auth modes
bigquery-setup.md -- Glue BIGQUERY type, GCP service accounts
discovery.md -- Finding existing connections and candidate sources
credential-security.md -- Secrets Manager and IAM DB auth
network-setup.md -- VPC, subnets, security groups, endpoints
troubleshooting.md -- Connection errors and diagnostic flow

Files

ATTRIBUTION.mdAdded by AutoGineer930 B
LICENSELicence file added by AutoGineer9.9 KiB
NOTICELicence file added by AutoGineer67 B
SKILL.md8.7 KiB
references/bigquery-setup.md2.6 KiB
references/credential-security.md4.3 KiB
references/discovery.md3.6 KiB
references/jdbc-setup.md3.5 KiB
references/network-setup.md3.9 KiB
references/snowflake-setup.md2.6 KiB
references/troubleshooting.md7.6 KiB

references/troubleshooting.md

Connection Troubleshooting

Diagnose Glue connection failures. Run checks in order: network → credentials → driver → SSL. Most failures are network.

Test Decision Tree
Network
Credentials
Driver
SSL
Smoke-Test Glue Job Template

Test Decision Tree

Run aws glue test-connection --connection-name <NAME>. If it fails, read the error message.
If error mentions timeout, unreachable, UnableToFindVpcEndpoint, or ENI -- go to Network.
If error mentions authentication, Access denied, invalid username/password, ORA-01017, 28000 -- go to Credentials.
If error mentions No suitable driver, ClassNotFoundException -- go to Driver.
If error mentions SSL handshake, certificate, TLS -- go to SSL.
If TestConnection passes but the engine-level smoke test fails, the issue is engine-specific (driver version, catalog config, Spark serialization). Run the smoke-test Glue job for a more informative error. See Smoke-Test Glue Job Template.

Network

Most connection failures are network. Check in order:

1. Subnet and routing

aws glue get-connection --name <NAME> \
  --query 'Connection.PhysicalConnectionRequirements'

Note the SubnetId. Check its route table:

aws ec2 describe-route-tables \
  --filters Name=association.subnet-id,Values=<SUBNET_ID>

Verify: route to source's VPC CIDR exists.

2. Security groups

Verify Glue SG allows outbound to source port AND has self-referencing rule:

aws ec2 describe-security-groups --group-ids <GLUE_SG>

Verify source SG allows inbound from Glue SG:

aws ec2 describe-security-groups --group-ids <SOURCE_SG>

3. S3 VPC endpoint

aws ec2 describe-vpc-endpoints \
  --filters Name=vpc-id,Values=<VPC_ID> Name=service-name,Values=com.amazonaws.<region>.s3

If missing and subnet has no NAT gateway, create the endpoint. See network-setup.md.

4. Test from EC2 in the same subnet

Launch or use an existing EC2 in the Glue subnet with the Glue SG attached:

telnet <source-host> <source-port>
nc -zv <source-host> <source-port>

If EC2 can't reach the source, fix routing/SG/NACL before blaming Glue.

5. Database firewall

Source-side ACLs beyond AWS SGs:

Oracle: listener.ora restricts connecting hosts
SQL Server: Windows Firewall on the host
PostgreSQL: pg_hba.conf
MySQL: user host restrictions (SELECT user, host FROM mysql.user)
Self-managed in a VPC: NACLs on the subnet

Credentials

Run through this checklist:

1. Secrets Manager access

# Impersonate the Glue role and fetch the secret
aws sts assume-role --role-arn <GLUE_ROLE_ARN> --role-session-name test \
  | jq -r '.Credentials'
# then with those creds:
aws secretsmanager get-secret-value --secret-id <SECRET_ID>

If AccessDenied: Glue role lacks secretsmanager:GetSecretValue on the secret ARN. See credential-security.md.

2. Secret contents match expected keys

JDBC: username, password
Snowflake: snowflakeUser, snowflakePassword
BigQuery: bare base64 string (no JSON keys)

3. IAM DB auth (if enabled)

Verify the Glue role has rds-db:connect on arn:aws:rds-db:<region>:<account>:dbuser:<resource-id>/<db-user>.

Verify the DB user exists with IDENTIFIED WITH AWSAuthenticationPlugin (MySQL) or GRANT rds_iam TO <user> (PostgreSQL).

4. Direct credential test

From EC2 in the Glue subnet:

# Oracle
sqlplus <user>/<password>@//host:1521/service
# PostgreSQL
PGPASSWORD=<password> psql -h host -U user -d db -c "SELECT 1"
# MySQL
mysql -h host -u user -p<password> -e "SELECT 1"

5. Password edge cases

Special characters (@, #, %, :) in the password can break JDBC URL parsing. Store in Secrets Manager (avoids URL encoding entirely).
Expired password: Oracle SELECT account_status FROM dba_users; MySQL / Postgres check user's password expiry.
Locked account: Oracle ALTER USER <user> ACCOUNT UNLOCK.

Driver

For built-in drivers (Oracle, SQL Server, PostgreSQL, MySQL, Redshift), no action needed.

For custom drivers:

1. JAR accessible

Verify the Glue role can read the JAR:

aws s3 head-object --bucket <SCRIPTS_BUCKET> --key jdbc-drivers/<driver>.jar

2. Driver class name matches

Engine	Correct class
Oracle	`oracle.jdbc.OracleDriver`
SQL Server	`com.microsoft.sqlserver.jdbc.SQLServerDriver`
PostgreSQL	`org.postgresql.Driver`
MySQL 8.x	`com.mysql.cj.jdbc.Driver`
MySQL 5.x	`com.mysql.jdbc.Driver` (deprecated but sometimes needed)
Redshift	`com.amazon.redshift.jdbc.Driver`

3. Driver version compatibility

Driver major version must match or exceed the database major version. Downgrading works for minor versions, not major.

SSL

1. Enforcement mismatch

Source requires SSL but connection doesn't enable it:

"JDBC_ENFORCE_SSL": "true"

2. Self-signed certificates

Source uses a cert not in the default Java truststore:

Import the cert into a custom truststore
Upload truststore to S3
Add to Glue job args: --extra-jars s3://... and JVM args pointing at the truststore

For AWS RDS and Aurora, the default truststore includes the RDS CA bundle.

3. TLS version

Older databases may require TLS 1.0/1.1; Glue 5.1 or higher defaults to 1.2+. Update database or use connection property to downgrade (not recommended).

Smoke-Test Glue Job Template

When test-connection passes but the engine-level verification fails (or when test-connection fails with an unhelpful message), a minimal Glue job produces a clearer error.

Save to s3://<scripts>/test-connection.py:

import sys
from awsglue.utils import getResolvedOptions
from awsglue.context import GlueContext
from pyspark.context import SparkContext

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'connection_name', 'source_type'])
sc = SparkContext()
glueContext = GlueContext(sc)

test_queries = {
    'oracle': '(SELECT 1 FROM DUAL) AS t',
    'sqlserver': '(SELECT 1) AS t',
    'postgresql': '(SELECT 1) AS t',
    'mysql': '(SELECT 1) AS t',
    'redshift': '(SELECT 1) AS t',
}

source_type = args['source_type']
if source_type not in test_queries:
    raise ValueError(
        f"Unsupported source_type '{source_type}'. "
        "This JDBC smoke test supports: oracle, sqlserver, postgresql, mysql, redshift. "
        "For Snowflake/BigQuery, use their native connection_type."
    )

try:
    df = glueContext.create_dynamic_frame.from_options(
        connection_type='jdbc',
        connection_options={
            'useConnectionProperties': 'true',
            'connectionName': args['connection_name'],
            'dbtable': test_queries[args['source_type']]
        }
    ).toDF()
    print(f"SUCCESS: {df.collect()}")
except Exception as e:
    print(f"FAIL: {type(e).__name__}: {e}")
    raise

Create and run the job:

aws glue create-job \
  --name test-connection-smoke \
  --role <GLUE_ROLE_ARN> \
  --command Name=glueetl,ScriptLocation=s3://<scripts>/test-connection.py,PythonVersion=3 \
  --connections Connections=<CONNECTION_NAME> \
  --glue-version 5.1 \
  --number-of-workers 2 \
  --worker-type G.1X

aws glue start-job-run \
  --job-name test-connection-smoke \
  --arguments '{"--connection_name":"<CONNECTION_NAME>","--source_type":"<TYPE>"}'

Read CloudWatch logs for the specific JDBC error. Most common errors are more descriptive in logs than in get-connection-test output.

Delete the test job after use.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an MCP server, Needs a command-line tool

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

AWS

Publisher

AWS (Official)

Source

aws/agent-toolkit-for-aws · plugins/aws-data-analytics/skills/connecting-to-data-source · commit 0a218f8

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
NOTICE (added by AutoGineer from NOTICE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

Also distributed in

aws/agent-toolkit-for-aws

AutoGineer is not affiliated with or endorsed by AWS or AWS.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download connecting-to-data-source.zip