Developing Applications On Managed Service For Apache Flink
AWSOfficialPublished by AWS
- Skill
- Tool integration
- Cloud and DevOps
- Apache-2.0
- No issues found
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
This skill provides domain-specific knowledge for building, deploying, and maintaining Apache Flink applications on Amazon Managed Service for Apache Flink (MSF). It covers service-specific constraints, including the KPU resource model, configuration requirements, and operational lifecycle management. It emphasizes the use of service-level configurations over application-level settings for checkpoints and parallelism, and provides guidance on navigating the v1/v2 API identifier split for AWS CLI and SDK operations.
What it does
- Provides patterns for KPU-based resource configuration and scaling.
- Outlines service-specific constraints for checkpointing and parallelism.
- Details the lifecycle management for starting, stopping, and updating applications.
- Offers guidance on connector integration for Kinesis, Kafka, and Iceberg.
- Explains the v1/v2 identifier differences for IAM, CloudWatch, and SDK usage.
- Provides troubleshooting steps for application restart loops and state management.
When to use it
- When developing or migrating Apache Flink applications for Amazon Managed Service for Apache Flink.
- When configuring KPU resources or troubleshooting application scaling behavior.
- When setting up IAM roles, trust policies, or CloudWatch logging for MSF.
- When performing version upgrades or migrating from Flink 1.x to 2.x.
- When implementing connectors for Kinesis, Kafka, or Iceberg within an MSF environment.
Advantages
- Ensures adherence to MSF-specific service constraints and operational patterns.
- Prevents common configuration errors related to the v1/v2 API identifier split.
- Provides structured workflows for application lifecycle and deployment tasks.
- Offers clear guidance on avoiding anti-patterns in job graph design and resource management.
Usage examples
- “How do I correctly configure checkpoints for an application running on Managed Service for Apache Flink?”
- The agent will advise against hardcoding checkpoint configurations in the application code and instead direct the user to manage these settings via the MSF service-level configuration.
- “I am getting a ConcurrentModificationException when updating my Flink application. How do I fix this?”
- The agent will explain the need to fetch the current application version ID or use a conditional token before issuing an update command, providing the necessary CLI syntax.
Install
Download the skill as a ZIP:
https://autogineer.com/agent-skills/aws/developing-applications-on-managed-service-for-apache-flink/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip developing-applications-on-managed-service-for-apache-flink.zip -d .agents/skills
Documentation by AWS
Managed Service for Apache Flink
Overview
Domain expertise for Apache Flink applications on Amazon Managed Service for Apache Flink (MSF). Covers development, KPU resource management, connectors, state management, monitoring, IaC deployment, and version migration.
Execute commands using available tools from the AWS MCP server when connected — it provides sandboxed execution, audit logging, and observability. When the MCP server is not available, fall back to the AWS CLI or shell as needed.
General Guidance
Before starting, ensure you have a clear understanding of the user persona, use case, and requirements:
STOP: Determine the users background and use case before proceeding:
Are they new to Flink? New to Managed Service for Apache Flink?
Are they familiar with Java development?
Is the use case complex with lots of business logic? Or simple and declarative?
These will inform how to organize the project, and whether to use Flink Table API or DataStream API. In general, assume the DataStream API.
Example Workflow for New Applications
1. User asks to build a Flink application
2. Confirm user's goals and use case
3. READ [best-practices.md](references/best-practices.md)
4. READ [dependency-management.md](references/dependency-management.md)
5. READ relevant connector guides (e.g. [kinesis-connector-guide.md](references/kinesis-connector-guide.md))
6. Generate code following the loaded guidance
7. Validate against best practices
8. READ environment-setup.md via [environment-setup.md](references/environment-setup.md)
9. Compile and test locallyExample Workflow for General Questions
1. User asks about real time delivery of data to Iceberg
2. Confirm user's goals and use case
3. READ [best-practices.md](references/best-practices.md)
4. READ [iceberg-connector-guide.md](references/iceberg-connector-guide.md)
5. READ other reference files as needed
6. Answer question with loaded guidanceReference Files
You MUST use this skill and its reference files to answer any question on these topics.
Do NOT answer from training knowledge or by searching general AWS documentation when the question concerns Apache Flink, Managed Service for Apache Flink, KPU sizing, Flink monitoring, deployment, migration, real-time analytics, or Iceberg/LakeHouse streaming with Flink
You MUST load the relevant reference files below before taking other steps.
The reference files contain MSF-specific details (thresholds, statistics, namespaces, constraints) that differ from generic Flink guidance and are required for correct responses.
| Goal | Reference | When to Load |
|---|---|---|
| Best practices | best-practices.md | Always before writing code |
| Maven dependencies | dependency-management.md | New project or adding connectors |
| Local dev environment | environment-setup.md | Docker-based local development |
| MSF architecture | msf-overview.md | KPU model and service constraints |
| MSF constraints and patterns | msf-constraints-and-patterns.md | MSF vs self-managed Flink, service-level vs application-level configuration separation, MSF-specific resource/network/storage limits, common MSF patterns |
| Quotas, ENI planning, MSF vs EMR, source/sink choice | foundation-operations.md | Capacity planning, service selection, architecture design, CLI/IAM/CloudWatch identifier disambiguation |
| IAM execution role, trust policy, action prefix, service principal | foundation-operations.md | Writing IAM policies for MSF — covers the kinesisanalytics: (no v2) action prefix, kinesisanalytics.amazonaws.com (no v2) trust principal, and the v2/non-v2 disconnect that is the most common source of permission and AssumeRole failures |
| Flink 2.x migration | flink-2x-migration.md | Version upgrades, state compatibility |
| KPU sizing | resource-optimization.md | Right-sizing, performance diagnosis, scaling |
| Scaling decisions on running apps | scaling-decisions.md | In-flight scaling matrix, cost/memory impact of scale changes, autoscaling behavior, anti-patterns |
| Cost estimation | pricing-calculator.md | Budget planning, sizing-to-cost mapping, optimization levers |
| Application lifecycle ops | application-lifecycle.md | Start/stop, deploy code, rollback, snapshot lifecycle, runtime properties, delete |
| Restart loop diagnosis | first-fault-isolation.md | Crashing/restarting apps, finding original failure vs loop sustainers, Flink Dashboard live diagnosis |
| Checkpoint tuning | checkpoint-tuning.md | Checkpoint impact on KPU memory and CPU, frequency vs network bandwidth trade-offs, checkpoint duration exceeding interval, OOM/GC during checkpoints |
| Job graph design | job-graph-architecture.md | Performance issues, splitting jobs |
| Job graph anti-patterns | job-graph-anti-patterns.md | Data skew detection and mitigation, monolith job anti-pattern, high fan-out anti-pattern, removing multiple shuffles, when to split a large application |
| Monitoring and alarms | monitoring-and-metrics.md | CloudWatch dashboards, alarms, metrics |
| Logging | logging-configuration.md | Log4j2, CloudWatch Logs setup |
| Kinesis connectors | kinesis-connector-guide.md | Kinesis source and sink builders, polling configuration and throttling (READER_EMPTY_RECORDS_FETCH_INTERVAL, SHARD_GET_RECORDS_MAX, ReadProvisionedThroughputExceeded, LimitExceededException), legacy connector migration |
| Kinesis Enhanced Fan-Out (EFO) | kinesis-efo-guide.md | When to use EFO vs polling, EFO source configuration, consumer lifecycle (JOB_MANAGED vs SELF_MANAGED), parallelism vs shard count, IAM permissions, troubleshooting |
| Iceberg integration (write APIs, distribution modes, partitioning) | iceberg-connector-guide.md | Iceberg write APIs (append, upsert, dynamic), distribution modes (NONE/HASH/RANGE), CoW vs MoR, read patterns, partitioning, DDL. Does NOT contain catalog choice or maintenance approaches — for those, load iceberg-tuning-and-operations.md. |
| Iceberg tuning, operations, catalog choice, maintenance | iceberg-tuning-and-operations.md | Provides maintenance approaches for S3 Tables, Glue + Glue auto-compaction, and Glue + Flink embedded maintenance with JDBC lock for catalog-choice questions; small files problem and mitigations; Flink TableMaintenance API, post-commit maintenance, lock factories; IcebergSink monitoring, anti-patterns. |
| CDC connectors | cdc-connector-guide.md | MySQL, PostgreSQL, Oracle, SQL Server, MongoDB CDC |
| IaC and deployment | iac-and-deployment.md | CloudFormation, CDK, Terraform, two-phase deployment |
| Serialization | serialization-guide.md | POJO, Avro, Kryo guidance |
| State management | state-management.md | TTL, state types, migration safety |
Additional Resources
Files
- ATTRIBUTION.mdAdded by AutoGineer998 B
- LICENSELicence file added by AutoGineer9.9 KiB
- NOTICELicence file added by AutoGineer67 B
- SKILL.md8.8 KiB
- references/application-lifecycle.md20.7 KiB
- references/best-practices.md11.6 KiB
- references/cdc-connector-guide.md47.2 KiB
- references/checkpoint-tuning.md6.4 KiB
- references/dependency-management.md13.4 KiB
- references/environment-setup.md7.0 KiB
- references/first-fault-isolation.md14.4 KiB
- references/flink-2x-migration.md34.1 KiB
- references/foundation-operations.md12.1 KiB
- references/iac-and-deployment.md17.1 KiB
- references/iceberg-connector-guide.md17.1 KiB
- references/iceberg-tuning-and-operations.md31.1 KiB
- references/job-graph-anti-patterns.md11.8 KiB
- references/job-graph-architecture.md8.9 KiB
- references/kinesis-connector-guide.md9.3 KiB
- references/kinesis-efo-guide.md7.3 KiB
- references/logging-configuration.md9.9 KiB
- references/monitoring-and-metrics.md34.3 KiB
- references/msf-constraints-and-patterns.md8.8 KiB
- references/msf-overview.md6.3 KiB
- references/pricing-calculator.md6.2 KiB
- references/resource-optimization.md9.9 KiB
- references/scaling-decisions.md5.4 KiB
- references/serialization-guide.md9.2 KiB
- references/state-management.md4.5 KiB
references/foundation-operations.md
Foundation Operations: Quotas, Service Selection, and Architecture
Overview
Cross-cutting operational knowledge: MSF service quotas, ENI capacity planning for VPC apps, MSF vs EMR Flink decision, and source/sink selection. Use when planning capacity, choosing the right streaming service, or designing a new pipeline.
CLI and CloudWatch Identifiers
| Item | Value | Common wrong values |
|---|---|---|
| AWS CLI service / SDK client | kinesisanalyticsv2 | ❌ flink, msf, kinesisanalytics (v1, deprecated SQL apps only) |
| Service Quotas service-code | kinesisanalytics (no v2) | ❌ kinesisanalyticsv2, flink, msf |
| IAM action prefix | kinesisanalytics: (no v2) | ❌ kinesisanalyticsv2: |
| CloudWatch namespace | AWS/KinesisAnalytics | ❌ AWS/Flink, AWS/ManagedFlink |
| Trust policy principal | kinesisanalytics.amazonaws.com | ❌ kinesisanalyticsv2.amazonaws.com |
The CLI/SDK is the only identifier that uses the v2 suffix. Service Quotas, IAM actions, the CloudWatch namespace, and the trust principal all use the legacy kinesisanalytics name. Treating the v2 form as the "default" and applying it everywhere is the single most common source of permission failures, empty metric results, missing service-quota lookups, and trust policy errors.
What Goes In the Execution Role (and What Does NOT)
The MSF execution role is assumed by the MSF service to access your data plane resources on behalf of the application. It is not used by the application code itself, and it does not call MSF's own control plane. The principle of least privilege follows from that:
| Permission | Required in execution role? | Why |
|---|---|---|
Source/sink data plane (e.g. kinesis:GetRecords, s3:PutObject, kafka:DescribeCluster) | Yes — scoped to specific stream/bucket/cluster ARNs | The service uses this role to read sources and write sinks |
logs:CreateLogStream, logs:PutLogEvents, logs:DescribeLogStreams on the configured log group | Yes | The service writes application logs to the configured CloudWatch Logs group |
EC2 ENI permissions (ec2:CreateNetworkInterface, ec2:DescribeNetworkInterfaces, ec2:DeleteNetworkInterface, ec2:CreateNetworkInterfacePermission, ec2:DescribeVpcs, ec2:DescribeSubnets, ec2:DescribeSecurityGroups) | Yes — only for VPC-enabled apps | The service creates ENIs on your behalf when VPC is configured |
kinesisanalytics:* actions | No | These are MSF control-plane actions consumed by humans/CI calling the MSF API, not by the service when it runs your application |
cloudwatch:PutMetricData | No | MSF publishes the standard metrics (cpuUtilization, downtime, numRecordsIn*, etc.) to the AWS/KinesisAnalytics namespace from the service plane, not via the execution role. Adding it does no harm if scoped to the namespace, but it's noise — leave it out for a clean least-privilege role. The exception: if your application code explicitly calls CloudWatchAsyncClient.putMetricData() to emit custom application metrics, then you do need to grant it (still scope to the custom namespace) |
secretsmanager:GetSecretValue | Only if you use Secrets Manager for connector credentials | Scope to the specific secret ARN |
kms:Decrypt | Only when reading from a KMS-encrypted source/sink or KMS-encrypted secret | Scope to the specific key ARN |
When generating an execution role, default to omitting kinesisanalytics:* and cloudwatch:PutMetricData unless the user explicitly says they emit custom metrics. The CloudWatch metrics you see in the AWS console come from the service, not from this role.
Service Quotas
| Quota | Default | Adjustable |
|---|---|---|
| Applications per region | 50 | Yes |
| KPUs per application | 64 (configurable to 250) | Yes |
| Application snapshots per application | 1000 | Yes |
| Parallelism per application | 256 | Yes |
Check current values:
aws service-quotas list-aws-default-service-quotas \
--service-code kinesisanalytics --region "$REGION"The Service Quotas service-code is kinesisanalytics (legacy name), not kinesisanalyticsv2. The KPUs-per-application quota code is L-3A88E041 (Apache Flink Kinesis Processing Units (KPUs)).
Request increases via request-service-quota-increase with the quota code from the list output. Increases for KPU and snapshot quotas typically approve within a business day.
ENI Capacity Planning (VPC Apps)
MSF creates one ENI per allocated KPU, in each subnet, for VPC-enabled applications. Subnet sizing and the regional ENI quota both matter.
ENIs_per_subnet = KPUs
required_subnet_IPs = ENIs_per_subnet + 20% headroom for scaling and rolling restartsExample: 16 KPU app 16 ENIs per subnet. Allow 20 available IPs per subnet to account for 20% buffer. A /28 subnet (11 usable IPs) is too small; use /27 (27 usable) or larger.
Regional ENI quota (vpc service code, Network interfaces per region) defaults to 5,000. Each VPC-enabled MSF KPU consumes one. Multiple large MSF apps in the same region can pressure this quota — check before deploying. Other services in the VPC (Lambda, ECS, RDS) also consume ENIs from the same quota.
Subnets should span at least 2 AZs for fault tolerance.
When VPC Is Required
| Source/Sink | VPC Required |
|---|---|
| Kinesis Data Streams | No (public endpoint) |
| Amazon S3 | No (public endpoint, S3 gateway endpoint optional) |
| DynamoDB | No (public endpoint) |
| Firehose | No |
| Amazon MSK (Kafka) | Yes |
| RDS / Aurora | Yes |
| ElastiCache | Yes |
| OpenSearch in VPC | Yes |
| Self-hosted Kafka | Yes if private |
| Public Kafka / external API | No, but needs NAT gateway in VPC subnets |
VPC apps without a NAT gateway lose access to public AWS endpoints (CloudWatch, S3 if not using gateway endpoint, Kinesis). Symptoms: silent failure of metric publishing, S3 checkpoint failures, deserialization errors trying to call schema registry.
MSF vs EMR Flink
| Factor | MSF | EMR Flink |
|---|---|---|
| Operations | Fully managed, no clusters | Self-managed EC2/EKS clusters |
| Scaling | KPU autoscaling (CPU-only) | Manual cluster scaling |
| Billing | Per KPU-hour | Per EC2-hour (+ EMR surcharge) |
| Flink version | AWS-managed (1.15, 1.18, 1.19, 1.20, 2.2) | Any Flink version, including custom builds |
| Custom connectors | Limited to bundled JARs / fat-JAR upload | Full Flink ecosystem |
| Job isolation | One job per application | Multiple jobs per cluster |
| Startup time | 1–3 min | 5–15 min cluster boot |
| Max parallelism | 256 (quota-adjustable) | Unlimited (cluster size) |
| State backend | Managed RocksDB | Self-managed RocksDB / heap |
Choose MSF when: zero infrastructure management, single-job-per-app is acceptable, parallelism fits within KPU limits, fast iteration matters.
Choose EMR Flink when: custom Flink connectors not in MSF, multiple Flink jobs sharing infrastructure, specific Flink version or patch control, parallelism exceeds 256, fine-grained CPU/memory ratios needed.
Cost crossover: EMR is typically cheaper at large scale (10+ KPU equivalent, 24/7) due to EC2 commitment savings, but the operational overhead (cluster patching, scaling, monitoring) typically erases the savings unless EMR expertise already exists.
Source Selection: KDS vs MSK vs S3
| Source | Best For | Throughput | VPC | Ordering | Retention |
|---|---|---|---|---|---|
| Kinesis Data Streams (KDS) | AWS-native ingestion, < 1 GB/s, Lambda integration | Per-shard (1 MB/s in, 2 MB/s out) | No | Per-shard | 1–365 days |
| Amazon MSK | Kafka ecosystem, complex routing, > 1 GB/s | Per-broker (hundreds MB/s) | Yes | Per-partition | Unlimited (storage-based) |
| Amazon S3 | Batch-to-stream replay, reprocessing | Bulk file scan | No | File ordering | Indefinite |
KDS pitfalls: shard count drives parallelism; shard count is hard to change post-creation. Cost scales linearly with shard count, not throughput.
MSK pitfalls: broker provisioning takes hours; cross-AZ replication doubles network cost; SASL/IAM auth requires careful security group setup.
S3 source pitfalls: no event-time ordering across files unless designed in; file enumeration is the bottleneck for large prefix counts.
For source-side EFO and Kinesis polling tradeoffs, see kinesis-efo-guide.md. For MSK setup, see the iac-and-deployment.md deployment patterns.
Sink Selection
| Sink | Best For | VPC | Ordering Preserved |
|---|---|---|---|
| S3 (Parquet/ORC) | Data lake, batch analytics, Athena | No | File-level only |
| Iceberg | Transactional data lake, schema evolution, time travel | No (catalog-dependent) | Yes (commit-order) |
| Kinesis Data Streams | Real-time downstream consumers | No | Per-shard |
| MSK | Kafka ecosystem | Yes | Per-partition |
| OpenSearch | Search, log analytics, dashboards | Yes (VPC mode) or No (public) | No (eventually consistent) |
| DynamoDB | Low-latency key-value lookups | No | No (last-write-wins per key) |
| RDS / Aurora | Relational writes, joins on results | Yes | Yes (transaction order) |
| Firehose | Managed delivery to S3/Redshift/OpenSearch with batching | No | Within batch |
S3 small files anti-pattern: Streaming writers commit on every checkpoint, creating one file per checkpoint per partition. With 60s checkpoints and 8 partitions, a job writes 11,520 files/day. Use Iceberg with compaction, or batch via Firehose, or increase checkpoint interval. See iceberg-tuning-and-operations.md.
RDS/Aurora sink anti-pattern: Per-record JDBC writes are 10–100× slower than batched writes. Use the JDBC sink with batching enabled. RDS connection pool limits (typically 5,000) cap effective parallelism.
Architecture Patterns
| Pattern | Use Case | VPC |
|---|---|---|
| KDS → MSF → S3/Iceberg | Data lake ingestion | No |
| MSK → MSF → MSK | Stream-to-stream enrichment | Yes |
| KDS → MSF → DynamoDB | Real-time aggregation serving | No |
| KDS → MSF → OpenSearch | Real-time search index | Optional |
| CDC source → MSF → Iceberg | Database replication to lake | Yes (DB side) |
| MSK → MSF → S3 + KDS | Fan-out to lake and downstream consumers | Yes (MSK side) |
Co-location and Cross-AZ Cost
For private connections (MSK, RDS, OpenSearch in VPC), MSF KPUs must be in the same VPC or peered VPC as the source/sink. Cross-AZ data transfer between MSF and MSK is always billed at $0.01/GB each direction — at 100 MB/s sustained that is $26K/year. Place MSF subnets in the same AZs as MSK brokers and use rack-aware producers/consumers where supported.
Common Mistakes
| Mistake | Impact |
|---|---|
Using kinesisanalytics for the CLI / SDK control plane | Returns v1 SQL apps only, not Flink — use kinesisanalyticsv2 for the CLI/SDK |
kinesisanalyticsv2 for the Service Quotas service-code | NoSuchResourceException — Service Quotas uses the legacy kinesisanalytics code |
kinesisanalyticsv2: IAM actions | All API calls denied — IAM uses kinesisanalytics: prefix |
AWS/Flink namespace in CloudWatch | Empty metric results |
kinesisanalyticsv2.amazonaws.com trust principal | MSF cannot AssumeRole — trust principal is kinesisanalytics.amazonaws.com |
| Hardcoding region lists for cross-region discovery | Disabled regions cause AccessDeniedException — use describe-regions |
| Sizing subnets without ENI headroom | Application stuck STARTING; ENIs cannot be created |
| Choosing EMR for a single-job pipeline | Operational overhead exceeds infrastructure savings |
| Mismatched MSF and source AZs | Persistent cross-AZ data transfer cost |
References
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Needs an MCP server
Security
No issues found by our static scan· No issues found
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Published by
- AWS
- Publisher
- AWS (Official)
- Source
- aws/agent-toolkit-for-aws · plugins/aws-data-analytics/skills/developing-applications-on-managed-service-for-apache-flink · commit 0a218f8
- Licence
- Apache License 2.0 (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE)
- NOTICE (added by AutoGineer from NOTICE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
- Also distributed in
- aws/agent-toolkit-for-aws
AutoGineer is not affiliated with or endorsed by AWS or AWS.
Related skills
Modal
K-Dense· Community
A serverless cloud platform for deploying and scaling Python applications, AI models, and GPU-accelerated workloads.
Cloud and DevOpsClaude CodeClaude appsOpenAI Codex+14 moreApache-2.0Low riskGitHub stars of the source repository56.4k
Cloud Infrastructure Network Engineer
Seth Hobson· Community
An expert network engineer persona specializing in cloud architecture, security, and performance optimization.
Cloud and DevOpsClaude CodeMITGitHub stars of the source repository42.3k
Deployment Validation Cloud Architect
Seth Hobson· Community
Expert cloud architect for multi-cloud infrastructure, cost optimization, and resilient system design.
Cloud and DevOpsClaude CodeMITGitHub stars of the source repository42.3k
Airflow Dag Patterns
Seth Hobson· Community
Best practices and design patterns for building and maintaining production-ready Apache Airflow DAGs.
Cloud and DevOpsClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository42.3k
Cost Optimization
Seth Hobson· Community
A comprehensive framework for reducing cloud infrastructure costs across AWS, Azure, GCP, and OCI.
Cloud and DevOpsClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository42.3k
Hybrid Cloud Networking
Seth Hobson· Community
A guide to configuring secure, high-performance network connectivity between on-premises infrastructure and major cloud providers.
Cloud and DevOpsClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository42.3k
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download developing-applications-on-managed-service-for-apache-flink.zip