Developing Applications On Managed Service For Apache Flink
AWSOfficialPublished by AWS
- Skill
- Tool integration
- Cloud and DevOps
- Apache-2.0
- No issues found
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
This skill provides domain-specific knowledge for building, deploying, and maintaining Apache Flink applications on Amazon Managed Service for Apache Flink (MSF). It covers service-specific constraints, including the KPU resource model, configuration requirements, and operational lifecycle management. It emphasizes the use of service-level configurations over application-level settings for checkpoints and parallelism, and provides guidance on navigating the v1/v2 API identifier split for AWS CLI and SDK operations.
What it does
- Provides patterns for KPU-based resource configuration and scaling.
- Outlines service-specific constraints for checkpointing and parallelism.
- Details the lifecycle management for starting, stopping, and updating applications.
- Offers guidance on connector integration for Kinesis, Kafka, and Iceberg.
- Explains the v1/v2 identifier differences for IAM, CloudWatch, and SDK usage.
- Provides troubleshooting steps for application restart loops and state management.
When to use it
- When developing or migrating Apache Flink applications for Amazon Managed Service for Apache Flink.
- When configuring KPU resources or troubleshooting application scaling behavior.
- When setting up IAM roles, trust policies, or CloudWatch logging for MSF.
- When performing version upgrades or migrating from Flink 1.x to 2.x.
- When implementing connectors for Kinesis, Kafka, or Iceberg within an MSF environment.
Advantages
- Ensures adherence to MSF-specific service constraints and operational patterns.
- Prevents common configuration errors related to the v1/v2 API identifier split.
- Provides structured workflows for application lifecycle and deployment tasks.
- Offers clear guidance on avoiding anti-patterns in job graph design and resource management.
Usage examples
- “How do I correctly configure checkpoints for an application running on Managed Service for Apache Flink?”
- The agent will advise against hardcoding checkpoint configurations in the application code and instead direct the user to manage these settings via the MSF service-level configuration.
- “I am getting a ConcurrentModificationException when updating my Flink application. How do I fix this?”
- The agent will explain the need to fetch the current application version ID or use a conditional token before issuing an update command, providing the necessary CLI syntax.
Install
Download the skill as a ZIP:
https://autogineer.com/agent-skills/aws/developing-applications-on-managed-service-for-apache-flink/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip developing-applications-on-managed-service-for-apache-flink.zip -d .agents/skills
Documentation by AWS
Managed Service for Apache Flink
Overview
Domain expertise for Apache Flink applications on Amazon Managed Service for Apache Flink (MSF). Covers development, KPU resource management, connectors, state management, monitoring, IaC deployment, and version migration.
Execute commands using available tools from the AWS MCP server when connected — it provides sandboxed execution, audit logging, and observability. When the MCP server is not available, fall back to the AWS CLI or shell as needed.
General Guidance
Before starting, ensure you have a clear understanding of the user persona, use case, and requirements:
STOP: Determine the users background and use case before proceeding:
Are they new to Flink? New to Managed Service for Apache Flink?
Are they familiar with Java development?
Is the use case complex with lots of business logic? Or simple and declarative?
These will inform how to organize the project, and whether to use Flink Table API or DataStream API. In general, assume the DataStream API.
Example Workflow for New Applications
1. User asks to build a Flink application
2. Confirm user's goals and use case
3. READ [best-practices.md](references/best-practices.md)
4. READ [dependency-management.md](references/dependency-management.md)
5. READ relevant connector guides (e.g. [kinesis-connector-guide.md](references/kinesis-connector-guide.md))
6. Generate code following the loaded guidance
7. Validate against best practices
8. READ environment-setup.md via [environment-setup.md](references/environment-setup.md)
9. Compile and test locallyExample Workflow for General Questions
1. User asks about real time delivery of data to Iceberg
2. Confirm user's goals and use case
3. READ [best-practices.md](references/best-practices.md)
4. READ [iceberg-connector-guide.md](references/iceberg-connector-guide.md)
5. READ other reference files as needed
6. Answer question with loaded guidanceReference Files
You MUST use this skill and its reference files to answer any question on these topics.
Do NOT answer from training knowledge or by searching general AWS documentation when the question concerns Apache Flink, Managed Service for Apache Flink, KPU sizing, Flink monitoring, deployment, migration, real-time analytics, or Iceberg/LakeHouse streaming with Flink
You MUST load the relevant reference files below before taking other steps.
The reference files contain MSF-specific details (thresholds, statistics, namespaces, constraints) that differ from generic Flink guidance and are required for correct responses.
| Goal | Reference | When to Load |
|---|---|---|
| Best practices | best-practices.md | Always before writing code |
| Maven dependencies | dependency-management.md | New project or adding connectors |
| Local dev environment | environment-setup.md | Docker-based local development |
| MSF architecture | msf-overview.md | KPU model and service constraints |
| MSF constraints and patterns | msf-constraints-and-patterns.md | MSF vs self-managed Flink, service-level vs application-level configuration separation, MSF-specific resource/network/storage limits, common MSF patterns |
| Quotas, ENI planning, MSF vs EMR, source/sink choice | foundation-operations.md | Capacity planning, service selection, architecture design, CLI/IAM/CloudWatch identifier disambiguation |
| IAM execution role, trust policy, action prefix, service principal | foundation-operations.md | Writing IAM policies for MSF — covers the kinesisanalytics: (no v2) action prefix, kinesisanalytics.amazonaws.com (no v2) trust principal, and the v2/non-v2 disconnect that is the most common source of permission and AssumeRole failures |
| Flink 2.x migration | flink-2x-migration.md | Version upgrades, state compatibility |
| KPU sizing | resource-optimization.md | Right-sizing, performance diagnosis, scaling |
| Scaling decisions on running apps | scaling-decisions.md | In-flight scaling matrix, cost/memory impact of scale changes, autoscaling behavior, anti-patterns |
| Cost estimation | pricing-calculator.md | Budget planning, sizing-to-cost mapping, optimization levers |
| Application lifecycle ops | application-lifecycle.md | Start/stop, deploy code, rollback, snapshot lifecycle, runtime properties, delete |
| Restart loop diagnosis | first-fault-isolation.md | Crashing/restarting apps, finding original failure vs loop sustainers, Flink Dashboard live diagnosis |
| Checkpoint tuning | checkpoint-tuning.md | Checkpoint impact on KPU memory and CPU, frequency vs network bandwidth trade-offs, checkpoint duration exceeding interval, OOM/GC during checkpoints |
| Job graph design | job-graph-architecture.md | Performance issues, splitting jobs |
| Job graph anti-patterns | job-graph-anti-patterns.md | Data skew detection and mitigation, monolith job anti-pattern, high fan-out anti-pattern, removing multiple shuffles, when to split a large application |
| Monitoring and alarms | monitoring-and-metrics.md | CloudWatch dashboards, alarms, metrics |
| Logging | logging-configuration.md | Log4j2, CloudWatch Logs setup |
| Kinesis connectors | kinesis-connector-guide.md | Kinesis source and sink builders, polling configuration and throttling (READER_EMPTY_RECORDS_FETCH_INTERVAL, SHARD_GET_RECORDS_MAX, ReadProvisionedThroughputExceeded, LimitExceededException), legacy connector migration |
| Kinesis Enhanced Fan-Out (EFO) | kinesis-efo-guide.md | When to use EFO vs polling, EFO source configuration, consumer lifecycle (JOB_MANAGED vs SELF_MANAGED), parallelism vs shard count, IAM permissions, troubleshooting |
| Iceberg integration (write APIs, distribution modes, partitioning) | iceberg-connector-guide.md | Iceberg write APIs (append, upsert, dynamic), distribution modes (NONE/HASH/RANGE), CoW vs MoR, read patterns, partitioning, DDL. Does NOT contain catalog choice or maintenance approaches — for those, load iceberg-tuning-and-operations.md. |
| Iceberg tuning, operations, catalog choice, maintenance | iceberg-tuning-and-operations.md | Provides maintenance approaches for S3 Tables, Glue + Glue auto-compaction, and Glue + Flink embedded maintenance with JDBC lock for catalog-choice questions; small files problem and mitigations; Flink TableMaintenance API, post-commit maintenance, lock factories; IcebergSink monitoring, anti-patterns. |
| CDC connectors | cdc-connector-guide.md | MySQL, PostgreSQL, Oracle, SQL Server, MongoDB CDC |
| IaC and deployment | iac-and-deployment.md | CloudFormation, CDK, Terraform, two-phase deployment |
| Serialization | serialization-guide.md | POJO, Avro, Kryo guidance |
| State management | state-management.md | TTL, state types, migration safety |
Additional Resources
Files
- ATTRIBUTION.mdAdded by AutoGineer998 B
- LICENSELicence file added by AutoGineer9.9 KiB
- NOTICELicence file added by AutoGineer67 B
- SKILL.md8.8 KiB
- references/application-lifecycle.md20.7 KiB
- references/best-practices.md11.6 KiB
- references/cdc-connector-guide.md47.2 KiB
- references/checkpoint-tuning.md6.4 KiB
- references/dependency-management.md13.4 KiB
- references/environment-setup.md7.0 KiB
- references/first-fault-isolation.md14.4 KiB
- references/flink-2x-migration.md34.1 KiB
- references/foundation-operations.md12.1 KiB
- references/iac-and-deployment.md17.1 KiB
- references/iceberg-connector-guide.md17.1 KiB
- references/iceberg-tuning-and-operations.md31.1 KiB
- references/job-graph-anti-patterns.md11.8 KiB
- references/job-graph-architecture.md8.9 KiB
- references/kinesis-connector-guide.md9.3 KiB
- references/kinesis-efo-guide.md7.3 KiB
- references/logging-configuration.md9.9 KiB
- references/monitoring-and-metrics.md34.3 KiB
- references/msf-constraints-and-patterns.md8.8 KiB
- references/msf-overview.md6.3 KiB
- references/pricing-calculator.md6.2 KiB
- references/resource-optimization.md9.9 KiB
- references/scaling-decisions.md5.4 KiB
- references/serialization-guide.md9.2 KiB
- references/state-management.md4.5 KiB
references/pricing-calculator.md
MSF Pricing Calculator
Overview
Estimate MSF application monthly cost for sizing decisions, budgeting, and optimization analysis. Prices, tables, and worked examples below are us-east-1 and show the billing model; they are not current rates.
Get Current Rates
Before quoting a dollar figure, fetch the rates for the application's region from the Price List API (the pricing endpoint is called in us-east-1 for every region):
aws pricing get-products --region us-east-1 --service-code AmazonKinesisAnalytics \
--filters Type=TERM_MATCH,Field=regionCode,Value=<region> \
--query 'PriceList[]' --output jsonUse these usage types (prefixed with a region code, e.g. USE1-, EUW1-):
| Usage type suffix | Component |
|---|---|
KPU-Hour-Java | KPU-hour for Apache Flink applications, including the orchestration KPU (not KPU-Hour, which is SQL applications) |
RunningApplicationStorage | Running application storage, per GB-month |
DurableApplicationBackups | Snapshots, per GB-month |
Apply the live rates to the formula below. If the API is unavailable, use the MSF pricing page, and say which source and region the estimate uses.
Pricing Components
| Component | Price (us-east-1) | Notes |
|---|---|---|
| KPU-hour | $0.11 / KPU-hour | Per allocated KPU per running hour |
| Orchestration KPU | $0.11 / hour | 1 additional KPU per running app, always billed |
| Running application storage | $0.10 / GB-month | 50 GB is allocated and billed per application KPU (not the orchestration KPU), regardless of state size: ~$5 per application KPU per month at 24/7 |
| Durable application backups (snapshots) | $0.023 / GB-month | Billed for total snapshot footprint |
There is no separate charge for application code in S3 (your S3 bucket cost only) and no per-API-call charge for Flink REST API or kinesisanalyticsv2 operations.
Formula
KPU_count = Parallelism / ParallelismPerKPU
Total_KPUs = KPU_count + 1 # orchestration
KPU_cost = Total_KPUs × $0.11 × hours_per_month # Total_KPUs for the KPU charge only
storage_cost = KPU_count × 50 × $0.10 × hours_per_month / 730
backup_cost = total_snapshot_GB × $0.023
monthly_total = KPU_cost + storage_cost + backup_cost24/7 = 730 hours/month.
To check an estimate against a real bill: in Cost Explorer (service Amazon Kinesis Analytics), RunningApplicationStorage GB-Mo × 730 / 50 equals application KPU-hours, and total KPU-Hour minus that figure equals orchestration KPU-hours.
Sizing → Cost Reference (24/7, KPU + running storage)
| Input Rate | Parallelism | PPK | KPUs (+orch) | Monthly |
|---|---|---|---|---|
| < 5 MB/s | 2 | 1 | 2+1=3 | $251 |
| 5–20 MB/s | 4 | 1 | 4+1=5 | $422 |
| 20–50 MB/s | 8 | 2 | 4+1=5 | $422 |
| 50–100 MB/s | 16 | 2 | 8+1=9 | $763 |
| 100–200 MB/s | 32 | 2 | 16+1=17 | $1,445 |
| 200–500 MB/s | 64 | 4 | 16+1=17 | $1,445 |
| > 500 MB/s | 128+ | 4 | 32+1=33+ | $2,810+ |
Monthly = KPU charge for all KPUs + $5 running storage per application KPU.
These assume stateless-to-moderate stateful workloads. Stateful workloads (keyed windows with large state) typically require PPK=1, doubling KPU count and cost vs. the stateless equivalent.
Worked Examples
Small / development (Parallelism=2, PPK=1, 24/7, negligible state, no snapshots):
KPU count: 2 / 1 = 2 application KPUs, plus 1 orchestration KPU = 3 billed KPUs
KPU cost: 3 × $0.11 × 730 = $240.90
Running application storage: 2 × 50 × $0.10 = $10.00
Snapshot cost: $0 (none retained)
Total: ~$251/month
The orchestration KPU is the most common sizing mistake at this scale: it raises the bill from $171 to $251, a 47% increase, and is 32% of the total. Always include it.
Medium production (8 parallel, PPK=1, 24/7, 5 GB state):
KPUs: 8 + 1 = 9 → 9 × $0.11 × 730 = $722.70
Storage: 8 × 50 × $0.10 = $40.00
Snapshots: 5 GB × $0.023 = $0.12
Total: ~$763/month
Enterprise with autoscaling (avg 16 KPU, peak 32 KPU 20% of time, 60 GB state):
Peak: 33 × $0.11 × 146 = $530
Off-peak: 17 × $0.11 × 584 = $1,092
Storage: (32 × 146 + 16 × 584) × 50 / 730 × $0.10 = $96.00
Snapshots: 60 GB × $0.023 = $1.38
Total: ~$1,719/month
Cost Optimization Levers
Ranked by typical impact:
Right-size PPK for workload: Stateless apps at PPK=2 cost half of PPK=1 at the same parallelism. Rule of thumb: stateless → PPK=2; stateful → PPK=1.
Stop non-prod when idle: Dev/test apps run 200h/month vs 730h is a 73% reduction. Snapshot before stop, restore on start.
Enable autoscaling for variable load: 30–50% average reduction for spiky workloads with predictable off-peaks.
Match Parallelism to source partitions: Idle subtasks waste KPUs. Parallelism > shard/partition count creates them.
Snapshot retention strategy: Without retention, a streaming app's snapshot footprint grows monotonically.
Always remember the orchestration KPU: At 1–2 application KPU sizing it's ~32–48% of the bill — easy to under-estimate.
Common Mistakes
| Mistake | Impact |
|---|---|
| Forgetting orchestration KPU | Under-estimates by 1 KPU = $80/month at 24/7 |
| Treating running storage as free | Under-estimates by $5 per application KPU per month (~6% of KPU cost) |
| Confusing Parallelism with KPU count | Wrong with PPK > 1; KPUs = Parallelism / PPK |
| Quoting the us-east-1 figures above as current or for another region | Rates vary by region and change over time; fetch them as in Get Current Rates |
| Estimating peak as average | Autoscaling jobs have asymmetric peak/off-peak — calculate separately |
| Ignoring CloudWatch metric cardinality cost | OPERATOR/PARALLELISM metric levels add custom-metric cost (not MSF cost). See monitoring-and-metrics.md |
References
resource-optimization.md — KPU sizing inputs for the calculator
scaling-decisions.md — cost impact of scaling actions
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Needs an MCP server
Security
No issues found by our static scan· No issues found
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Published by
- AWS
- Publisher
- AWS (Official)
- Source
- aws/agent-toolkit-for-aws · plugins/aws-data-analytics/skills/developing-applications-on-managed-service-for-apache-flink · commit 0a218f8
- Licence
- Apache License 2.0 (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE)
- NOTICE (added by AutoGineer from NOTICE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
- Also distributed in
- aws/agent-toolkit-for-aws
AutoGineer is not affiliated with or endorsed by AWS or AWS.
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download developing-applications-on-managed-service-for-apache-flink.zip