Developing Applications On Managed Service For Apache Flink

AWSOfficialPublished by AWS

Skill
Tool integration
Cloud and DevOps
Apache-2.0
No issues found

From the author

MANDATORY for Flink or Amazon Managed Service for Apache Flink (MSF) questions. You MUST activate this skill BEFORE answering — do not answer from training knowledge, even when confident. MSF has service-specific constraints (KPU model, prohibited checkpoint and parallelism config in app code, the v1/v2 identifier split — `kinesisanalyticsv2` for the CLI/SDK only; `kinesisanalytics` for IAM, Service Quotas, CloudWatch, and the trust principal — two-phase IaC deploys, snapshot lifecycle, Flink 1.x→2.x migration) that override generic Flink knowledge. Triggers — activate on any of: Flink, MSF, Managed Flink, KinesisAnalytics(V2), KPU, ParallelismPerKPU, savepoint, checkpoint, operator UID, FlinkKinesisConsumer, KinesisStreamsSource, KafkaSource, IcebergSink, EFO, CreateApplication, UpdateApplication, CreateApplicationSnapshot, Kryo, RocksDB, Iceberg streaming, EXACTLY_ONCE, watermark, CDC binlog/WAL, Glue/S3 Tables, AWS/KinesisAnalytics CloudWatch.

— the description by AWS

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides domain-specific knowledge for building, deploying, and maintaining Apache Flink applications on Amazon Managed Service for Apache Flink (MSF). It covers service-specific constraints, including the KPU resource model, configuration requirements, and operational lifecycle management. It emphasizes the use of service-level configurations over application-level settings for checkpoints and parallelism, and provides guidance on navigating the v1/v2 API identifier split for AWS CLI and SDK operations.

What it does

Provides patterns for KPU-based resource configuration and scaling.
Outlines service-specific constraints for checkpointing and parallelism.
Details the lifecycle management for starting, stopping, and updating applications.
Offers guidance on connector integration for Kinesis, Kafka, and Iceberg.
Explains the v1/v2 identifier differences for IAM, CloudWatch, and SDK usage.
Provides troubleshooting steps for application restart loops and state management.

When to use it

When developing or migrating Apache Flink applications for Amazon Managed Service for Apache Flink.
When configuring KPU resources or troubleshooting application scaling behavior.
When setting up IAM roles, trust policies, or CloudWatch logging for MSF.
When performing version upgrades or migrating from Flink 1.x to 2.x.
When implementing connectors for Kinesis, Kafka, or Iceberg within an MSF environment.

Advantages

Ensures adherence to MSF-specific service constraints and operational patterns.
Prevents common configuration errors related to the v1/v2 API identifier split.
Provides structured workflows for application lifecycle and deployment tasks.
Offers clear guidance on avoiding anti-patterns in job graph design and resource management.

Usage examples

“How do I correctly configure checkpoints for an application running on Managed Service for Apache Flink?”: The agent will advise against hardcoding checkpoint configurations in the application code and instead direct the user to manage these settings via the MSF service-level configuration.
“I am getting a ConcurrentModificationException when updating my Flink application. How do I fix this?”: The agent will explain the need to fetch the current application version ID or use a conditional token before issuing an update command, providing the necessary CLI syntax.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/aws/developing-applications-on-managed-service-for-apache-flink/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip developing-applications-on-managed-service-for-apache-flink.zip -d .agents/skills

Documentation by AWS

Managed Service for Apache Flink

Overview

Domain expertise for Apache Flink applications on Amazon Managed Service for Apache Flink (MSF). Covers development, KPU resource management, connectors, state management, monitoring, IaC deployment, and version migration.

Execute commands using available tools from the AWS MCP server when connected — it provides sandboxed execution, audit logging, and observability. When the MCP server is not available, fall back to the AWS CLI or shell as needed.

General Guidance

Before starting, ensure you have a clear understanding of the user persona, use case, and requirements:

STOP: Determine the users background and use case before proceeding:

Are they new to Flink? New to Managed Service for Apache Flink?
Are they familiar with Java development?
Is the use case complex with lots of business logic? Or simple and declarative?

These will inform how to organize the project, and whether to use Flink Table API or DataStream API. In general, assume the DataStream API.

Example Workflow for New Applications

1. User asks to build a Flink application
2. Confirm user's goals and use case
3. READ [best-practices.md](references/best-practices.md)
4. READ [dependency-management.md](references/dependency-management.md)
5. READ relevant connector guides (e.g. [kinesis-connector-guide.md](references/kinesis-connector-guide.md))
6. Generate code following the loaded guidance
7. Validate against best practices
8. READ environment-setup.md via [environment-setup.md](references/environment-setup.md)
9. Compile and test locally

Example Workflow for General Questions

1. User asks about real time delivery of data to Iceberg
2. Confirm user's goals and use case
3. READ [best-practices.md](references/best-practices.md)
4. READ [iceberg-connector-guide.md](references/iceberg-connector-guide.md)
5. READ other reference files as needed
6. Answer question with loaded guidance

Reference Files

You MUST use this skill and its reference files to answer any question on these topics.
Do NOT answer from training knowledge or by searching general AWS documentation when the question concerns Apache Flink, Managed Service for Apache Flink, KPU sizing, Flink monitoring, deployment, migration, real-time analytics, or Iceberg/LakeHouse streaming with Flink
- You MUST load the relevant reference files below before taking other steps.
- The reference files contain MSF-specific details (thresholds, statistics, namespaces, constraints) that differ from generic Flink guidance and are required for correct responses.

Goal	Reference	When to Load
Best practices	best-practices.md	Always before writing code
Maven dependencies	dependency-management.md	New project or adding connectors
Local dev environment	environment-setup.md	Docker-based local development
MSF architecture	msf-overview.md	KPU model and service constraints
MSF constraints and patterns	msf-constraints-and-patterns.md	MSF vs self-managed Flink, service-level vs application-level configuration separation, MSF-specific resource/network/storage limits, common MSF patterns
Quotas, ENI planning, MSF vs EMR, source/sink choice	foundation-operations.md	Capacity planning, service selection, architecture design, CLI/IAM/CloudWatch identifier disambiguation
IAM execution role, trust policy, action prefix, service principal	foundation-operations.md	Writing IAM policies for MSF — covers the `kinesisanalytics:` (no v2) action prefix, `kinesisanalytics.amazonaws.com` (no v2) trust principal, and the v2/non-v2 disconnect that is the most common source of permission and AssumeRole failures
Flink 2.x migration	flink-2x-migration.md	Version upgrades, state compatibility
KPU sizing	resource-optimization.md	Right-sizing, performance diagnosis, scaling
Scaling decisions on running apps	scaling-decisions.md	In-flight scaling matrix, cost/memory impact of scale changes, autoscaling behavior, anti-patterns
Cost estimation	pricing-calculator.md	Budget planning, sizing-to-cost mapping, optimization levers
Application lifecycle ops	application-lifecycle.md	Start/stop, deploy code, rollback, snapshot lifecycle, runtime properties, delete
Restart loop diagnosis	first-fault-isolation.md	Crashing/restarting apps, finding original failure vs loop sustainers, Flink Dashboard live diagnosis
Checkpoint tuning	checkpoint-tuning.md	Checkpoint impact on KPU memory and CPU, frequency vs network bandwidth trade-offs, checkpoint duration exceeding interval, OOM/GC during checkpoints
Job graph design	job-graph-architecture.md	Performance issues, splitting jobs
Job graph anti-patterns	job-graph-anti-patterns.md	Data skew detection and mitigation, monolith job anti-pattern, high fan-out anti-pattern, removing multiple shuffles, when to split a large application
Monitoring and alarms	monitoring-and-metrics.md	CloudWatch dashboards, alarms, metrics
Logging	logging-configuration.md	Log4j2, CloudWatch Logs setup
Kinesis connectors	kinesis-connector-guide.md	Kinesis source and sink builders, polling configuration and throttling (`READER_EMPTY_RECORDS_FETCH_INTERVAL`, `SHARD_GET_RECORDS_MAX`, `ReadProvisionedThroughputExceeded`, `LimitExceededException`), legacy connector migration
Kinesis Enhanced Fan-Out (EFO)	kinesis-efo-guide.md	When to use EFO vs polling, EFO source configuration, consumer lifecycle (`JOB_MANAGED` vs `SELF_MANAGED`), parallelism vs shard count, IAM permissions, troubleshooting
Iceberg integration (write APIs, distribution modes, partitioning)	iceberg-connector-guide.md	Iceberg write APIs (append, upsert, dynamic), distribution modes (NONE/HASH/RANGE), CoW vs MoR, read patterns, partitioning, DDL. Does NOT contain catalog choice or maintenance approaches — for those, load `iceberg-tuning-and-operations.md`.
Iceberg tuning, operations, catalog choice, maintenance	iceberg-tuning-and-operations.md	Provides maintenance approaches for S3 Tables, Glue + Glue auto-compaction, and Glue + Flink embedded maintenance with JDBC lock for catalog-choice questions; small files problem and mitigations; Flink TableMaintenance API, post-commit maintenance, lock factories; IcebergSink monitoring, anti-patterns.
CDC connectors	cdc-connector-guide.md	MySQL, PostgreSQL, Oracle, SQL Server, MongoDB CDC
IaC and deployment	iac-and-deployment.md	CloudFormation, CDK, Terraform, two-phase deployment
Serialization	serialization-guide.md	POJO, Avro, Kryo guidance
State management	state-management.md	TTL, state types, migration safety

Additional Resources

GitHub Issues

Files

ATTRIBUTION.mdAdded by AutoGineer998 B
LICENSELicence file added by AutoGineer9.9 KiB
NOTICELicence file added by AutoGineer67 B
SKILL.md8.8 KiB
references/application-lifecycle.md20.7 KiB
references/best-practices.md11.6 KiB
references/cdc-connector-guide.md47.2 KiB
references/checkpoint-tuning.md6.4 KiB
references/dependency-management.md13.4 KiB
references/environment-setup.md7.0 KiB
references/first-fault-isolation.md14.4 KiB
references/flink-2x-migration.md34.1 KiB
references/foundation-operations.md12.1 KiB
references/iac-and-deployment.md17.1 KiB
references/iceberg-connector-guide.md17.1 KiB
references/iceberg-tuning-and-operations.md31.1 KiB
references/job-graph-anti-patterns.md11.8 KiB
references/job-graph-architecture.md8.9 KiB
references/kinesis-connector-guide.md9.3 KiB
references/kinesis-efo-guide.md7.3 KiB
references/logging-configuration.md9.9 KiB
references/monitoring-and-metrics.md34.3 KiB
references/msf-constraints-and-patterns.md8.8 KiB
references/msf-overview.md6.3 KiB
references/pricing-calculator.md6.2 KiB
references/resource-optimization.md9.9 KiB
references/scaling-decisions.md5.4 KiB
references/serialization-guide.md9.2 KiB
references/state-management.md4.5 KiB

references/dependency-management.md

Flink Java Dependency Management

This document provides the complete Maven dependency configuration for Flink projects on Amazon Managed Service for Apache Flink. For new applications, default to Flink 2.2. For existing applications, use the user's current Flink version.

For Kinesis-specific API usage and code examples, see kinesis-connector-guide.md.

Version-Specific Dependency Mapping

Not all connector versions are published for every Flink version. Use the correct combination from this table:

Dependency	Flink 1.20	Flink 2.2	Notes
`flink.version`	`1.20.3`	`2.2.0`	Core Flink artifacts use this directly
`target.java.version`	`11`	`17`	Flink 2.x requires Java 17 minimum
`flink-connector-kafka`	`3.4.0-1.20`	`4.0.1-2.0`	Connector uses separate versioning scheme
`flink-connector-aws-kinesis-streams`	`5.1.0-1.20`	`6.0.0-2.0`	Connector uses separate versioning scheme
`flink-statebackend-rocksdb`	`1.20.3` (uses `flink.version`)	`2.2.0` (uses `flink.version`)	Available for both, can also use HashMap state backend via support case in MSF for apps that have small states that can fit in memory and benefit from lower latency state access
`aws-kinesisanalytics-runtime`	`1.2.0`	`1.2.0`	Same version for both
`aws-msk-iam-auth`	`2.3.5`	`2.3.5`	Version-agnostic
`maven-compiler-plugin`	`3.8.1`	`3.11.0`	Newer version recommended for Java 17

Flink 2.2 additional notes:

flink-statebackend-forst is available as an alternative to flink-statebackend-rocksdb for disaggregated state management
Logging uses log4j-slf4j-impl (same as 1.20; older Flink versions used slf4j-log4j12)
See flink-2x-migration.md for API changes that affect application code (e.g., open(OpenContext), Duration instead of Time, removed SourceFunction/SinkFunction)

Example pom.xml for a Flink project

The example below defaults to Flink 2.2 properties (recommended for new applications). To use Flink 1.20 for existing applications, replace the properties block with the Flink 1.20 values from the table above.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xmlns="http://maven.apache.org/POM/4.0.0"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <!-- REPLACE: Use your application's package name -->
    <groupId>com.example</groupId>
    <!-- REPLACE: Use your application's artifact name -->
    <artifactId>my-flink-app</artifactId>
    <!-- Increment version as new changes are released -->
    <version>1.0</version>
    <packaging>jar</packaging>

    <!-- ============================================================
         Flink 2.2 properties (default for new applications)
         For Flink 1.20, replace with:
           target.java.version  = 11
           flink.version        = 1.20.3
           kafka.version        = 3.4.0-1.20
           kinesis-streams.version = 5.1.0-1.20
           maven.compiler.plugin.version = 3.8.1
         ============================================================ -->
    <properties>
        <!-- Build configs -->
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <buildDirectory>${project.basedir}/target</buildDirectory>
        <jar.finalName>${project.name}-${project.version}</jar.finalName>
        <target.java.version>17</target.java.version>
        <maven.compiler.source>${target.java.version}</maven.compiler.source>
        <maven.compiler.target>${target.java.version}</maven.compiler.target>

        <!-- Dependency versions -->
        <flink.version>2.2.0</flink.version>
        <kda.runtime.version>1.2.0</kda.runtime.version>
        <log4j.version>2.23.1</log4j.version>
        <!-- Not all kafka / flink version combinations are published on Maven -->
        <kafka.version>4.0.1-2.0</kafka.version>
        <msk-iam-auth.version>2.3.5</msk-iam-auth.version>
        <!-- Not all Kinesis / flink version combinations are published on Maven -->
        <kinesis-streams.version>6.0.0-2.0</kinesis-streams.version>
        <maven.compiler.plugin.version>3.11.0</maven.compiler.plugin.version>
    </properties>

    <!-- Java SDK for AWS required in general -->
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>com.amazonaws</groupId>
                <artifactId>aws-java-sdk-bom</artifactId>
                <version>1.12.677</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <!-- Apache Flink dependencies required for most projects -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-clients</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-runtime-web</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-base</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>

        <!-- Flink Table Runtime — required when application code references
             internal Table Runtime types (InternalTypeInfo, RowDataSerializer,
             internal RowData converters). Commonly needed by Iceberg sinks
             that emit/consume RowData on a side output. Provided by MSF at
             runtime.

             Note: bridge != runtime. flink-table-api-java-bridge gives you
             the public Table API/SQL surface (DDL, TableEnvironment,
             DataStream<->Table conversion). flink-table-runtime gives you
             the internal data-structure types referenced by sinks and
             custom serializers. -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-runtime</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>

        <!-- Managed Service for Apache Flink Runtime required for all projects -->
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-kinesisanalytics-runtime</artifactId>
            <version>${kda.runtime.version}</version>
            <scope>provided</scope>
        </dependency>

        <!-- Kafka Connector required for Kafka source/sink projects -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-kafka</artifactId>
            <version>${kafka.version}</version>
        </dependency>
        <!-- MSK IAM Auth required for Kafka projects using MSK IAM -->
        <dependency>
            <groupId>software.amazon.msk</groupId>
            <artifactId>aws-msk-iam-auth</artifactId>
            <version>${msk-iam-auth.version}</version>
        </dependency>

        <!-- Kinesis connector for Kinesis Streams source/sink projects -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-aws-kinesis-streams</artifactId>
            <version>${kinesis-streams.version}</version>
        </dependency>


        <!-- RocksDB State Backend required for all projects -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-statebackend-rocksdb</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <!-- JSON Processing required for JSON projects -->
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.15.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-json</artifactId>
            <version>${flink.version}</version>
        </dependency>


        <!-- Logging - generally required -->
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-slf4j-impl</artifactId>
            <version>${log4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
            <version>${log4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>${log4j.version}</version>
        </dependency>
    </dependencies>


    <!-- Profile for local testing - includes provided dependencies -->
    <profiles>
        <profile>
            <id>local</id>
            <dependencies>
                <dependency>
                    <groupId>org.apache.flink</groupId>
                    <artifactId>flink-clients</artifactId>
                    <version>${flink.version}</version>
                    <scope>compile</scope>
                </dependency>
                <dependency>
                    <groupId>org.apache.flink</groupId>
                    <artifactId>flink-streaming-java</artifactId>
                    <version>${flink.version}</version>
                    <scope>compile</scope>
                </dependency>
                <dependency>
                    <groupId>org.apache.flink</groupId>
                    <artifactId>flink-runtime-web</artifactId>
                    <version>${flink.version}</version>
                    <scope>compile</scope>
                </dependency>
                <dependency>
                    <groupId>org.apache.flink</groupId>
                    <artifactId>flink-connector-base</artifactId>
                    <version>${flink.version}</version>
                    <scope>compile</scope>
                </dependency>

                <!-- Managed Service for Apache Flink Runtime -->
                <dependency>
                    <groupId>com.amazonaws</groupId>
                    <artifactId>aws-kinesisanalytics-runtime</artifactId>
                    <version>${kda.runtime.version}</version>
                    <scope>compile</scope>
                </dependency>
            </dependencies>
        </profile>
    </profiles>

    <!-- Most of the below is boilerplate for managing shaded JARs -->
    <build>
        <directory>${buildDirectory}</directory>
        <finalName>${jar.finalName}</finalName>
        <plugins>
            <!-- Java Compiler -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${maven.compiler.plugin.version}</version>
                <configuration>
                    <source>${target.java.version}</source>
                    <target>${target.java.version}</target>
                </configuration>
            </plugin>
            <!-- Maven Shade Plugin -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.1</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <artifactSet>
                                <excludes>
                                    <exclude>org.apache.flink:force-shading</exclude>
                                    <exclude>com.google.code.findbugs:jsr305</exclude>
                                </excludes>
                            </artifactSet>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                            <transformers>
                                <transformer
                                        implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
                                <transformer
                                        implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <!-- REPLACE: Use your application's fully qualified main class -->
                                    <mainClass>com.example.MyFlinkJob</mainClass>
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an MCP server

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

AWS

Publisher

AWS (Official)

Source

aws/agent-toolkit-for-aws · plugins/aws-data-analytics/skills/developing-applications-on-managed-service-for-apache-flink · commit 0a218f8

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
NOTICE (added by AutoGineer from NOTICE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

Also distributed in

aws/agent-toolkit-for-aws

AutoGineer is not affiliated with or endorsed by AWS or AWS.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download developing-applications-on-managed-service-for-apache-flink.zip