Support Flink 1.17.1 by gfmio · Pull Request #332 · apache/flink-statefun

gfmio · 2023-08-31T10:23:58Z

This PR updates the project to use and be compatible with Flink 1.17.1.

We've been running this in production for a few months now without any issues.

…n into support-flink-1.17.1

MartijnVisser · 2023-09-05T07:53:12Z

@gfmio Thanks a lot for the PRs, but can you please follow the code contribution process as explained on https://flink.apache.org/how-to-contribute/contribute-code/ ? Specifically talking about filing a Jira, following the right commit structure and naming conventions etc.

gfmio · 2023-09-05T07:54:07Z

Apologies, sure, will do.

dannyporrello · 2023-11-20T17:03:22Z

Hey, any update on this? @MartijnVisser @gfmio

MartijnVisser

@gfmio I've looked over the PR, but it looks like there are changes in here which aren't necessary for the upgrade to Flink 1.17.1 and should be a separate ticket or at the very least, a separate commit. Could you rebase and make the necessary changes?

MartijnVisser · 2023-11-27T10:46:02Z

pom.xml

        <scala.version>2.12.7</scala.version>
        <lz4-java.version>1.8.0</lz4-java.version>
-        <flink-shaded-jackson.version>2.12.4-15.0</flink-shaded-jackson.version>
+        <flink-shaded-jackson.version>2.14.2-17.0</flink-shaded-jackson.version>


This is the wrong version for Flink 1.17.1

MartijnVisser · 2023-11-27T10:46:31Z

pom.xml

+        <flink-connector-kinesis.version>4.1.0-1.17</flink-connector-kinesis.version>
+        <flink-connector-aws-kinesis-streams.version>4.1.0-1.17</flink-connector-aws-kinesis-streams.version>
+        <okhttp.version>3.14.6</okhttp.version>
+        <flink-shaded-netty.version>4.1.82.Final-16.1</flink-shaded-netty.version>
+        <junit.version>4.12</junit.version>
+        <hamcrest-all.version>1.3</hamcrest-all.version>
+        <kryo.version>2.24.0</kryo.version>
+        <jackson-databind.version>2.13.2.2</jackson-databind.version>
+        <flink-shaded-netty.version>4.1.82.Final-16.1</flink-shaded-netty.version>
+        <flink-shaded-force-shading.version>16.1</flink-shaded-force-shading.version>
+        <commons-codec.version>1.15</commons-codec.version>
+        <commons-logging.version>1.2</commons-logging.version>
+        <slf4j-api.version>1.7.36</slf4j-api.version>


It doesn't appear as we re-use these variables anywhere else, should we just leave them as they were?

MartijnVisser · 2023-11-27T10:47:06Z

pom.xml

+            <dependency>
+                <groupId>org.apache.flink</groupId>
+                <artifactId>flink-shaded-netty</artifactId>
+                <version>${flink-shaded-netty.version}</version>
+            </dependency>
+
+            <dependency>
+                <groupId>org.apache.flink</groupId>
+                <artifactId>flink-core</artifactId>
+                <version>${flink.version}</version>
+            </dependency>
+
+            <dependency>
+                <groupId>org.apache.flink</groupId>
+                <artifactId>flink-table-common</artifactId>
+                <version>${flink.version}</version>
+            </dependency>
+
+            <dependency>
+                <groupId>org.apache.flink</groupId>
+                <artifactId>flink-connector-base</artifactId>
+                <version>${flink.version}</version>
+            </dependency>
+
+            <dependency>
+                <groupId>org.apache.flink</groupId>
+                <artifactId>flink-shaded-force-shading</artifactId>
+                <version>${flink-shaded-force-shading.version}</version>
+            </dependency>
+
+            <dependency>
+                <groupId>org.apache.flink</groupId>
+                <artifactId>flink-shaded-jackson</artifactId>
+                <version>${flink-shaded-jackson.version}</version>
+            </dependency>
+
+            <dependency>
+                <groupId>commons-codec</groupId>
+                <artifactId>commons-codec</artifactId>
+                <version>${commons-codec.version}</version>
+            </dependency>
+
+            <dependency>
+                <groupId>commons-logging</groupId>
+                <artifactId>commons-logging</artifactId>
+                <version>${commons-logging.version}</version>
+            </dependency>
+
+            <dependency>
+                <groupId>org.slf4j</groupId>
+                <artifactId>slf4j-api</artifactId>
+                <version>${slf4j-api.version}</version>


Why are we adding these? That shouldn't be necessary in an upgrade to 1.17.1

MartijnVisser · 2023-11-27T10:47:22Z

pom.xml

                                    <outputTarget>
                                        <type>descriptor</type>
                                        <outputDirectory>${basedir}/target/test-classes</outputDirectory>
+                                        <addSources>main</addSources>


Is this needed for the 1.171. upgrade?

MartijnVisser · 2023-11-27T10:47:47Z

...n-flink-core/src/main/java/org/apache/flink/statefun/flink/core/StatefulFunctionsConfig.java

  public StatefulFunctionsUniverseProvider getProvider(ClassLoader cl) {
    try {
-      return InstantiationUtil.deserializeObject(universeInitializerClassBytes, cl, false);
+      return InstantiationUtil.deserializeObject(universeInitializerClassBytes, cl);


Is this needed for the Flink upgrade?

Edit: yes it is :)

MartijnVisser · 2023-11-27T10:48:16Z

statefun-flink/statefun-flink-io-bundle/pom.xml

+            <version>${flink-connector-kinesis.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.flink</groupId>
+            <artifactId>flink-connector-aws-kinesis-streams</artifactId>
+            <version>${flink-connector-aws-kinesis-streams.version}</version>


Is this necessary for the Flink upgrade itself?

MartijnVisser · 2023-11-27T10:48:40Z

tools/docker/Dockerfile

 # limitations under the License.

-FROM apache/flink:1.15.2-scala_2.12-java8
+FROM flink:1.17.1-scala_2.12-java11


Statefun still requires Java 8, so I don't think we should bump it as part of this PR

MartijnVisser · 2023-11-27T14:07:52Z

@gfmio For context, I've made MartijnVisser#3 which would probably be sufficient

EnneS · 2024-10-04T13:01:01Z

hey, any news on this? I used your branch @MartijnVisser and the changes are working as expected, even with flink 1.19.1 (I haven't tested any later versions).

gfmio added 5 commits August 31, 2023 10:22

support flink 1.17.1

1a6ae6e

add vscode and devcontainer config

44c638f

add taskfile

a09af38

ignore bin

450842b

support flink 1.17.1

6478acd

gfmio force-pushed the support-flink-1.17.1 branch from 1a6ae6e to 6478acd Compare August 31, 2023 10:57

gfmio marked this pull request as ready for review August 31, 2023 11:10

gfmio changed the title ~~use flink 1.17.1~~ Support Flink 1.17.1 Aug 31, 2023

gfmio added 2 commits August 31, 2023 12:20

Merge branch 'support-flink-1.17.1' of github.com:gfmio/flink-statefu…

800219c

…n into support-flink-1.17.1

remove accidentally configured files

2d69991

MartijnVisser requested changes Nov 27, 2023

View reviewed changes

Conversation

gfmio commented Aug 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MartijnVisser commented Sep 5, 2023

Uh oh!

gfmio commented Sep 5, 2023

Uh oh!

dannyporrello commented Nov 20, 2023

Uh oh!

MartijnVisser left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MartijnVisser commented Nov 27, 2023

Uh oh!

EnneS commented Oct 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gfmio commented Aug 31, 2023 •

edited

Loading