Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add code examples using flink-scala-api #142

Merged
merged 7 commits into from
Aug 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,8 @@ target
.metals
.vscode
metals.sbt
.scala-build
.scala-build
.ammonite/*
*.jar
*checkpoint
sink-table
11 changes: 11 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,17 @@ If you want to create new project easily check this __Giter8 template__ out: [no

We suggest to remove the official `flink-scala` and `flink-streaming-scala` dependencies altogether to simplify the migration and do not to mix two flavors of API in the same project. But it's technically possible and not required.

## Examples

There is a wide range of [code examples](https://github.com/flink-extended/flink-scala-api/tree/master/modules/examples) to introduce you to flink-scala-api, both using Scala scripts and multimodule applications. These examples include:

- Flink jobs built using Scala 3 with Ammonite and Scala CLI;
- A complete application for fraud detection;
- Examples using Datastream and Table APIs;
- Simple job developed interactively via Jupyter notebooks;
- Word count reading text from a web socket;
- Example usage of DataGen connector and Kafka sink;
- And more;

## Differences with the Official Flink Scala API

Expand Down
39 changes: 39 additions & 0 deletions build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@ lazy val rootScalaVersion = "3.3.3"
lazy val flinkVersion = System.getProperty("flinkVersion", "1.18.1")

lazy val root = (project in file("."))
.aggregate(`scala-api`, `examples`)
.settings(
publish / skip := true
)

lazy val `scala-api` = (project in file("modules/scala-api"))
.settings(ReleaseProcess.releaseSettings(flinkVersion) *)
.settings(
name := "flink-scala-api",
Expand Down Expand Up @@ -108,3 +114,36 @@ lazy val root = (project in file("."))
mdocIn := new File("README.md")
)
.enablePlugins(MdocPlugin)

val flinkMajorAndMinorVersion =
flinkVersion.split("\\.").toList.take(2).mkString(".")

lazy val `examples` = (project in file("modules/examples"))
.settings(
scalaVersion := rootScalaVersion,
Test / fork := true,
libraryDependencies ++= Seq(
"org.flinkextended" %% "flink-scala-api" % "1.18.1_1.1.6",
"org.apache.flink" % "flink-runtime-web" % "1.18.1" % Provided,
"org.apache.flink" % "flink-clients" % "1.18.1" % Provided,
"org.apache.flink" % "flink-state-processor-api" % "1.18.1" % Provided,
"org.apache.flink" % "flink-connector-kafka" % "3.0.2-1.18" % Provided,
"org.apache.flink" % "flink-connector-files" % "1.18.1" % Provided,
"org.apache.flink" % "flink-table-runtime" % "1.18.1" % Provided,
"org.apache.flink" % "flink-table-planner-loader" % "1.18.1" % Provided,
"io.bullet" %% "borer-core" % "1.14.0" % Provided,
"ch.qos.logback" % "logback-classic" % "1.4.14" % Provided,
"org.apache.flink" % "flink-test-utils" % "1.18.1" % Test,
"org.apache.flink" % "flink-streaming-java" % "1.18.1" % Test classifier "tests",
"org.scalatest" %% "scalatest" % "3.2.15" % Test
),
Compile / run := Defaults
.runTask(
Compile / fullClasspath,
Compile / run / mainClass,
Compile / run / runner
)
.evaluated,
Compile / run / fork := true
)
.enablePlugins(ProtobufPlugin)
Loading
Loading