flink

Estimated reading time: 7 minutes

Apache Flink® is a powerful open-source distributed stream and batch processing framework.

GitHub repo: https://github.com/docker-flink/docker-flink

Library reference

This content is imported from the official Docker Library docs, and is provided by the original uploader. You can view the Docker Store page for this image at https://store.docker.com/images/flink

Supported tags and respective Dockerfile links

Quick reference

What is Apache Flink?

Apache Flink is an open source stream processing framework with powerful stream- and batch-processing capabilities.

Learn more about Flink at https://flink.apache.org/

wikipedia.org/wiki/Apache_Flink

logo

How to use this Docker image

To run a single Flink local cluster:

$ docker run --name flink_local -p 8081:8081 -t flink local

Then with a web browser go to http://localhost:8081/ to see the Flink Web Dashboard (adjust the hostname for your Docker host).

To use Flink, you can submit a job to the cluster using the Web UI or you can also do it from a different Flink container, for example:

$ docker run --rm -t flink flink run -m <jobmanager:port> -c <your_class> <your_jar> <your_params>

Running a JobManager or a TaskManager

You can run a JobManager (master).

$ docker run --name flink_jobmanager -d -t flink jobmanager

You can also run a TaskManager (worker). Notice that workers need to register with the JobManager directly or via ZooKeeper so the master starts to send them tasks to execute.

$ docker run --name flink_taskmanager -d -t flink taskmanager

Running a cluster using Docker Compose

With Docker Compose you can create a Flink cluster:

version: "2.1"
services:
  jobmanager:
    image: ${FLINK_DOCKER_IMAGE_NAME:-flink}
    expose:
      - "6123"
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

  taskmanager:
    image: ${FLINK_DOCKER_IMAGE_NAME:-flink}
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager
    command: taskmanager
    links:
      - "jobmanager:jobmanager"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

and just run docker-compose up.

Scale the cluster up or down to N TaskManagers:

docker-compose scale taskmanager=<N>

Configuration

These are the default ports used by the Flink image:

  • The Web Client is on port 8081
  • JobManager RPC port 6123
  • TaskManagers RPC port 6122
  • TaskManagers Data port 6121

About this repository

This repository is available on github.com/docker-flink/docker-flink, and the official build is on the Docker Hub.

This repository is maintained by members of the Apache Flink community.

Image Variants

The flink images come in many flavors, each designed for a specific use case.

flink:<version>

This is the defacto image. If you are unsure about what your needs are, you probably want to use this one. It is designed to be used both as a throw away container (mount your source code and start the container to start your app), as well as the base to build other images off of.

flink:alpine

This image is based on the popular Alpine Linux project, available in the alpine official image. Alpine Linux is much smaller than most distribution base images (~5MB), and thus leads to much slimmer images in general.

This variant is highly recommended when final image size being as small as possible is desired. The main caveat to note is that it does use musl libc instead of glibc and friends, so certain software might run into issues depending on the depth of their libc requirements. However, most software doesn’t have an issue with this, so this variant is usually a very safe choice. See this Hacker News comment thread for more discussion of the issues that might arise and some pro/con comparisons of using Alpine-based images.

To minimize image size, it’s uncommon for additional related tools (such as git or bash) to be included in Alpine-based images. Using this image as a base, add the things you need in your own Dockerfile (see the alpine image description for examples of how to install packages if you are unfamiliar).

License

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

      https://www.apache.org/licenses/LICENSE-2.0

Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.

As with all Docker images, these likely also contain other software which may be under other licenses (such as Bash, etc from the base distribution, along with any direct or indirect dependencies of the primary software being contained).

Some additional license information which was able to be auto-detected might be found in the repo-info repository’s flink/ directory.

As for any pre-built image usage, it is the image user’s responsibility to ensure that any use of this image complies with any relevant licenses for all software contained within.

library, sample, flink