DataFlow Pipeline

Overview

Real-Time Streaming Data Platform

DataFlow Pipeline is a real-time data processing platform built on Apache Flink + Kafka. Users configure data processing tasks through a Vue frontend, which are orchestrated into DAG graphs and executed by the Flink engine, with Kafka as the message middleware for efficient data flow.

The platform supports multiple data source integrations (MySQL, Redis, Kafka Topics) with built-in operators for data cleaning, aggregation, and window computation. The Java backend provides RESTful APIs for the frontend, with Redis caching for real-time task status queries. As a core big data course project, it successfully processed million-level daily simulated data streams.

Tech Stack

Architecture

Full-stack big data solution with stream computing + visual monitoring

Java

Python

Apache Flink

Apache Kafka

MySQL

Redis

Docker

Vue

Key Features

Core Capabilities

🔄

Stream Processing Engine

High-performance stream processing engine based on Flink, supporting window computation, state management, and Exactly-Once semantic guarantees

📊

Visual Monitoring

Vue frontend displaying real-time task status, throughput, and latency metrics with task start/stop and configuration management

🛡️

Fault Tolerance and Recovery

Automatic task failure recovery based on Flink Checkpoint mechanism, with Kafka persistent messaging ensuring zero data loss

📈

Multidimensional Analytics

Built-in aggregation, filtering, Join, window and other data processing operators with custom UDF extension support

Performance

Performance Metrics

500K+

TPS

99.9%

Availability

100ms

Latency

15+

Jobs

Code Sample

Flink Streaming Job

Core Flink job definition for real-time Kafka data stream processing

StreamJob.java

import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer;
// Define the Flink streaming job
public class StreamJob {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env =
            StreamExecutionEnvironment.getExecutionEnvironment();
        // Kafka source with auto-commit
        FlinkKafkaConsumer<String> consumer =
            new FlinkKafkaConsumer<>(
                "sensor-data",
                new SimpleStringSchema(),
                kafkaProps
            );
        env.addSource(consumer)
           .map(new DataParser())
           .keyBy(DataEvent::getDeviceId)
           .window(TumblingProcessingTimeWindows.of(
               Time.seconds(10)))
           .aggregate(new AvgAggregator())
           .addSink(new MySQLSink());
        env.execute("DataFlow Pipeline");
    }
}