Flink 1.12.2 源码浅析 : AbstractUdfStreamOperator

一 .前言

AbstractUdfStreamOperator 是所有算子的基类, 所以过来瞄一下…

• 以StreamMap 举例, 看一下的类的关系图…
  
  嗯, 这个图有点复杂, 处理一下, 清掉繁琐的内容. 保留核心主体架构.

在这里我们看到看到StreamMap 其实分两部分,
一个是输入实现了OneInputStreamOperator 接口.
另外一个是输出继承了AbstractUdfStreamOperator 抽象类.

二 .StreamOperator

Basic interface for stream operators.

StreamOperator是一个接口类, 实现了 CheckpointListener, KeyContext, Disposable, Serializable四个接口.

方法清单如下:

2.1. 生命周期相关

2.2. 快照状态相关

2.3. 其他接口

1. Disposable接口类 : 自由一个dispose方法, 无论operator成功还是失败都会调用.

2. Serializable 序列化,不做赘述

3. KeyContext 设置key相关, 里面有关于key的setCurrentKey/getCurrentKey方法

4. CheckpointListener ,Checkpoint监听相关. 有notifyCheckpointComplete(通知完成)和notifyCheckpointAborted(通知终止)方法.

。

2.4.OneInputStreamOperator

这个就不细说了, 就是一个接口继承了StreamOperator类和Input 接口.
并且Input接口重写了setKeyContextElement方法.

• Input接口方法

/**
 * Interface for stream operators with one input. Use {@link
 * org.apache.flink.streaming.api.operators.AbstractStreamOperator} as a base class if you want to
 * implement a custom operator.
 *
 * @param <IN> The input type of the operator
 * @param <OUT> The output type of the operator
 */
@PublicEvolving
public interface OneInputStreamOperator<IN, OUT> extends StreamOperator<OUT>, Input<IN> {
    @Override
    default void setKeyContextElement(StreamRecord<IN> record) throws Exception {
        setKeyContextElement1(record);
    }
}

三.AbstractStreamOperator

AbstractStreamOperator 是一个抽象类, 实现了StreamOperator 接口.

3.1. 属性

    /** The logger used by the operator class and its subclasses. */
    protected static final Logger LOG = LoggerFactory.getLogger(AbstractStreamOperator.class);

    // ----------- configuration properties -------------

    // 定义operator的链接方案。
    // ChainingStrategy.HEAD : 当前算子允许被后置算子Chain，但不会与前置算子进行Chain
    // A sane default for most operators
    protected ChainingStrategy chainingStrategy = ChainingStrategy.HEAD;

    // ---------------- runtime fields ------------------

    //
    /**
     * 包含此运算符（以及同一链中的其他运算符）的任务。
     *
     * The task that contains this operator
     * (and other operators in the same chain).
     * */
    private transient StreamTask<?, ?> container;


    protected transient StreamConfig config;

    // 输出相关
    protected transient Output<StreamRecord<OUT>> output;

    /** The runtime context for UDFs. */

    private transient StreamingRuntimeContext runtimeContext;

    // ---------------- key/value state ------------------

    /**
     * {@code KeySelector} for extracting a key from an element being processed. This is used to
     * scope keyed state to a key. This is null if the operator is not a keyed operator.
     *
     * <p>This is for elements from the first input.
     */
    private transient KeySelector<?, ?> stateKeySelector1;

    /**
     * {@code KeySelector} for extracting a key from an element being processed. This is used to
     * scope keyed state to a key. This is null if the operator is not a keyed operator.
     *
     * <p>This is for elements from the second input.
     */
    private transient KeySelector<?, ?> stateKeySelector2;

    private transient StreamOperatorStateHandler stateHandler;

    private transient InternalTimeServiceManager<?> timeServiceManager;

    // --------------- Metrics ---------------------------

    /** Metric group for the operator. */
    protected transient OperatorMetricGroup metrics;

    protected transient LatencyStats latencyStats;

    // ---------------- time handler ------------------

    protected transient ProcessingTimeService processingTimeService;

    // ---------------- two-input operator watermarks ------------------

    // We keep track of watermarks from both inputs, the combined input is the minimum
    // Once the minimum advances we emit a new watermark for downstream operators
    private long combinedWatermark = Long.MIN_VALUE;
    private long input1Watermark = Long.MIN_VALUE;
    private long input2Watermark = Long.MIN_VALUE;

    // ------------------------------------------------------------------------
    //  Life Cycle
    // ------------------------------------------------------------------------

3.2. 源自父类相关方法实现

四 .AbstractUdfStreamOperator

具有用户定义function的 operators的基础类.
作为operator生命周期的一部分，此类处理用户定义函数的opening和closing。

4.1. 属性

AbstractUdfStreamOperator 抽象类只有一个属性protected final F userFunction;

4.2. 构造方法

构造方法只是为了给属性userFunction赋值.

    public AbstractUdfStreamOperator(F userFunction) {
        this.userFunction = requireNonNull(userFunction);
        checkUdfCheckpointingPreconditions();
    }

五 .StreamMap

StreamMap 没有属性, 只有一个构造方法和一个processElement 方法…

5.1. 构造方法

    public StreamMap(MapFunction<IN, OUT> mapper) {
        super(mapper);
        chainingStrategy = ChainingStrategy.ALWAYS;
    }

5.2. processElement

    @Override
    public void processElement(StreamRecord<IN> element) throws Exception {
        // userFunction.map(element.getValue()) 就是用户定义的MapFunction里面的map方法
        // 将element.getValue() 用用户自定义的map方法里面的内容进行处理...
        output.collect(element.replace(userFunction.map(element.getValue())));
    }

六 .StreamFlatMap

StreamFlatMap 跟StreamMap一样,都是集成了AbstractUdfStreamOperator 抽象类和实现了OneInputStreamOperator .
区别是 StreamFlatMap 中 有一个属性 private transient TimestampedCollector<OUT> collector; .
在open方法中构建

• 构造方法

    public StreamFlatMap(FlatMapFunction<IN, OUT> flatMapper) {
        super(flatMapper);
        chainingStrategy = ChainingStrategy.ALWAYS;
    }

• open方法 :

    @Override
    public void open() throws Exception {
        super.open();
        collector = new TimestampedCollector<>(output);
    }

• 处理数据 :

@Override
public void processElement(StreamRecord<IN> element) throws Exception {
    collector.setTimestamp(element);
    // 使用用户自定义的flatMap函数处理数据
    userFunction.flatMap(element.getValue(), collector);
}