Guide on writing Flink plugins

[Kien Truong]

Hi all,

We want to write a Flink plugins to integrate Flink jobs with our in-house monitoring system. Are there any guide or tutorial that we can follow to write a Flink plugins ? 

The official documents are a bit bare bone.

Regards,

Kien

[大森林]

I want to simulate the scene"back pressure"

could you tell me

what's the example for datastream datagen?

Thanks for your help.

[Chesnay Schepler]

I would recommend looking at one of the existing reporter modules, like flink-metrics-slf4j.

The important bit plugin-wise is to have a factory class and the META-INF/services entry.

On 10/6/2020 5:23 AM, Kien Truong wrote:

Hi all,

We want to write a Flink plugins to integrate Flink jobs with our in-house monitoring system. Are there any guide or tutorial that we can follow to write a Flink plugins ? 

The official documents are a bit bare bone.

Regards,

Kien

[Shengkai Fang]

Hi, I think you can take a look at org.apache.flink.table.planner.plan.nodes.common.CommonPhysicalTableSourceScan#createSourceTransformation, which will tell you how to get transformations by source function and stream execution environment. In datastream api, we also have a DataGen that is org.apache.flink.streaming.api.functions.source.datagen.DataGeneratorSource. Therefore, I think we can do as follows to create a DataGen: 

new DataStream(env, env.addSource(new DataGeneratorSource(RandomGenerator.longGenerator(0, 1000))).getTransformation()) 

I am not sure whether this is the best method. Welcome for any options.
 

大森林 <[hidden email]> 于2020年10月6日周二 上午11:44写道：

I want to simulate the scene"back pressure"

could you tell me

what's the example for datastream datagen?

Thanks for your help.

[大森林]

My java is poor

I have tried like this,but it's wrong,Could you tell me how to fix this...

import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.LocalStreamEnvironment;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.source.datagen.DataGeneratorSource;
import org.apache.flink.streaming.api.functions.source.datagen.RandomGenerator;

public class datagen {


    public static void main(String[] args) throws Exception {

        LocalStreamEnvironment env = StreamExecutionEnvironment.createLocalEnvironment();


        DataStream datasets = new DataStream(env, env.addSource(new DataGeneratorSource(RandomGenerator.longGenerator(0, 1000))).getTransformation());
    }

}

I want to use this to simulate "back pressure" scene,do you have some advices or guide?thanks~!

------------------ 原始邮件 ------------------

发件人: "Shengkai Fang" <[hidden email]>;

发送时间: 2020年10月6日(星期二) 晚上11:06

收件人: "大森林"<[hidden email]>;

抄送: "user"<[hidden email]>;

主题: Re: what's the example for datastream data generator?

Hi, I think you can take a look at org.apache.flink.table.planner.plan.nodes.common.CommonPhysicalTableSourceScan#createSourceTransformation, which will tell you how to get transformations by source function and stream execution environment. In datastream api, we also have a DataGen that is org.apache.flink.streaming.api.functions.source.datagen.DataGeneratorSource. Therefore, I think we can do as follows to create a DataGen: 

new DataStream(env, env.addSource(new DataGeneratorSource(RandomGenerator.longGenerator(0, 1000))).getTransformation()) 

I am not sure whether this is the best method. Welcome for any options.
 

大森林 <[hidden email]> 于2020年10月6日周二 上午11:44写道：

I want to simulate the scene"back pressure"

could you tell me

what's the example for datastream datagen?

Thanks for your help.