Yahoo Streaming Benchmark on a Flink 1.5 cluster

[Naum Gjorgjeski]

Hi,

I am trying to run the data Artisans version  of the Yahoo Streaming Benchmark. The benchmark applications are  written for Flink 1.0.1. However, I need them to run on a Flink 1.5  cluster. When I try to build the benchmark applications with any version  of Flink from 1.3.0 or higher, I get many compile errors. The compile  errors state that some of the classes and methods cannot be found  (because part of the Flink API has changed in recent versions).

The classes that cannot be found are:
org.apache.flink.streaming.connectors.kafka.partitioner.FixedPartitioner
org.apache.flink.api.common.state.OperatorState
org.apache.flink.api.common.state.StateBackend
org.apache.flink.runtime.state.AsynchronousStateHandle
org.apache.flink.runtime.state.StateHandle
org.apache.flink.streaming.runtime.tasks.StreamTaskState
org.apache.flink.runtime.state.AbstractStateBackend.CheckpointStateOutputView

The methods that cannot be found are:
org.apache.flink.streaming.api.operators.StreamOperator.snapshotOperatorState(long,long)
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStateBackend()
org.apache.flink.runtime.state.AbstractStateBackend.createCheckpointStateOutputView(long,long)

I was able to find a substitution for few of them (e.g. using FlinkFixedPartitioner  instead of FlinkPartitioner), but for most of them there are no  straightforward substitutions. Could you please give me an advice on how  to resolve this problem? Thank you in advance.

Best regards,
Naum

[Naum Gjorgjeski]

Hi all,

I still cannot resolve the problem. Can I please get any advice for it? Thank you.

Best regards,

Naum Gjorgjeski

---

From: Naum Gjorgjeski <[hidden email]>
Sent: Wednesday, August 1, 2018 1:05 AM
To: [hidden email]
Subject: Yahoo Streaming Benchmark on a Flink 1.5 cluster

 

Hi,

I am trying to run the data Artisans version  of the Yahoo Streaming Benchmark. The benchmark applications are  written for Flink 1.0.1. However, I need them to run on a Flink 1.5  cluster. When I try to build the benchmark applications with any version  of Flink from 1.3.0 or higher, I get many compile errors. The compile  errors state that some of the classes and methods cannot be found  (because part of the Flink API has changed in recent versions).

The classes that cannot be found are:
org.apache.flink.streaming.connectors.kafka.partitioner.FixedPartitioner
org.apache.flink.api.common.state.OperatorState
org.apache.flink.api.common.state.StateBackend
org.apache.flink.runtime.state.AsynchronousStateHandle
org.apache.flink.runtime.state.StateHandle
org.apache.flink.streaming.runtime.tasks.StreamTaskState
org.apache.flink.runtime.state.AbstractStateBackend.CheckpointStateOutputView

The methods that cannot be found are:
org.apache.flink.streaming.api.operators.StreamOperator.snapshotOperatorState(long,long)
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStateBackend()
org.apache.flink.runtime.state.AbstractStateBackend.createCheckpointStateOutputView(long,long)

I was able to find a substitution for few of them (e.g. using FlinkFixedPartitioner  instead of FlinkPartitioner), but for most of them there are no  straightforward substitutions. Could you please give me an advice on how  to resolve this problem? Thank you in advance.

Best regards,
Naum

[vino yang]

Hi Namu,

I don't think you need to pay attention to the internals of the Flink API. 

Its interface is backward compatible. If you update the dependent version of the API and the corresponding version of the Flink system so that their versions are consistent, there should be no problems.

Please ensure that the version of Flink is consistent with the version of the API you are using.

Thanks, vino.

Naum Gjorgjeski <[hidden email]> 于2018年8月11日周六 上午4:25写道：

Hi all,

I still cannot resolve the problem. Can I please get any advice for it? Thank you.

Best regards,

Naum Gjorgjeski

---

From: Naum Gjorgjeski <[hidden email]>
Sent: Wednesday, August 1, 2018 1:05 AM
To: [hidden email]
Subject: Yahoo Streaming Benchmark on a Flink 1.5 cluster

 

Hi,

I am trying to run the data Artisans version  of the Yahoo Streaming Benchmark. The benchmark applications are  written for Flink 1.0.1. However, I need them to run on a Flink 1.5  cluster. When I try to build the benchmark applications with any version  of Flink from 1.3.0 or higher, I get many compile errors. The compile  errors state that some of the classes and methods cannot be found  (because part of the Flink API has changed in recent versions).

The classes that cannot be found are:
org.apache.flink.streaming.connectors.kafka.partitioner.FixedPartitioner
org.apache.flink.api.common.state.OperatorState
org.apache.flink.api.common.state.StateBackend
org.apache.flink.runtime.state.AsynchronousStateHandle
org.apache.flink.runtime.state.StateHandle
org.apache.flink.streaming.runtime.tasks.StreamTaskState
org.apache.flink.runtime.state.AbstractStateBackend.CheckpointStateOutputView

The methods that cannot be found are:
org.apache.flink.streaming.api.operators.StreamOperator.snapshotOperatorState(long,long)
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStateBackend()
org.apache.flink.runtime.state.AbstractStateBackend.createCheckpointStateOutputView(long,long)

I was able to find a substitution for few of them (e.g. using FlinkFixedPartitioner  instead of FlinkPartitioner), but for most of them there are no  straightforward substitutions. Could you please give me an advice on how  to resolve this problem? Thank you in advance.

Best regards,
Naum

[Chesnay Schepler]

@vino Please take a deeper look at the question before answering them.

The benchmark implements custom operators that make use of several non-Public APIs,
so API backwards compatibility simply doesn't apply here.

Updating the benchmark will require a bit of work. Below are some hints i gathered, but given that we're jumping
5 versions at once (and 1.0 being really old) I can't guarantee anything about the correctness.

The replacement for StateBackend is StateBinder. (apparently)

The OperatorState usage should be replaced with a ValueState usage.

The code in QueryableWindowOperatorEvicting#snapshotOperatorState can be ported to a keyed ValueState,
have a look at the WindowOperator#open().

The StateHandle/StreamTaskState/CheckpointStateOutputView stuff should become irrelevant after porting the operator to 1.5.

On 11.08.2018 04:14, vino yang wrote:

Hi Namu,

I don't think you need to pay attention to the internals of the Flink API. 

Its interface is backward compatible. If you update the dependent version of the API and the corresponding version of the Flink system so that their versions are consistent, there should be no problems.

Please ensure that the version of Flink is consistent with the version of the API you are using.

Thanks, vino.

Naum Gjorgjeski <[hidden email]> 于2018年8月11日周六 上午4:25写道：

Hi all,

I still cannot resolve the problem. Can I please get any advice for it? Thank you.

Best regards,

Naum Gjorgjeski

---

From: Naum Gjorgjeski <[hidden email]>
Sent: Wednesday, August 1, 2018 1:05 AM
To: [hidden email]
Subject: Yahoo Streaming Benchmark on a Flink 1.5 cluster

 

Hi,

I am trying to run the data Artisans version  of the Yahoo Streaming Benchmark. The benchmark applications are  written for Flink 1.0.1. However, I need them to run on a Flink 1.5  cluster. When I try to build the benchmark applications with any version  of Flink from 1.3.0 or higher, I get many compile errors. The compile  errors state that some of the classes and methods cannot be found  (because part of the Flink API has changed in recent versions).

The classes that cannot be found are:
org.apache.flink.streaming.connectors.kafka.partitioner.FixedPartitioner
org.apache.flink.api.common.state.OperatorState
org.apache.flink.api.common.state.StateBackend
org.apache.flink.runtime.state.AsynchronousStateHandle
org.apache.flink.runtime.state.StateHandle
org.apache.flink.streaming.runtime.tasks.StreamTaskState
org.apache.flink.runtime.state.AbstractStateBackend.CheckpointStateOutputView

The methods that cannot be found are:
org.apache.flink.streaming.api.operators.StreamOperator.snapshotOperatorState(long,long)
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStateBackend()
org.apache.flink.runtime.state.AbstractStateBackend.createCheckpointStateOutputView(long,long)

I was able to find a substitution for few of them (e.g. using FlinkFixedPartitioner  instead of FlinkPartitioner), but for most of them there are no  straightforward substitutions. Could you please give me an advice on how  to resolve this problem? Thank you in advance.

Best regards,
Naum