Comparing Storm and Flink resource requirements

[Gyula Fóra]

Hi All!

I would like to ask the community for any experience regarding migration from Storm to Flink production applications.

Specifically I am interested in your experience related to the resource requirements for the same pipeline as implemented in Flink vs in Storm. The design of the applications might be significantly different for the 2 systems (state, added complexity by new features enabled) so I don't expect precise numbers, estimates would be completely fine :)

I understand that this is a quite complex question and I would really appreciate any information you can provide.

Thank you all!

Gyula

[vino yang]

Hi Gyula,

Based on our previous experience switching from Storm to Flink. For the same business, resources of the same size are completely sufficient, and the performance indicators are slightly better than Storm. As you said, this may be related to using some of Flink's special features like state. Our state was small at the time, and the main business was real-time ETL. If it is a different type of business, the problem may be more complicated and may require a specific analysis of the specific problem.

Best,

Vino

Gyula Fóra <[hidden email]> 于2019年10月21日周一 下午8:15写道：

Hi All!

I would like to ask the community for any experience regarding migration from Storm to Flink production applications.

Specifically I am interested in your experience related to the resource requirements for the same pipeline as implemented in Flink vs in Storm. The design of the applications might be significantly different for the 2 systems (state, added complexity by new features enabled) so I don't expect precise numbers, estimates would be completely fine :)

I understand that this is a quite complex question and I would really appreciate any information you can provide.

Thank you all!

Gyula

[Gyula Fóra]

Thanks Vino, this is very helpful. 

At the end of the day we are looking for something like: 

A job that requires N nodes in Storm, only requires K nodes when implemented in Flink, where hopefully K < N :)

Cheers,

Gyula

On Tue, Oct 22, 2019 at 2:31 PM vino yang <[hidden email]> wrote:

Hi Gyula,

Based on our previous experience switching from Storm to Flink. For the same business, resources of the same size are completely sufficient, and the performance indicators are slightly better than Storm. As you said, this may be related to using some of Flink's special features like state. Our state was small at the time, and the main business was real-time ETL. If it is a different type of business, the problem may be more complicated and may require a specific analysis of the specific problem.

Best,

Vino

Gyula Fóra <[hidden email]> 于2019年10月21日周一 下午8:15写道：

Hi All!

I would like to ask the community for any experience regarding migration from Storm to Flink production applications.

Specifically I am interested in your experience related to the resource requirements for the same pipeline as implemented in Flink vs in Storm. The design of the applications might be significantly different for the 2 systems (state, added complexity by new features enabled) so I don't expect precise numbers, estimates would be completely fine :)

I understand that this is a quite complex question and I would really appreciate any information you can provide.

Thank you all!

Gyula