Flink+YARN HDFS replication factor

[Piper Piper]

Hello,

When using Flink+YARN (with HDFS) and having a long running Flink session (mode) cluster with a Flink client submitting jobs, the HDFS could have a replication factor greater than 1 (example 3).

So, I would like to know when and how any of the data (like event-data or batch-data) or code (like JAR) in a Flink job is saved to the HDFS and is replicated in the entire YARN cluster of nodes?

For example, in streaming applications, would all the event-data only be in memory (RAM) until it reaches the DAG's sink and then must be saved into HDFS?

Thank you,

Piper

[Till Rohrmann]

Hi Piper,

in general, Flink does not store transient data such as event data on HDFS. Event data (data which is sent between the TaskManager's to process it) is only kept in memory and if becoming too big spilled by some operators to local disk.

What Flink stores on HDFS (given it is configured this way), is the state data which is part of the jobs checkpoints. Moreover, Flink stores the job information such as the JobGraph and the corresponding blobs (Jars and job artifacts) on HDFS if configured so.

Cheers,

Till

On Wed, Jan 29, 2020 at 7:07 AM Piper Piper <[hidden email]> wrote:

Hello,

When using Flink+YARN (with HDFS) and having a long running Flink session (mode) cluster with a Flink client submitting jobs, the HDFS could have a replication factor greater than 1 (example 3).

So, I would like to know when and how any of the data (like event-data or batch-data) or code (like JAR) in a Flink job is saved to the HDFS and is replicated in the entire YARN cluster of nodes?

For example, in streaming applications, would all the event-data only be in memory (RAM) until it reaches the DAG's sink and then must be saved into HDFS?

Thank you,

Piper

[Piper Piper]

Hi Till,

Thank you for the information!

In case of wide transformations, Spark stores input data onto disk between shuffles. So, I was wondering if Flink does that as well (even for windows of streaming data), and whether that "storing to disk" is persisted to the HDFS and honors the replication factor.

Best,

Pankaj

On Wed, Jan 29, 2020 at 9:56 AM Till Rohrmann <[hidden email]> wrote:

Hi Piper,

in general, Flink does not store transient data such as event data on HDFS. Event data (data which is sent between the TaskManager's to process it) is only kept in memory and if becoming too big spilled by some operators to local disk.

What Flink stores on HDFS (given it is configured this way), is the state data which is part of the jobs checkpoints. Moreover, Flink stores the job information such as the JobGraph and the corresponding blobs (Jars and job artifacts) on HDFS if configured so.

Cheers,

Till

On Wed, Jan 29, 2020 at 7:07 AM Piper Piper <[hidden email]> wrote:

Hello,

When using Flink+YARN (with HDFS) and having a long running Flink session (mode) cluster with a Flink client submitting jobs, the HDFS could have a replication factor greater than 1 (example 3).

So, I would like to know when and how any of the data (like event-data or batch-data) or code (like JAR) in a Flink job is saved to the HDFS and is replicated in the entire YARN cluster of nodes?

For example, in streaming applications, would all the event-data only be in memory (RAM) until it reaches the DAG's sink and then must be saved into HDFS?

Thank you,

Piper

[Piper Piper]

Please disregard my previous email. I found the answer online.

I thought writing data to local disk automatically meant the data would be persisted to HDFS. However, Spark writes data (in between shuffles) to local disk only.

Thanks

On Thu, Jan 30, 2020, 2:00 PM Piper Piper <[hidden email]> wrote:

Hi Till,

Thank you for the information!

In case of wide transformations, Spark stores input data onto disk between shuffles. So, I was wondering if Flink does that as well (even for windows of streaming data), and whether that "storing to disk" is persisted to the HDFS and honors the replication factor.

Best,

Pankaj

On Wed, Jan 29, 2020 at 9:56 AM Till Rohrmann <[hidden email]> wrote:

Hi Piper,

in general, Flink does not store transient data such as event data on HDFS. Event data (data which is sent between the TaskManager's to process it) is only kept in memory and if becoming too big spilled by some operators to local disk.

What Flink stores on HDFS (given it is configured this way), is the state data which is part of the jobs checkpoints. Moreover, Flink stores the job information such as the JobGraph and the corresponding blobs (Jars and job artifacts) on HDFS if configured so.

Cheers,

Till

On Wed, Jan 29, 2020 at 7:07 AM Piper Piper <[hidden email]> wrote:

Hello,

When using Flink+YARN (with HDFS) and having a long running Flink session (mode) cluster with a Flink client submitting jobs, the HDFS could have a replication factor greater than 1 (example 3).

So, I would like to know when and how any of the data (like event-data or batch-data) or code (like JAR) in a Flink job is saved to the HDFS and is replicated in the entire YARN cluster of nodes?

For example, in streaming applications, would all the event-data only be in memory (RAM) until it reaches the DAG's sink and then must be saved into HDFS?

Thank you,

Piper

[Till Rohrmann]

The same applies to Flink. Transient data will only be stored on local disks.

Cheers,

Till

On Thu, Jan 30, 2020 at 9:10 PM Piper Piper <[hidden email]> wrote:

Please disregard my previous email. I found the answer online.

I thought writing data to local disk automatically meant the data would be persisted to HDFS. However, Spark writes data (in between shuffles) to local disk only.

Thanks

On Thu, Jan 30, 2020, 2:00 PM Piper Piper <[hidden email]> wrote:

Hi Till,

Thank you for the information!

In case of wide transformations, Spark stores input data onto disk between shuffles. So, I was wondering if Flink does that as well (even for windows of streaming data), and whether that "storing to disk" is persisted to the HDFS and honors the replication factor.

Best,

Pankaj

On Wed, Jan 29, 2020 at 9:56 AM Till Rohrmann <[hidden email]> wrote:

Hi Piper,

in general, Flink does not store transient data such as event data on HDFS. Event data (data which is sent between the TaskManager's to process it) is only kept in memory and if becoming too big spilled by some operators to local disk.

What Flink stores on HDFS (given it is configured this way), is the state data which is part of the jobs checkpoints. Moreover, Flink stores the job information such as the JobGraph and the corresponding blobs (Jars and job artifacts) on HDFS if configured so.

Cheers,

Till

On Wed, Jan 29, 2020 at 7:07 AM Piper Piper <[hidden email]> wrote:

Hello,

When using Flink+YARN (with HDFS) and having a long running Flink session (mode) cluster with a Flink client submitting jobs, the HDFS could have a replication factor greater than 1 (example 3).

So, I would like to know when and how any of the data (like event-data or batch-data) or code (like JAR) in a Flink job is saved to the HDFS and is replicated in the entire YARN cluster of nodes?

For example, in streaming applications, would all the event-data only be in memory (RAM) until it reaches the DAG's sink and then must be saved into HDFS?

Thank you,

Piper