How to verify if checkpoints are asynchronous or sync

[RKandoji]

Hi,

I'm using Flink 1.9, BlinkPlanner and rocksDB for backend with checkpointing enabled.

I understand that checkpointing is async by default for RocksDB. But I'm curious if there is way to verify if the checkpoints are happening asynchronously or synchronously.

Please let me know.

Thanks,

RKandoji

[William C]

Can you enable debug log to check with that?

regards.

on 2020/1/8 6:36, RKandoji wrote:
> But I'm curious if there is way to verify if the checkpoints are
> happening asynchronously or synchronously.
>

[RKandoji]

Thanks for the reply.

I will check and enable debug logs specifically for the class that contains this log.

But in general logs are already too huge and I'm trying to suppress some of them, so wondering if there is any other way?

Thanks,

RKandoji

On Tue, Jan 7, 2020 at 7:50 PM William C <[hidden email]> wrote:

Can you enable debug log to check with that?

regards.

on 2020/1/8 6:36, RKandoji wrote:
> But I'm curious if there is way to verify if the checkpoints are
> happening asynchronously or synchronously.
>

[Zhijiang(wangzhijiang999)]

The log way is simple for tracing and you can also grep some keywords to find your requirement messages to avoid skimming through the whole large logs.

I am not quite sure what's your specific motivation for doing this. Besides the log way, you can also monitor the thread stack for confirming whether it is happening, but maybe it is not very convenient.

Another possible way is via the checkpoint metrics which would record the sync/async duration time, maybe it can also satisfy your requirements.

Best,

Zhijiang 

------------------------------------------------------------------

From:RKandoji <[hidden email]>

Send Time:2020 Jan. 8 (Wed.) 10:23

To:William C <[hidden email]>

Cc:user <[hidden email]>

Subject:Re: How to verify if checkpoints are asynchronous or sync

Thanks for the reply.

I will check and enable debug logs specifically for the class that contains this log.

But in general logs are already too huge and I'm trying to suppress some of them, so wondering if there is any other way?

Thanks,

RKandoji

On Tue, Jan 7, 2020 at 7:50 PM William C <[hidden email]> wrote:

Can you enable debug log to check with that?

regards.

on 2020/1/8 6:36, RKandoji wrote:
> But I'm curious if there is way to verify if the checkpoints are
> happening asynchronously or synchronously.
>

[Congxian Qiu]

Hi

RocksDB supports Incremental and full snapshot, both are async, do you want to verify whether it's incremental or full snapshot? but I don't know an easy to get this information currently

Zhijiang <[hidden email]> 于2020年1月8日周三 上午10:56写道：

The log way is simple for tracing and you can also grep some keywords to find your requirement messages to avoid skimming through the whole large logs.

I am not quite sure what's your specific motivation for doing this. Besides the log way, you can also monitor the thread stack for confirming whether it is happening, but maybe it is not very convenient.

Another possible way is via the checkpoint metrics which would record the sync/async duration time, maybe it can also satisfy your requirements.

Best,

Zhijiang 

------------------------------------------------------------------

From:RKandoji <[hidden email]>

Send Time:2020 Jan. 8 (Wed.) 10:23

To:William C <[hidden email]>

Cc:user <[hidden email]>

Subject:Re: How to verify if checkpoints are asynchronous or sync

Thanks for the reply.

I will check and enable debug logs specifically for the class that contains this log.

But in general logs are already too huge and I'm trying to suppress some of them, so wondering if there is any other way?

Thanks,

RKandoji

On Tue, Jan 7, 2020 at 7:50 PM William C <[hidden email]> wrote:

Can you enable debug log to check with that?

regards.

on 2020/1/8 6:36, RKandoji wrote:
> But I'm curious if there is way to verify if the checkpoints are
> happening asynchronously or synchronously.
>

[RKandoji]

Sorry for not providing the context earlier.

I'm running my job on a EC2 instance with 32 cores and according to the documentation I tried to use as many task slots the number of cores, numOfTaskSlots=32 and parallelism=32. But I noticed that the performance is slightly degrading when I'm using 32 task slots. Performance seems better at 26 task slots than >26 task slots. So I was trying to understand if additional CPU cores are being utilized by checkpointing or any other async (or background operations, in the process I was trying to verify if the checkpointing is async.

Thanks,

RKandoji

On Tue, Jan 7, 2020 at 10:16 PM Congxian Qiu <[hidden email]> wrote:

Hi

RocksDB supports Incremental and full snapshot, both are async, do you want to verify whether it's incremental or full snapshot? but I don't know an easy to get this information currently

Best,

Congxian

Zhijiang <[hidden email]> 于2020年1月8日周三 上午10:56写道：

The log way is simple for tracing and you can also grep some keywords to find your requirement messages to avoid skimming through the whole large logs.

I am not quite sure what's your specific motivation for doing this. Besides the log way, you can also monitor the thread stack for confirming whether it is happening, but maybe it is not very convenient.

Another possible way is via the checkpoint metrics which would record the sync/async duration time, maybe it can also satisfy your requirements.

Best,

Zhijiang 

------------------------------------------------------------------

From:RKandoji <[hidden email]>

Send Time:2020 Jan. 8 (Wed.) 10:23

To:William C <[hidden email]>

Cc:user <[hidden email]>

Subject:Re: How to verify if checkpoints are asynchronous or sync

Thanks for the reply.

I will check and enable debug logs specifically for the class that contains this log.

But in general logs are already too huge and I'm trying to suppress some of them, so wondering if there is any other way?

Thanks,

RKandoji

On Tue, Jan 7, 2020 at 7:50 PM William C <[hidden email]> wrote:

Can you enable debug log to check with that?

regards.

on 2020/1/8 6:36, RKandoji wrote:
> But I'm curious if there is way to verify if the checkpoints are
> happening asynchronously or synchronously.
>

[William C]

Hallo

on 2020/1/8 11:31, RKandoji wrote:
> I'm running my job on a EC2 instance with 32 cores and according to the
> documentation I tried to use as many task slots the number of cores,
> numOfTaskSlots=32 and parallelism=32. But I noticed that the performance
> is slightly degrading when I'm using 32 task slots. Performance seems
> better at 26 task slots than >26 task slots. So I was trying to
> understand if additional CPU cores are being utilized by checkpointing
> or any other async (or background operations, in the process I was
> trying to verify if the checkpointing is async.

How did you evaluate the performance?
It's may due to busy IO or thread competition or something else similiar.
You'd better dig into more details via logging.

Regards.

[Congxian Qiu]

If you want to figure out the performance problem, maybe async-profile[1] can be helpful

[1] https://github.com/jvm-profiling-tools/async-profiler

William C <[hidden email]> 于2020年1月8日周三 上午11:37写道：

Hallo

on 2020/1/8 11:31, RKandoji wrote:
> I'm running my job on a EC2 instance with 32 cores and according to the
> documentation I tried to use as many task slots the number of cores,
> numOfTaskSlots=32 and parallelism=32. But I noticed that the performance
> is slightly degrading when I'm using 32 task slots. Performance seems
> better at 26 task slots than >26 task slots. So I was trying to
> understand if additional CPU cores are being utilized by checkpointing
> or any other async (or background operations, in the process I was
> trying to verify if the checkpointing is async.

How did you evaluate the performance?
It's may due to busy IO or thread competition or something else similiar.
You'd better dig into more details via logging.

Regards.

[RKandoji]

I evaluated performance by looking at the number of input records processed over 10min and 30mins duration.

Thanks,

R

On Wed, Jan 8, 2020 at 2:21 AM Congxian Qiu <[hidden email]> wrote:

If you want to figure out the performance problem, maybe async-profile[1] can be helpful

[1] https://github.com/jvm-profiling-tools/async-profiler

Best,

Congxian

William C <[hidden email]> 于2020年1月8日周三 上午11:37写道：

Hallo

on 2020/1/8 11:31, RKandoji wrote:
> I'm running my job on a EC2 instance with 32 cores and according to the
> documentation I tried to use as many task slots the number of cores,
> numOfTaskSlots=32 and parallelism=32. But I noticed that the performance
> is slightly degrading when I'm using 32 task slots. Performance seems
> better at 26 task slots than >26 task slots. So I was trying to
> understand if additional CPU cores are being utilized by checkpointing
> or any other async (or background operations, in the process I was
> trying to verify if the checkpointing is async.

How did you evaluate the performance?
It's may due to busy IO or thread competition or something else similiar.
You'd better dig into more details via logging.

Regards.