(DEPRECATED) Apache Flink User Mailing List archive.

No data issued by flink window after a few hours

Classic

List

Threaded

17 messages Options

潘功森

No data issued by flink window after a few hours

Hi all,

I used flink window, and when the job begins, we could get the results of windiow.But there’re no results issued after a few hours.

I found the job is still running and no errors, and the data not used window all can be issued.

By the way, I used Flink 1.3.2 and ram to cache the distinct data about sliding window.

Yours,

September

vino yang

Re: No data issued by flink window after a few hours

Hi,

Can you explain what "ram to cache the distinct data about sliding window" mean?

The information you provide is too small and will not help others to help you analyze the problem and provide advice.

In addition, regarding the usage of Flink related issues, please only send mail to the user mailing list.

The dev mailing list is mainly used to discuss development related issues.

Thanks vino.

? ?? <[hidden email]> 于2018年10月10日周三上午10:37写道：

Hi all,
I used flink window, and when the job begins, we could get the results of windiow.But there’re no results issued after a few hours.
I found the job is still running and no errors, and the data not used window all can be issued.
By the way, I used Flink 1.3.2 and ram to cache the distinct data about sliding window.

Yours,
September

潘功森

答复: No data issued by flink window after a few hours

Hi,

"ram to cache the distinct data about sliding window" means I used computer momery not the third part db to cache the data need used in window.

“the data need used in window” means ：such as the sliding window is 1 hour, and I need to count the distinct users, I need to cache the user id about one hour.

Cause there’re no related errors.

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 10:49:43 AM
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Can you explain what "ram to cache the distinct data about sliding window" mean?

The information you provide is too small and will not help others to help you analyze the problem and provide advice.

In addition, regarding the usage of Flink related issues, please only send mail to the user mailing list.

The dev mailing list is mainly used to discuss development related issues.

Thanks vino.

? ?? <[hidden email]> 于2018年10月10日周三上午10:37写道：

Hi all,
I used flink window, and when the job begins, we could get the results of windiow.But there’re no results issued after a few hours.
I found the job is still running and no errors, and the data not used window all can be issued.
By the way, I used Flink 1.3.2 and ram to cache the distinct data about sliding window.

Yours,
September

vino yang

Re: No data issued by flink window after a few hours

Hi,

Did you mean "computer momery" referring to Memory Statebackend?

The Flink window mechanism is internally based on State, and this is done for fault tolerance.

If you introduce external storage, it will break its design and bring other problems.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:02写道：

Hi,

"ram to cache the distinct data about sliding window" means I used computer momery not the third part db to cache the data need used in window.

“the data need used in window” means ：such as the sliding window is 1 hour, and I need to count the distinct users, I need to cache the user id about one hour.

Cause there’re no related errors.

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 10:49:43 AM
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Can you explain what "ram to cache the distinct data about sliding window" mean?

The information you provide is too small and will not help others to help you analyze the problem and provide advice.

In addition, regarding the usage of Flink related issues, please only send mail to the user mailing list.

The dev mailing list is mainly used to discuss development related issues.

Thanks vino.

? ?? <[hidden email]> 于2018年10月10日周三上午10:37写道：

Hi all,
I used flink window, and when the job begins, we could get the results of windiow.But there’re no results issued after a few hours.
I found the job is still running and no errors, and the data not used window all can be issued.
By the way, I used Flink 1.3.2 and ram to cache the distinct data about sliding window.

Yours,
September

vino yang

Re: No data issued by flink window after a few hours

Hi,

I saw the exception image you provided. Based on the exception message, it seems you used the default max state size (5MB).

You can specify the max state size to override the default value. Try :

MemoryStateBackend stateBackend = new MemoryStateBackend(theSizeOfBytes);

Please note that you need to reserve enough memory for Flink.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:36写道：

Please have a look about my last mail.

When the cached window data is too large, how?

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 11:33:48 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Did you mean "computer momery" referring to Memory Statebackend?

The Flink window mechanism is internally based on State, and this is done for fault tolerance.

If you introduce external storage, it will break its design and bring other problems.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:02写道：

Hi,

"ram to cache the distinct data about sliding window" means I used computer momery not the third part db to cache the data need used in window.

“the data need used in window” means ：such as the sliding window is 1 hour, and I need to count the distinct users, I need to cache the user id about one hour.

Cause there’re no related errors.

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 10:49:43 AM
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Can you explain what "ram to cache the distinct data about sliding window" mean?

The information you provide is too small and will not help others to help you analyze the problem and provide advice.

In addition, regarding the usage of Flink related issues, please only send mail to the user mailing list.

The dev mailing list is mainly used to discuss development related issues.

Thanks vino.

? ?? <[hidden email]> 于2018年10月10日周三上午10:37写道：

Hi all,
I used flink window, and when the job begins, we could get the results of windiow.But there’re no results issued after a few hours.
I found the job is still running and no errors, and the data not used window all can be issued.
By the way, I used Flink 1.3.2 and ram to cache the distinct data about sliding window.

Yours,
September

潘功森

答复: No data issued by flink window after a few hours

Hi，

Cause default state size in one hour is too small，and the max window size is 24 hours, so I used 500M.

MemoryStateBackend stateBackend = new MemoryStateBackend(MAX_STATE_SIZE);//500M
env.setStateBackend(stateBackend);

And I found Irrespective of the configured maximal state size, the state cannot be larger than the akka frame size.

So I add a config in flink-comf.yaml:

akka.framesize: 524288000b

What else do I have to pay attention to?

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 11:45:31 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

I saw the exception image you provided. Based on the exception message, it seems you used the default max state size (5MB).

You can specify the max state size to override the default value. Try :

MemoryStateBackend stateBackend = new MemoryStateBackend(theSizeOfBytes);

Please note that you need to reserve enough memory for Flink.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:36写道：

Please have a look about my last mail.

When the cached window data is too large, how?

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 11:33:48 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Did you mean "computer momery" referring to Memory Statebackend?

The Flink window mechanism is internally based on State, and this is done for fault tolerance.

If you introduce external storage, it will break its design and bring other problems.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:02写道：

Hi,

"ram to cache the distinct data about sliding window" means I used computer momery not the third part db to cache the data need used in window.

“the data need used in window” means ：such as the sliding window is 1 hour, and I need to count the distinct users, I need to cache the user id about one hour.

Cause there’re no related errors.

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 10:49:43 AM
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Can you explain what "ram to cache the distinct data about sliding window" mean?

The information you provide is too small and will not help others to help you analyze the problem and provide advice.

In addition, regarding the usage of Flink related issues, please only send mail to the user mailing list.

The dev mailing list is mainly used to discuss development related issues.

Thanks vino.

? ?? <[hidden email]> 于2018年10月10日周三上午10:37写道：

Hi all,
I used flink window, and when the job begins, we could get the results of windiow.But there’re no results issued after a few hours.
I found the job is still running and no errors, and the data not used window all can be issued.
By the way, I used Flink 1.3.2 and ram to cache the distinct data about sliding window.

Yours,
September

潘功森

答复: No data issued by flink window after a few hours

Hi，

I changed as below configurations，and it looks fine when job started.

But there’re no results issued when window ends after running about six hours, and no errors and exceptions.

How can I position the question?

Yours,

September

发件人: 潘功森 <[hidden email]>
发送时间: Wednesday, October 10, 2018 2:44:48 PM
收件人: vino yang
抄送: user
主题: 答复: No data issued by flink window after a few hours

Hi，

Cause default state size in one hour is too small，and the max window size is 24 hours, so I used 500M.

MemoryStateBackend stateBackend = new MemoryStateBackend(MAX_STATE_SIZE);//500M
env.setStateBackend(stateBackend);

And I found Irrespective of the configured maximal state size, the state cannot be larger than the akka frame size.

So I add a config in flink-comf.yaml:

akka.framesize: 524288000b

What else do I have to pay attention to?

Yours,

September

Hi,

I saw the exception image you provided. Based on the exception message, it seems you used the default max state size (5MB).

You can specify the max state size to override the default value. Try :

MemoryStateBackend stateBackend = new MemoryStateBackend(theSizeOfBytes);

Please note that you need to reserve enough memory for Flink.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:36写道：

Please have a look about my last mail.

When the cached window data is too large, how?

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 11:33:48 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Did you mean "computer momery" referring to Memory Statebackend?

The Flink window mechanism is internally based on State, and this is done for fault tolerance.

If you introduce external storage, it will break its design and bring other problems.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:02写道：

Hi,

"ram to cache the distinct data about sliding window" means I used computer momery not the third part db to cache the data need used in window.

“the data need used in window” means ：such as the sliding window is 1 hour, and I need to count the distinct users, I need to cache the user id about one hour.

Cause there’re no related errors.

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 10:49:43 AM
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Can you explain what "ram to cache the distinct data about sliding window" mean?

The information you provide is too small and will not help others to help you analyze the problem and provide advice.

In addition, regarding the usage of Flink related issues, please only send mail to the user mailing list.

The dev mailing list is mainly used to discuss development related issues.

Thanks vino.

? ?? <[hidden email]> 于2018年10月10日周三上午10:37写道：

Hi all,
I used flink window, and when the job begins, we could get the results of windiow.But there’re no results issued after a few hours.
I found the job is still running and no errors, and the data not used window all can be issued.
By the way, I used Flink 1.3.2 and ram to cache the distinct data about sliding window.

Yours,
September

vino yang

Re: No data issued by flink window after a few hours

Hi gongsen,

Have you used event time as time semantics? If so, then the possible problem is related to watermark.

Since I don't know the details of your program, it's hard to make a conclusion. You can check if your watermark is firing normally.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月11日周四下午12:12写道：

Hi，

I changed as below configurations，and it looks fine when job started.

But there’re no results issued when window ends after running about six hours, and no errors and exceptions.

How can I position the question?

Yours,

September

发件人: 潘功森 <[hidden email]>
发送时间: Wednesday, October 10, 2018 2:44:48 PM
收件人: vino yang
抄送: user
主题: 答复: No data issued by flink window after a few hours
Hi，

Cause default state size in one hour is too small，and the max window size is 24 hours, so I used 500M.
MemoryStateBackend stateBackend = new MemoryStateBackend(MAX_STATE_SIZE);//500M
env.setStateBackend(stateBackend);
And I found Irrespective of the configured maximal state size, the state cannot be larger than the akka frame size.

So I add a config in flink-comf.yaml:

akka.framesize: 524288000b

What else do I have to pay attention to?

Yours,

September
发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 11:45:31 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

I saw the exception image you provided. Based on the exception message, it seems you used the default max state size (5MB).

You can specify the max state size to override the default value. Try :

MemoryStateBackend stateBackend = new MemoryStateBackend(theSizeOfBytes);

Please note that you need to reserve enough memory for Flink.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:36写道：

Please have a look about my last mail.

When the cached window data is too large, how?

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 11:33:48 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Did you mean "computer momery" referring to Memory Statebackend?

The Flink window mechanism is internally based on State, and this is done for fault tolerance.

If you introduce external storage, it will break its design and bring other problems.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:02写道：

Hi,

"ram to cache the distinct data about sliding window" means I used computer momery not the third part db to cache the data need used in window.

“the data need used in window” means ：such as the sliding window is 1 hour, and I need to count the distinct users, I need to cache the user id about one hour.

Cause there’re no related errors.

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 10:49:43 AM
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Can you explain what "ram to cache the distinct data about sliding window" mean?

The information you provide is too small and will not help others to help you analyze the problem and provide advice.

In addition, regarding the usage of Flink related issues, please only send mail to the user mailing list.

The dev mailing list is mainly used to discuss development related issues.

Thanks vino.

? ?? <[hidden email]> 于2018年10月10日周三上午10:37写道：

Hi all,
I used flink window, and when the job begins, we could get the results of windiow.But there’re no results issued after a few hours.
I found the job is still running and no errors, and the data not used window all can be issued.
By the way, I used Flink 1.3.2 and ram to cache the distinct data about sliding window.

Yours,
September

Dawid Wysakowicz-2

Re: 答复: No data issued by flink window after a few hours

In reply to this post by 潘功森

Hi,

I agree with Vino, that you should check if the watermark is progressing for all subtasks, if you are using event time semantics. If this is not the problem it would help if you could share the code of your job. By the way have you tried reproducing the problem with collection source?

Best,

Dawid

On 10/10/18 08:44, 潘功森 wrote:

Hi，

Cause default state size in one hour is too small，and the max window size is 24 hours, so I used 500M.
MemoryStateBackend stateBackend = new MemoryStateBackend(MAX_STATE_SIZE);//500M
env.setStateBackend(stateBackend);
And I found Irrespective of the configured maximal state size, the state cannot be larger than the akka frame size.

So I add a config in flink-comf.yaml:

akka.framesize: 524288000b

What else do I have to pay attention to?

Yours,

September
发件人: vino yang [hidden email]
发送时间: Wednesday, October 10, 2018 11:45:31 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

I saw the exception image you provided. Based on the exception message, it seems you used the default max state size (5MB).

You can specify the max state size to override the default value. Try :

MemoryStateBackend stateBackend = new MemoryStateBackend(theSizeOfBytes);

Please note that you need to reserve enough memory for Flink.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:36写道：

Please have a look about my last mail.

When the cached window data is too large, how?

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 11:33:48 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Did you mean "computer momery" referring to Memory Statebackend?

The Flink window mechanism is internally based on State, and this is done for fault tolerance.

If you introduce external storage, it will break its design and bring other problems.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:02写道：

Hi,

"ram to cache the distinct data about sliding window" means I used computer momery not the third part db to cache the data need used in window.

“the data need used in window” means ：such as the sliding window is 1 hour, and I need to count the distinct users, I need to cache the user id about one hour.

Cause there’re no related errors.

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 10:49:43 AM
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Can you explain what "ram to cache the distinct data about sliding window" mean?

The information you provide is too small and will not help others to help you analyze the problem and provide advice.

In addition, regarding the usage of Flink related issues, please only send mail to the user mailing list.

The dev mailing list is mainly used to discuss development related issues.

Thanks vino.

? ?? <[hidden email]> 于2018年10月10日周三上午10:37写道：

Hi all,
I used flink window, and when the job begins, we could get the results of windiow.But there’re no results issued after a few hours.
I found the job is still running and no errors, and the data not used window all can be issued.
By the way, I used Flink 1.3.2 and ram to cache the distinct data about sliding window.

Yours,
September

signature.asc (849 bytes) Download Attachment

潘功森

答复: No data issued by flink window after a few hours

In reply to this post by vino yang

Hi，

Yes, I used event time.

I found watermarks fired normally when the job started, but it stopped and no changed after running hours.

Step1. As I used memory state backend and make max_state_sie and akka frame size bigger,and It also will stop after a few hours.

I found the error below:

And the job restarted:

Step 2.Then I configured fs state below:

cid:image001.png@01D461B2.9B8C6790

But the file is null, I think the configuration is wrong?

Could you give me your advice?

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Thursday, October 11, 2018 2:56:40 PM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi gongsen,

Have you used event time as time semantics? If so, then the possible problem is related to watermark.

Since I don't know the details of your program, it's hard to make a conclusion. You can check if your watermark is firing normally.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月11日周四下午12:12写道：

Hi，

I changed as below configurations，and it looks fine when job started.

But there’re no results issued when window ends after running about six hours, and no errors and exceptions.

How can I position the question?

Yours,

September

发件人: 潘功森 <[hidden email]>
发送时间: Wednesday, October 10, 2018 2:44:48 PM
收件人: vino yang
抄送: user
主题: 答复: No data issued by flink window after a few hours
Hi，

Cause default state size in one hour is too small，and the max window size is 24 hours, so I used 500M.
MemoryStateBackend stateBackend = new MemoryStateBackend(MAX_STATE_SIZE);//500M
env.setStateBackend(stateBackend);
And I found Irrespective of the configured maximal state size, the state cannot be larger than the akka frame size.

So I add a config in flink-comf.yaml:

akka.framesize: 524288000b

What else do I have to pay attention to?

Yours,

September
发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 11:45:31 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

I saw the exception image you provided. Based on the exception message, it seems you used the default max state size (5MB).

You can specify the max state size to override the default value. Try :

MemoryStateBackend stateBackend = new MemoryStateBackend(theSizeOfBytes);

Please note that you need to reserve enough memory for Flink.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:36写道：

Please have a look about my last mail.

When the cached window data is too large, how?

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 11:33:48 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Did you mean "computer momery" referring to Memory Statebackend?

The Flink window mechanism is internally based on State, and this is done for fault tolerance.

If you introduce external storage, it will break its design and bring other problems.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:02写道：

Hi,

"ram to cache the distinct data about sliding window" means I used computer momery not the third part db to cache the data need used in window.

“the data need used in window” means ：such as the sliding window is 1 hour, and I need to count the distinct users, I need to cache the user id about one hour.

Cause there’re no related errors.

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 10:49:43 AM
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Can you explain what "ram to cache the distinct data about sliding window" mean?

The information you provide is too small and will not help others to help you analyze the problem and provide advice.

In addition, regarding the usage of Flink related issues, please only send mail to the user mailing list.

The dev mailing list is mainly used to discuss development related issues.

Thanks vino.

? ?? <[hidden email]> 于2018年10月10日周三上午10:37写道：

Hi all,
I used flink window, and when the job begins, we could get the results of windiow.But there’re no results issued after a few hours.
I found the job is still running and no errors, and the data not used window all can be issued.
By the way, I used Flink 1.3.2 and ram to cache the distinct data about sliding window.

Yours,
September

潘功森

答复: 答复: No data issued by flink window after a few hours

In reply to this post by Dawid Wysakowicz-2

The second question looks fine.

cid:image004.png@01D461B6.93D6FF70

Yours,

September

发件人: [hidden email]
发送时间: 2018年10月11日 15:13
收件人: [hidden email]; [hidden email]
抄送: [hidden email]
主题: Re: 答复: No data issued by flink window after a few hours

Hi,

Best,

Dawid

On 10/10/18 08:44, 潘功森 wrote:

Hi，

Cause default state size in one hour is too small，and the max window size is 24 hours, so I used 500M.
MemoryStateBackend stateBackend = new MemoryStateBackend(MAX_STATE_SIZE);//500M
env.setStateBackend(stateBackend);
And I found Irrespective of the configured maximal state size, the state cannot be larger than the akka frame size.

So I add a config in flink-comf.yaml:

akka.framesize: 524288000b

What else do I have to pay attention to?

Yours,

September

发件人: vino yang [hidden email]
发送时间: Wednesday, October 10, 2018 11:45:31 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

I saw the exception image you provided. Based on the exception message, it seems you used the default max state size (5MB).

You can specify the max state size to override the default value. Try :

MemoryStateBackend stateBackend = new MemoryStateBackend(theSizeOfBytes);

Please note that you need to reserve enough memory for Flink.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:36写道：

Please have a look about my last mail.

When the cached window data is too large, how?

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 11:33:48 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Did you mean "computer momery" referring to Memory Statebackend?

The Flink window mechanism is internally based on State, and this is done for fault tolerance.

If you introduce external storage, it will break its design and bring other problems.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:02写道：

Hi,

"ram to cache the distinct data about sliding window" means I used computer momery not the third part db to cache the data need used in window.

“the data need used in window” means ：such as the sliding window is 1 hour, and I need to count the distinct users, I need to cache the user id about one hour.

Cause there’re no related errors.

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 10:49:43 AM
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Can you explain what "ram to cache the distinct data about sliding window" mean?

The information you provide is too small and will not help others to help you analyze the problem and provide advice.

In addition, regarding the usage of Flink related issues, please only send mail to the user mailing list.

The dev mailing list is mainly used to discuss development related issues.

Thanks vino.

? ?? <[hidden email]> 于2018年10月10日周三上午10:37写道：

Hi all,
I used flink window, and when the job begins, we could get the results of windiow.But there’re no results issued after a few hours.
I found the job is still running and no errors, and the data not used window all can be issued.
By the way, I used Flink 1.3.2 and ram to cache the distinct data about sliding window.

Yours,
September

潘功森

答复: No data issued by flink window after a few hours

In reply to this post by vino yang

Hi,

I found the pictures maybe too big and the net here not so good, so the mail I wrote is not sent sucsessfully last night.

Yes, I used event time.

I found watermarks fired normally when the job started, but it stopped and no changed after running hours.

And I changed as fs state backend, I configured at flink-conf.yaml below:

State.backend: filesystem

State.backend.fs.checkpointdir: file:///xxx/checkpoints

I found there are many checkpoints saved in the file.

But the watermarks also will be stoppped.

And there’re two enviroments.

The local running well, and the problem occurred on site.

There’re two differences:

1. Data amount very small in local and very huge on site.

2. On site configurations:

Akka.ask.timeout: 2min

Akka.lookup.timeout: 2min

Akka.client.timeout: 3min

Akka.log.lifecycle.events: on

Akka.tcp.timeout: 60s

But not configured at local.

So there’re any influences?

Please help me…

Yours,

September

发件人: [hidden email]
发送时间: 2018年10月11日 14:56
收件人: [hidden email]
抄送: [hidden email]
主题: Re: No data issued by flink window after a few hours

Hi gongsen,

Have you used event time as time semantics? If so, then the possible problem is related to watermark.

Since I don't know the details of your program, it's hard to make a conclusion. You can check if your watermark is firing normally.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月11日周四下午12:12写道：

Hi，

I changed as below configurations，and it looks fine when job started.

But there’re no results issued when window ends after running about six hours, and no errors and exceptions.

How can I position the question?

Yours,

September

发件人: 潘功森 <[hidden email]>
发送时间: Wednesday, October 10, 2018 2:44:48 PM
收件人: vino yang
抄送: user
主题: 答复: No data issued by flink window after a few hours
Hi，

Cause default state size in one hour is too small，and the max window size is 24 hours, so I used 500M.
MemoryStateBackend stateBackend = new MemoryStateBackend(MAX_STATE_SIZE);//500M
env.setStateBackend(stateBackend);
And I found Irrespective of the configured maximal state size, the state cannot be larger than the akka frame size.

So I add a config in flink-comf.yaml:

akka.framesize: 524288000b

What else do I have to pay attention to?

Yours,

September
发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 11:45:31 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

I saw the exception image you provided. Based on the exception message, it seems you used the default max state size (5MB).

You can specify the max state size to override the default value. Try :

MemoryStateBackend stateBackend = new MemoryStateBackend(theSizeOfBytes);

Please note that you need to reserve enough memory for Flink.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:36写道：

Please have a look about my last mail.

When the cached window data is too large, how?

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 11:33:48 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Did you mean "computer momery" referring to Memory Statebackend?

The Flink window mechanism is internally based on State, and this is done for fault tolerance.

If you introduce external storage, it will break its design and bring other problems.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:02写道：

Hi,

"ram to cache the distinct data about sliding window" means I used computer momery not the third part db to cache the data need used in window.

“the data need used in window” means ：such as the sliding window is 1 hour, and I need to count the distinct users, I need to cache the user id about one hour.

Cause there’re no related errors.

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 10:49:43 AM
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Can you explain what "ram to cache the distinct data about sliding window" mean?

The information you provide is too small and will not help others to help you analyze the problem and provide advice.

In addition, regarding the usage of Flink related issues, please only send mail to the user mailing list.

The dev mailing list is mainly used to discuss development related issues.

Thanks vino.

? ?? <[hidden email]> 于2018年10月10日周三上午10:37写道：

Hi all,
I used flink window, and when the job begins, we could get the results of windiow.But there’re no results issued after a few hours.
I found the job is still running and no errors, and the data not used window all can be issued.
By the way, I used Flink 1.3.2 and ram to cache the distinct data about sliding window.

Yours,
September

潘功森

答复: No data issued by flink window after a few hours

Please look at the mail below, the others were out of data cause the bad network.

Yours,

September

发件人: 潘功森 <[hidden email]>
发送时间: Friday, October 12, 2018 11:05:49 AM
收件人: vino yang
抄送: user
主题: 答复: No data issued by flink window after a few hours

Hi,

I found the pictures maybe too big and the net here not so good, so the mail I wrote is not sent sucsessfully last night.

Yes, I used event time.

I found watermarks fired normally when the job started, but it stopped and no changed after running hours.

And I changed as fs state backend, I configured at flink-conf.yaml below:

State.backend: filesystem

State.backend.fs.checkpointdir: file:///xxx/checkpoints

I found there are many checkpoints saved in the file.

But the watermarks also will be stoppped.

And there’re two enviroments.

The local running well, and the problem occurred on site.

There’re two differences:

1. Data amount very small in local and very huge on site.

2. On site configurations:

Akka.ask.timeout: 2min

Akka.lookup.timeout: 2min

Akka.client.timeout: 3min

Akka.log.lifecycle.events: on

Akka.tcp.timeout: 60s

But not configured at local.

So there’re any influences?

Please help me…

Yours,

September

发件人: [hidden email]
发送时间: 2018年10月11日 14:56
收件人: [hidden email]
抄送: [hidden email]
主题: Re: No data issued by flink window after a few hours

Hi gongsen,

Have you used event time as time semantics? If so, then the possible problem is related to watermark.

Since I don't know the details of your program, it's hard to make a conclusion. You can check if your watermark is firing normally.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月11日周四下午12:12写道：

Hi，

I changed as below configurations，and it looks fine when job started.

But there’re no results issued when window ends after running about six hours, and no errors and exceptions.

How can I position the question?

Yours,

September

发件人: 潘功森 <[hidden email]>
发送时间: Wednesday, October 10, 2018 2:44:48 PM
收件人: vino yang
抄送: user
主题: 答复: No data issued by flink window after a few hours
Hi，

Cause default state size in one hour is too small，and the max window size is 24 hours, so I used 500M.
MemoryStateBackend stateBackend = new MemoryStateBackend(MAX_STATE_SIZE);//500M
env.setStateBackend(stateBackend);
And I found Irrespective of the configured maximal state size, the state cannot be larger than the akka frame size.

So I add a config in flink-comf.yaml:

akka.framesize: 524288000b

What else do I have to pay attention to?

Yours,

September
发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 11:45:31 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

I saw the exception image you provided. Based on the exception message, it seems you used the default max state size (5MB).

You can specify the max state size to override the default value. Try :

MemoryStateBackend stateBackend = new MemoryStateBackend(theSizeOfBytes);

Please note that you need to reserve enough memory for Flink.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:36写道：

Please have a look about my last mail.

When the cached window data is too large, how?

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 11:33:48 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Did you mean "computer momery" referring to Memory Statebackend?

The Flink window mechanism is internally based on State, and this is done for fault tolerance.

If you introduce external storage, it will break its design and bring other problems.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:02写道：

Hi,

"ram to cache the distinct data about sliding window" means I used computer momery not the third part db to cache the data need used in window.

“the data need used in window” means ：such as the sliding window is 1 hour, and I need to count the distinct users, I need to cache the user id about one hour.

Cause there’re no related errors.

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 10:49:43 AM
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Can you explain what "ram to cache the distinct data about sliding window" mean?

The information you provide is too small and will not help others to help you analyze the problem and provide advice.

In addition, regarding the usage of Flink related issues, please only send mail to the user mailing list.

The dev mailing list is mainly used to discuss development related issues.

Thanks vino.

? ?? <[hidden email]> 于2018年10月10日周三上午10:37写道：

vino yang

Re: No data issued by flink window after a few hours

Hi gongsen,

Since you are running well locally, it should not be a configuration issue.

You can refer to the Flink UI to see if your checkpoint is delayed.

I hope that you can follow the instructions in the documentation[1] and provide some screenshots that will help the community help locate the problem.

Thanks, vino.

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/debugging_event_time.html#monitoring-current-event-time

潘功森 <[hidden email]> 于2018年10月12日周五上午11:10写道：

Please look at the mail below, the others were out of data cause the bad network.

Yours,

September

发件人: 潘功森 <[hidden email]>
发送时间: Friday, October 12, 2018 11:05:49 AM
收件人: vino yang
抄送: user
主题: 答复: No data issued by flink window after a few hours
Hi,

I found the pictures maybe too big and the net here not so good, so the mail I wrote is not sent sucsessfully last night.

Yes, I used event time.

I found watermarks fired normally when the job started, but it stopped and no changed after running hours.

And I changed as fs state backend, I configured at flink-conf.yaml below:

State.backend: filesystem

State.backend.fs.checkpointdir: file:///xxx/checkpoints

I found there are many checkpoints saved in the file.

But the watermarks also will be stoppped.

And there’re two enviroments.

The local running well, and the problem occurred on site.

There’re two differences:

1.     Data amount very small in local and very huge on site.

2.     On site configurations:

Akka.ask.timeout: 2min

Akka.lookup.timeout: 2min

Akka.client.timeout: 3min

Akka.log.lifecycle.events: on

Akka.tcp.timeout: 60s

But not configured at local.

So there’re any influences?

Please help me…



Yours,

September

发件人: [hidden email]
发送时间: 2018年10月11日 14:56
收件人: [hidden email]
抄送: [hidden email]
主题: Re: No data issued by flink window after a few hours

Hi gongsen,

Have you used event time as time semantics? If so, then the possible problem is related to watermark.

Since I don't know the details of your program, it's hard to make a conclusion. You can check if your watermark is firing normally.

Thanks, vino.
潘功森 <[hidden email]> 于2018年10月11日周四下午12:12写道：
Hi，

I changed as below configurations，and it looks fine when job started.

But there’re no results issued when window ends after running about six hours, and no errors and exceptions.

How can I position the question?

Yours,

September

发件人: 潘功森 <[hidden email]>
发送时间: Wednesday, October 10, 2018 2:44:48 PM
收件人: vino yang
抄送: user
主题: 答复: No data issued by flink window after a few hours
Hi，

Cause default state size in one hour is too small，and the max window size is 24 hours, so I used 500M.
MemoryStateBackend stateBackend = new MemoryStateBackend(MAX_STATE_SIZE);//500M
env.setStateBackend(stateBackend);
And I found Irrespective of the configured maximal state size, the state cannot be larger than the akka frame size.

So I add a config in flink-comf.yaml:

akka.framesize: 524288000b

What else do I have to pay attention to?

Yours,

September
发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 11:45:31 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

I saw the exception image you provided. Based on the exception message, it seems you used the default max state size (5MB).

You can specify the max state size to override the default value. Try :

MemoryStateBackend stateBackend = new MemoryStateBackend(theSizeOfBytes);

Please note that you need to reserve enough memory for Flink.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:36写道：

Please have a look about my last mail.

When the cached window data is too large, how?

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 11:33:48 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Did you mean "computer momery" referring to Memory Statebackend?

The Flink window mechanism is internally based on State, and this is done for fault tolerance.

If you introduce external storage, it will break its design and bring other problems.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:02写道：

Hi,

"ram to cache the distinct data about sliding window" means I used computer momery not the third part db to cache the data need used in window.

“the data need used in window” means ：such as the sliding window is 1 hour, and I need to count the distinct users, I need to cache the user id about one hour.

Cause there’re no related errors.

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 10:49:43 AM
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Can you explain what "ram to cache the distinct data about sliding window" mean?

The information you provide is too small and will not help others to help you analyze the problem and provide advice.

In addition, regarding the usage of Flink related issues, please only send mail to the user mailing list.

The dev mailing list is mainly used to discuss development related issues.

Thanks vino.

? ?? <[hidden email]> 于2018年10月10日周三上午10:37写道：
Hi all,
I used flink window, and when the job begins, we could get the results of windiow.But there’re no results issued after a few hours.
I found the job is still running and no errors, and the data not used window all can be issued.
By the way, I used Flink 1.3.2 and ram to cache the distinct data about sliding window.

Yours,
September

0CCF36EC30E5474594E9BC3BFDD09BC3.png (212 bytes) Download Attachment

B48BCB6811C14A918C4846147B465745.png (212 bytes) Download Attachment

ABAF44AD8564406EBA4D4A3DADA1BA8C.png (212 bytes) Download Attachment

0CCF36EC30E5474594E9BC3BFDD09BC3.png (212 bytes) Download Attachment

潘功森

答复: No data issued by flink window after a few hours

Hi,

I chose the ten seconds and fifteen minutes windows, and it looks fine when running few minutes as below pictures.

But I found there‘re no changes between 13:34 and 13:36, but it also could be issued , and I can tolerate it.

If no results issued after windows ends, I will take pictures again for you.

Yours,

September

发件人: [hidden email]
发送时间: 2018年10月12日 13:04
主题: Re: No data issued by flink window after a few hours

Hi gongsen,

Since you are running well locally, it should not be a configuration issue.

You can refer to the Flink UI to see if your checkpoint is delayed.

I hope that you can follow the instructions in the documentation[1] and provide some screenshots that will help the community help locate the problem.

Thanks, vino.

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/debugging_event_time.html#monitoring-current-event-time

潘功森 <[hidden email]> 于2018年10月12日周五上午11:10写道：

Please look at the mail below, the others were out of data cause the bad network.

Yours,

September

发件人: 潘功森 <[hidden email]>
发送时间: Friday, October 12, 2018 11:05:49 AM
收件人: vino yang
抄送: user
主题: 答复: No data issued by flink window after a few hours
Hi,

I found the pictures maybe too big and the net here not so good, so the mail I wrote is not sent sucsessfully last night.

Yes, I used event time.

I found watermarks fired normally when the job started, but it stopped and no changed after running hours.

And I changed as fs state backend, I configured at flink-conf.yaml below:

State.backend: filesystem

State.backend.fs.checkpointdir: file:///xxx/checkpoints

I found there are many checkpoints saved in the file.

But the watermarks also will be stoppped.

And there’re two enviroments.

The local running well, and the problem occurred on site.

There’re two differences:

1.     Data amount very small in local and very huge on site.

2.     On site configurations:

Akka.ask.timeout: 2min

Akka.lookup.timeout: 2min

Akka.client.timeout: 3min

Akka.log.lifecycle.events: on

Akka.tcp.timeout: 60s

But not configured at local.

So there’re any influences?

Please help me…



Yours,

September

发件人: [hidden email]
发送时间: 2018年10月11日 14:56
收件人: [hidden email]
抄送: [hidden email]
主题: Re: No data issued by flink window after a few hours

Hi gongsen,

Have you used event time as time semantics? If so, then the possible problem is related to watermark.

Since I don't know the details of your program, it's hard to make a conclusion. You can check if your watermark is firing normally.

Thanks, vino.
潘功森 <[hidden email]> 于2018年10月11日周四下午12:12写道：
Hi，

I changed as below configurations，and it looks fine when job started.

But there’re no results issued when window ends after running about six hours, and no errors and exceptions.

How can I position the question?

Yours,

September

发件人: 潘功森 <[hidden email]>
发送时间: Wednesday, October 10, 2018 2:44:48 PM
收件人: vino yang
抄送: user
主题: 答复: No data issued by flink window after a few hours
Hi，

Cause default state size in one hour is too small，and the max window size is 24 hours, so I used 500M.
MemoryStateBackend stateBackend = new MemoryStateBackend(MAX_STATE_SIZE);//500M
env.setStateBackend(stateBackend);
And I found Irrespective of the configured maximal state size, the state cannot be larger than the akka frame size.

So I add a config in flink-comf.yaml:

akka.framesize: 524288000b

What else do I have to pay attention to?

Yours,

September
发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 11:45:31 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

I saw the exception image you provided. Based on the exception message, it seems you used the default max state size (5MB).

You can specify the max state size to override the default value. Try :

MemoryStateBackend stateBackend = new MemoryStateBackend(theSizeOfBytes);

Please note that you need to reserve enough memory for Flink.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:36写道：

Please have a look about my last mail.

When the cached window data is too large, how?

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 11:33:48 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Did you mean "computer momery" referring to Memory Statebackend?

The Flink window mechanism is internally based on State, and this is done for fault tolerance.

If you introduce external storage, it will break its design and bring other problems.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:02写道：

Hi,

"ram to cache the distinct data about sliding window" means I used computer momery not the third part db to cache the data need used in window.

“the data need used in window” means ：such as the sliding window is 1 hour, and I need to count the distinct users, I need to cache the user id about one hour.

Cause there’re no related errors.

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 10:49:43 AM
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Can you explain what "ram to cache the distinct data about sliding window" mean?

The information you provide is too small and will not help others to help you analyze the problem and provide advice.

In addition, regarding the usage of Flink related issues, please only send mail to the user mailing list.

The dev mailing list is mainly used to discuss development related issues.

Thanks vino.

? ?? <[hidden email]> 于2018年10月10日周三上午10:37写道：
Hi all,
I used flink window, and when the job begins, we could get the results of windiow.But there’re no results issued after a few hours.
I found the job is still running and no errors, and the data not used window all can be issued.
By the way, I used Flink 1.3.2 and ram to cache the distinct data about sliding window.

Yours,
September

潘功森

答复: No data issued by flink window after a few hours

Hi,

It stopped to update now.

Yours,

September

发件人: 潘功森 <[hidden email]>
发送时间: Friday, October 12, 2018 1:49:58 PM
收件人: vino yang
抄送: user; [hidden email]
主题: 答复: No data issued by flink window after a few hours

Hi,

I chose the ten seconds and fifteen minutes windows, and it looks fine when running few minutes as below pictures.

But I found there‘re no changes between 13:34 and 13:36, but it also could be issued , and I can tolerate it.

If no results issued after windows ends, I will take pictures again for you.

Yours,

September

发件人: [hidden email]
发送时间: 2018年10月12日 13:04
主题: Re: No data issued by flink window after a few hours

Hi gongsen,

Since you are running well locally, it should not be a configuration issue.

You can refer to the Flink UI to see if your checkpoint is delayed.

I hope that you can follow the instructions in the documentation[1] and provide some screenshots that will help the community help locate the problem.

Thanks, vino.

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/debugging_event_time.html#monitoring-current-event-time

潘功森 <[hidden email]> 于2018年10月12日周五上午11:10写道：

Please look at the mail below, the others were out of data cause the bad network.

Yours,

September

发件人: 潘功森 <[hidden email]>
发送时间: Friday, October 12, 2018 11:05:49 AM
收件人: vino yang
抄送: user
主题: 答复: No data issued by flink window after a few hours
Hi,

I found the pictures maybe too big and the net here not so good, so the mail I wrote is not sent sucsessfully last night.

Yes, I used event time.

I found watermarks fired normally when the job started, but it stopped and no changed after running hours.

And I changed as fs state backend, I configured at flink-conf.yaml below:

State.backend: filesystem

State.backend.fs.checkpointdir: file:///xxx/checkpoints

I found there are many checkpoints saved in the file.

But the watermarks also will be stoppped.

And there’re two enviroments.

The local running well, and the problem occurred on site.

There’re two differences:

1.     Data amount very small in local and very huge on site.

2.     On site configurations:

Akka.ask.timeout: 2min

Akka.lookup.timeout: 2min

Akka.client.timeout: 3min

Akka.log.lifecycle.events: on

Akka.tcp.timeout: 60s

But not configured at local.

So there’re any influences?

Please help me…



Yours,

September

发件人: [hidden email]
发送时间: 2018年10月11日 14:56
收件人: [hidden email]
抄送: [hidden email]
主题: Re: No data issued by flink window after a few hours

Hi gongsen,

Have you used event time as time semantics? If so, then the possible problem is related to watermark.

Since I don't know the details of your program, it's hard to make a conclusion. You can check if your watermark is firing normally.

Thanks, vino.
潘功森 <[hidden email]> 于2018年10月11日周四下午12:12写道：
Hi，

I changed as below configurations，and it looks fine when job started.

But there’re no results issued when window ends after running about six hours, and no errors and exceptions.

How can I position the question?

Yours,

September

发件人: 潘功森 <[hidden email]>
发送时间: Wednesday, October 10, 2018 2:44:48 PM
收件人: vino yang
抄送: user
主题: 答复: No data issued by flink window after a few hours
Hi，

Cause default state size in one hour is too small，and the max window size is 24 hours, so I used 500M.
MemoryStateBackend stateBackend = new MemoryStateBackend(MAX_STATE_SIZE);//500M
env.setStateBackend(stateBackend);
And I found Irrespective of the configured maximal state size, the state cannot be larger than the akka frame size.

So I add a config in flink-comf.yaml:

akka.framesize: 524288000b

What else do I have to pay attention to?

Yours,

September
发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 11:45:31 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

I saw the exception image you provided. Based on the exception message, it seems you used the default max state size (5MB).

You can specify the max state size to override the default value. Try :

MemoryStateBackend stateBackend = new MemoryStateBackend(theSizeOfBytes);

Please note that you need to reserve enough memory for Flink.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:36写道：

Please have a look about my last mail.

When the cached window data is too large, how?

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 11:33:48 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Did you mean "computer momery" referring to Memory Statebackend?

The Flink window mechanism is internally based on State, and this is done for fault tolerance.

If you introduce external storage, it will break its design and bring other problems.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:02写道：

Hi,

"ram to cache the distinct data about sliding window" means I used computer momery not the third part db to cache the data need used in window.

“the data need used in window” means ：such as the sliding window is 1 hour, and I need to count the distinct users, I need to cache the user id about one hour.

Cause there’re no related errors.

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 10:49:43 AM
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Can you explain what "ram to cache the distinct data about sliding window" mean?

The information you provide is too small and will not help others to help you analyze the problem and provide advice.

In addition, regarding the usage of Flink related issues, please only send mail to the user mailing list.

The dev mailing list is mainly used to discuss development related issues.

Thanks vino.

? ?? <[hidden email]> 于2018年10月10日周三上午10:37写道：
Hi all,
I used flink window, and when the job begins, we could get the results of windiow.But there’re no results issued after a few hours.
I found the job is still running and no errors, and the data not used window all can be issued.
By the way, I used Flink 1.3.2 and ram to cache the distinct data about sliding window.

Yours,
September

潘功森

答复: No data issued by flink window after a few hours

Hi vino,

I found when job run a period of time, the currentLowWatermark will be like a time far from now like 1539618594000 (2018/10/15 23:49:54), but now is 2018/10/15 14:29:00 .

I checked event time from data, there’re no event time far from now.

Yours,

September

发件人: 潘功森 <[hidden email]>
发送时间: Friday, October 12, 2018 2:05:33 PM
收件人: vino yang
抄送: user; [hidden email]
主题: 答复: No data issued by flink window after a few hours

Hi,

It stopped to update now.

Yours,

September

Hi,

I chose the ten seconds and fifteen minutes windows, and it looks fine when running few minutes as below pictures.

But I found there‘re no changes between 13:34 and 13:36, but it also could be issued , and I can tolerate it.

If no results issued after windows ends, I will take pictures again for you.

Yours,

September

发件人: [hidden email]
发送时间: 2018年10月12日 13:04
主题: Re: No data issued by flink window after a few hours

Hi gongsen,

Since you are running well locally, it should not be a configuration issue.

You can refer to the Flink UI to see if your checkpoint is delayed.

I hope that you can follow the instructions in the documentation[1] and provide some screenshots that will help the community help locate the problem.

Thanks, vino.

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/debugging_event_time.html#monitoring-current-event-time

潘功森 <[hidden email]> 于2018年10月12日周五上午11:10写道：

Please look at the mail below, the others were out of data cause the bad network.

Yours,

September

发件人: 潘功森 <[hidden email]>
发送时间: Friday, October 12, 2018 11:05:49 AM
收件人: vino yang
抄送: user
主题: 答复: No data issued by flink window after a few hours
Hi,

I found the pictures maybe too big and the net here not so good, so the mail I wrote is not sent sucsessfully last night.

Yes, I used event time.

I found watermarks fired normally when the job started, but it stopped and no changed after running hours.

And I changed as fs state backend, I configured at flink-conf.yaml below:

State.backend: filesystem

State.backend.fs.checkpointdir: file:///xxx/checkpoints

I found there are many checkpoints saved in the file.

But the watermarks also will be stoppped.

And there’re two enviroments.

The local running well, and the problem occurred on site.

There’re two differences:

1.     Data amount very small in local and very huge on site.

2.     On site configurations:

Akka.ask.timeout: 2min

Akka.lookup.timeout: 2min

Akka.client.timeout: 3min

Akka.log.lifecycle.events: on

Akka.tcp.timeout: 60s

But not configured at local.

So there’re any influences?

Please help me…



Yours,

September

发件人: [hidden email]
发送时间: 2018年10月11日 14:56
收件人: [hidden email]
抄送: [hidden email]
主题: Re: No data issued by flink window after a few hours

Hi gongsen,

Have you used event time as time semantics? If so, then the possible problem is related to watermark.

Since I don't know the details of your program, it's hard to make a conclusion. You can check if your watermark is firing normally.

Thanks, vino.
潘功森 <[hidden email]> 于2018年10月11日周四下午12:12写道：
Hi，

I changed as below configurations，and it looks fine when job started.

But there’re no results issued when window ends after running about six hours, and no errors and exceptions.

How can I position the question?

Yours,

September

发件人: 潘功森 <[hidden email]>
发送时间: Wednesday, October 10, 2018 2:44:48 PM
收件人: vino yang
抄送: user
主题: 答复: No data issued by flink window after a few hours
Hi，

Cause default state size in one hour is too small，and the max window size is 24 hours, so I used 500M.
MemoryStateBackend stateBackend = new MemoryStateBackend(MAX_STATE_SIZE);//500M
env.setStateBackend(stateBackend);
And I found Irrespective of the configured maximal state size, the state cannot be larger than the akka frame size.

So I add a config in flink-comf.yaml:

akka.framesize: 524288000b

What else do I have to pay attention to?

Yours,

September
发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 11:45:31 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

I saw the exception image you provided. Based on the exception message, it seems you used the default max state size (5MB).

You can specify the max state size to override the default value. Try :

MemoryStateBackend stateBackend = new MemoryStateBackend(theSizeOfBytes);

Please note that you need to reserve enough memory for Flink.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:36写道：

Please have a look about my last mail.

When the cached window data is too large, how?

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 11:33:48 AM
收件人: [hidden email]
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Did you mean "computer momery" referring to Memory Statebackend?

The Flink window mechanism is internally based on State, and this is done for fault tolerance.

If you introduce external storage, it will break its design and bring other problems.

Thanks, vino.

潘功森 <[hidden email]> 于2018年10月10日周三上午11:02写道：

Hi,

"ram to cache the distinct data about sliding window" means I used computer momery not the third part db to cache the data need used in window.

“the data need used in window” means ：such as the sliding window is 1 hour, and I need to count the distinct users, I need to cache the user id about one hour.

Cause there’re no related errors.

Yours,

September

发件人: vino yang <[hidden email]>
发送时间: Wednesday, October 10, 2018 10:49:43 AM
抄送: user
主题: Re: No data issued by flink window after a few hours

Hi,

Can you explain what "ram to cache the distinct data about sliding window" mean?

The information you provide is too small and will not help others to help you analyze the problem and provide advice.

In addition, regarding the usage of Flink related issues, please only send mail to the user mailing list.

The dev mailing list is mainly used to discuss development related issues.

Thanks vino.

? ?? <[hidden email]> 于2018年10月10日周三上午10:37写道：
Hi all,
I used flink window, and when the job begins, we could get the results of windiow.But there’re no results issued after a few hours.
I found the job is still running and no errors, and the data not used window all can be issued.
By the way, I used Flink 1.3.2 and ram to cache the distinct data about sliding window.

Yours,
September