Re: MODERATE for [e-mail]

[Jordan Kuan]

Dear Flink Dev Team

I have encountered a problem and can't find any solution in Google.

And I have created a thread in stackoverflow.com but no response.

https://stackoverflow.com/questions/47123371/flink-windows-ha

I would really appreciate it if you could give some suggestions to me.

On Tue, Nov 7, 2017 at 1:53 AM, Robert Metzger <[hidden email]> wrote:

Hi,

I would suggest to send your question to the [hidden email] list (make sure to subscribe first)

---------- Forwarded message ----------
From: <[hidden email]>
Date: Mon, Nov 6, 2017 at 5:23 PM
Subject: MODERATE for [hidden email]
To:
Cc: dev-allow-tc.1509985381.dcccgaimcaiiefbkiapi-jordan.kuan=[hidden email]



To approve:
   [hidden email]
To reject:
   [hidden email]
To give a reason to reject:
%%% Start comment
%%% End comment



---------- Forwarded message ----------
From: Jordan Kuan <[hidden email]>
To: [hidden email]
Cc: 
Bcc: 
Date: Tue, 7 Nov 2017 00:22:53 +0800
Subject: Flink Windows HA Issue.

Dear Flink Dev Team

I have encountered a problem and can't find any solution in Google.

And I have created a thread in stackoverflow.com but no response.

https://stackoverflow.com/questions/47123371/flink-windows-ha

I would really appreciate it if you could give some suggestions to me.

--

[Ufuk Celebi]

As answered by David on SO, the files need to be accessible by all
nodes. In your setup this seems not to be the case, therefore it won't
work. You need a distributed file system (e.g. NFS or HDFS) or object
store (e.g. S3) that is accessible from all nodes.

– Ufuk


On Tue, Nov 7, 2017 at 3:34 AM, Jordan Kuan <[hidden email]> wrote:

> Dear Flink Dev Team
>
> I have encountered a problem and can't find any solution in Google.
> And I have created a thread in stackoverflow.com but no response.
> https://stackoverflow.com/questions/47123371/flink-windows-ha
>
> I would really appreciate it if you could give some suggestions to me.
>
> Thanks,
>
> Jordan
>
> On Tue, Nov 7, 2017 at 1:53 AM, Robert Metzger <[hidden email]> wrote:
>>
>> Hi,
>> I would suggest to send your question to the [hidden email] list
>> (make sure to subscribe first)
>>
>> ---------- Forwarded message ----------
>> From: <[hidden email]>
>> Date: Mon, Nov 6, 2017 at 5:23 PM
>> Subject: MODERATE for [hidden email]
>> To:
>> Cc:
>> dev-allow-tc.1509985381.dcccgaimcaiiefbkiapi-jordan.kuan=[hidden email]
>>
>>
>>
>> To approve:
>>    [hidden email]
>> To reject:
>>    [hidden email]
>> To give a reason to reject:
>> %%% Start comment
>> %%% End comment
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: Jordan Kuan <[hidden email]>
>> To: [hidden email]
>> Cc:
>> Bcc:
>> Date: Tue, 7 Nov 2017 00:22:53 +0800
>> Subject: Flink Windows HA Issue.
>> Dear Flink Dev Team
>>
>> I have encountered a problem and can't find any solution in Google.
>> And I have created a thread in stackoverflow.com but no response.
>> https://stackoverflow.com/questions/47123371/flink-windows-ha
>>
>> I would really appreciate it if you could give some suggestions to me.
>>
>> Thanks,
>>
>> Jordan
>>
>>
>>
>
>
>
> --
> Best Regards,
> Jordan Kuan

[Jordan Kuan]

Dear Ufuk,

Thank you for your reply.

All the cluster Flink servers are able to access network drive, and it mapped as drive Y in all nodes.
Do I need to provide more information?

Thanks,
Jordan

> On 7 Nov 2017, at 6:36 PM, Ufuk Celebi <[hidden email]> wrote:
>
> As answered by David on SO, the files need to be accessible by all
> nodes. In your setup this seems not to be the case, therefore it won't
> work. You need a distributed file system (e.g. NFS or HDFS) or object
> store (e.g. S3) that is accessible from all nodes.
>
> – Ufuk
>
>
> On Tue, Nov 7, 2017 at 3:34 AM, Jordan Kuan <[hidden email]> wrote:
>> Dear Flink Dev Team
>>
>> I have encountered a problem and can't find any solution in Google.
>> And I have created a thread in stackoverflow.com but no response.
>> https://stackoverflow.com/questions/47123371/flink-windows-ha
>>
>> I would really appreciate it if you could give some suggestions to me.
>>
>> Thanks,
>>
>> Jordan
>>
>> On Tue, Nov 7, 2017 at 1:53 AM, Robert Metzger <[hidden email]> wrote:
>>>
>>> Hi,
>>> I would suggest to send your question to the [hidden email] list
>>> (make sure to subscribe first)
>>>
>>> ---------- Forwarded message ----------
>>> From: <[hidden email]>
>>> Date: Mon, Nov 6, 2017 at 5:23 PM
>>> Subject: MODERATE for [hidden email]
>>> To:
>>> Cc:
>>> dev-allow-tc.1509985381.dcccgaimcaiiefbkiapi-jordan.kuan=[hidden email]
>>>
>>>
>>>
>>> To approve:
>>>   [hidden email]
>>> To reject:
>>>   [hidden email]
>>> To give a reason to reject:
>>> %%% Start comment
>>> %%% End comment
>>>
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Jordan Kuan <[hidden email]>
>>> To: [hidden email]
>>> Cc:
>>> Bcc:
>>> Date: Tue, 7 Nov 2017 00:22:53 +0800
>>> Subject: Flink Windows HA Issue.
>>> Dear Flink Dev Team
>>>
>>> I have encountered a problem and can't find any solution in Google.
>>> And I have created a thread in stackoverflow.com but no response.
>>> https://stackoverflow.com/questions/47123371/flink-windows-ha
>>>
>>> I would really appreciate it if you could give some suggestions to me.
>>>
>>> Thanks,
>>>
>>> Jordan
>>>
>>>
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Jordan Kuan

[Ufuk Celebi]

Hey Jordan,

yeah, that should just work. Check out the state backend configuration
here: https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/state/state_backends.html

– Ufuk

On Tue, Nov 7, 2017 at 11:44 AM, Jordan Kuan <[hidden email]> wrote:

> Dear Ufuk,
>
> Thank you for your reply.
>
> All the cluster Flink servers are able to access network drive, and it mapped as drive Y in all nodes.
> Do I need to provide more information?
>
> Thanks,
> Jordan
>
>
>> On 7 Nov 2017, at 6:36 PM, Ufuk Celebi <[hidden email]> wrote:
>>
>> As answered by David on SO, the files need to be accessible by all
>> nodes. In your setup this seems not to be the case, therefore it won't
>> work. You need a distributed file system (e.g. NFS or HDFS) or object
>> store (e.g. S3) that is accessible from all nodes.
>>
>> – Ufuk
>>
>>
>> On Tue, Nov 7, 2017 at 3:34 AM, Jordan Kuan <[hidden email]> wrote:
>>> Dear Flink Dev Team
>>>
>>> I have encountered a problem and can't find any solution in Google.
>>> And I have created a thread in stackoverflow.com but no response.
>>> https://stackoverflow.com/questions/47123371/flink-windows-ha
>>>
>>> I would really appreciate it if you could give some suggestions to me.
>>>
>>> Thanks,
>>>
>>> Jordan
>>>
>>> On Tue, Nov 7, 2017 at 1:53 AM, Robert Metzger <[hidden email]> wrote:
>>>>
>>>> Hi,
>>>> I would suggest to send your question to the [hidden email] list
>>>> (make sure to subscribe first)
>>>>
>>>> ---------- Forwarded message ----------
>>>> From: <[hidden email]>
>>>> Date: Mon, Nov 6, 2017 at 5:23 PM
>>>> Subject: MODERATE for [hidden email]
>>>> To:
>>>> Cc:
>>>> dev-allow-tc.1509985381.dcccgaimcaiiefbkiapi-jordan.kuan=[hidden email]
>>>>
>>>>
>>>>
>>>> To approve:
>>>>   [hidden email]
>>>> To reject:
>>>>   [hidden email]
>>>> To give a reason to reject:
>>>> %%% Start comment
>>>> %%% End comment
>>>>
>>>>
>>>>
>>>> ---------- Forwarded message ----------
>>>> From: Jordan Kuan <[hidden email]>
>>>> To: [hidden email]
>>>> Cc:
>>>> Bcc:
>>>> Date: Tue, 7 Nov 2017 00:22:53 +0800
>>>> Subject: Flink Windows HA Issue.
>>>> Dear Flink Dev Team
>>>>
>>>> I have encountered a problem and can't find any solution in Google.
>>>> And I have created a thread in stackoverflow.com but no response.
>>>> https://stackoverflow.com/questions/47123371/flink-windows-ha
>>>>
>>>> I would really appreciate it if you could give some suggestions to me.
>>>>
>>>> Thanks,
>>>>
>>>> Jordan
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Jordan Kuan
>

[Jordan Kuan]

Dear Ufuk,

Sorry, I still can’t get that work, I have double checked all the nodes can access the mapped drive Y. 

Here is my state backend configuration, and it writes files to local C drive.

“state.backend.fs.checkpointdir: file:///cygdrive/Y/flink-checkpoint-dir

state.checkpoints.dir: /cygdrive/Y/flink-checkpoints

state.backend: filesystem

high-availability.storageDir: file:///cygdrive/Y/flink-recovery

"

I have attached the flink-conf.yaml and screenshots, Is here any wrong setting? 

Thanks,

Jordan

On 7 Nov 2017, at 6:49 PM, Ufuk Celebi <[hidden email]> wrote:

Hey Jordan,

yeah, that should just work. Check out the state backend configuration
here: https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/state/state_backends.html

– Ufuk

On Tue, Nov 7, 2017 at 11:44 AM, Jordan Kuan <[hidden email]> wrote:

Dear Ufuk,

Thank you for your reply.

All the cluster Flink servers are able to access network drive, and it mapped as drive Y in all nodes.
Do I need to provide more information?

Thanks,
Jordan

On 7 Nov 2017, at 6:36 PM, Ufuk Celebi <[hidden email]> wrote:

As answered by David on SO, the files need to be accessible by all
nodes. In your setup this seems not to be the case, therefore it won't
work. You need a distributed file system (e.g. NFS or HDFS) or object
store (e.g. S3) that is accessible from all nodes.

– Ufuk


On Tue, Nov 7, 2017 at 3:34 AM, Jordan Kuan <[hidden email]> wrote:

Dear Flink Dev Team

I have encountered a problem and can't find any solution in Google.
And I have created a thread in stackoverflow.com but no response.
https://stackoverflow.com/questions/47123371/flink-windows-ha

I would really appreciate it if you could give some suggestions to me.

Thanks,

Jordan

On Tue, Nov 7, 2017 at 1:53 AM, Robert Metzger <[hidden email]> wrote:

Hi,
I would suggest to send your question to the [hidden email] list
(make sure to subscribe first)

---------- Forwarded message ----------
From: <[hidden email]>
Date: Mon, Nov 6, 2017 at 5:23 PM
Subject: MODERATE for [hidden email]
To:
Cc:
dev-allow-tc.1509985381.dcccgaimcaiiefbkiapi-jordan.kuan=[hidden email]



To approve:
 [hidden email]
To reject:
 [hidden email]
To give a reason to reject:
%%% Start comment
%%% End comment



---------- Forwarded message ----------
From: Jordan Kuan <[hidden email]>
To: [hidden email]
Cc:
Bcc:
Date: Tue, 7 Nov 2017 00:22:53 +0800
Subject: Flink Windows HA Issue.
Dear Flink Dev Team

I have encountered a problem and can't find any solution in Google.
And I have created a thread in stackoverflow.com but no response.
https://stackoverflow.com/questions/47123371/flink-windows-ha

I would really appreciate it if you could give some suggestions to me.

Thanks,

Jordan

--
Best Regards,
Jordan Kuan

[Chesnay Schepler]

Have you tried removing the "cygdrive" portion from the path?

Something along the lines of

“state.backend.fs.checkpointdir: file:///Y:/flink-checkpoint-dir

state.checkpoints.dir: Y:/flink-checkpoints

state.backend: filesystem

high-availability.storageDir: file:///Y:/flink-recovery

"

On 07.11.2017 12:36, Jordan Kuan wrote:

Dear Ufuk,

Sorry, I still can’t get that work, I have double checked all the nodes can access the mapped drive Y. 

Here is my state backend configuration, and it writes files to local C drive.

“state.backend.fs.checkpointdir: file:///cygdrive/Y/flink-checkpoint-dir

state.checkpoints.dir: /cygdrive/Y/flink-checkpoints

state.backend: filesystem

high-availability.storageDir: file:///cygdrive/Y/flink-recovery

"

I have attached the flink-conf.yaml and screenshots, Is here any wrong setting? 

Thanks,

Jordan

On 7 Nov 2017, at 6:49 PM, Ufuk Celebi <[hidden email]> wrote:

Hey Jordan,

yeah, that should just work. Check out the state backend configuration
here: https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/state/state_backends.html

– Ufuk

On Tue, Nov 7, 2017 at 11:44 AM, Jordan Kuan <[hidden email]> wrote:

Dear Ufuk,

Thank you for your reply.

All the cluster Flink servers are able to access network drive, and it mapped as drive Y in all nodes.
Do I need to provide more information?

Thanks,
Jordan

On 7 Nov 2017, at 6:36 PM, Ufuk Celebi <[hidden email]> wrote:

As answered by David on SO, the files need to be accessible by all
nodes. In your setup this seems not to be the case, therefore it won't
work. You need a distributed file system (e.g. NFS or HDFS) or object
store (e.g. S3) that is accessible from all nodes.

– Ufuk


On Tue, Nov 7, 2017 at 3:34 AM, Jordan Kuan <[hidden email]> wrote:

Dear Flink Dev Team

I have encountered a problem and can't find any solution in Google.
And I have created a thread in stackoverflow.com but no response.
https://stackoverflow.com/questions/47123371/flink-windows-ha

I would really appreciate it if you could give some suggestions to me.

Thanks,

Jordan

On Tue, Nov 7, 2017 at 1:53 AM, Robert Metzger <[hidden email]> wrote:

Hi,
I would suggest to send your question to the [hidden email] list
(make sure to subscribe first)

---------- Forwarded message ----------
From: <[hidden email]>
Date: Mon, Nov 6, 2017 at 5:23 PM
Subject: MODERATE for [hidden email]
To:
Cc:
dev-allow-tc.1509985381.dcccgaimcaiiefbkiapi-jordan.kuan=[hidden email]



To approve:
 [hidden email]
To reject:
 [hidden email]
To give a reason to reject:
%%% Start comment
%%% End comment



---------- Forwarded message ----------
From: Jordan Kuan <[hidden email]>
To: [hidden email]
Cc:
Bcc:
Date: Tue, 7 Nov 2017 00:22:53 +0800
Subject: Flink Windows HA Issue.
Dear Flink Dev Team

I have encountered a problem and can't find any solution in Google.
And I have created a thread in stackoverflow.com but no response.
https://stackoverflow.com/questions/47123371/flink-windows-ha

I would really appreciate it if you could give some suggestions to me.

Thanks,

Jordan

--
Best Regards,
Jordan Kuan