Flink REST API side effect?

[Tomasz Dudziak]

Hi,

 

I found an interesting behaviour of the REST API in my automated system tests using that API where I was getting status of a purposefully failing job.

 

If you query job details immediately after job submission, subsequent queries will return its status as RUNNING for a moment until Flink’s restart strategies are exhausted. If you don’t and take a brief moment before your first query of  the job status, it will be reported in a more fine-grained fashion, e.g. RESTARTING or FAILING.

 

Has anyone else noticed that behaviour?

 

Thanks,

Tomasz

 

Tomasz Dudziak | Marshall Wace LLP, George House, 131 Sloane Street, London, SW1X 9AT | E-mail: [hidden email] | Tel: [phone]

 

 

This e-mail and any attachments are confidential to the addressee(s) and may contain information that is legally privileged and/or confidential. Please refer to http://www.mwam.com/email-disclaimer-uk for important disclosures regarding this email. If we collect and use your personal data we will use it in accordance with our privacy policy, which can be reviewed at https://www.mwam.com/privacy-policy.

Marshall Wace LLP is authorised and regulated by the Financial Conduct Authority. Marshall Wace LLP is a limited liability partnership registered in England and Wales with registered number OC302228 and registered office at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this e-mail as a client, or an investor in an investment vehicle, managed or advised by Marshall Wace North America L.P., the sender of this e-mail is communicating with you in the sender's capacity as an associated or related person of Marshall Wace North America L.P., which is registered with the US Securities and Exchange Commission as an investment adviser.

[Chesnay Schepler]

This is expected, the backing data structure is cached for a while so we never hammer the JobManager with requests.

IIRC this is controlled via "web.refresh-interval", with the default being 3 seconds.

On 11/05/2020 14:10, Tomasz Dudziak wrote:

Hi,

 

I found an interesting behaviour of the REST API in my automated system tests using that API where I was getting status of a purposefully failing job.

 

If you query job details immediately after job submission, subsequent queries will return its status as RUNNING for a moment until Flink’s restart strategies are exhausted. If you don’t and take a brief moment before your first query of  the job status, it will be reported in a more fine-grained fashion, e.g. RESTARTING or FAILING.

 

Has anyone else noticed that behaviour?

 

Thanks,

Tomasz

 

Tomasz Dudziak | Marshall Wace LLP, George House, 131 Sloane Street, London, SW1X 9AT | E-mail: [hidden email] | Tel: [phone]

 

 

This e-mail and any attachments are confidential to the addressee(s) and may contain information that is legally privileged and/or confidential. Please refer to http://www.mwam.com/email-disclaimer-uk for important disclosures regarding this email. If we collect and use your personal data we will use it in accordance with our privacy policy, which can be reviewed at https://www.mwam.com/privacy-policy.

Marshall Wace LLP is authorised and regulated by the Financial Conduct Authority. Marshall Wace LLP is a limited liability partnership registered in England and Wales with registered number OC302228 and registered office at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this e-mail as a client, or an investor in an investment vehicle, managed or advised by Marshall Wace North America L.P., the sender of this e-mail is communicating with you in the sender's capacity as an associated or related person of Marshall Wace North America L.P., which is registered with the US Securities and Exchange Commission as an investment adviser.

[Tomasz Dudziak]

Thanks for reply. So do I understand correctly if I say that whenever you query job status it gets cached for a configurable amount of time and subsequent queries within that time period will not show any change?

 

From: Chesnay Schepler <[hidden email]>
Sent: 11 May 2020 13:20
To: Tomasz Dudziak <[hidden email]>; [hidden email]
Subject: Re: Flink REST API side effect?

 

This is expected, the backing data structure is cached for a while so we never hammer the JobManager with requests.

 

IIRC this is controlled via "web.refresh-interval", with the default being 3 seconds.

 

On 11/05/2020 14:10, Tomasz Dudziak wrote:

Hi,

 

I found an interesting behaviour of the REST API in my automated system tests using that API where I was getting status of a purposefully failing job.

 

If you query job details immediately after job submission, subsequent queries will return its status as RUNNING for a moment until Flink’s restart strategies are exhausted. If you don’t and take a brief moment before your first query of  the job status, it will be reported in a more fine-grained fashion, e.g. RESTARTING or FAILING.

 

Has anyone else noticed that behaviour?

 

Thanks,

Tomasz

 

Tomasz Dudziak | Marshall Wace LLP, George House, 131 Sloane Street, London, SW1X 9AT | E-mail: [hidden email] | Tel: [phone]

 

 

This e-mail and any attachments are confidential to the addressee(s) and may contain information that is legally privileged and/or confidential. Please refer to http://www.mwam.com/email-disclaimer-uk for important disclosures regarding this email. If we collect and use your personal data we will use it in accordance with our privacy policy, which can be reviewed at https://www.mwam.com/privacy-policy.

Marshall Wace LLP is authorised and regulated by the Financial Conduct Authority. Marshall Wace LLP is a limited liability partnership registered in England and Wales with registered number OC302228 and registered office at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this e-mail as a client, or an investor in an investment vehicle, managed or advised by Marshall Wace North America L.P., the sender of this e-mail is communicating with you in the sender's capacity as an associated or related person of Marshall Wace North America L.P., which is registered with the US Securities and Exchange Commission as an investment adviser.

 

 

This e-mail and any attachments are confidential to the addressee(s) and may contain information that is legally privileged and/or confidential. Please refer to http://www.mwam.com/email-disclaimer-uk for important disclosures regarding this email. If we collect and use your personal data we will use it in accordance with our privacy policy, which can be reviewed at https://www.mwam.com/privacy-policy.

Marshall Wace LLP is authorised and regulated by the Financial Conduct Authority. Marshall Wace LLP is a limited liability partnership registered in England and Wales with registered number OC302228 and registered office at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this e-mail as a client, or an investor in an investment vehicle, managed or advised by Marshall Wace North America L.P., the sender of this e-mail is communicating with you in the sender's capacity as an associated or related person of Marshall Wace North America L.P., which is registered with the US Securities and Exchange Commission as an investment adviser.

[Chesnay Schepler]

yes, that is correct.

On 11/05/2020 14:28, Tomasz Dudziak wrote:

Thanks for reply. So do I understand correctly if I say that whenever you query job status it gets cached for a configurable amount of time and subsequent queries within that time period will not show any change?

 

From: Chesnay Schepler [hidden email]
Sent: 11 May 2020 13:20
To: Tomasz Dudziak [hidden email]; [hidden email]
Subject: Re: Flink REST API side effect?

 

This is expected, the backing data structure is cached for a while so we never hammer the JobManager with requests.

 

IIRC this is controlled via "web.refresh-interval", with the default being 3 seconds.

 

On 11/05/2020 14:10, Tomasz Dudziak wrote:

Hi,

 

I found an interesting behaviour of the REST API in my automated system tests using that API where I was getting status of a purposefully failing job.

 

If you query job details immediately after job submission, subsequent queries will return its status as RUNNING for a moment until Flink’s restart strategies are exhausted. If you don’t and take a brief moment before your first query of  the job status, it will be reported in a more fine-grained fashion, e.g. RESTARTING or FAILING.

 

Has anyone else noticed that behaviour?

 

Thanks,

Tomasz

 

Tomasz Dudziak | Marshall Wace LLP, George House, 131 Sloane Street, London, SW1X 9AT | E-mail: [hidden email] | Tel: [phone]

 

 

This e-mail and any attachments are confidential to the addressee(s) and may contain information that is legally privileged and/or confidential. Please refer to http://www.mwam.com/email-disclaimer-uk for important disclosures regarding this email. If we collect and use your personal data we will use it in accordance with our privacy policy, which can be reviewed at https://www.mwam.com/privacy-policy.

Marshall Wace LLP is authorised and regulated by the Financial Conduct Authority. Marshall Wace LLP is a limited liability partnership registered in England and Wales with registered number OC302228 and registered office at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this e-mail as a client, or an investor in an investment vehicle, managed or advised by Marshall Wace North America L.P., the sender of this e-mail is communicating with you in the sender's capacity as an associated or related person of Marshall Wace North America L.P., which is registered with the US Securities and Exchange Commission as an investment adviser.

 

 

This e-mail and any attachments are confidential to the addressee(s) and may contain information that is legally privileged and/or confidential. Please refer to http://www.mwam.com/email-disclaimer-uk for important disclosures regarding this email. If we collect and use your personal data we will use it in accordance with our privacy policy, which can be reviewed at https://www.mwam.com/privacy-policy.

Marshall Wace LLP is authorised and regulated by the Financial Conduct Authority. Marshall Wace LLP is a limited liability partnership registered in England and Wales with registered number OC302228 and registered office at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this e-mail as a client, or an investor in an investment vehicle, managed or advised by Marshall Wace North America L.P., the sender of this e-mail is communicating with you in the sender's capacity as an associated or related person of Marshall Wace North America L.P., which is registered with the US Securities and Exchange Commission as an investment adviser.