(DEPRECATED) Apache Flink User Mailing List archive.

How to add caching to async function?

classic Classic list List threaded Threaded
♦
♦
 6 messages Options
William Saar-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

How to add caching to async function?

William Saar-2
Hi,
I am trying to implement an async function that looks up a value in a cache or, if the value doesn't exist in the cache, queries a web service, but I'm having trouble creating the cache. I've tried to create a RichAsyncFunction and add a map state as cache, but I'm getting: State is not supported in rich async functions.

What is the best practice for doing this? I guess I could have a previous step with state and send the responses from the rich function back as an iteration, but I would guess that's the wrong approach...

Thanks,
William
Fabian Hueske-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: How to add caching to async function?

Fabian Hueske-2
Hi William,

Does the cache need to be fault tolerant?
If not you could use a regular in-memory map as cache (+some LRU cleaning).
Or do you expect the cache to group too large for the memory?

Best, Fabian


Am Mo., 4. Feb. 2019 um 18:00 Uhr schrieb William Saar <[hidden email]>:
Hi,
I am trying to implement an async function that looks up a value in a cache or, if the value doesn't exist in the cache, queries a web service, but I'm having trouble creating the cache. I've tried to create a RichAsyncFunction and add a map state as cache, but I'm getting: State is not supported in rich async functions.

What is the best practice for doing this? I guess I could have a previous step with state and send the responses from the rich function back as an iteration, but I would guess that's the wrong approach...

Thanks,
William
Lasse Nedergaard
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: How to add caching to async function?

Lasse Nedergaard
Hi William

We have created a solution that do it. Please take a look at my presentation from Flink forward. 
Hopefully you can get inspired. 

Med venlig hilsen / Best regards
Lasse Nedergaard


Den 4. feb. 2019 kl. 19.04 skrev Fabian Hueske <[hidden email]>:

Hi William,

Does the cache need to be fault tolerant?
If not you could use a regular in-memory map as cache (+some LRU cleaning).
Or do you expect the cache to group too large for the memory?

Best, Fabian


Am Mo., 4. Feb. 2019 um 18:00 Uhr schrieb William Saar <[hidden email]>:
Hi,
I am trying to implement an async function that looks up a value in a cache or, if the value doesn't exist in the cache, queries a web service, but I'm having trouble creating the cache. I've tried to create a RichAsyncFunction and add a map state as cache, but I'm getting: State is not supported in rich async functions.

What is the best practice for doing this? I guess I could have a previous step with state and send the responses from the rich function back as an iteration, but I would guess that's the wrong approach...

Thanks,
William
William Saar-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: How to add caching to async function?

William Saar-2
Thanks! Looks like iterations is indeed the way to go for now then...



----- Original Message -----
From:
"Lasse Nedergaard" <[hidden email]>

To:
"Fabian Hueske" <[hidden email]>
Cc:
"William Saar" <[hidden email]>, "user" <[hidden email]>
Sent:
Mon, 4 Feb 2019 20:20:30 +0100
Subject:
Re: How to add caching to async function?


Hi William

We have created a solution that do it. Please take a look at my presentation from Flink forward. 
Hopefully you can get inspired. 

Med venlig hilsen / Best regards
Lasse Nedergaard


Den 4. feb. 2019 kl. 19.04 skrev Fabian Hueske <[hidden email]>:

Hi William,

Does the cache need to be fault tolerant?
If not you could use a regular in-memory map as cache (+some LRU cleaning).
Or do you expect the cache to group too large for the memory?

Best, Fabian


Am Mo., 4. Feb. 2019 um 18:00 Uhr schrieb William Saar <[hidden email]>:
Hi,
I am trying to implement an async function that looks up a value in a cache or, if the value doesn't exist in the cache, queries a web service, but I'm having trouble creating the cache. I've tried to create a RichAsyncFunction and add a map state as cache, but I'm getting: State is not supported in rich async functions.

What is the best practice for doing this? I guess I could have a previous step with state and send the responses from the rich function back as an iteration, but I would guess that's the wrong approach...

Thanks,
William
Lasse Nedergaard
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: How to add caching to async function?

Lasse Nedergaard
Hi William

No iterations isn’t the solution as you can (will) end up in a deadlock. We concluded that storing the results from external lookup in Kafka and use these data as input to the cache was the only way

Med venlig hilsen / Best regards
Lasse Nedergaard


Den 5. feb. 2019 kl. 10.22 skrev William Saar <[hidden email]>:

Thanks! Looks like iterations is indeed the way to go for now then...



----- Original Message -----
From:
"Lasse Nedergaard" <[hidden email]>

To:
"Fabian Hueske" <[hidden email]>
Cc:
"William Saar" <[hidden email]>, "user" <[hidden email]>
Sent:
Mon, 4 Feb 2019 20:20:30 +0100
Subject:
Re: How to add caching to async function?


Hi William

We have created a solution that do it. Please take a look at my presentation from Flink forward. 
Hopefully you can get inspired. 

Med venlig hilsen / Best regards
Lasse Nedergaard


Den 4. feb. 2019 kl. 19.04 skrev Fabian Hueske <[hidden email]>:

Hi William,

Does the cache need to be fault tolerant?
If not you could use a regular in-memory map as cache (+some LRU cleaning).
Or do you expect the cache to group too large for the memory?

Best, Fabian


Am Mo., 4. Feb. 2019 um 18:00 Uhr schrieb William Saar <[hidden email]>:
Hi,
I am trying to implement an async function that looks up a value in a cache or, if the value doesn't exist in the cache, queries a web service, but I'm having trouble creating the cache. I've tried to create a RichAsyncFunction and add a map state as cache, but I'm getting: State is not supported in rich async functions.

What is the best practice for doing this? I guess I could have a previous step with state and send the responses from the rich function back as an iteration, but I would guess that's the wrong approach...

Thanks,
William
William Saar-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: How to add caching to async function?

William Saar-2
Ah, thanks, missed it when I only looked at the slides. Yes, have heard deadlocks are a problem with iterations but hoped it had been fixed. Pity, had been hoping to replace an external service with the Flink job, but will keep the service around for the caching,



----- Original Message -----
From:
"Lasse Nedergaard" <[hidden email]>

To:
"William Saar" <[hidden email]>
Cc:
"Fabian Hueske" <[hidden email]>, "user" <[hidden email]>
Sent:
Tue, 5 Feb 2019 10:41:41 +0100
Subject:
Re: How to add caching to async function?


Hi William

No iterations isn’t the solution as you can (will) end up in a deadlock. We concluded that storing the results from external lookup in Kafka and use these data as input to the cache was the only way

Med venlig hilsen / Best regards
Lasse Nedergaard


Den 5. feb. 2019 kl. 10.22 skrev William Saar <[hidden email]>:

Thanks! Looks like iterations is indeed the way to go for now then...



----- Original Message -----
From:
"Lasse Nedergaard" <[hidden email]>

To:
"Fabian Hueske" <[hidden email]>
Cc:
"William Saar" <[hidden email]>, "user" <[hidden email]>
Sent:
Mon, 4 Feb 2019 20:20:30 +0100
Subject:
Re: How to add caching to async function?


Hi William

We have created a solution that do it. Please take a look at my presentation from Flink forward. 
Hopefully you can get inspired. 

Med venlig hilsen / Best regards
Lasse Nedergaard


Den 4. feb. 2019 kl. 19.04 skrev Fabian Hueske <[hidden email]>:

Hi William,

Does the cache need to be fault tolerant?
If not you could use a regular in-memory map as cache (+some LRU cleaning).
Or do you expect the cache to group too large for the memory?

Best, Fabian


Am Mo., 4. Feb. 2019 um 18:00 Uhr schrieb William Saar <[hidden email]>:
Hi,
I am trying to implement an async function that looks up a value in a cache or, if the value doesn't exist in the cache, queries a web service, but I'm having trouble creating the cache. I've tried to create a RichAsyncFunction and add a map state as cache, but I'm getting: State is not supported in rich async functions.

What is the best practice for doing this? I guess I could have a previous step with state and send the responses from the rich function back as an iteration, but I would guess that's the wrong approach...

Thanks,
William
Free forum by Nabble Edit this page