|
|
Hi Vishal,
Sorry, I didn't notice this requirement, but I can't come up with another solution, unless disable checkpointing or customize your own kafka source function. For the first case, you may have to give up storing states in flink's state backend. For the second one, write your own implementation with kafka client and always seek to the latest position when the job begin to run.
Best, Tony Wei I think I can set . a new uuid but it seems `allowNonRestoreState` is a CLI hint. I need the "automatic" restart on failure to use the new uuid. Our use case has no use of data on Kafka that is not current.
Hi Vishal,
AFAIK, the current behavior of kafka source will always use checkpoint state as the start position, ignoring other configuration. A workaround solution I can come up with is to set a new uuid to your kafka source and restore your job with `allowNonRestoreState`. Therefore, you can use the way that Rong provided to set the initial start position.
cc. Gordon who know more about the details of kafka source.
Best, Tony Wei
Hi Vishal,
You can probably try using similar offset configuration as a service consumer. Maybe this will be useful to look at [1]
Thanks, Rong
Hi Vishal,
No, there is no way to do this currently.
On Wed, Nov 21, 2018 at 10:22 AM Vishal Santoshi < [hidden email]> wrote: Any one ?
On Tue, Nov 20, 2018 at 12:48 PM Vishal Santoshi < [hidden email]> wrote: Is it possible to have checkpointing but reset the kafka offsets to latest on restart on failure ?
|
Locked
