SteamingFileSink with TwoPhaseCommit

[Taher Koitawala]

Hi All,

          As per my understanding and the API of StreamingFileSink, TwoPhaseCommit is not being used. Can someone please confirm is that's right?              Also if StreamingFileSink does not support TwoPhaseCommits what is the best way to implement this?

[Kostas Kloudas-2]

Hi Taher,

The StreamingFileSink implements a version of TwoPhaseCommit. Can you elaborate a bit on what do you mean by " TwoPhaseCommit is not being used"?

Cheers,

Kostas

On Thu, Jan 10, 2019 at 9:29 AM Taher Koitawala <[hidden email]> wrote:

Hi All,

          As per my understanding and the API of StreamingFileSink, TwoPhaseCommit is not being used. Can someone please confirm is that's right?              Also if StreamingFileSink does not support TwoPhaseCommits what is the best way to implement this?

[Taher Koitawala]

StreamingFileSink extends RichSinkFunction and implements CheckpointedFunction, CheckpointListener and ProcessingTimeCallback however TwoPhaseCommitSinkFunction is never used by StreamingFileSink.  Hence I had a question if the sink uses the TwoPhaseCommit protocol or not. 

Regards,

Taher Koitawala 

On Thu 10 Jan, 2019, 2:40 PM Kostas Kloudas <[hidden email] wrote:

Hi Taher,

The StreamingFileSink implements a version of TwoPhaseCommit. Can you elaborate a bit on what do you mean by " TwoPhaseCommit is not being used"?

Cheers,

Kostas

On Thu, Jan 10, 2019 at 9:29 AM Taher Koitawala <[hidden email]> wrote:

Hi All,

          As per my understanding and the API of StreamingFileSink, TwoPhaseCommit is not being used. Can someone please confirm is that's right?              Also if StreamingFileSink does not support TwoPhaseCommits what is the best way to implement this?

[Kostas Kloudas-2]

That is correct. 

The StreamingFileSink does not extend the TwoPhaseCommitSinkFunction because of some internal implementation details.

But this, does not mean that it does not implement a two phase commit protocol (which is independent of the implementation).

Cheers,

Kostas

On Thu, Jan 10, 2019 at 10:33 AM Taher Koitawala <[hidden email]> wrote:

StreamingFileSink extends RichSinkFunction and implements CheckpointedFunction, CheckpointListener and ProcessingTimeCallback however TwoPhaseCommitSinkFunction is never used by StreamingFileSink.  Hence I had a question if the sink uses the TwoPhaseCommit protocol or not. 

Regards,

Taher Koitawala 

On Thu 10 Jan, 2019, 2:40 PM Kostas Kloudas <[hidden email] wrote:

Hi Taher,

The StreamingFileSink implements a version of TwoPhaseCommit. Can you elaborate a bit on what do you mean by " TwoPhaseCommit is not being used"?

Cheers,

Kostas

On Thu, Jan 10, 2019 at 9:29 AM Taher Koitawala <[hidden email]> wrote:

Hi All,

          As per my understanding and the API of StreamingFileSink, TwoPhaseCommit is not being used. Can someone please confirm is that's right?              Also if StreamingFileSink does not support TwoPhaseCommits what is the best way to implement this?

[Taher Koitawala]

Hi Kostas,

               Thanks you for the clarification, also can you please point how StreamingFileSink uses TwoPhaseCommit. Can you also point out the implementing class for that?

On Thu, Jan 10, 2019 at 3:10 PM Kostas Kloudas <[hidden email]> wrote:

That is correct. 

The StreamingFileSink does not extend the TwoPhaseCommitSinkFunction because of some internal implementation details.

But this, does not mean that it does not implement a two phase commit protocol (which is independent of the implementation).

Cheers,

Kostas

On Thu, Jan 10, 2019 at 10:33 AM Taher Koitawala <[hidden email]> wrote:

StreamingFileSink extends RichSinkFunction and implements CheckpointedFunction, CheckpointListener and ProcessingTimeCallback however TwoPhaseCommitSinkFunction is never used by StreamingFileSink.  Hence I had a question if the sink uses the TwoPhaseCommit protocol or not. 

Regards,

Taher Koitawala 

On Thu 10 Jan, 2019, 2:40 PM Kostas Kloudas <[hidden email] wrote:

Hi Taher,

The StreamingFileSink implements a version of TwoPhaseCommit. Can you elaborate a bit on what do you mean by " TwoPhaseCommit is not being used"?

Cheers,

Kostas

On Thu, Jan 10, 2019 at 9:29 AM Taher Koitawala <[hidden email]> wrote:

Hi All,

          As per my understanding and the API of StreamingFileSink, TwoPhaseCommit is not being used. Can someone please confirm is that's right?              Also if StreamingFileSink does not support TwoPhaseCommits what is the best way to implement this?

[Kostas Kloudas-3]

Hi Taher,

Well, I would say there is no single class that implements it. 

In a nutshell, it is the StreamingFileSink that (through Buckets) tells the responsible Bucket what to do at each step of the lifecycle of the Flink operator

(mainly on element, on checkpoint, on checkpoint completed and on restore).

So I would suggest that you should have a look in these classes.

In essence, the main steps of the "choreography" are similar to the ones that the older BucketingSink was doing.

Cheers,

Kostas

On Thu, Jan 10, 2019 at 10:47 AM Taher Koitawala <[hidden email]> wrote:

Hi Kostas,

               Thanks you for the clarification, also can you please point how StreamingFileSink uses TwoPhaseCommit. Can you also point out the implementing class for that?

On Thu, Jan 10, 2019 at 3:10 PM Kostas Kloudas <[hidden email]> wrote:

That is correct. 

The StreamingFileSink does not extend the TwoPhaseCommitSinkFunction because of some internal implementation details.

But this, does not mean that it does not implement a two phase commit protocol (which is independent of the implementation).

Cheers,

Kostas

On Thu, Jan 10, 2019 at 10:33 AM Taher Koitawala <[hidden email]> wrote:

StreamingFileSink extends RichSinkFunction and implements CheckpointedFunction, CheckpointListener and ProcessingTimeCallback however TwoPhaseCommitSinkFunction is never used by StreamingFileSink.  Hence I had a question if the sink uses the TwoPhaseCommit protocol or not. 

Regards,

Taher Koitawala 

On Thu 10 Jan, 2019, 2:40 PM Kostas Kloudas <[hidden email] wrote:

Hi Taher,

The StreamingFileSink implements a version of TwoPhaseCommit. Can you elaborate a bit on what do you mean by " TwoPhaseCommit is not being used"?

Cheers,

Kostas

On Thu, Jan 10, 2019 at 9:29 AM Taher Koitawala <[hidden email]> wrote:

Hi All,

          As per my understanding and the API of StreamingFileSink, TwoPhaseCommit is not being used. Can someone please confirm is that's right?              Also if StreamingFileSink does not support TwoPhaseCommits what is the best way to implement this?

[Taher Koitawala]

Ok, thanks for the clarification. Really appreciate your help Kostas

On Thu 10 Jan, 2019, 6:19 PM Kostas Kloudas <[hidden email] wrote:

Hi Taher,

Well, I would say there is no single class that implements it. 

In a nutshell, it is the StreamingFileSink that (through Buckets) tells the responsible Bucket what to do at each step of the lifecycle of the Flink operator

(mainly on element, on checkpoint, on checkpoint completed and on restore).

So I would suggest that you should have a look in these classes.

In essence, the main steps of the "choreography" are similar to the ones that the older BucketingSink was doing.

Cheers,

Kostas

On Thu, Jan 10, 2019 at 10:47 AM Taher Koitawala <[hidden email]> wrote:

Hi Kostas,

               Thanks you for the clarification, also can you please point how StreamingFileSink uses TwoPhaseCommit. Can you also point out the implementing class for that?

On Thu, Jan 10, 2019 at 3:10 PM Kostas Kloudas <[hidden email]> wrote:

That is correct. 

The StreamingFileSink does not extend the TwoPhaseCommitSinkFunction because of some internal implementation details.

But this, does not mean that it does not implement a two phase commit protocol (which is independent of the implementation).

Cheers,

Kostas

On Thu, Jan 10, 2019 at 10:33 AM Taher Koitawala <[hidden email]> wrote:

StreamingFileSink extends RichSinkFunction and implements CheckpointedFunction, CheckpointListener and ProcessingTimeCallback however TwoPhaseCommitSinkFunction is never used by StreamingFileSink.  Hence I had a question if the sink uses the TwoPhaseCommit protocol or not. 

Regards,

Taher Koitawala 

On Thu 10 Jan, 2019, 2:40 PM Kostas Kloudas <[hidden email] wrote:

Hi Taher,

The StreamingFileSink implements a version of TwoPhaseCommit. Can you elaborate a bit on what do you mean by " TwoPhaseCommit is not being used"?

Cheers,

Kostas

On Thu, Jan 10, 2019 at 9:29 AM Taher Koitawala <[hidden email]> wrote:

Hi All,

          As per my understanding and the API of StreamingFileSink, TwoPhaseCommit is not being used. Can someone please confirm is that's right?              Also if StreamingFileSink does not support TwoPhaseCommits what is the best way to implement this?