Accessing elements from DataStream

[Piyush Shrivastava]

Hi all,

Can we access individual elements from a DataStream through an iterator like we can in a WindowedStream with the apply function?

I am able to access the elements of a WindowedStream using the apply function and using the Iterable and Collector interfaces:

val ds = ws.apply((K, W, input: Iterable[T], output: Collector[R]) => {

   val it = input.toIterator

   while(it.hasNext){

      ...

   }

})

the output of this function is a DataStream. I want to perform a keyBy on this DataStream and then again iterate through it's individual elements. We cannot use the apply function on DataStreams like in WindowedStreams. Is there any other way?

 

[rmetzger0]

Hi,

you can just use a flatMap() on a DataStream to access individual elements from a stream.

On Thu, May 5, 2016 at 1:00 PM, Piyush Shrivastava <[hidden email]> wrote:

Hi all,

Can we access individual elements from a DataStream through an iterator like we can in a WindowedStream with the apply function?

I am able to access the elements of a WindowedStream using the apply function and using the Iterable and Collector interfaces:

val ds = ws.apply((K, W, input: Iterable[T], output: Collector[R]) => {

   val it = input.toIterator

   while(it.hasNext){

      ...

   }

})

the output of this function is a DataStream. I want to perform a keyBy on this DataStream and then again iterate through it's individual elements. We cannot use the apply function on DataStreams like in WindowedStreams. Is there any other way?

 

Thanks and Regards,

[hidden email]

http://webograffiti.com

[Piyush Shrivastava]

Hi Robert,

Can you share an example where flatmap is used to access elements?

 

On Thursday, 5 May 2016 4:45 PM, Robert Metzger <[hidden email]> wrote:

Hi,

you can just use a flatMap() on a DataStream to access individual elements from a stream.

On Thu, May 5, 2016 at 1:00 PM, Piyush Shrivastava <[hidden email]> wrote:

Hi all,

Can we access individual elements from a DataStream through an iterator like we can in a WindowedStream with the apply function?

I am able to access the elements of a WindowedStream using the apply function and using the Iterable and Collector interfaces:

val ds = ws.apply((K, W, input: Iterable[T], output: Collector[R]) => {

   val it = input.toIterator

   while(it.hasNext){

      ...

   }

})

the output of this function is a DataStream. I want to perform a keyBy on this DataStream and then again iterate through it's individual elements. We cannot use the apply function on DataStreams like in WindowedStreams. Is there any other way?

 

Thanks and Regards,

[hidden email]

http://webograffiti.com

[rmetzger0]

Yes. Here: https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/index.html#example-program

On Thu, May 5, 2016 at 1:16 PM, Piyush Shrivastava <[hidden email]> wrote:

Hi Robert,

Can you share an example where flatmap is used to access elements?

 

Thanks and Regards,

[hidden email]

http://webograffiti.com

On Thursday, 5 May 2016 4:45 PM, Robert Metzger <[hidden email]> wrote:

Hi,

you can just use a flatMap() on a DataStream to access individual elements from a stream.

On Thu, May 5, 2016 at 1:00 PM, Piyush Shrivastava <[hidden email]> wrote:

Hi all,

Can we access individual elements from a DataStream through an iterator like we can in a WindowedStream with the apply function?

I am able to access the elements of a WindowedStream using the apply function and using the Iterable and Collector interfaces:

val ds = ws.apply((K, W, input: Iterable[T], output: Collector[R]) => {

   val it = input.toIterator

   while(it.hasNext){

      ...

   }

})

the output of this function is a DataStream. I want to perform a keyBy on this DataStream and then again iterate through it's individual elements. We cannot use the apply function on DataStreams like in WindowedStreams. Is there any other way?

 

Thanks and Regards,

[hidden email]

http://webograffiti.com