Data point goes missing within iteration

Posted by Biplob Biswas on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Data-point-goes-missing-within-iteration-tp7776.html

Hi,

I am reading data points from a file and then i have to perform iterations over it.

When I just check the data points before the iteration as follows,

tuples.flatMap(new CheckData())

and print count inside CheckData() then I get 2500 data points each over 4 partitions, i.e. 10000 datapoints overall.


Now when I do the following :

ConnectedIterativeStreams<Point, Centroid[]> inputsAndMicroCluster = tuples.iterate()
                                                                                        .withFeedbackType(Centroid[].class);

DataStream<Centroid[]> updatedMicroCluster =
                                inputsAndMicroCluster
                                .flatMap(new MyCoFlatmap());

inputsAndMicroCluster.closeWith(updatedMicroCluster.broadcast());


Then, inside the MyCoFlatmap() method in map1, I check the count and its less than 1/5th of the total number of datapoints.


I don't really know what is happening here exactly, why would the number of data points reduce like that suddenly?

Thanks and Regards
Biplob Biswas