(DEPRECATED) Apache Flink User Mailing List archive.

Question Regarding a sink..

classic Classic list List threaded Threaded
♦
♦
 2 messages Options
Steve Jerman
Reply | Threaded
Open this post in threaded view
♦
♦
|

Question Regarding a sink..

Steve Jerman
Hi,

I have a sink writing data to InfluxDB. I’ve noticed that the sink gets multiple copies of upstream records..

Why does this happen, and how can I avoid it… ?

Below is a trace …showing 2 records (I have a parallelism of two) for each record in the ‘.printToError’ for the same stream.

Any help/suggestions appreciated.

Steve


1> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=2F:A4:BD:56:EC:4D, sessionTime=15000}]
2> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=17:8E:FC:7E:F7:20, sessionTime=20000}]
1> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=40:07:2D:CB:41:07, sessionTime=5000}]
2> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=28:24:0B:B6:42:CA, sessionTime=5000}]
1> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=8F:13:AC:4A:DA:93, sessionTime=5000}]
2> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=43:7D:8A:D4:7D:D7, sessionTime=5000}]
2> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=77:CD:BD:48:EE:D8, sessionTime=5000}]
18:41:27,351 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=2F:A4:BD:56:EC:4D, sessionTime=15000}]
18:41:27,351 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=2F:A4:BD:56:EC:4D, sessionTime=15000}]
18:41:27,351 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=17:8E:FC:7E:F7:20, sessionTime=20000}]
18:41:27,351 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=17:8E:FC:7E:F7:20, sessionTime=20000}]
18:41:27,351 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=40:07:2D:CB:41:07, sessionTime=5000}]
18:41:27,351 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=40:07:2D:CB:41:07, sessionTime=5000}]
18:41:27,351 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=28:24:0B:B6:42:CA, sessionTime=5000}]
18:41:27,351 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=28:24:0B:B6:42:CA, sessionTime=5000}]
18:41:27,352 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=8F:13:AC:4A:DA:93, sessionTime=5000}]
18:41:27,352 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=8F:13:AC:4A:DA:93, sessionTime=5000}]
18:41:27,352 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=43:7D:8A:D4:7D:D7, sessionTime=5000}]
18:41:27,352 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=43:7D:8A:D4:7D:D7, sessionTime=5000}]
18:41:27,352 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=77:CD:BD:48:EE:D8, sessionTime=5000}]
18:41:27,352 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=77:CD:BD:48:EE:D8, sessionTime=5000}]
Tzu-Li (Gordon) Tai
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Question Regarding a sink..

Tzu-Li (Gordon) Tai
Hi Steve,

This normally shouldn’t happen, unless there simply is two copies of the data.

What is the source of the topology? Also, this might be obvious, but if you have broadcasted your input stream to the sink, then each sink instance would then get all records in the input stream.

Cheers,
Gordon

On March 24, 2017 at 9:11:35 AM, Steve Jerman ([hidden email]) wrote:

Hi,

I have a sink writing data to InfluxDB. I’ve noticed that the sink gets multiple copies of upstream records..

Why does this happen, and how can I avoid it… ?

Below is a trace …showing 2 records (I have a parallelism of two) for each record in the ‘.printToError’ for the same stream.

Any help/suggestions appreciated.

Steve


1> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=2F:A4:BD:56:EC:4D, sessionTime=15000}]
2> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=17:8E:FC:7E:F7:20, sessionTime=20000}]
1> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=40:07:2D:CB:41:07, sessionTime=5000}]
2> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=28:24:0B:B6:42:CA, sessionTime=5000}]
1> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=8F:13:AC:4A:DA:93, sessionTime=5000}]
2> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=43:7D:8A:D4:7D:D7, sessionTime=5000}]
2> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=77:CD:BD:48:EE:D8, sessionTime=5000}]
18:41:27,351 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=2F:A4:BD:56:EC:4D, sessionTime=15000}]
18:41:27,351 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=2F:A4:BD:56:EC:4D, sessionTime=15000}]
18:41:27,351 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=17:8E:FC:7E:F7:20, sessionTime=20000}]
18:41:27,351 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=17:8E:FC:7E:F7:20, sessionTime=20000}]
18:41:27,351 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=40:07:2D:CB:41:07, sessionTime=5000}]
18:41:27,351 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=40:07:2D:CB:41:07, sessionTime=5000}]
18:41:27,351 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=28:24:0B:B6:42:CA, sessionTime=5000}]
18:41:27,351 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=28:24:0B:B6:42:CA, sessionTime=5000}]
18:41:27,352 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=8F:13:AC:4A:DA:93, sessionTime=5000}]
18:41:27,352 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=8F:13:AC:4A:DA:93, sessionTime=5000}]
18:41:27,352 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=43:7D:8A:D4:7D:D7, sessionTime=5000}]
18:41:27,352 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=43:7D:8A:D4:7D:D7, sessionTime=5000}]
18:41:27,352 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=77:CD:BD:48:EE:D8, sessionTime=5000}]
18:41:27,352 INFO  com.kloudspot.flink.sink.InfluxSink                           - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=77:CD:BD:48:EE:D8, sessionTime=5000}]
Free forum by Nabble Edit this page