datastream.print() doesn't works

[AndreaKinn]

Hi,
I have a simple datastream of a Tuple2. Unfortunately when I call the print() method. No one output is showed although no errors or exceptions are raised.
I want to highlight that I have also other data streams which are correctly printed.

This is the stream:

DataStream<Tuple2<Double,Double>> result = HTM.learn(kafkaStream, new Harness.AnomalyNetwork())
                                .select(new InferenceSelectFunction<Harness.KafkaRecord, Tuple2<Double, Double>>() {
                                          @Override
                                          public Tuple2<Double,Double> select(Tuple2<Harness.KafkaRecord, NetworkInference> inference) throws Exception {
                                          return new Tuple2<Double, Double>(3.14444, inference.f1.getAnomalyScore());
                                         }
});

I'm quite sure the problem isn't in the particular function described above because effectively in inner functions of getAnomalyScore for example the return value exists.

Anyway I'm not able to find the problem which doesn't allow to print the stream so I thought to write here to discover if there are typical situation where streams are not printed. How can I verify and debug the problem? Some hints about it ? I really tried everything, also tried to print something in the function select() for example, with no results

[Chesnay Schepler]

The easiest explanation is that there is nothing to print.

Since print statements within the select function don't appear in the
logs I assume
that the result of HTM.learn is empty.

Please check via the webUI or metrics whether any of these operations
actually return records.

On 29.08.2017 13:19, AndreaKinn wrote:

> Hi,
> I have a simple datastream of a Tuple2. Unfortunately when I call the
> print() method. No one output is showed although no errors or exceptions are
> raised.
> I want to highlight that I have also other data streams which are correctly
> printed.
>
> This is the stream:
>
> /DataStream<Tuple2&lt;Double,Double>> result = HTM.learn(kafkaStream, new
> Harness.AnomalyNetwork())
> .select(new InferenceSelectFunction<Harness.KafkaRecord,
> Tuple2&lt;Double, Double>>() {
>  @Override
>                                            public Tuple2<Double,Double>
> select(Tuple2<Harness.KafkaRecord, NetworkInference> inference) throws
> Exception {
>                          return new Tuple2<Double, Double>(3.14444,
> inference.f1.getAnomalyScore());
>                                           }
> });/
>
> I'm quite sure the problem isn't in the particular function described above
> because effectively in inner functions of getAnomalyScore for example the
> return value exists.
>
> Anyway I'm not able to find the problem which doesn't allow to print the
> stream so I thought to write here to discover if there are typical situation
> where streams are not printed. How can I verify and debug the problem? Some
> hints about it ? I really tried everything, also tried to print something in
> the function select() for example, with no results
>
>
>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/datastream-print-doesn-t-works-tp15223.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
>

[Timo Walther]

Don't forget to call env.execute() at the end and make sure you have
configured your logger correctly.

Regards,
Timo

Am 29.08.17 um 14:59 schrieb Chesnay Schepler:

> The easiest explanation is that there is nothing to print.
>
> Since print statements within the select function don't appear in the
> logs I assume
> that the result of HTM.learn is empty.
>
> Please check via the webUI or metrics whether any of these operations
> actually return records.
>
> On 29.08.2017 13:19, AndreaKinn wrote:
>> Hi,
>> I have a simple datastream of a Tuple2. Unfortunately when I call the
>> print() method. No one output is showed although no errors or
>> exceptions are
>> raised.
>> I want to highlight that I have also other data streams which are
>> correctly
>> printed.
>>
>> This is the stream:
>>
>> /DataStream<Tuple2&lt;Double,Double>> result = HTM.learn(kafkaStream,
>> new
>> Harness.AnomalyNetwork())
>>                 .select(new InferenceSelectFunction<Harness.KafkaRecord,
>> Tuple2&lt;Double, Double>>() {
>>                       @Override
>>                                            public Tuple2<Double,Double>
>> select(Tuple2<Harness.KafkaRecord, NetworkInference> inference) throws
>> Exception {
>>                                   return new Tuple2<Double,
>> Double>(3.14444,
>> inference.f1.getAnomalyScore());
>>                                           }
>> });/
>>
>> I'm quite sure the problem isn't in the particular function described
>> above
>> because effectively in inner functions of getAnomalyScore for example
>> the
>> return value exists.
>>
>> Anyway I'm not able to find the problem which doesn't allow to print the
>> stream so I thought to write here to discover if there are typical
>> situation
>> where streams are not printed. How can I verify and debug the
>> problem? Some
>> hints about it ? I really tried everything, also tried to print
>> something in
>> the function select() for example, with no results
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/datastream-print-doesn-t-works-tp15223.html
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive at Nabble.com.
>>

[AndreaKinn]

Chesnay Schepler wrote

The easiest explanation is that there is nothing to print.

Since print statements within the select function don't appear in the
logs I assume
that the result of HTM.learn is empty.

Please check via the webUI or metrics whether any of these operations
actually return records.

On 29.08.2017 13:19, AndreaKinn wrote:
> Hi,
> I have a simple datastream of a Tuple2. Unfortunately when I call the
> print() method. No one output is showed although no errors or exceptions are
> raised.
> I want to highlight that I have also other data streams which are correctly
> printed.
>
> This is the stream:
>
> /DataStream<Tuple2&lt;Double,Double>> result = HTM.learn(kafkaStream, new
> Harness.AnomalyNetwork())
> .select(new InferenceSelectFunction<Harness.KafkaRecord,
> Tuple2<Double, Double>>() {
>  @Override
>                                            public Tuple2<Double,Double>
> select(Tuple2<Harness.KafkaRecord, NetworkInference> inference) throws
> Exception {
>                          return new Tuple2<Double, Double>(3.14444,
> inference.f1.getAnomalyScore());
>                                           }
> });/
>
> I'm quite sure the problem isn't in the particular function described above
> because effectively in inner functions of getAnomalyScore for example the
> return value exists.
>
> Anyway I'm not able to find the problem which doesn't allow to print the
> stream so I thought to write here to discover if there are typical situation
> where streams are not printed. How can I verify and debug the problem? Some
> hints about it ? I really tried everything, also tried to print something in
> the function select() for example, with no results
>
>
>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/datastream-print-doesn-t-works-tp15223.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
>

I'm losing my head in 6 hours...
To try to start flink job in webUI I tried to generate the jar of my flink program using mvn clean package command.
I had to solve several problems about external libraries, then the command worked but now in Eclipse everything is fucked. I have many errors something like ClassNotFoundException. I tried everything but I'm not able to rewind the program to the last functioning configuration. I'm desperate...

I'm not an expert of maven, what it does exactly and how can I cancel the clean operation to go back to the previous config?

[Chesnay Schepler]

I'm afraid I don't know anything about eclipse.

Running mvn clean package will first delete all files in the /target
directory, and then recompile your code to that very location.
It shouldn't affect an IDE in a way that isn't resolvable by rebuilding
the project in it.
You could also try re-importing your code as a new project and see if it
helps.

Note that you can start a Flink job with a functioning web UI from the
IDE by calling
`StreamExecutionEnvironment#createLocalEnvironmentWithWebUI()`, it is
not necessary to build a jar.

On 29.08.2017 22:00, AndreaKinn wrote:

> Chesnay Schepler wrote
>> The easiest explanation is that there is nothing to print.
>>
>> Since print statements within the select function don't appear in the
>> logs I assume
>> that the result of HTM.learn is empty.
>>
>> Please check via the webUI or metrics whether any of these operations
>> actually return records.
>>
>> On 29.08.2017 13:19, AndreaKinn wrote:
>>> Hi,
>>> I have a simple datastream of a Tuple2. Unfortunately when I call the
>>> print() method. No one output is showed although no errors or exceptions
>>> are
>>> raised.
>>> I want to highlight that I have also other data streams which are
>>> correctly
>>> printed.
>>>
>>> This is the stream:
>>>
>>> /DataStream&lt;Tuple2&amp;lt;Double,Double&gt;> result =
>>> HTM.learn(kafkaStream, new
>>> Harness.AnomalyNetwork())
>>> .select(new InferenceSelectFunction&lt;Harness.KafkaRecord,
>> &gt; Tuple2&lt;Double, Double>>() {
>>>  @Override
>>>                                             public
>>> Tuple2&lt;Double,Double&gt;
>>> select(Tuple2&lt;Harness.KafkaRecord, NetworkInference&gt; inference)
>>> throws
>>> Exception {
>>>                          return new Tuple2&lt;Double,
>>> Double&gt;(3.14444,
>>> inference.f1.getAnomalyScore());
>>>                                            }
>>> });/
>>>
>>> I'm quite sure the problem isn't in the particular function described
>>> above
>>> because effectively in inner functions of getAnomalyScore for example the
>>> return value exists.
>>>
>>> Anyway I'm not able to find the problem which doesn't allow to print the
>>> stream so I thought to write here to discover if there are typical
>>> situation
>>> where streams are not printed. How can I verify and debug the problem?
>>> Some
>>> hints about it ? I really tried everything, also tried to print something
>>> in
>>> the function select() for example, with no results
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/datastream-print-doesn-t-works-tp15223.html
>>> Sent from the Apache Flink User Mailing List archive. mailing list
>>> archive at Nabble.com.
>>>
> I'm losing my head in 6 hours...
> To try to start flink job in webUI I tried to generate the jar of my flink
> program using mvn clean package command.
> I had to solve several problems about external libraries, then the command
> worked but now in Eclipse everything is fucked. I have many errors something
> like ClassNotFoundException. I tried everything but I'm not able to rewind
> the program to the last functioning configuration. I'm desperate...
>
> I'm not an expert of maven, what it does exactly and how can I cancel the
> clean operation to go back to the previous config?
>
>
>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/datastream-print-doesn-t-works-tp15223p15237.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
>

[AndreaKinn]

Hi, in the night uninstalling and re-installing maven and flink I solved my
issue.

I started the web dashboard using start-local.sh script and used
/createLocalEnvironmentWithWebUI(new Configuration())/ as you suggested.
Anyway when I start it in eclipse in the ui dashboard no running jobs are
showed (as I expected).
To run it via console with the console command:

/./bin/flink run -c org.apache.flink.quickstart.StreamingJob
target/flink-java-project-0.1.jar/

but an error appears:

/No operators defined in streaming topology. Cannot execute.
/

Strangely, if I execute it in eclipse no errors appear. Do you know which
operators it refers??



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[AndreaKinn]

I call env.execute().
What do you mean for "configure the logger?"


Timo Walther wrote

> Don't forget to call env.execute() at the end and make sure you have
> configured your logger correctly.
>
> Regards,
> Timo
>
> Am 29.08.17 um 14:59 schrieb Chesnay Schepler:
>> The easiest explanation is that there is nothing to print.
>>
>> Since print statements within the select function don't appear in the
>> logs I assume
>> that the result of HTM.learn is empty.
>>
>> Please check via the webUI or metrics whether any of these operations
>> actually return records.
>>
>> On 29.08.2017 13:19, AndreaKinn wrote:
>>> Hi,
>>> I have a simple datastream of a Tuple2. Unfortunately when I call the
>>> print() method. No one output is showed although no errors or
>>> exceptions are
>>> raised.
>>> I want to highlight that I have also other data streams which are
>>> correctly
>>> printed.
>>>
>>> This is the stream:
>>>
>>> /DataStream&lt;Tuple2&amp;lt;Double,Double&gt;> result =
>>> HTM.learn(kafkaStream,
>>> new
>>> Harness.AnomalyNetwork())
>>>                 .select(new
>>> InferenceSelectFunction&lt;Harness.KafkaRecord,
> &gt;> Tuple2&lt;Double, Double>>() {
>>>                       @Override
>>>                                            public
>>> Tuple2&lt;Double,Double&gt;
>>> select(Tuple2&lt;Harness.KafkaRecord, NetworkInference&gt; inference)
>>> throws
>>> Exception {
>>>                                   return new Tuple2&lt;Double,
> &gt;> Double>(3.14444,
>>> inference.f1.getAnomalyScore());
>>>                                           }
>>> });/
>>>
>>> I'm quite sure the problem isn't in the particular function described
>>> above
>>> because effectively in inner functions of getAnomalyScore for example
>>> the
>>> return value exists.
>>>
>>> Anyway I'm not able to find the problem which doesn't allow to print the
>>> stream so I thought to write here to discover if there are typical
>>> situation
>>> where streams are not printed. How can I verify and debug the
>>> problem? Some
>>> hints about it ? I really tried everything, also tried to print
>>> something in
>>> the function select() for example, with no results
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/datastream-print-doesn-t-works-tp15223.html
>>> Sent from the Apache Flink User Mailing List archive. mailing list
>>> archive at Nabble.com.
>>>

--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Chesnay Schepler]

If you call createLocalEnvironmentWithWebUI you don't need to start a
cluster with the start-local.sh script,
you can run it from the IDE and it will start the web UI. If you submit
a job to a cluster that was started outside the IDE
you can call getExecutionEnvironment as usual.

Not sure why the errors are different, but the message generally occurs
when you call execute() on an environment
where you never defined any operators on. Please check that you're not
creating multiple environments.

If possible it might also be helpful to provide us (or me personally)
with the code of your program.

On 30.08.2017 19:08, AndreaKinn wrote:

> Hi, in the night uninstalling and re-installing maven and flink I solved my
> issue.
>
> I started the web dashboard using start-local.sh script and used
> /createLocalEnvironmentWithWebUI(new Configuration())/ as you suggested.
> Anyway when I start it in eclipse in the ui dashboard no running jobs are
> showed (as I expected).
> To run it via console with the console command:
>
> /./bin/flink run -c org.apache.flink.quickstart.StreamingJob
> target/flink-java-project-0.1.jar/
>
> but an error appears:
>
> /No operators defined in streaming topology. Cannot execute.
> /
>
> Strangely, if I execute it in eclipse no errors appear. Do you know which
> operators it refers??
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>

[AndreaKinn]

I verified I use just one environment. Unfortunately, (also without using
start-local.sh)  /callingcreateLocalEnvironmentWithWebUI()/ and run the
program from the IDE anyway no one running jobs is listed in the dashboard
at /http://localhost:8081/#/overview/.

In the ide it is correctly executed meanwhile.

I pasted my code here  pastebin <https://pastebin.com/raw/1WFFvfJF>  . Tell
if you need other classes.

Thank you



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/