Iterations problem in command line

[Marcela Charfuelan]

Hello,

I implemented an algorithm that includes iterations (EM algorithm) and I
am getting different results when running in eclipse (Luna Release
(4.4.0)) and when running in the command line using Flink run; the
program does not crash is just that after the first iteration the
results are different (wrong in the command line).

The solution I am getting in eclipse, for each iteration, is the same
that I would get if running the algorithm in octave for example, so I am
sure the solution is correct.

I have tried using java plus the command-line arguments of eclipse on
the command line and that also works ok (local in ubuntu).

Has anybody experienced something similar? Any idea why this could
happen? how can I fix this?

Regards,
Marcela.

[Fabian Hueske-2]

Hi Marcela,

do you run the algorithm in both setups with the same parallelism?

Best, Fabian

2016-02-26 16:52 GMT+01:00 Marcela Charfuelan <[hidden email]>:

Hello,

I implemented an algorithm that includes iterations (EM algorithm) and I am getting different results when running in eclipse (Luna Release (4.4.0)) and when running in the command line using Flink run; the program does not crash is just that after the first iteration the results are different (wrong in the command line).

The solution I am getting in eclipse, for each iteration, is the same that I would get if running the algorithm in octave for example, so I am sure the solution is correct.

I have tried using java plus the command-line arguments of eclipse on the command line and that also works ok (local in ubuntu).

Has anybody experienced something similar? Any idea why this could happen? how can I fix this?

Regards,
Marcela.

[Marcela Charfuelan]

Thanks Fabian,
I am using in both default options, since I am not testing in a cluster
yet, just local in ubuntu, I am not specifying any parallelism.
just to test I set in the program env.setParallelism(1) and running with
-p 1 (which I guess I would not need) but I am still getting the same issue.

Regards,
MArcela.


On 29.02.2016 16:44, Fabian Hueske wrote:

> Hi Marcela,
>
> do you run the algorithm in both setups with the same parallelism?
>
> Best, Fabian
>
> 2016-02-26 16:52 GMT+01:00 Marcela Charfuelan
> <[hidden email] <mailto:[hidden email]>>:
>
>     Hello,
>
>     I implemented an algorithm that includes iterations (EM algorithm)
>     and I am getting different results when running in eclipse (Luna
>     Release (4.4.0)) and when running in the command line using Flink
>     run; the program does not crash is just that after the first
>     iteration the results are different (wrong in the command line).
>
>     The solution I am getting in eclipse, for each iteration, is the
>     same that I would get if running the algorithm in octave for
>     example, so I am sure the solution is correct.
>
>     I have tried using java plus the command-line arguments of eclipse
>     on the command line and that also works ok (local in ubuntu).
>
>     Has anybody experienced something similar? Any idea why this could
>     happen? how can I fix this?
>
>     Regards,
>     Marcela.
>
>

[Fabian Hueske-2]

Yes, env.setParallelism(1) fixes the parallelism of all operators to 1 (unless an operator overrides this setting).

Can you identify at which position in the data flow the results start to diverge?

Best, Fabian

2016-02-29 17:57 GMT+01:00 Marcela Charfuelan <[hidden email]>:

Thanks Fabian,
I am using in both default options, since I am not testing in a cluster yet, just local in ubuntu, I am not specifying any parallelism.
just to test I set in the program env.setParallelism(1) and running with -p 1 (which I guess I would not need) but I am still getting the same issue.

Regards,
MArcela.


On 29.02.2016 16:44, Fabian Hueske wrote:

Hi Marcela,

do you run the algorithm in both setups with the same parallelism?

Best, Fabian

2016-02-26 16:52 GMT+01:00 Marcela Charfuelan
<[hidden email] <mailto:[hidden email]>>:

    Hello,

    I implemented an algorithm that includes iterations (EM algorithm)
    and I am getting different results when running in eclipse (Luna
    Release (4.4.0)) and when running in the command line using Flink
    run; the program does not crash is just that after the first
    iteration the results are different (wrong in the command line).

    The solution I am getting in eclipse, for each iteration, is the
    same that I would get if running the algorithm in octave for
    example, so I am sure the solution is correct.

    I have tried using java plus the command-line arguments of eclipse
    on the command line and that also works ok (local in ubuntu).

    Has anybody experienced something similar? Any idea why this could
    happen? how can I fix this?

    Regards,
    Marcela.

[Marcela Charfuelan]

Hi,
the iteration looks like:
DataSet<GMM> gmms = getInitialGMMDataSet(env);
               
IterativeDataSet<GMM> loop = gmms.iterate(50);

DataSet<GMM> newGMMs = features.map(new
Estep_ExpectationMaximisation()).withBroadcastSet(loop, "gmms")
                       .reduceGroup(new
Mstep_ExpectationMaximisation()).withBroadcastSet(loop, "gmms");
       
DataSet<GMM> finalGMMs = loop.closeWith(newGMMs)

in every iteration the gmms parameters should be updated, I have noticed
that the first iteration is ok, but afterwards start to get wrong... at
least in the command line (for example gmms.coeff for the three gmms
here should not sum up more that 1)

I have put the code here in case it helps:
https://github.com/marcelach1/EmExercise

Regards,
Marcela.


On 01.03.2016 10:58, Fabian Hueske wrote:

> Yes, env.setParallelism(1) fixes the parallelism of all operators to 1
> (unless an operator overrides this setting).
> Can you identify at which position in the data flow the results start to
> diverge?
>
> Best, Fabian
>
> 2016-02-29 17:57 GMT+01:00 Marcela Charfuelan
> <[hidden email] <mailto:[hidden email]>>:
>
>     Thanks Fabian,
>     I am using in both default options, since I am not testing in a
>     cluster yet, just local in ubuntu, I am not specifying any parallelism.
>     just to test I set in the program env.setParallelism(1) and running
>     with -p 1 (which I guess I would not need) but I am still getting
>     the same issue.
>
>     Regards,
>     MArcela.
>
>
>     On 29.02.2016 16:44, Fabian Hueske wrote:
>
>         Hi Marcela,
>
>         do you run the algorithm in both setups with the same parallelism?
>
>         Best, Fabian
>
>         2016-02-26 16:52 GMT+01:00 Marcela Charfuelan
>         <[hidden email]
>         <mailto:[hidden email]>
>         <mailto:[hidden email]
>         <mailto:[hidden email]>>>:
>
>              Hello,
>
>              I implemented an algorithm that includes iterations (EM
>         algorithm)
>              and I am getting different results when running in eclipse
>         (Luna
>              Release (4.4.0)) and when running in the command line using
>         Flink
>              run; the program does not crash is just that after the first
>              iteration the results are different (wrong in the command
>         line).
>
>              The solution I am getting in eclipse, for each iteration,
>         is the
>              same that I would get if running the algorithm in octave for
>              example, so I am sure the solution is correct.
>
>              I have tried using java plus the command-line arguments of
>         eclipse
>              on the command line and that also works ok (local in ubuntu).
>
>              Has anybody experienced something similar? Any idea why
>         this could
>              happen? how can I fix this?
>
>              Regards,
>              Marcela.
>
>
>
>

--
Dr. Marcela Charfuelan, Senior Researcher
TU Berlin, School of Electrical Engineering and Computer Sciences
Database Systems and Information Management (DIMA)
EN7, Einsteinufer 17, D-10587 Berlin
Room: EN 725  Phone: [phone]
URL: http://www.user.tu-berlin.de/charfuelan