Re: 回复：Re: flink one transformation end,the next transformation start

[Tzu-Li (Gordon) Tai]

Sorry, I just realized our previous conversation on this question was done via private email and not to [hidden email]

Forwarding the previous content of the conversation back to the mailing list:

On March 30, 2017 at 4:15:46 PM, [hidden email] ([hidden email]) wrote:

the job can run success,but the result is error.

the record 1 and the record 14 is same,so the vector compute cos value is 1,but on the yarn the value is not 1,and others are different from the result which run on local.

so,i guess,the step:

1 val data = env.readTextFile("hdfs:///....")//DataSet[(String,Array[String])]

2 val dataVec = computeDataVect(data)//DataSet[(String,Int,Array[(Int,Double)])]

3 val rescomm = computeCosSims (dataVec)//DataSet[(String,Array[(String,Double)])]

the record is from 1,2,3;but the step 3 must start when step 2 is end,because step 3 compute all record cos sim value must use all data.so is there some operate can set the step 3 start when step 2 is end.

----- 原始邮件 -----
发件人："Tzu-Li (Gordon) Tai" <[hidden email]>
收件人：[hidden email]
主题：Re: flink one transformation end,the next transformation start
日期：2017年03月30日 15点54分

Hi,

What exactly is the error you’re running into on YARN? You should be able to find them in the TM logs.

It’ll be helpful to understand the problem if you can provide them (just the relevant parts of the error will do).

Otherwise, I currently can not tell much from the information here.

Cheers,

Gordon

On March 30, 2017 at 3:33:53 PM, [hidden email] ([hidden email]) wrote:

hi,all,

i run a job,it is :

---------------------------------------------------------

val data = env.readTextFile("hdfs:///....")//DataSet[(String,Array[String])]

val dataVec = computeDataVect(data)//DataSet[(String,Int,Array[(Int,Double)])]

val rescomm = computeCosSims (dataVec)//DataSet[(String,Array[(String,Double)])]

but when run on the yarn cluster,the result was error,the job can success;and run on the local,in eclipse on my computer,the result is correct.

so,i run twice,

first:

val data = env.readTextFile("hdfs:///....")//DataSet[(String,Array[String])]

val dataVec = computeDataVect(data)//DataSet[(String,Int,Array[(Int,Double)])]

dataVec.writeAsText("hdfs///vec")//the vector is correct,

second:

val readVec = env.readTextFile("hdfs:///vec").map(...)//DataSet[(String,Int,Array[(Int,Double)])]

val rescomm = computeCosSims (dataVec)//DataSet[(String,Array[(String,Double)])]

and the result is correct,is the same as on local,in eclispe.

----------------------------------

someone can solve the problem?