Building Flink with hadoop 2.6
Locked
Building Flink with hadoop 2.6
 
|
Hi ; We need to test some things with flink and hadoop 2.6 (the trunc method). We’ve set up a build task on our Jenkins and everything seem okay. However when we replace the original jar from your 0.10-SNAPSHOT distribution by ours there are some missing dependencies (log4j, curator, and maybe others) and we get some ClassNotFoundException at runtime. Are we missing some build parameters ? Thanks in advance, B.R. |
|
|
Hi Gwen, can you tell us the "mvn" command you're using for building Flink? On Wed, Oct 14, 2015 at 4:37 PM, Gwenhael Pasquiers <[hidden email]> wrote:
|
RE: Building Flink with hadoop 2.6
|
|
Hi Robert ! I’m using ” mvn clean install -DskipTests -Dhadoop.version=2.6.0 “. From: Robert Metzger [mailto:[hidden email]]
Hi Gwen, can you tell us the "mvn" command you're using for building Flink? On Wed, Oct 14, 2015 at 4:37 PM, Gwenhael Pasquiers <[hidden email]> wrote: Hi ; We need to test some things with flink and hadoop 2.6 (the trunc method). We’ve set up a build task on our Jenkins and everything seem okay. However when we replace the original jar from your 0.10-SNAPSHOT distribution by ours there are some missing dependencies (log4j, curator, and maybe others) and
we get some ClassNotFoundException at runtime. Are we missing some build parameters ?
Thanks in advance, B.R. |
|
|
Great. Which classes can it not find at runtime? I'll try to build and run Flink with exactly the command you've provided. On Wed, Oct 14, 2015 at 4:49 PM, Gwenhael Pasquiers <[hidden email]> wrote:
|
RE: Building Flink with hadoop 2.6
|
|
The first class that it can not find is : org.apache.log4j.Level The org.apache.log4j package is not present in the fat jar I get from the mvn command, but it is in the one you distributed on your website. From: Robert Metzger [mailto:[hidden email]]
Great. Which classes can it not find at runtime? I'll try to build and run Flink with exactly the command you've provided. On Wed, Oct 14, 2015 at 4:49 PM, Gwenhael Pasquiers <[hidden email]> wrote: Hi Robert ! I’m using ” mvn clean install -DskipTests -Dhadoop.version=2.6.0 “. From: Robert
Metzger [mailto:[hidden email]]
Hi Gwen, can you tell us the "mvn" command you're using for building Flink? On Wed, Oct 14, 2015 at 4:37 PM, Gwenhael Pasquiers <[hidden email]> wrote: Hi ; We need to test some things with flink and hadoop 2.6 (the trunc method). We’ve set up a build task on our Jenkins and everything seem okay. However when we replace the original jar from your 0.10-SNAPSHOT distribution by ours there are some missing dependencies (log4j, curator, and maybe others) and
we get some ClassNotFoundException at runtime. Are we missing some build parameters ?
Thanks in advance, B.R. |
|
|
Ah, I know what's causing this issue. In the latest 0.10-SNAPSHOT, we have removed log4j from the fat jar. Can you copy everything from the lib/ folder from your maven build into the lib/ folder of your flink installation? Log4j is now in a separate jar in the lib/ folder . What about the curator dependency issue? On Wed, Oct 14, 2015 at 4:56 PM, Gwenhael Pasquiers <[hidden email]> wrote:
|
|
|
One more thing regarding the truncate method: Its supported as of Hadoop 2.7.0 (https://issues.apache.org/jira/browse/HDFS-3107) On Wed, Oct 14, 2015 at 5:00 PM, Robert Metzger <[hidden email]> wrote:
|
RE: Building Flink with hadoop 2.6
|
|
Yes … You’re right. Anyway, adding the log4j jar solved the issue and our app is working properly, thanks ! About curator, I just observed that it was not there anymore when comparing the old and new fatjars. But it’s probably now in another dependency,
anyway there is no curator-related error so it just probably moved. Thanks ! Gwen’ From: Robert Metzger [mailto:[hidden email]]
One more thing regarding the truncate method: Its supported as of Hadoop 2.7.0 (https://issues.apache.org/jira/browse/HDFS-3107) On Wed, Oct 14, 2015 at 5:00 PM, Robert Metzger <[hidden email]> wrote: Ah, I know what's causing this issue. In the latest 0.10-SNAPSHOT, we have removed log4j from the fat jar. Can you copy everything from the lib/ folder from your maven build into the lib/ folder of your flink installation? Log4j is now in a separate jar in the lib/ folder . What about the curator dependency issue? On Wed, Oct 14, 2015 at 4:56 PM, Gwenhael Pasquiers <[hidden email]> wrote: The first class that it can not find is : org.apache.log4j.Level The org.apache.log4j package is not present in the fat jar I get from the mvn command,
but it is in the one you distributed on your website. From: Robert
Metzger [mailto:[hidden email]]
Great. Which classes can it not find at runtime? I'll try to build and run Flink with exactly the command you've provided. On Wed, Oct 14, 2015 at 4:49 PM, Gwenhael Pasquiers <[hidden email]> wrote: Hi Robert ! I’m using ” mvn clean install -DskipTests -Dhadoop.version=2.6.0 “. From: Robert
Metzger [mailto:[hidden email]]
Hi Gwen, can you tell us the "mvn" command you're using for building Flink? On Wed, Oct 14, 2015 at 4:37 PM, Gwenhael Pasquiers <[hidden email]> wrote: Hi ; We need to test some things with flink and hadoop 2.6 (the trunc method). We’ve set up a build task on our Jenkins and everything seem okay. However when we replace the original jar from your 0.10-SNAPSHOT distribution by ours there are some missing dependencies (log4j, curator, and maybe others) and
we get some ClassNotFoundException at runtime. Are we missing some build parameters ?
Thanks in advance, B.R. |
|
|
Great. We are shading curator now into a different location, that's why you can't find it anymore. I suspect you're trying out our new exactly-once filesystem sinks. Please let us know how well its working for you and if you're missing something. Its a pretty new feature :) Also note that you can use the fs sinks with hadoop versions below 2.7.0, then we'll write some metadata containing the valid offsets. On Wed, Oct 14, 2015 at 5:18 PM, Gwenhael Pasquiers <[hidden email]> wrote:
|
RE: Building Flink with hadoop 2.6
|
|
Yes, we’re onto the exactly-once ; trying to write RCFiles (Parquet and ORCFiles are not compatible because of their footer). It seems to be working perfectly. As expected, Flink is falling back to .valid-length metadata on HDFS 2.6 (and 2.3). From: Robert Metzger [mailto:[hidden email]]
Great. We are shading curator now into a different location, that's why you can't find it anymore. I suspect you're trying out our new exactly-once filesystem sinks. Please let us know how well its working for you and if you're missing something. Its a pretty new feature :) Also note that you can use the fs sinks with hadoop versions below 2.7.0, then we'll write some metadata containing the valid offsets. On Wed, Oct 14, 2015 at 5:18 PM, Gwenhael Pasquiers <[hidden email]> wrote: Yes … You’re right. Anyway, adding the log4j jar solved the issue and our app is working properly, thanks ! About curator, I just observed that it was not there anymore when comparing the old
and new fatjars. But it’s probably now in another dependency, anyway there is no curator-related error so it just probably moved. Thanks ! Gwen’ From: Robert
Metzger [mailto:[hidden email]]
One more thing regarding the truncate method: Its supported as of Hadoop 2.7.0 (https://issues.apache.org/jira/browse/HDFS-3107) On Wed, Oct 14, 2015 at 5:00 PM, Robert Metzger <[hidden email]> wrote: Ah, I know what's causing this issue. In the latest 0.10-SNAPSHOT, we have removed log4j from the fat jar. Can you copy everything from the lib/ folder from your maven build into the lib/ folder of your flink installation? Log4j is now in a separate jar in the lib/ folder . What about the curator dependency issue? On Wed, Oct 14, 2015 at 4:56 PM, Gwenhael Pasquiers <[hidden email]> wrote: The first class that it can not find is : org.apache.log4j.Level The org.apache.log4j package is not present in the fat jar I get from the mvn command,
but it is in the one you distributed on your website. From: Robert
Metzger [mailto:[hidden email]]
Great. Which classes can it not find at runtime? I'll try to build and run Flink with exactly the command you've provided. On Wed, Oct 14, 2015 at 4:49 PM, Gwenhael Pasquiers <[hidden email]> wrote: Hi Robert ! I’m using ” mvn clean install -DskipTests -Dhadoop.version=2.6.0 “. From: Robert
Metzger [mailto:[hidden email]]
Hi Gwen, can you tell us the "mvn" command you're using for building Flink? On Wed, Oct 14, 2015 at 4:37 PM, Gwenhael Pasquiers <[hidden email]> wrote: Hi ; We need to test some things with flink and hadoop 2.6 (the trunc method). We’ve set up a build task on our Jenkins and everything seem okay. However when we replace the original jar from your 0.10-SNAPSHOT distribution by ours there are some missing dependencies (log4j, curator, and maybe others) and
we get some ClassNotFoundException at runtime. Are we missing some build parameters ?
Thanks in advance, B.R. |
| Free forum by Nabble | Edit this page |