add FLINK_LIB_DIR to classpath on yarn
Locked
add FLINK_LIB_DIR to classpath on yarn
 
|
Hello. I have a flink cluster on yarn.I wanna add FLINK_LIB_DIR to classpath. because hibernate.cfg.xml need to be on the classpath. when i'm using stand alone cluster, just add FLINK_LIB_DIR to FLINK_CLASSPATH. but on yarn, Fixing config.sh, yarn-session.sh and flink-daemon.sh is not working. Best Regards, Dong-iL, Kim. -- <A HREF="http://www.kiva.org" TARGET="_top">
<IMG SRC="http://www.kiva.org/images/bannerlong.png" WIDTH="460" HEIGHT="60" ALT="Kiva - loans that change lives" BORDER="0" ALIGN="BOTTOM"></A> |
Re: add FLINK_LIB_DIR to classpath on yarn -> add properties file to class path on yarn
|
|
I’saw the source code of flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java Flink ships the FLINK_LIB_DIR and add to classpath only jar files. I want to know how to add resource file to classpath. Best Regards, Dong-iL, Kim.
|
Re: add FLINK_LIB_DIR to classpath on yarn -> add properties file to class path on yarn
|
|
Looping in Max directly because he probably knows the Yarn stuff best.
@Max: Do you have any idea how to do this? On Fri, 22 Jul 2016 at 05:46 김동일 <[hidden email]> wrote:
|
Re: add FLINK_LIB_DIR to classpath on yarn -> add properties file to class path on yarn
|
|
Hi!
In the latest master and in the upcoming 1.1, all files in the lib folder will be shipped to the Yarn cluster and added to the class path. In Flink version <= 1.0.x no files will be added to the ship files by default (only the flink-dist*.jar will be shipped). Regardless of the version, if you use the `addShipFiles(..)` method on YarnClusterDescriptor or --yarnship parameter on the CLI, then all files specified will be shipped. Cheers, Max On Mon, Jul 25, 2016 at 1:39 PM, Aljoscha Krettek <[hidden email]> wrote: > Looping in Max directly because he probably knows the Yarn stuff best. > > @Max: Do you have any idea how to do this? > > On Fri, 22 Jul 2016 at 05:46 김동일 <[hidden email]> wrote: >> >> I’saw the source code of >> flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java >> Flink ships the FLINK_LIB_DIR and add to classpath only jar files. >> I want to know how to add resource file to classpath. >> >> Best Regards, >> Dong-iL, Kim. >> >> On Jul 22, 2016, at 4:28 AM, Dong iL, Kim <[hidden email]> wrote: >> >> Hello. >> I have a flink cluster on yarn. >> I wanna add FLINK_LIB_DIR to classpath. >> because hibernate.cfg.xml need to be on the classpath. >> when i'm using stand alone cluster, just add FLINK_LIB_DIR to >> FLINK_CLASSPATH. >> but on yarn, Fixing config.sh, yarn-session.sh and flink-daemon.sh is not >> working. >> >> Best Regards, >> Dong-iL, Kim. >> >> -- >> <A HREF="http://www.kiva.org" TARGET="_top"> >> <IMG SRC="http://www.kiva.org/images/bannerlong.png" WIDTH="460" >> HEIGHT="60" ALT="Kiva - loans that change lives" BORDER="0" >> ALIGN="BOTTOM"></A> |
Re: add FLINK_LIB_DIR to classpath on yarn -> add properties file to class path on yarn
|
|
Hi Max,
As discussed here , I have put my yaml file in the flink lib directory, but still I am not able to get this file from classpath. I am using Flink 1.1.1 and cfg4j to load the file from classpath. Running the job on YARN in EMR using the below command: ./bin/flink run <jar_name> Can you please let me know what I am doing wrong here. Regards, Vinay Patil |
|
|
Hi Vinay, the JobManager and TaskManager logs contain the classpath used when starting a container on YARN. Can you check if the yaml file is in the classpath? On Tue, Oct 25, 2016 at 8:28 AM, vinay patil <[hidden email]> wrote: Hi Max, |
Re: add FLINK_LIB_DIR to classpath on yarn -> add properties file to class path on yarn
|
|
Hi Robert, Sorry for the late response. I checked the TM and JM logs but am not able to see that file added in the classpath. I have deployed Flink on EMR, and placed the yaml file in /usr/lib/flink/lib folder. What am I doing wrong here ? Regards, Vinay Patil On Wed, Oct 26, 2016 at 1:25 PM, rmetzger0 [via Apache Flink User Mailing List archive.] <[hidden email]> wrote:
|
Re: add FLINK_LIB_DIR to classpath on yarn -> add properties file to class path on yarn
|
|
The JVM only accepts Jar files in the classpath. You will have to load
your custom files from the working directory of the node where the lib directory is shipped. By the way, the /lib directory is meant for Jar files. If you want to ship a custom file, it's better to add it to the ship files in the cluster descriptor or use the --yarnship option from the command-line. -Max On Thu, Nov 17, 2016 at 5:37 AM, vinay patil <[hidden email]> wrote: > Hi Robert, > > Sorry for the late response. > > I checked the TM and JM logs but am not able to see that file added in the > classpath. > > I have deployed Flink on EMR, and placed the yaml file in /usr/lib/flink/lib > folder. > > What am I doing wrong here ? > > Regards, > Vinay Patil > > On Wed, Oct 26, 2016 at 1:25 PM, rmetzger0 [via Apache Flink User Mailing > List archive.] <[hidden email]> wrote: >> >> Hi Vinay, >> >> the JobManager and TaskManager logs contain the classpath used when >> starting a container on YARN. Can you check if the yaml file is in the >> classpath? >> >> On Tue, Oct 25, 2016 at 8:28 AM, vinay patil <[hidden email]> wrote: >>> >>> Hi Max, >>> >>> As discussed here , I have put my yaml file in the flink lib directory, >>> but >>> still I am not able to get this file from classpath. >>> >>> I am using Flink 1.1.1 and cfg4j to load the file from classpath. >>> >>> Running the job on YARN in EMR using the below command: >>> >>> ./bin/flink run <jar_name> >>> >>> Can you please let me know what I am doing wrong here. >>> >>> Regards, >>> Vinay Patil >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/add-FLINK-LIB-DIR-to-classpath-on-yarn-tp8097p9702.html >>> Sent from the Apache Flink User Mailing List archive. mailing list >>> archive at Nabble.com. >> >> >> >> >> ________________________________ >> If you reply to this email, your message will be added to the discussion >> below: >> >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/add-FLINK-LIB-DIR-to-classpath-on-yarn-tp8097p9719.html >> To start a new topic under Apache Flink User Mailing List archive., email >> [hidden email] >> To unsubscribe from Apache Flink User Mailing List archive., click here. >> NAML > > > > ________________________________ > View this message in context: Re: add FLINK_LIB_DIR to classpath on yarn -> > add properties file to class path on yarn > > Sent from the Apache Flink User Mailing List archive. mailing list archive > at Nabble.com. |
Re: add FLINK_LIB_DIR to classpath on yarn -> add properties file to class path on yarn
|
|
Hi Max, When I provided the config file using --yarnship option , it worked. Thank you. However I did not get this : " it's better to add it to the ship files in the cluster descriptor" Can you please tell me how to do this ? Regards, Vinay Patil On Fri, Nov 18, 2016 at 4:12 AM, Maximilian Michels [via Apache Flink User Mailing List archive.] <[hidden email]> wrote: The JVM only accepts Jar files in the classpath. You will have to load |
Re: add FLINK_LIB_DIR to classpath on yarn -> add properties file to class path on yarn
|
|
Hi Vinay,
I was referring to setting up an deploying a YARN cluster directly through a Java program, instead of using the command-line interface. When you do that, you typically construct a YarnClusterDescriptor and parameterize it, then you call deploy() to deploy the cluster and create a YarnClusterClient for interaction. -Max On Fri, Nov 18, 2016 at 12:20 PM, vinay patil <[hidden email]> wrote: > Hi Max, > > When I provided the config file using --yarnship option , it worked. Thank > you. > > However I did not get this : " it's better to add it to the ship files in > the > cluster descriptor" Can you please tell me how to do this ? > > Regards, > Vinay Patil > > On Fri, Nov 18, 2016 at 4:12 AM, Maximilian Michels [via Apache Flink User > Mailing List archive.] <[hidden email]> wrote: >> >> The JVM only accepts Jar files in the classpath. You will have to load >> your custom files from the working directory of the node where the lib >> directory is shipped. >> >> By the way, the /lib directory is meant for Jar files. If you want to >> ship a custom file, it's better to add it to the ship files in the >> cluster descriptor or use the --yarnship option from the command-line. >> >> -Max >> >> >> On Thu, Nov 17, 2016 at 5:37 AM, vinay patil <[hidden email]> wrote: >> >> > Hi Robert, >> > >> > Sorry for the late response. >> > >> > I checked the TM and JM logs but am not able to see that file added in >> > the >> > classpath. >> > >> > I have deployed Flink on EMR, and placed the yaml file in >> > /usr/lib/flink/lib >> > folder. >> > >> > What am I doing wrong here ? >> > >> > Regards, >> > Vinay Patil >> > >> > On Wed, Oct 26, 2016 at 1:25 PM, rmetzger0 [via Apache Flink User >> > Mailing >> > List archive.] <[hidden email]> wrote: >> >> >> >> >> Hi Vinay, >> >> >> >> the JobManager and TaskManager logs contain the classpath used when >> >> starting a container on YARN. Can you check if the yaml file is in the >> >> classpath? >> >> >> >> On Tue, Oct 25, 2016 at 8:28 AM, vinay patil <[hidden email]> wrote: >> >>> >> >>> Hi Max, >> >>> >> >>> As discussed here , I have put my yaml file in the flink lib >> >>> directory, >> >>> but >> >>> still I am not able to get this file from classpath. >> >>> >> >>> I am using Flink 1.1.1 and cfg4j to load the file from classpath. >> >>> >> >>> Running the job on YARN in EMR using the below command: >> >>> >> >>> ./bin/flink run <jar_name> >> >>> >> >>> Can you please let me know what I am doing wrong here. >> >>> >> >>> Regards, >> >>> Vinay Patil >> >>> >> >>> >> >>> >> >>> -- >> >>> View this message in context: >> >>> >> >>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/add-FLINK-LIB-DIR-to-classpath-on-yarn-tp8097p9702.html >> >>> Sent from the Apache Flink User Mailing List archive. mailing list >> >>> archive at Nabble.com. >> >> >> >> >> >> >> >> >> >> ________________________________ >> >> If you reply to this email, your message will be added to the >> >> discussion >> >> below: >> >> >> >> >> >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/add-FLINK-LIB-DIR-to-classpath-on-yarn-tp8097p9719.html >> >> To start a new topic under Apache Flink User Mailing List archive., >> >> [hidden email] >> >> To unsubscribe from Apache Flink User Mailing List archive., click >> >> here. >> >> NAML >> > >> > >> > >> > ________________________________ >> > View this message in context: Re: add FLINK_LIB_DIR to classpath on yarn >> > -> >> > add properties file to class path on yarn >> > >> > Sent from the Apache Flink User Mailing List archive. mailing list >> > archive >> > at Nabble.com. >> >> >> ________________________________ >> If you reply to this email, your message will be added to the discussion >> below: >> >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/add-FLINK-LIB-DIR-to-classpath-on-yarn-tp8097p10187.html >> To start a new topic under Apache Flink User Mailing List archive., email >> [hidden email] >> To unsubscribe from Apache Flink User Mailing List archive., click here. >> NAML > > > > ________________________________ > View this message in context: Re: add FLINK_LIB_DIR to classpath on yarn -> > add properties file to class path on yarn > Sent from the Apache Flink User Mailing List archive. mailing list archive > at Nabble.com. |
Re: add FLINK_LIB_DIR to classpath on yarn -> add properties file to class path on yarn
|
|
OK, got that, thank you for your solution, it worked for me with --yarnship option Regards, Vinay Patil On Tue, Nov 22, 2016 at 4:44 PM, Maximilian Michels [via Apache Flink User Mailing List archive.] <[hidden email]> wrote: Hi Vinay, |
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |