Re: Recursive Traversal of the Input Path Directory, Not working

Posted by Stefan Richter on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Recursive-Traversal-of-the-Input-Path-Directory-Not-working-tp13926p13961.html

No, that doesn’t make a difference and also works.

Am 23.06.2017 um 11:40 schrieb Adarsh Jain <[hidden email]>:

I am using "val env = ExecutionEnvironment.getExecutionEnvironment", can this be the problem?

With "import org.apache.flink.api.scala.ExecutionEnvironment"

Using scala in my program.

Regards,

Adarsh 

On Fri, Jun 23, 2017 at 3:01 PM, Stefan Richter <[hidden email]> wrote:

I just copy pasted your code, adding the missing "val env = LocalEnvironment.createLocalEnvironment()" and exchanged the string with a local directory for some test files that I created. No other changes.

Am 23.06.2017 um 11:25 schrieb Adarsh Jain <[hidden email]>:

<img width="0" height="0" class="m_1627718801731494630mailtrack-img" style="float:right" alt="" src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7">Hi Stefan,

Thanks for your efforts in checking the same, still doesn't work for me. 

Can you copy paste the code you used maybe I am doing some silly mistake and am not able to figure out the same.

Thanks again.

Regards,

Adarsh

On Fri, Jun 23, 2017 at 2:32 PM, Stefan Richter <[hidden email]> wrote:

Hi,

I tried this out on the current master and the 1.3 release and both work for me everything works exactly as expected, for file names, a directory, and even nested directories.

Best,

Stefan

Am 22.06.2017 um 21:13 schrieb Adarsh Jain <[hidden email]>:

Hi Stefan,

Yes your understood right, when I give full path till the filename it works fine however when I give path till 

directory it does not read the data, doesn't print any exceptions too ... I am also not sure why it is behaving like this.

Should be easily replicable, in case you can try. Will be really helpful.

Regards,

Adarsh

On Thu, Jun 22, 2017 at 9:00 PM, Stefan Richter <[hidden email]> wrote:

Hi,

I am not sure I am getting the problem right: the code works if you use a file name, but it does not work for directories? What exactly is not working? Do you get any exceptions?

Best,

Stefan

Am 22.06.2017 um 17:01 schrieb Adarsh Jain <[hidden email]>:

Hi,

I am trying to use "Recursive Traversal of the Input Path Directory" in Flink 1.3 using scala. Snippet of my code below. If I give exact file name it is working fine. Ref https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/batch/index.html

import org.apache.flink.api.java.utils.ParameterTool

import org.apache.flink.api.java.{DataSet, ExecutionEnvironment}

import org.apache.flink.configuration.Configuration

val config = new Configuration

    config.setBoolean("recursive.file.enumeration",true)

val featuresSource: String = "file:///Users/adarsh/Documents/testData/featurecsv/31c710ac40/2017/06/22"

val testInput = env.readTextFile(featuresSource).withParameters(config)

testInput.print()

Please guide how to fix this.

Regards,

Adarsh