Re: Recursive Traversal of the Input Path Directory, Not working

Posted by Stefan Richter on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Recursive-Traversal-of-the-Input-Path-Directory-Not-working-tp13926p13954.html

Hi,

I tried this out on the current master and the 1.3 release and both work for me everything works exactly as expected, for file names, a directory, and even nested directories.

Best,

Stefan

Am 22.06.2017 um 21:13 schrieb Adarsh Jain <[hidden email]>:

Hi Stefan,

Yes your understood right, when I give full path till the filename it works fine however when I give path till 

directory it does not read the data, doesn't print any exceptions too ... I am also not sure why it is behaving like this.

Should be easily replicable, in case you can try. Will be really helpful.

Regards,

Adarsh

On Thu, Jun 22, 2017 at 9:00 PM, Stefan Richter <[hidden email]> wrote:

Hi,

I am not sure I am getting the problem right: the code works if you use a file name, but it does not work for directories? What exactly is not working? Do you get any exceptions?

Best,

Stefan

Am 22.06.2017 um 17:01 schrieb Adarsh Jain <[hidden email]>:

<img width="0" height="0" class="m_8394359328080938236mailtrack-img" style="float:right" alt="" src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7">Hi,

I am trying to use "Recursive Traversal of the Input Path Directory" in Flink 1.3 using scala. Snippet of my code below. If I give exact file name it is working fine. Ref https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/batch/index.html

import org.apache.flink.api.java.utils.ParameterTool

import org.apache.flink.api.java.{DataSet, ExecutionEnvironment}

import org.apache.flink.configuration.Configuration

val config = new Configuration

    config.setBoolean("recursive.file.enumeration",true)

val featuresSource: String = "file:///Users/adarsh/Documents/testData/featurecsv/31c710ac40/2017/06/22"

val testInput = env.readTextFile(featuresSource).withParameters(config)

testInput.print()

Please guide how to fix this.

Regards,

Adarsh