Read every file in a directory at once

[Flavio Pompermaier]

Hi to all,

I want to apply a map function to every file in a folder. Is there an easy way (or an already existing InputFormat) to do that?

Best,

Flavio

[Flavio Pompermaier]

My current solution is:

List<String> paths = new ArrayList<String>();
File dir = new File(BASE_DIR);
for (File f : dir.listFiles()) {
      paths.add(f.getName());
}
 DataSet<String> mail = env.fromCollection(paths).map(new FileToString(BASE_DIR)).

The FileToString does basically a map that return FileUtils.toString(new File(baseDir, filePath));

I hope this could help someone else..

On Thu, Feb 18, 2016 at 3:48 PM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I want to apply a map function to every file in a folder. Is there an easy way (or an already existing InputFormat) to do that?

Best,

Flavio

[Stephan Ewen]

Thanks for sharing this solution!

On Thu, Feb 18, 2016 at 4:02 PM, Flavio Pompermaier <[hidden email]> wrote:

My current solution is:

List<String> paths = new ArrayList<String>();
File dir = new File(BASE_DIR);
for (File f : dir.listFiles()) {
      paths.add(f.getName());
}
 DataSet<String> mail = env.fromCollection(paths).map(new FileToString(BASE_DIR)).

The FileToString does basically a map that return FileUtils.toString(new File(baseDir, filePath));

I hope this could help someone else..

On Thu, Feb 18, 2016 at 3:48 PM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I want to apply a map function to every file in a folder. Is there an easy way (or an already existing InputFormat) to do that?

Best,

Flavio