Sink - writeAsText problem

[Punit Naik]

Hello

I executed my Flink code in eclipse and it properly generated the output by creating a folder (as specified in the string) and placing output files in them.

But when I exported the project as JAR and ran the same code using ./flink run, it generated the output, but instead of creating a folder with files in it, it just created a single file (as specified in the string) (output was correct though).

Why does this happen? I want Flink to write its output in folder.

--

[Fabian Hueske-2]

Did you specify a parallelism? The default parallelism of a Flink instance is 1 [1].

You can set a different default parallelism in ./conf/flink-conf.yaml or pass a job specific parallelism with ./bin/flink using  the -p flag [2]. More options to define parallelism are in the docs [3].

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html#common-options

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/cli.html

[3] https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/common/index.html#parallel-execution

2016-05-03 9:26 GMT+02:00 Punit Naik <[hidden email]>:

Hello

I executed my Flink code in eclipse and it properly generated the output by creating a folder (as specified in the string) and placing output files in them.

But when I exported the project as JAR and ran the same code using ./flink run, it generated the output, but instead of creating a folder with files in it, it just created a single file (as specified in the string) (output was correct though).

Why does this happen? I want Flink to write its output in folder.

--

Thank You

Regards

Punit Naik

[Stephan Ewen]

Hi!

There is the option to always create a directory: "fs.output.always-create-directory"

See https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html#file-systems

Greetings,
Stephan

On Tue, May 3, 2016 at 9:26 AM, Punit Naik <[hidden email]> wrote:

Hello

I executed my Flink code in eclipse and it properly generated the output by creating a folder (as specified in the string) and placing output files in them.

But when I exported the project as JAR and ran the same code using ./flink run, it generated the output, but instead of creating a folder with files in it, it just created a single file (as specified in the string) (output was correct though).

Why does this happen? I want Flink to write its output in folder.

--

Thank You

Regards

Punit Naik

[Punit Naik]

Hi Stephen, Fabian

setting "fs.output.always-create-directory" to true in flink-config.yml worked!

On Tue, May 3, 2016 at 2:27 PM, Stephan Ewen <[hidden email]> wrote:

Hi!

There is the option to always create a directory: "fs.output.always-create-directory"

See https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html#file-systems

Greetings,
Stephan

On Tue, May 3, 2016 at 9:26 AM, Punit Naik <[hidden email]> wrote:

Hello

I executed my Flink code in eclipse and it properly generated the output by creating a folder (as specified in the string) and placing output files in them.

But when I exported the project as JAR and ran the same code using ./flink run, it generated the output, but instead of creating a folder with files in it, it just created a single file (as specified in the string) (output was correct though).

Why does this happen? I want Flink to write its output in folder.

--

Thank You

Regards

Punit Naik

--

[Fabian Hueske-2]

Yes, but be aware that your program runs with parallelism 1 if you do not configure the parallelism.

2016-05-03 11:07 GMT+02:00 Punit Naik <[hidden email]>:

Hi Stephen, Fabian

setting "fs.output.always-create-directory" to true in flink-config.yml worked!

On Tue, May 3, 2016 at 2:27 PM, Stephan Ewen <[hidden email]> wrote:

Hi!

There is the option to always create a directory: "fs.output.always-create-directory"

See https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html#file-systems

Greetings,
Stephan

On Tue, May 3, 2016 at 9:26 AM, Punit Naik <[hidden email]> wrote:

Hello

I executed my Flink code in eclipse and it properly generated the output by creating a folder (as specified in the string) and placing output files in them.

But when I exported the project as JAR and ran the same code using ./flink run, it generated the output, but instead of creating a folder with files in it, it just created a single file (as specified in the string) (output was correct though).

Why does this happen? I want Flink to write its output in folder.

--

Thank You

Regards

Punit Naik

--

Thank You

Regards

Punit Naik

[Punit Naik]

Yeah thanks for letting me know.

On 03-May-2016 2:40 PM, "Fabian Hueske" <[hidden email]> wrote:

Yes, but be aware that your program runs with parallelism 1 if you do not configure the parallelism.

2016-05-03 11:07 GMT+02:00 Punit Naik <[hidden email]>:

Hi Stephen, Fabian

setting "fs.output.always-create-directory" to true in flink-config.yml worked!

On Tue, May 3, 2016 at 2:27 PM, Stephan Ewen <[hidden email]> wrote:

Hi!

There is the option to always create a directory: "fs.output.always-create-directory"

See https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html#file-systems

Greetings,
Stephan

On Tue, May 3, 2016 at 9:26 AM, Punit Naik <[hidden email]> wrote:

Hello

I executed my Flink code in eclipse and it properly generated the output by creating a folder (as specified in the string) and placing output files in them.

But when I exported the project as JAR and ran the same code using ./flink run, it generated the output, but instead of creating a folder with files in it, it just created a single file (as specified in the string) (output was correct though).

Why does this happen? I want Flink to write its output in folder.

--

Thank You

Regards

Punit Naik

--

Thank You

Regards

Punit Naik