Using FileInputFormat - org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit

[Mohit Anchlia]

 In trying to use this code I get the following error. Is it asking me to implement additional interface?

streamEnv.readFile(format, args[0], FileProcessingMode.PROCESS_CONTINUOUSLY, 2000).print();

[main] INFO com.s.flink.example.PDFInputFormat - Start streaming
[main] INFO org.apache.flink.api.java.typeutils.TypeExtractor - class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit does not contain a setter for field modificationTime
[main] INFO org.apache.flink.api.java.typeutils.TypeExtractor - class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit is not a valid POJO type because not all fields are valid POJO fields.

[Mohit Anchlia]

I even tried existing format but still same error:

FileInputFormat<String> fileInputFormat = new TextInputFormat(new Path(args[0]));

fileInputFormat.setNestedFileEnumeration(true);

streamEnv.readFile(fileInputFormat, args[0],

FileProcessingMode.PROCESS_CONTINUOUSLY, 10000L).print();

[main] INFO org.apache.flink.api.java.typeutils.TypeExtractor - class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit does not contain a setter for field modificationTime
[main] INFO org.apache.flink.api.java.typeutils.TypeExtractor - c

On Mon, Jul 31, 2017 at 1:07 PM, Mohit Anchlia <[hidden email]> wrote:

 In trying to use this code I get the following error. Is it asking me to implement additional interface?

streamEnv.readFile(format, args[0], FileProcessingMode.PROCESS_CONTINUOUSLY, 2000).print();

[main] INFO com.s.flink.example.PDFInputFormat - Start streaming
[main] INFO org.apache.flink.api.java.typeutils.TypeExtractor - class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit does not contain a setter for field modificationTime
[main] INFO org.apache.flink.api.java.typeutils.TypeExtractor - class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit is not a valid POJO type because not all fields are valid POJO fields.

[Fabian Hueske-2]

Hi Mohit,

these are just INFO log statements that do not necessarily indicate a problem.

Is the program working otherwise or do you observe other problems?

Best, Fabian

2017-08-01 0:32 GMT+02:00 Mohit Anchlia <[hidden email]>:

I even tried existing format but still same error:

FileInputFormat<String> fileInputFormat = new TextInputFormat(new Path(args[0]));

fileInputFormat.setNestedFileEnumeration(true);

streamEnv.readFile(fileInputFormat, args[0],

FileProcessingMode.PROCESS_CONTINUOUSLY, 10000L).print();

[main] INFO org.apache.flink.api.java.typeutils.TypeExtractor - class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit does not contain a setter for field modificationTime
[main] INFO org.apache.flink.api.java.typeutils.TypeExtractor - c

On Mon, Jul 31, 2017 at 1:07 PM, Mohit Anchlia <[hidden email]> wrote:

 In trying to use this code I get the following error. Is it asking me to implement additional interface?

streamEnv.readFile(format, args[0], FileProcessingMode.PROCESS_CONTINUOUSLY, 2000).print();

[main] INFO com.s.flink.example.PDFInputFormat - Start streaming
[main] INFO org.apache.flink.api.java.typeutils.TypeExtractor - class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit does not contain a setter for field modificationTime
[main] INFO org.apache.flink.api.java.typeutils.TypeExtractor - class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit is not a valid POJO type because not all fields are valid POJO fields.

[Mohit Anchlia]

I don't see the print output.

On Tue, Aug 1, 2017 at 2:08 AM, Fabian Hueske <[hidden email]> wrote:

Hi Mohit,

these are just INFO log statements that do not necessarily indicate a problem.

Is the program working otherwise or do you observe other problems?

Best, Fabian

2017-08-01 0:32 GMT+02:00 Mohit Anchlia <[hidden email]>:

I even tried existing format but still same error:

FileInputFormat<String> fileInputFormat = new TextInputFormat(new Path(args[0]));

fileInputFormat.setNestedFileEnumeration(true);

streamEnv.readFile(fileInputFormat, args[0],

FileProcessingMode.PROCESS_CONTINUOUSLY, 10000L).print();

[main] INFO org.apache.flink.api.java.typeutils.TypeExtractor - class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit does not contain a setter for field modificationTime
[main] INFO org.apache.flink.api.java.typeutils.TypeExtractor - c

On Mon, Jul 31, 2017 at 1:07 PM, Mohit Anchlia <[hidden email]> wrote:

 In trying to use this code I get the following error. Is it asking me to implement additional interface?

streamEnv.readFile(format, args[0], FileProcessingMode.PROCESS_CONTINUOUSLY, 2000).print();

[main] INFO com.s.flink.example.PDFInputFormat - Start streaming
[main] INFO org.apache.flink.api.java.typeutils.TypeExtractor - class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit does not contain a setter for field modificationTime
[main] INFO org.apache.flink.api.java.typeutils.TypeExtractor - class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit is not a valid POJO type because not all fields are valid POJO fields.

[Mohit Anchlia]

This doesn't work even with TextInputFormat. Not sure what's wrong.

On Tue, Aug 1, 2017 at 9:53 AM, Mohit Anchlia <[hidden email]> wrote:

I don't see the print output.

On Tue, Aug 1, 2017 at 2:08 AM, Fabian Hueske <[hidden email]> wrote:

Hi Mohit,

these are just INFO log statements that do not necessarily indicate a problem.

Is the program working otherwise or do you observe other problems?

Best, Fabian

2017-08-01 0:32 GMT+02:00 Mohit Anchlia <[hidden email]>:

I even tried existing format but still same error:

FileInputFormat<String> fileInputFormat = new TextInputFormat(new Path(args[0]));

fileInputFormat.setNestedFileEnumeration(true);

streamEnv.readFile(fileInputFormat, args[0],

FileProcessingMode.PROCESS_CONTINUOUSLY, 10000L).print();

[main] INFO org.apache.flink.api.java.typeutils.TypeExtractor - class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit does not contain a setter for field modificationTime
[main] INFO org.apache.flink.api.java.typeutils.TypeExtractor - c

On Mon, Jul 31, 2017 at 1:07 PM, Mohit Anchlia <[hidden email]> wrote:

 In trying to use this code I get the following error. Is it asking me to implement additional interface?

streamEnv.readFile(format, args[0], FileProcessingMode.PROCESS_CONTINUOUSLY, 2000).print();

[main] INFO com.s.flink.example.PDFInputFormat - Start streaming
[main] INFO org.apache.flink.api.java.typeutils.TypeExtractor - class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit does not contain a setter for field modificationTime
[main] INFO org.apache.flink.api.java.typeutils.TypeExtractor - class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit is not a valid POJO type because not all fields are valid POJO fields.

[Mohit Anchlia]

This was user induced problem - me. I wasn't calling streamenv.execute() :(

On Tue, Aug 1, 2017 at 1:29 PM, Mohit Anchlia <[hidden email]> wrote:

This doesn't work even with TextInputFormat. Not sure what's wrong.

On Tue, Aug 1, 2017 at 9:53 AM, Mohit Anchlia <[hidden email]> wrote:

I don't see the print output.

On Tue, Aug 1, 2017 at 2:08 AM, Fabian Hueske <[hidden email]> wrote:

Hi Mohit,

these are just INFO log statements that do not necessarily indicate a problem.

Is the program working otherwise or do you observe other problems?

Best, Fabian

2017-08-01 0:32 GMT+02:00 Mohit Anchlia <[hidden email]>:

I even tried existing format but still same error:

FileInputFormat<String> fileInputFormat = new TextInputFormat(new Path(args[0]));

fileInputFormat.setNestedFileEnumeration(true);

streamEnv.readFile(fileInputFormat, args[0],

FileProcessingMode.PROCESS_CONTINUOUSLY, 10000L).print();

[main] INFO org.apache.flink.api.java.typeutils.TypeExtractor - class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit does not contain a setter for field modificationTime
[main] INFO org.apache.flink.api.java.typeutils.TypeExtractor - c

On Mon, Jul 31, 2017 at 1:07 PM, Mohit Anchlia <[hidden email]> wrote:

 In trying to use this code I get the following error. Is it asking me to implement additional interface?

streamEnv.readFile(format, args[0], FileProcessingMode.PROCESS_CONTINUOUSLY, 2000).print();

[main] INFO com.s.flink.example.PDFInputFormat - Start streaming
[main] INFO org.apache.flink.api.java.typeutils.TypeExtractor - class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit does not contain a setter for field modificationTime
[main] INFO org.apache.flink.api.java.typeutils.TypeExtractor - class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit is not a valid POJO type because not all fields are valid POJO fields.