rowTime from json nested timestamp field in SQL-Client

[Ashwin Sinha]

Hi Users,

In Flink1.5 SQL CLient, we are trying to define rowTime from a nested JSON element, but struggling with syntax.

JSON data format: https://pastebin.com/ByCLhEnF
YML table config: https://pastebin.com/cgEtQPDQ

Now, in above config, we want to access payload.after.modifiedon as rowTime column. We tried SQL query with aggregation on 'payload.after.modifiedon' as time but get this error.

Is there anyway where we can register nested timestamp field as rowTime for the source table?

--

::DISCLAIMER::

----------------------------------------------------------------------------------------------------------------------------------------------------

This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.

[Timo Walther]

Hi Ashwin,

the SQL Client is in an early development stage right now and has some limitations. Your problem is one of them. I files an issue for this: https://issues.apache.org/jira/browse/FLINK-9864

There is no easy solution to fix this problem. Maybe you can use processing-time for your windows?

Regards,
Timo

Am 16.07.18 um 14:25 schrieb Ashwin Sinha:

Hi Users,

In Flink1.5 SQL CLient, we are trying to define rowTime from a nested JSON element, but struggling with syntax.

JSON data format: https://pastebin.com/ByCLhEnF
YML table config: https://pastebin.com/cgEtQPDQ

Now, in above config, we want to access payload.after.modifiedon as rowTime column. We tried SQL query with aggregation on 'payload.after.modifiedon' as time but get this error.

Is there anyway where we can register nested timestamp field as rowTime for the source table?

--

::DISCLAIMER::

----------------------------------------------------------------------------------------------------------------------------------------------------

This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.

[Ashwin Sinha]

Thanks Timo for the clarification, but our processing actually involves aggregations on huge past data also, which won't be served by processing time. 

Is this a WIP feature?

On Mon, Jul 16, 2018 at 7:29 PM Timo Walther <[hidden email]> wrote:

Hi Ashwin,

the SQL Client is in an early development stage right now and has some limitations. Your problem is one of them. I files an issue for this: https://issues.apache.org/jira/browse/FLINK-9864

There is no easy solution to fix this problem. Maybe you can use processing-time for your windows?

Regards,
Timo

Am 16.07.18 um 14:25 schrieb Ashwin Sinha:

Hi Users,

In Flink1.5 SQL CLient, we are trying to define rowTime from a nested JSON element, but struggling with syntax.

JSON data format: https://pastebin.com/ByCLhEnF
YML table config: https://pastebin.com/cgEtQPDQ

Now, in above config, we want to access payload.after.modifiedon as rowTime column. We tried SQL query with aggregation on 'payload.after.modifiedon' as time but get this error.

Is there anyway where we can register nested timestamp field as rowTime for the source table?

--

::DISCLAIMER::

----------------------------------------------------------------------------------------------------------------------------------------------------

This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.

--

::DISCLAIMER::

----------------------------------------------------------------------------------------------------------------------------------------------------

This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.

[Ashwin Sinha]

Hi Timo,

We want to add this functionality in a forked branch. Can you guide us with steps to quickly apply a patch/fix for the same?

On Mon, Jul 16, 2018 at 9:06 PM Ashwin Sinha <[hidden email]> wrote:

Thanks Timo for the clarification, but our processing actually involves aggregations on huge past data also, which won't be served by processing time. 

Is this a WIP feature?

On Mon, Jul 16, 2018 at 7:29 PM Timo Walther <[hidden email]> wrote:

Hi Ashwin,

the SQL Client is in an early development stage right now and has some limitations. Your problem is one of them. I files an issue for this: https://issues.apache.org/jira/browse/FLINK-9864

There is no easy solution to fix this problem. Maybe you can use processing-time for your windows?

Regards,
Timo

Am 16.07.18 um 14:25 schrieb Ashwin Sinha:

Hi Users,

In Flink1.5 SQL CLient, we are trying to define rowTime from a nested JSON element, but struggling with syntax.

JSON data format: https://pastebin.com/ByCLhEnF
YML table config: https://pastebin.com/cgEtQPDQ

Now, in above config, we want to access payload.after.modifiedon as rowTime column. We tried SQL query with aggregation on 'payload.after.modifiedon' as time but get this error.

Is there anyway where we can register nested timestamp field as rowTime for the source table?

--

::DISCLAIMER::

----------------------------------------------------------------------------------------------------------------------------------------------------

This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.

--

--

::DISCLAIMER::

----------------------------------------------------------------------------------------------------------------------------------------------------

This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.

[Timo Walther]

Hi Ashwin,

if you quickly want to make this work you can look into org.apache.flink.table.descriptors.RowtimeValidator#getRowtimeComponents.

This is the component that converts the string property into a org.apache.flink.table.sources.tsextractors.TimestampExtractor. You can implement your custom timestamp extractor that performs some logic.

Regards,
Timo


Am 17.07.18 um 14:37 schrieb Ashwin Sinha:

Hi Timo,

We want to add this functionality in a forked branch. Can you guide us with steps to quickly apply a patch/fix for the same?

On Mon, Jul 16, 2018 at 9:06 PM Ashwin Sinha <[hidden email]> wrote:

Thanks Timo for the clarification, but our processing actually involves aggregations on huge past data also, which won't be served by processing time. 

Is this a WIP feature?

On Mon, Jul 16, 2018 at 7:29 PM Timo Walther <[hidden email]> wrote:

Hi Ashwin,

the SQL Client is in an early development stage right now and has some limitations. Your problem is one of them. I files an issue for this: https://issues.apache.org/jira/browse/FLINK-9864

There is no easy solution to fix this problem. Maybe you can use processing-time for your windows?

Regards,
Timo

Am 16.07.18 um 14:25 schrieb Ashwin Sinha:

Hi Users,

In Flink1.5 SQL CLient, we are trying to define rowTime from a nested JSON element, but struggling with syntax.

JSON data format: https://pastebin.com/ByCLhEnF
YML table config: https://pastebin.com/cgEtQPDQ

Now, in above config, we want to access payload.after.modifiedon as rowTime column. We tried SQL query with aggregation on 'payload.after.modifiedon' as time but get this error.

Is there anyway where we can register nested timestamp field as rowTime for the source table?

--

::DISCLAIMER::

----------------------------------------------------------------------------------------------------------------------------------------------------

This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.

--

--

::DISCLAIMER::

----------------------------------------------------------------------------------------------------------------------------------------------------

This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.