Timestamp data type mismatch

[Satyam Shekhar]

Hello,

I am running into an issue while trying to create a TableSource with rowtime attribute. I have configured TableSource to return produced type of Row(DataTypes.BIGINT, DataTypes.TIMESTAMP) via DataType TableSource<T>::getProducedDataType(). The returned DataStream has a flatmap operator that implements ResultTypeQueryable and returns typeinfo RowTypeInfo({Types.LONG, Types.SQL_TIMESTAMP}, {...}). 

Queries on this table source fail with the following error -

TableSource of type io.netspring.blaze.eval.BlazeTableSource returned a DataStream of data type Row(C0: Long, blaze_itime: Timestamp) that does not match with the data type ROW<`C0` BIGINT, `blaze_itime` TIMESTAMP(6)> declared by the TableSource.getProducedDataType() method. Please validate the implementation of the TableSource.

Queries on DataStream without the timestamp column work. I was also able to somewhat make it work with the timestamp column by changing the DataStream to return Types.LOCAL_DATE_TIME. However, I am curious to know why Types.TIMESTAMP does not match with DataTypes.TIMESTAMP.

Regards,

Satyam

[Dawid Wysakowicz-2]

Hi Satyam,

The thing is that Types.SQL_TIMESTAMP uses java.sql.Timestamp and serializes it as long (millis since epoch) and thus have milliseconds precision. The default precision for a DataTypes.TIMESTAMP is 6 and the default bridging class is LocalDataTime.

It should work if you return DataTypes.TIMESTAMP(3).bridgedTo(java.sql.Timestamp.class) in the getProducedDataType.

Best,

Dawid

On 08/06/2020 10:08, Satyam Shekhar wrote:

Hello,

I am running into an issue while trying to create a TableSource with rowtime attribute. I have configured TableSource to return produced type of Row(DataTypes.BIGINT, DataTypes.TIMESTAMP) via DataType TableSource<T>::getProducedDataType(). The returned DataStream has a flatmap operator that implements ResultTypeQueryable and returns typeinfo RowTypeInfo({Types.LONG, Types.SQL_TIMESTAMP}, {...}). 

Queries on this table source fail with the following error -

TableSource of type io.netspring.blaze.eval.BlazeTableSource returned a DataStream of data type Row(C0: Long, blaze_itime: Timestamp) that does not match with the data type ROW<`C0` BIGINT, `blaze_itime` TIMESTAMP(6)> declared by the TableSource.getProducedDataType() method. Please validate the implementation of the TableSource.

Queries on DataStream without the timestamp column work. I was also able to somewhat make it work with the timestamp column by changing the DataStream to return Types.LOCAL_DATE_TIME. However, I am curious to know why Types.TIMESTAMP does not match with DataTypes.TIMESTAMP.

Regards,

Satyam

[Satyam Shekhar]

Thanks Dawid for the explanation.

Your suggested approach makes things work.

Regards,

Satyam

On Mon, Jun 8, 2020 at 1:18 AM Dawid Wysakowicz <[hidden email]> wrote:

Hi Satyam,

The thing is that Types.SQL_TIMESTAMP uses java.sql.Timestamp and serializes it as long (millis since epoch) and thus have milliseconds precision. The default precision for a DataTypes.TIMESTAMP is 6 and the default bridging class is LocalDataTime.

It should work if you return DataTypes.TIMESTAMP(3).bridgedTo(java.sql.Timestamp.class) in the getProducedDataType.

Best,

Dawid

On 08/06/2020 10:08, Satyam Shekhar wrote:

Hello,

I am running into an issue while trying to create a TableSource with rowtime attribute. I have configured TableSource to return produced type of Row(DataTypes.BIGINT, DataTypes.TIMESTAMP) via DataType TableSource<T>::getProducedDataType(). The returned DataStream has a flatmap operator that implements ResultTypeQueryable and returns typeinfo RowTypeInfo({Types.LONG, Types.SQL_TIMESTAMP}, {...}). 

Queries on this table source fail with the following error -

TableSource of type io.netspring.blaze.eval.BlazeTableSource returned a DataStream of data type Row(C0: Long, blaze_itime: Timestamp) that does not match with the data type ROW<`C0` BIGINT, `blaze_itime` TIMESTAMP(6)> declared by the TableSource.getProducedDataType() method. Please validate the implementation of the TableSource.

Queries on DataStream without the timestamp column work. I was also able to somewhat make it work with the timestamp column by changing the DataStream to return Types.LOCAL_DATE_TIME. However, I am curious to know why Types.TIMESTAMP does not match with DataTypes.TIMESTAMP.

Regards,

Satyam