(DEPRECATED) Apache Flink User Mailing List archive.

State Migration with RocksDB MapState

classic Classic list List threaded Threaded
♦
♦
 5 messages Options
Cliff Resnick
Reply | Threaded
Open this post in threaded view
♦
♦
|

State Migration with RocksDB MapState

Cliff Resnick
After trying out state migration in 1.8 rc2 I ran into this hard stop below. The comment does not give an indication why rocksdb map state cannot be migrated, and I'm wondering what the status is, since we do need this functionality and would like to know if this is a long-term blocker or not. Anyone know?
Tzu-Li (Gordon) Tai
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: State Migration with RocksDB MapState

Tzu-Li (Gordon) Tai
Hi Cliff,

Thanks for bringing this up!
AFAIK, this wouldn't be a long-term blocker. I've just opened a JIRA to track this [1].

As explained in the JIRA ticket, the main reason this is disallowed in the initial support for state schema evolution was due to how migration was implemented in the RocksDB state backend.
Technically speaking, enabling this in the future is definitely possible.

Cheers,
Gordon

[1]  https://issues.apache.org/jira/browse/FLINK-11947

On Mon, Mar 18, 2019 at 11:20 AM Cliff Resnick <[hidden email]> wrote:
After trying out state migration in 1.8 rc2 I ran into this hard stop below. The comment does not give an indication why rocksdb map state cannot be migrated, and I'm wondering what the status is, since we do need this functionality and would like to know if this is a long-term blocker or not. Anyone know?
Cliff Resnick
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: State Migration with RocksDB MapState

Cliff Resnick
Hi Gordon,

I noticed there has been no movement on this issue and I'm wondering if I can find some way to work around this.
My MapState value is a case class container of Avro-generated SpecificRecords. If one SpecificRecord changes I am stuck. 

From the issue It seems like the blocker is around evolving the MapState key type.  That may be a nasty problem, but my key type is stable and will never change. What do you think the level of difficulty would be to add support for evolving only the value?

Also, if I use GenericRecord instead of SpecificRecord will the need for schema evolution still be triggered? Or does it actually go down to the avro schema rather than just the class serialVersionUID?






On Mon, Mar 18, 2019 at 1:10 AM Tzu-Li (Gordon) Tai <[hidden email]> wrote:
Hi Cliff,

Thanks for bringing this up!
AFAIK, this wouldn't be a long-term blocker. I've just opened a JIRA to track this [1].

As explained in the JIRA ticket, the main reason this is disallowed in the initial support for state schema evolution was due to how migration was implemented in the RocksDB state backend.
Technically speaking, enabling this in the future is definitely possible.

Cheers,
Gordon

[1]  https://issues.apache.org/jira/browse/FLINK-11947

On Mon, Mar 18, 2019 at 11:20 AM Cliff Resnick <[hidden email]> wrote:
After trying out state migration in 1.8 rc2 I ran into this hard stop below. The comment does not give an indication why rocksdb map state cannot be migrated, and I'm wondering what the status is, since we do need this functionality and would like to know if this is a long-term blocker or not. Anyone know?
Tzu-Li (Gordon) Tai
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: State Migration with RocksDB MapState

Tzu-Li (Gordon) Tai
Hi Cliff,

Thanks for bringing this up again.

I think it would make sense to at least move this forward be only exclusively checking the schema for user keys in MapState, and allow value schema evolution.
I'll comment on the JIRA about this, and also make it a blocker for 1.9.0 to make sure it will be resolved by then.

Concerning your question on GenericRecord:
The actual schema resolution is still performed using the Avro schema, so you will still bump into the same issue.

Best,
Gordon

On Wed, Apr 24, 2019 at 7:45 PM Cliff Resnick <[hidden email]> wrote:
Hi Gordon,

I noticed there has been no movement on this issue and I'm wondering if I can find some way to work around this.
My MapState value is a case class container of Avro-generated SpecificRecords. If one SpecificRecord changes I am stuck. 

From the issue It seems like the blocker is around evolving the MapState key type.  That may be a nasty problem, but my key type is stable and will never change. What do you think the level of difficulty would be to add support for evolving only the value?

Also, if I use GenericRecord instead of SpecificRecord will the need for schema evolution still be triggered? Or does it actually go down to the avro schema rather than just the class serialVersionUID?






On Mon, Mar 18, 2019 at 1:10 AM Tzu-Li (Gordon) Tai <[hidden email]> wrote:
Hi Cliff,

Thanks for bringing this up!
AFAIK, this wouldn't be a long-term blocker. I've just opened a JIRA to track this [1].

As explained in the JIRA ticket, the main reason this is disallowed in the initial support for state schema evolution was due to how migration was implemented in the RocksDB state backend.
Technically speaking, enabling this in the future is definitely possible.

Cheers,
Gordon

[1]  https://issues.apache.org/jira/browse/FLINK-11947

On Mon, Mar 18, 2019 at 11:20 AM Cliff Resnick <[hidden email]> wrote:
After trying out state migration in 1.8 rc2 I ran into this hard stop below. The comment does not give an indication why rocksdb map state cannot be migrated, and I'm wondering what the status is, since we do need this functionality and would like to know if this is a long-term blocker or not. Anyone know?
Cliff Resnick
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: State Migration with RocksDB MapState

Cliff Resnick
Great news! Thanks

On Thu, Apr 25, 2019, 2:59 AM Tzu-Li (Gordon) Tai <[hidden email]> wrote:
Hi Cliff,

Thanks for bringing this up again.

I think it would make sense to at least move this forward be only exclusively checking the schema for user keys in MapState, and allow value schema evolution.
I'll comment on the JIRA about this, and also make it a blocker for 1.9.0 to make sure it will be resolved by then.

Concerning your question on GenericRecord:
The actual schema resolution is still performed using the Avro schema, so you will still bump into the same issue.

Best,
Gordon

On Wed, Apr 24, 2019 at 7:45 PM Cliff Resnick <[hidden email]> wrote:
Hi Gordon,

I noticed there has been no movement on this issue and I'm wondering if I can find some way to work around this.
My MapState value is a case class container of Avro-generated SpecificRecords. If one SpecificRecord changes I am stuck. 

From the issue It seems like the blocker is around evolving the MapState key type.  That may be a nasty problem, but my key type is stable and will never change. What do you think the level of difficulty would be to add support for evolving only the value?

Also, if I use GenericRecord instead of SpecificRecord will the need for schema evolution still be triggered? Or does it actually go down to the avro schema rather than just the class serialVersionUID?






On Mon, Mar 18, 2019 at 1:10 AM Tzu-Li (Gordon) Tai <[hidden email]> wrote:
Hi Cliff,

Thanks for bringing this up!
AFAIK, this wouldn't be a long-term blocker. I've just opened a JIRA to track this [1].

As explained in the JIRA ticket, the main reason this is disallowed in the initial support for state schema evolution was due to how migration was implemented in the RocksDB state backend.
Technically speaking, enabling this in the future is definitely possible.

Cheers,
Gordon

[1]  https://issues.apache.org/jira/browse/FLINK-11947

On Mon, Mar 18, 2019 at 11:20 AM Cliff Resnick <[hidden email]> wrote:
After trying out state migration in 1.8 rc2 I ran into this hard stop below. The comment does not give an indication why rocksdb map state cannot be migrated, and I'm wondering what the status is, since we do need this functionality and would like to know if this is a long-term blocker or not. Anyone know?
Free forum by Nabble Edit this page