Keyed function type erasure problem.

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Keyed function type erasure problem.

yunfan123
This post was updated on .
Just small change from PojoExample:
My problem is why the class like SelectorContainer can't as keyed function in flink
Full code please look up PojoExample.java
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/t921/PojoExample.java
public class PojoExample<KEY> {
        private SelectorContainer<KEY> selectorContainer;
        // KeySelector class like this can't be used in flink !!!
        public static class SelectorContainer<K> implements KeySelector<Word ,K> {
                public KeySelector<Word, K> keySelector;
                public SelectorContainer(KeySelector<Word, K> keySelector) {
                        this.keySelector = keySelector;
                }

                @Override
                public K getKey(Word value) throws Exception {
                        return keySelector.getKey(value);
                }
        }
        public void run() {
              dataStream.keyed(selectorContainer);
        }
       
}

This will cause:
Type of TypeVariable 'K' in 'class
org.apache.flink.streaming.examples.wordcount.PojoExample$SelectorContainer'
could not be determined. This is most likely a type erasure problem. The
type extraction currently supports types with generic variables only in
cases where all variables in the return type can be deduced from the input
type(s).

org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:915)

org.apache.flink.api.java.typeutils.TypeExtractor.privateCreateTypeInfo(TypeExtractor.java:836)

org.apache.flink.api.java.typeutils.TypeExtractor.getUnaryOperatorReturnType(TypeExtractor.java:622)

org.apache.flink.api.java.typeutils.TypeExtractor.getKeySelectorTypes(TypeExtractor.java:443)

org.apache.flink.api.java.typeutils.TypeExtractor.getKeySelectorTypes(TypeExtractor.java:436)

org.apache.flink.streaming.api.datastream.KeyedStream.<init>(KeyedStream.java:108)

org.apache.flink.streaming.api.datastream.DataStream.keyBy(DataStream.java:263)

org.apache.flink.streaming.examples.wordcount.PojoExample.execute(PojoExample.java:86)

org.apache.flink.streaming.examples.wordcount.PojoExample.main(PojoExample.java:108)

It sames can't use a class contains the KeySelector ?



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: Keyed function type erasure problem.

Fabian Hueske-2
Hi,

The problem is that Flink cannot determine the key type produced by the KeySelector because the type is a generic.
The type information is lost at runtime due to Java's type erasure.

You can try to implement the ResultTypeQueryable interface with the KeySelector and tell Flink the key type directly.

Best,
Fabian

2017-09-11 12:38 GMT+02:00 yunfan123 <[hidden email]>:
Just small change from PojoExample:
PojoExample.java
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/t921/PojoExample.java>
public class PojoExample<KEY> {
        private SelectorContainer<KEY> selectorContainer;
        // KeySelector class like this can't be used in flink !!!
        public static class SelectorContainer<K> implements KeySelector<Word ,K> {
                public KeySelector<Word, K> keySelector;
                public SelectorContainer(KeySelector<Word, K> keySelector) {
                        this.keySelector = keySelector;
                }

                @Override
                public K getKey(Word value) throws Exception {
                        return keySelector.getKey(value);
                }
        }
        public void run() {
              dataStream.keyed(selectorContainer);
        }

}

This will cause:
Type of TypeVariable 'K' in 'class
org.apache.flink.streaming.examples.wordcount.PojoExample$SelectorContainer'
could not be determined. This is most likely a type erasure problem. The
type extraction currently supports types with generic variables only in
cases where all variables in the return type can be deduced from the input
type(s).

org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:915)

org.apache.flink.api.java.typeutils.TypeExtractor.privateCreateTypeInfo(TypeExtractor.java:836)

org.apache.flink.api.java.typeutils.TypeExtractor.getUnaryOperatorReturnType(TypeExtractor.java:622)

org.apache.flink.api.java.typeutils.TypeExtractor.getKeySelectorTypes(TypeExtractor.java:443)

org.apache.flink.api.java.typeutils.TypeExtractor.getKeySelectorTypes(TypeExtractor.java:436)

org.apache.flink.streaming.api.datastream.KeyedStream.<init>(KeyedStream.java:108)

org.apache.flink.streaming.api.datastream.DataStream.keyBy(DataStream.java:263)

org.apache.flink.streaming.examples.wordcount.PojoExample.execute(PojoExample.java:86)

org.apache.flink.streaming.examples.wordcount.PojoExample.main(PojoExample.java:108)

It sames can't use a class contains the KeySelector ?



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Reply | Threaded
Open this post in threaded view
|

Re: Keyed function type erasure problem.

yunfan123