Java Maps and Type Information

[Simone Robutti]

Hello,

to my knowledge is not possible to use a java.util.Map for example in a FlatMapFunction<java.util.Map, java.util.Map>. Is that correct? It gives a typer error at runtime and it doesn't work even with explicit TypeInformation hints.

Is there any way to make it work?

Thanks,

Simone

[Aljoscha Krettek]

Hi,
what kind of program are you writing? I just wrote a quick example using the DataStream API where I’m using Map<String, Tuple2<String, Integer>> as the output type of one of my MapFunctions.

Cheers,
Aljoscha

> On 01 Mar 2016, at 16:33, Simone Robutti <[hidden email]> wrote:
>
> Hello,
>
> to my knowledge is not possible to use a java.util.Map for example in a FlatMapFunction<java.util.Map, java.util.Map>. Is that correct? It gives a typer error at runtime and it doesn't work even with explicit TypeInformation hints.
>
> Is there any way to make it work?
>
> Thanks,
>
> Simone

[Simone Robutti]

I tried to simplify it to the bones but I'm actually defining a custom MapFunction<java.util.Map<String,Object>,java.util.Map<String,Object>> that even with a simple identity function fails at runtime giving me the following error:

>Exception in thread "main" org.apache.flink.api.common.functions.InvalidTypesException: The return type of function 'main(Main.java:45)' could not be determined automatically, due to type erasure. You can give type information hints by using the returns(...) method on the result of the transformation call, or by letting your function implement the 'ResultTypeQueryable' interface.

where the line 45 is the line where I invoke the map function.

Here the piece of code:

ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();

        env.setParallelism(2);

        Map<String,Object> inputMap = new HashMap<String,Object>();

        inputMap.put("sepal_width",2.0);

        inputMap.put("sepal_length",2.0);

        inputMap.put("petal_width",2.0);

        inputMap.put("petal_length",2.0);

        MapFunction operator=new MapFunction<Map<String,Object>,Map<String,Object>>(){

            @Override

            public Map<String, Object> map(Map<String, Object> stringObjectMap) throws Exception {

                return stringObjectMap;

            }

        };

        List<Map<String, Object>> input = new LinkedList<>();

        input.add(inputMap);

        DataSource<Map<String, Object>> dataset = env.fromCollection(input);

        List<java.util.Map<FieldName, Object>> collectedResult = dataset.map(operator).collect();

2016-03-01 16:42 GMT+01:00 Aljoscha Krettek <[hidden email]>:

Hi,
what kind of program are you writing? I just wrote a quick example using the DataStream API where I’m using Map<String, Tuple2<String, Integer>> as the output type of one of my MapFunctions.

Cheers,
Aljoscha

> On 01 Mar 2016, at 16:33, Simone Robutti <[hidden email]> wrote:
>
> Hello,
>
> to my knowledge is not possible to use a java.util.Map for example in a FlatMapFunction<java.util.Map, java.util.Map>. Is that correct? It gives a typer error at runtime and it doesn't work even with explicit TypeInformation hints.
>
> Is there any way to make it work?
>
> Thanks,
>
> Simone

[Simone Robutti]

Ok, I made it work but there's still an issue. I used .returns(java.util.Map.class) after the "map" call and it works with this simple function but it doesn't compile with my CustomMapFunction that extends MapFunction. It gives a compilation error on the .returns() call. 

This is the case only if the variable operator is of type CustomMapFunction but if I do

> MapFunction operator = new CustomMapFunction();

it works again.

If I go back to 

> CustomMapFunction operator = new CustomMapFunction(); 

it gives this error:

>Error:(43, 87) java: no suitable method found for returns(java.lang.Class<java.util.Map>)

    method 

Should I open an issue?

2016-03-01 21:45 GMT+01:00 Simone Robutti <[hidden email]>:

I tried to simplify it to the bones but I'm actually defining a custom MapFunction<java.util.Map<String,Object>,java.util.Map<String,Object>> that even with a simple identity function fails at runtime giving me the following error:

>Exception in thread "main" org.apache.flink.api.common.functions.InvalidTypesException: The return type of function 'main(Main.java:45)' could not be determined automatically, due to type erasure. You can give type information hints by using the returns(...) method on the result of the transformation call, or by letting your function implement the 'ResultTypeQueryable' interface.

where the line 45 is the line where I invoke the map function.

Here the piece of code:

ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();

        env.setParallelism(2);

        Map<String,Object> inputMap = new HashMap<String,Object>();

        inputMap.put("sepal_width",2.0);

        inputMap.put("sepal_length",2.0);

        inputMap.put("petal_width",2.0);

        inputMap.put("petal_length",2.0);

        MapFunction operator=new MapFunction<Map<String,Object>,Map<String,Object>>(){

            @Override

            public Map<String, Object> map(Map<String, Object> stringObjectMap) throws Exception {

                return stringObjectMap;

            }

        };

        List<Map<String, Object>> input = new LinkedList<>();

        input.add(inputMap);

        DataSource<Map<String, Object>> dataset = env.fromCollection(input);

        List<java.util.Map<FieldName, Object>> collectedResult = dataset.map(operator).collect();

2016-03-01 16:42 GMT+01:00 Aljoscha Krettek <[hidden email]>:

Hi,
what kind of program are you writing? I just wrote a quick example using the DataStream API where I’m using Map<String, Tuple2<String, Integer>> as the output type of one of my MapFunctions.

Cheers,
Aljoscha

> On 01 Mar 2016, at 16:33, Simone Robutti <[hidden email]> wrote:
>
> Hello,
>
> to my knowledge is not possible to use a java.util.Map for example in a FlatMapFunction<java.util.Map, java.util.Map>. Is that correct? It gives a typer error at runtime and it doesn't work even with explicit TypeInformation hints.
>
> Is there any way to make it work?
>
> Thanks,
>
> Simone

[Aljoscha Krettek]

I’m afraid so, yes. I also tried out your example and it works if I add a .returns() after the .map() (as you did). Somehow the TypeExtractor seems to be acting up.

> On 02 Mar 2016, at 11:12, Simone Robutti <[hidden email]> wrote:
>
> >Error:(43, 87) java: no suitable method found for returns(java.lang.Class<java.util.Map>)
>     method