Table program cannot be compiled

[shkob1]

Hey,

While running a SQL query i get an OutOfMemoryError exception and "Table
program cannot be compiled" [2].
In my scenario i'm trying to enrich an event using an array of tags, each
tag has a boolean classification (like a WHERE clause) and with a custom
function i'm filtering the array to keep only TRUE results.
While i cannot publish the actual query the form of the query is as follows:

SELECT originalEvent, conditionalArray('Tag1', boolean_condition1, 'Tag2',
boolean_condition2).. FROM..
(more info at the use case here[1] )

I do have quite a lot of those tags (counting 126 tags now) but i had
something similar in the past without this error.
Any idea about it and if you can think of a workaround to this issue?

[1]
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Reconstruct-object-through-partial-select-query-td27782.html

[2]
org.apache.flink.api.common.InvalidProgramException: Table program cannot
be compiled. This is a bug. Please file an issue.
        at org.apache.flink.table.codegen.Compiler$class.compile(Compiler.scala:36)
        at
org.apache.flink.table.runtime.CRowProcessRunner.compile(CRowProcessRunner.scala:35)
        at
org.apache.flink.table.runtime.CRowProcessRunner.open(CRowProcessRunner.scala:49)
        at
org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
        at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
        at
org.apache.flink.streaming.api.operators.ProcessOperator.open(ProcessOperator.java:56)
        at
org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:424)
        at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:290)
        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.HashMap.newNode(HashMap.java:1734)
        at java.util.HashMap.putVal(HashMap.java:641)
        at java.util.HashMap.putMapEntries(HashMap.java:514)
        at java.util.HashMap.putAll(HashMap.java:784)
        at
org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:3322)
        at org.codehaus.janino.UnitCompiler.access$4900(UnitCompiler.java:212)
        at
org.codehaus.janino.UnitCompiler$8.visitLocalVariableDeclarationStatement(UnitCompiler.java:3207)
        at
org.codehaus.janino.UnitCompiler$8.visitLocalVariableDeclarationStatement(UnitCompiler.java:3175)
        at
org.codehaus.janino.Java$LocalVariableDeclarationStatement.accept(Java.java:3348)
        at
org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:3174)
        at
org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:3231)
        at org.codehaus.janino.UnitCompiler.access$3800(UnitCompiler.java:212)
        at org.codehaus.janino.UnitCompiler$8.visitBlock(UnitCompiler.java:3193)
        at org.codehaus.janino.UnitCompiler$8.visitBlock(UnitCompiler.java:3175)
        at org.codehaus.janino.Java$Block.accept(Java.java:2753)
        at
org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:3174)
        at
org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:3267)
        at org.codehaus.janino.UnitCompiler.access$4200(UnitCompiler.java:212)
        at
org.codehaus.janino.UnitCompiler$8.visitIfStatement(UnitCompiler.java:3197)
        at
org.codehaus.janino.UnitCompiler$8.visitIfStatement(UnitCompiler.java:3175)
        at org.codehaus.janino.Java$IfStatement.accept(Java.java:2923)
        at
org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:3174)
        at
org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:3163)
        at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2986)
        at
org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1313)
        at
org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1286)
        at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:785)
        at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:436)
        at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:212)
        at
org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:390)
        at
org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:385)
        at
org.codehaus.janino.Java$PackageMemberClassDeclaration.accept(Java.java:1405)



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[shkob1]

In a subsequent run i get
Caused by: org.codehaus.janino.JaninoRuntimeException: Code of method
"split$3681$(LDataStreamCalcRule$3682;)V" of class "DataStreamCalcRule$3682"
grows beyond 64 KB
 



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[shkob1]

BTW looking at past posts on this issue[1] it should have been fixed? i'm
using version 1.7.2
Also the recommendation was to use a custom function, though that's exactly
what im doing with the conditionalArray function[2]

Thanks!

[1]
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DataStreamCalcRule-1802-quot-grows-beyond-64-KB-when-execute-long-sql-td20832.html#a20841

[2]
public class ConditionalArrayFunction extends ScalarFunction {

    public static final String NAME = "conditionalArray";

    public String[] eval(Object... keyValues) {
        if (keyValues.length == 0) {
            return new String[]{};
        }
        final List keyValuesList = Arrays.asList(keyValues);
        List<String> trueItems = Lists.newArrayList();
        for (int i = 0; i < keyValuesList.size(); i = i + 2){
            final String key = (String)keyValuesList.get(i);
            final Object value = keyValuesList.get(i + 1);

            if (value != null && (boolean)value)
                trueItems.add(key);
        }
        return trueItems.toArray(new String[0]);
    }
}




--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Andrey Zagrebin-3]

Hi, I am looping in Timo and Dawid to look at the problem.

On Tue, May 14, 2019 at 9:12 PM shkob1 <[hidden email]> wrote:

BTW looking at past posts on this issue[1] it should have been fixed? i'm
using version 1.7.2
Also the recommendation was to use a custom function, though that's exactly
what im doing with the conditionalArray function[2]

Thanks!

[1]
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DataStreamCalcRule-1802-quot-grows-beyond-64-KB-when-execute-long-sql-td20832.html#a20841

[2]
public class ConditionalArrayFunction extends ScalarFunction {

    public static final String NAME = "conditionalArray";

    public String[] eval(Object... keyValues) {
        if (keyValues.length == 0) {
            return new String[]{};
        }
        final List keyValuesList = Arrays.asList(keyValues);
        List<String> trueItems = Lists.newArrayList();
        for (int i = 0; i < keyValuesList.size(); i = i + 2){
            final String key = (String)keyValuesList.get(i);
            final Object value = keyValuesList.get(i + 1);

            if (value != null && (boolean)value)
                trueItems.add(key);
        }
        return trueItems.toArray(new String[0]);
    }
}




--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Timo Walther]

Hi,

too many arguments for calling a UDF could currently lead to "grows beyond 64 KB" and maybe also causes the GC exception. This is a known issue covered in https://issues.apache.org/jira/browse/FLINK-8921.

Could you also add the tags to the function itself? Maybe as a static map for constant time access outside of the eval method?

Regards,

Timo

Am 15.05.19 um 17:10 schrieb Andrey Zagrebin:

Hi, I am looping in Timo and Dawid to look at the problem.

On Tue, May 14, 2019 at 9:12 PM shkob1 <[hidden email]> wrote:

BTW looking at past posts on this issue[1] it should have been fixed? i'm
using version 1.7.2
Also the recommendation was to use a custom function, though that's exactly
what im doing with the conditionalArray function[2]

Thanks!

[1]
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DataStreamCalcRule-1802-quot-grows-beyond-64-KB-when-execute-long-sql-td20832.html#a20841

[2]
public class ConditionalArrayFunction extends ScalarFunction {

    public static final String NAME = "conditionalArray";

    public String[] eval(Object... keyValues) {
        if (keyValues.length == 0) {
            return new String[]{};
        }
        final List keyValuesList = Arrays.asList(keyValues);
        List<String> trueItems = Lists.newArrayList();
        for (int i = 0; i < keyValuesList.size(); i = i + 2){
            final String key = (String)keyValuesList.get(i);
            final Object value = keyValuesList.get(i + 1);

            if (value != null && (boolean)value)
                trueItems.add(key);
        }
        return trueItems.toArray(new String[0]);
    }
}




--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[shkob1]

Hi Timo,

Thanks for the link.
Not sure i understand your suggestion though, is the goal here reducing the
amount of parameters coming to the UDF? if thats the case i can maybe have
the tag names there, but still need the expressions to get evaluated before
entering the eval. Do you see this in a different way?

I tried moving the tag names to be a member within the function instead of a
parameter, but apparently i still have too many arguments.

Let me know if this is not what you meant.

Thanks!
Shahar



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Timo Walther]

Hi Shahar,

yes the number of parameters should be the issue for a cannot compile
exception. If you moved most of the constants to a member in the
function, it should actually work.

Do you have a little reproducible example somewhere?

Thanks,
Timo



Am 16.05.19 um 19:59 schrieb shkob1:

> Hi Timo,
>
> Thanks for the link.
> Not sure i understand your suggestion though, is the goal here reducing the
> amount of parameters coming to the UDF? if thats the case i can maybe have
> the tag names there, but still need the expressions to get evaluated before
> entering the eval. Do you see this in a different way?
>
> I tried moving the tag names to be a member within the function instead of a
> parameter, but apparently i still have too many arguments.
>
> Let me know if this is not what you meant.
>
> Thanks!
> Shahar
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/