Gelly vertex ID type requirements?
Locked
Gelly vertex ID type requirements?
 
|
Hello,
I am having some trouble building a graph where the vertex ID type is a POJO. Specifically, it would be a class that has two fields: a long field, and a field which is of another class that has four byte fields. (Both classes implement the Comparable interface, as the Gelly guide specifies.) If the outer class has a default constructor, then it is recognized as a POJO, but I get the following exception: Exception in thread "main" java.lang.IllegalArgumentException: Wrong field type: PojoType<malom.GameState, fields = [board: Long, sid: PojoType<malom.SectorId, fields = [b: Byte, bf: Byte, w: Byte, wf: Byte]>]> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122) at org.apache.flink.api.java.operators.Keys$ExpressionKeys.<init>(Keys.java:241) at org.apache.flink.api.java.operators.Keys$ExpressionKeys.<init>(Keys.java:203) at org.apache.flink.api.java.operators.CoGroupOperator$CoGroupOperatorSets.where(CoGroupOperator.java:458) at org.apache.flink.graph.Graph.inDegrees(Graph.java:701) at org.apache.flink.graph.spargel.VertexCentricIteration.createResultVerticesWithDegrees(VertexCentricIteration.java:610) at org.apache.flink.graph.spargel.VertexCentricIteration.createResult(VertexCentricIteration.java:180) at org.apache.flink.api.java.DataSet.runOperation(DataSet.java:1044) at org.apache.flink.graph.Graph.runVertexCentricIteration(Graph.java:1312) at malom.Retrograde.run(Retrograde.java:64) at malom.Solver.main(Solver.java:32) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140) Note, that originally the exception just said "Wrong field type", from which I had no idea what type is it referring to, so I modified line 241 of Keys.java to include the type in the msg like this: Preconditions.checkArgument(fieldType instanceof AtomicType, "Wrong field type: " + fieldType.toString()); On the other hand, if I comment out the default constructor of the outer class, then it is not a POJO, but only a GenericTypeInfo is created from it, which implements AtomicType, so the previous exception does not appear. Does this mean that the Vertex IDs cannot be POJOs? I am not sure if this is the intended behaviour. After all, if we have a POJO, that should always be better then if we have a generic type, right? (I am just guessing here, but maybe CompositeType could also be regarded as an AtomicType: the only method declared by the AtomicType interface is createComparator, which is also defined in CompositeType (of which PojoTypeInfo is a subclass), but with different parameters, but maybe CompositeType could implement AtomicType by delegating the createComparator call with all of the fields specified?) I encountered another problem, which is may or may not be related to the above: without the default constructor (the GenericTypeInfo case), the VertexCentricIteration "eats" my graph, that is, the result graph has zero vertices. I traced this problem to the first join in VertexCentricIteration.createResultVerticesWithDegrees, where the "degrees" DataSet is created: both inputs of the join (outDegrees and inDegrees) contains the correct data, but the result (degrees) is empty. Interestingly, this problem disappears, if I add JoinHint.REPARTITION_SORT_MERGE. Best regards, Gabor |
Re: Gelly vertex ID type requirements?
|
|
Thanks for reporting this issue. The "Wrong field type" error looks like a bug to me. This happens, because PojoType is neither a TupleType nor an AtomicType. To me it looks like the TupleTypeInfoBase condition should be generalized to CompositeType.I will look into this. Cheers, Fabian 2015-07-30 14:18 GMT+02:00 Gábor Gévay <[hidden email]>: Hello, |
Re: Gelly vertex ID type requirements?
|
|
Thanks for the response.
As a temporary workaround, I tried to change these problematic lines: } else { Preconditions.checkArgument(fieldType instanceof AtomicType, "Wrong field type: " + fieldType.toString()); keyFields.add(new FlatFieldDescriptor(keyId, fieldType)); } into this: } else if (fieldType instanceof AtomicType) { keyFields.add(new FlatFieldDescriptor(keyId, fieldType)); } else { Preconditions.checkArgument(fieldType instanceof PojoTypeInfo, "Wrong field type: " + fieldType.toString()); ((PojoTypeInfo)fieldType).getFlatFields("*", 0, keyFields); } But then I ran into another problem: The TypeExtractor creates the TupleTypeInfoBase for the Edge type of my graph with the following types: 0 = {PojoTypeInfo@1067} "PojoType<malom.GameState, fields = [board: Long, sid: PojoType<malom.SectorId, fields = [b: Byte, bf: Byte, w: Byte, wf: Byte]>]>" 1 = {GenericTypeInfo@1068} "GenericType<malom.GameState>" 2 = {ValueTypeInfo@1069} "ValueType<NullValue>" The problem here is that the first two types should clearly be the same, as the Edge type looks like this: public class Edge<K, V> extends Tuple3<K, K, V> I did a bit of debugging on this, and the source of the problem seems to be the mechanism in TypeExtractor that would detect recursive types (see the "alreadySeen" field in TypeExtractor), as it mistakes the second appearance of malom.GameState with a recursive field. Specifically the following happens: createTypeInfoWithTypeHierarchy starts to process the Edge type, and in line 433 it calls itself for the first field, which proceeds into the privateGetForClass case which correctly detects that it is a POJO, and correctly returns a PojoTypeInfo; but in the meantime in line 1190, privateGetForClass adds malom.GameState to "alreadySeen". Then the outer createTypeInfoWithTypeHierarchy approaches the second field, goes into privateGetForClass, which mistakenly returns a GenericTypeInfo, as it thinks in line 1186, that a recursive type is being processed. Should I open a Jira for this? A possible solution would be to change the alreadySeen field into a parameter of all the various type extraction methods, and make it contain only those types that are ancestors in the nesting hierarchy. Best regards, Gabor 2015-07-30 14:32 GMT+02:00 Fabian Hueske <[hidden email]>: > Thanks for reporting this issue. > The "Wrong field type" error looks like a bug to me. > This happens, because PojoType is neither a TupleType nor an AtomicType. To > me it looks like the TupleTypeInfoBase condition should be generalized to > CompositeType. > > I will look into this. > > Cheers, Fabian > > 2015-07-30 14:18 GMT+02:00 Gábor Gévay <[hidden email]>: >> >> Hello, >> >> I am having some trouble building a graph where the vertex ID type is >> a POJO. Specifically, it would be a class that has two fields: a long >> field, and a field which is of another class that has four byte >> fields. (Both classes implement the Comparable interface, as the Gelly >> guide specifies.) If the outer class has a default constructor, then >> it is recognized as a POJO, but I get the following exception: >> >> Exception in thread "main" java.lang.IllegalArgumentException: Wrong >> field type: PojoType<malom.GameState, fields = [board: Long, sid: >> PojoType<malom.SectorId, fields = [b: Byte, bf: Byte, w: Byte, wf: >> Byte]>]> >> at >> com.google.common.base.Preconditions.checkArgument(Preconditions.java:122) >> at >> org.apache.flink.api.java.operators.Keys$ExpressionKeys.<init>(Keys.java:241) >> at >> org.apache.flink.api.java.operators.Keys$ExpressionKeys.<init>(Keys.java:203) >> at >> org.apache.flink.api.java.operators.CoGroupOperator$CoGroupOperatorSets.where(CoGroupOperator.java:458) >> at org.apache.flink.graph.Graph.inDegrees(Graph.java:701) >> at >> org.apache.flink.graph.spargel.VertexCentricIteration.createResultVerticesWithDegrees(VertexCentricIteration.java:610) >> at >> org.apache.flink.graph.spargel.VertexCentricIteration.createResult(VertexCentricIteration.java:180) >> at org.apache.flink.api.java.DataSet.runOperation(DataSet.java:1044) >> at >> org.apache.flink.graph.Graph.runVertexCentricIteration(Graph.java:1312) >> at malom.Retrograde.run(Retrograde.java:64) >> at malom.Solver.main(Solver.java:32) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:497) >> at >> com.intellij.rt.execution.application.AppMain.main(AppMain.java:140) >> >> Note, that originally the exception just said "Wrong field type", from >> which I had no idea what type is it referring to, so I modified line >> 241 of Keys.java to include the type in the msg like this: >> Preconditions.checkArgument(fieldType instanceof AtomicType, "Wrong >> field type: " + fieldType.toString()); >> >> On the other hand, if I comment out the default constructor of the >> outer class, then it is not a POJO, but only a GenericTypeInfo is >> created from it, which implements AtomicType, so the previous >> exception does not appear. Does this mean that the Vertex IDs cannot >> be POJOs? I am not sure if this is the intended behaviour. After all, >> if we have a POJO, that should always be better then if we have a >> generic type, right? >> >> (I am just guessing here, but maybe CompositeType could also be >> regarded as an AtomicType: the only method declared by the AtomicType >> interface is createComparator, which is also defined in CompositeType >> (of which PojoTypeInfo is a subclass), but with different parameters, >> but maybe CompositeType could implement AtomicType by delegating the >> createComparator call with all of the fields specified?) >> >> I encountered another problem, which is may or may not be related to >> the above: without the default constructor (the GenericTypeInfo case), >> the VertexCentricIteration "eats" my graph, that is, the result graph >> has zero vertices. I traced this problem to the first join in >> VertexCentricIteration.createResultVerticesWithDegrees, where the >> "degrees" DataSet is created: both inputs of the join (outDegrees and >> inDegrees) contains the correct data, but the result (degrees) is >> empty. Interestingly, this problem disappears, if I add >> JoinHint.REPARTITION_SORT_MERGE. >> >> Best regards, >> Gabor > > |
Re: Gelly vertex ID type requirements?
|
|
Hi, I opened a JIRA (FLINK-2442) and submitted a PR (#963) for the "Wrong field type" problem.2015-07-30 16:29 GMT+02:00 Gábor Gévay <[hidden email]>: Thanks for the response. |
Re: Gelly vertex ID type requirements?
|
|
Hi,
> I opened a JIRA (FLINK-2442) and submitted a PR (#963) for the "Wrong field > type" problem. Thanks for the fix! > Is the other problem is addressed in FLINK-2437? Unfortunately no. FLINK-2437 is a minor thing compared to the other problem. I opened a Jira for it (FLINK-2447). Best, Gabor 2015-07-31 0:29 GMT+02:00 Fabian Hueske <[hidden email]>: > Hi, > > I opened a JIRA (FLINK-2442) and submitted a PR (#963) for the "Wrong field > type" problem. > Is the other problem is addressed in FLINK-2437? > > Cheers, Fabian > > 2015-07-30 16:29 GMT+02:00 Gábor Gévay <[hidden email]>: >> >> Thanks for the response. >> As a temporary workaround, I tried to change these problematic lines: >> >> } else { >> Preconditions.checkArgument(fieldType instanceof AtomicType, "Wrong >> field type: " + fieldType.toString()); >> keyFields.add(new FlatFieldDescriptor(keyId, fieldType)); >> } >> >> into this: >> >> } else if (fieldType instanceof AtomicType) { >> keyFields.add(new FlatFieldDescriptor(keyId, fieldType)); >> } else { >> Preconditions.checkArgument(fieldType instanceof PojoTypeInfo, >> "Wrong field type: " + fieldType.toString()); >> ((PojoTypeInfo)fieldType).getFlatFields("*", 0, keyFields); >> } >> >> >> But then I ran into another problem: The TypeExtractor creates the >> TupleTypeInfoBase for the Edge type of my graph with the following >> types: >> >> 0 = {PojoTypeInfo@1067} "PojoType<malom.GameState, fields = [board: >> Long, sid: PojoType<malom.SectorId, fields = [b: Byte, bf: Byte, w: >> Byte, wf: Byte]>]>" >> 1 = {GenericTypeInfo@1068} "GenericType<malom.GameState>" >> 2 = {ValueTypeInfo@1069} "ValueType<NullValue>" >> >> The problem here is that the first two types should clearly be the >> same, as the Edge type looks like this: >> public class Edge<K, V> extends Tuple3<K, K, V> >> >> I did a bit of debugging on this, and the source of the problem seems >> to be the mechanism in TypeExtractor that would detect recursive types >> (see the "alreadySeen" field in TypeExtractor), as it mistakes the >> second appearance of malom.GameState with a recursive field. >> >> Specifically the following happens: createTypeInfoWithTypeHierarchy >> starts to process the Edge type, and in line 433 it calls itself for >> the first field, which proceeds into the privateGetForClass case which >> correctly detects that it is a POJO, and correctly returns a >> PojoTypeInfo; but in the meantime in line 1190, privateGetForClass >> adds malom.GameState to "alreadySeen". Then the outer >> createTypeInfoWithTypeHierarchy approaches the second field, goes into >> privateGetForClass, which mistakenly returns a GenericTypeInfo, as it >> thinks in line 1186, that a recursive type is being processed. >> >> Should I open a Jira for this? >> >> A possible solution would be to change the alreadySeen field into a >> parameter of all the various type extraction methods, and make it >> contain only those types that are ancestors in the nesting hierarchy. >> >> Best regards, >> Gabor >> >> 2015-07-30 14:32 GMT+02:00 Fabian Hueske <[hidden email]>: >> > Thanks for reporting this issue. >> > The "Wrong field type" error looks like a bug to me. >> > This happens, because PojoType is neither a TupleType nor an AtomicType. >> > To >> > me it looks like the TupleTypeInfoBase condition should be generalized >> > to >> > CompositeType. >> > >> > I will look into this. >> > >> > Cheers, Fabian >> > >> > 2015-07-30 14:18 GMT+02:00 Gábor Gévay <[hidden email]>: >> >> >> >> Hello, >> >> >> >> I am having some trouble building a graph where the vertex ID type is >> >> a POJO. Specifically, it would be a class that has two fields: a long >> >> field, and a field which is of another class that has four byte >> >> fields. (Both classes implement the Comparable interface, as the Gelly >> >> guide specifies.) If the outer class has a default constructor, then >> >> it is recognized as a POJO, but I get the following exception: >> >> >> >> Exception in thread "main" java.lang.IllegalArgumentException: Wrong >> >> field type: PojoType<malom.GameState, fields = [board: Long, sid: >> >> PojoType<malom.SectorId, fields = [b: Byte, bf: Byte, w: Byte, wf: >> >> Byte]>]> >> >> at >> >> >> >> com.google.common.base.Preconditions.checkArgument(Preconditions.java:122) >> >> at >> >> >> >> org.apache.flink.api.java.operators.Keys$ExpressionKeys.<init>(Keys.java:241) >> >> at >> >> >> >> org.apache.flink.api.java.operators.Keys$ExpressionKeys.<init>(Keys.java:203) >> >> at >> >> >> >> org.apache.flink.api.java.operators.CoGroupOperator$CoGroupOperatorSets.where(CoGroupOperator.java:458) >> >> at org.apache.flink.graph.Graph.inDegrees(Graph.java:701) >> >> at >> >> >> >> org.apache.flink.graph.spargel.VertexCentricIteration.createResultVerticesWithDegrees(VertexCentricIteration.java:610) >> >> at >> >> >> >> org.apache.flink.graph.spargel.VertexCentricIteration.createResult(VertexCentricIteration.java:180) >> >> at >> >> org.apache.flink.api.java.DataSet.runOperation(DataSet.java:1044) >> >> at >> >> org.apache.flink.graph.Graph.runVertexCentricIteration(Graph.java:1312) >> >> at malom.Retrograde.run(Retrograde.java:64) >> >> at malom.Solver.main(Solver.java:32) >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >> at >> >> >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> >> at >> >> >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:497) >> >> at >> >> com.intellij.rt.execution.application.AppMain.main(AppMain.java:140) >> >> >> >> Note, that originally the exception just said "Wrong field type", from >> >> which I had no idea what type is it referring to, so I modified line >> >> 241 of Keys.java to include the type in the msg like this: >> >> Preconditions.checkArgument(fieldType instanceof AtomicType, "Wrong >> >> field type: " + fieldType.toString()); >> >> >> >> On the other hand, if I comment out the default constructor of the >> >> outer class, then it is not a POJO, but only a GenericTypeInfo is >> >> created from it, which implements AtomicType, so the previous >> >> exception does not appear. Does this mean that the Vertex IDs cannot >> >> be POJOs? I am not sure if this is the intended behaviour. After all, >> >> if we have a POJO, that should always be better then if we have a >> >> generic type, right? >> >> >> >> (I am just guessing here, but maybe CompositeType could also be >> >> regarded as an AtomicType: the only method declared by the AtomicType >> >> interface is createComparator, which is also defined in CompositeType >> >> (of which PojoTypeInfo is a subclass), but with different parameters, >> >> but maybe CompositeType could implement AtomicType by delegating the >> >> createComparator call with all of the fields specified?) >> >> >> >> I encountered another problem, which is may or may not be related to >> >> the above: without the default constructor (the GenericTypeInfo case), >> >> the VertexCentricIteration "eats" my graph, that is, the result graph >> >> has zero vertices. I traced this problem to the first join in >> >> VertexCentricIteration.createResultVerticesWithDegrees, where the >> >> "degrees" DataSet is created: both inputs of the join (outDegrees and >> >> inDegrees) contains the correct data, but the result (degrees) is >> >> empty. Interestingly, this problem disappears, if I add >> >> JoinHint.REPARTITION_SORT_MERGE. >> >> >> >> Best regards, >> >> Gabor >> > >> > > > |
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |