Manual creation of TypedValue instances in flink statefun
Locked
Manual creation of TypedValue instances in flink statefun
 
|
Hello list, we are manually building TypedValue instances to be sent to a python remote function (with a reqreply function builder). We create the typed value as follows (in Kotlin): override fun map(value: Tuple2<Long, MutableIterable>): TypedValue {
return TypedValue.newBuilder()
.setValue(getProtoValueByteString(value.f0, value.f1))
.setTypename(typeName)
.setHasValue(true)
.build()
}
However, when running our job on a flink cluster (Flink 1.12.3, ververica platform) the job is terminated with this exception: java.lang.NoSuchMethodError: org.apache.flink.statefun.sdk.reqreply.generated.TypedValue$Builder.setValue(Lcom/google/protobuf/ByteString;)Lorg/apache/flink/statefun/sdk/reqreply/generated/TypedValue$Builder although the setValue method is actually defined in the
TypedValue class. We checked whether the issue could be related to some conflict on protobuf versions, but our classes are generated with protoc 3.7.1, which is aligned with what is reported in the flink-statefun (v3.0.0) parent pom. Any idea on what could cause the issue?
Thanks and best regards Dario Bonino -- Ing. Dario Bonino, Ph.D e-m@il: [hidden email] www: https://www.linkedin.com/in/dariobonino <foaf:Person> <foaf:firstName>Dario</foaf:firstName> <foaf:surname>Bonino</foaf:surname> <foaf:msnChatID>[hidden email]</foaf:msnChatID> </foaf:Person><script type="application/ld+json"> { "@context": "http://schema.org", "@type": "Person", "name": "Dario Bonino", "jobTitle": "POst-doc Researcher", "affiliation": "PHPower s.r.l.", "additionalName": "Dario", "url": "https://www.linkedin.com/in/dariobonino", } </script> |
Re: Manual creation of TypedValue instances in flink statefun
|
|
Hi Bonino, If you've also included the statefun-sdk-java into the class path then, make sure that you are not using accidentally the shaded version of a TypedValue (check the package name) Let me know if that helped, and we'll try to debug this together otherwise. Cheers, Igal. On Mon, May 24, 2021 at 6:46 PM Bonino Dario <[hidden email]> wrote:
|
Re: Manual creation of TypedValue instances in flink statefun
|
|
Dear Igal, we are actually analyzing the issue. Our imported package is import
org.apache.flink.statefun.sdk.reqreply.generated.TypedValue however at runtime (we analyzed the method signature via reflection on the typed value builder), the signature of the TypedValue.Builder.setValue method is: org.apache.flink.statefun.sdk.reqreply.generated.TypedValue$Builder org.apache.flink.statefun.sdk.reqreply.generated.TypedValue$Builder.setValue(org.apache.flink.statefun.sdk.shaded.com.google.protobuf.ByteString) and as you can notice, uses the shaded version of protobuf ByteString. Instead we are attempting to pass a non-shaded version of ByteString as a parameter and this leads to the reported error. Any idea on how to overcome this issue? Cheers, Dario On 5/25/21 11:22 AM, Igal Shilman
wrote:
-- Ing. Dario Bonino, Ph.D e-m@il: [hidden email] www: https://www.linkedin.com/in/dariobonino <foaf:Person> <foaf:firstName>Dario</foaf:firstName> <foaf:surname>Bonino</foaf:surname> <foaf:msnChatID>[hidden email]</foaf:msnChatID> </foaf:Person><script type="application/ld+json"> { "@context": "http://schema.org", "@type": "Person", "name": "Dario Bonino", "jobTitle": "POst-doc Researcher", "affiliation": "PHPower s.r.l.", "additionalName": "Dario", "url": "https://www.linkedin.com/in/dariobonino", } </script> |
Re: Manual creation of TypedValue instances in flink statefun
|
|
Do you have the statefun-sdk-java as a dependency, if you are not using it can you remove it? On Tue, May 25, 2021 at 11:28 AM Bonino Dario <[hidden email]> wrote:
|
Re: Manual creation of TypedValue instances in flink statefun
|
|
And also, if you are using reflection to debug this, can you also print the full package name of the class TypedValue$Builder extends? My guess is that you are using the shaded version of the builder. The shaded version is part of the: statefun-java-sdk, if you can remove that dependency I believe it should be fine. Good luck, Igal. On Tue, May 25, 2021 at 12:04 PM Igal Shilman <[hidden email]> wrote:
|
Re: Manual creation of TypedValue instances in flink statefun
|
|
In reply to this post by Igal Shilman
Dear Igal, in the meanwhile we already tried to remove the statefun-sdk-java as a dependency and
actually the error is gone. Are there any guidelines to understand when the dependency is
mandatory and how to solve conflicts as such in those cases? Best regards Dario On 5/25/21 12:04 PM, Igal Shilman
wrote:
-- Ing. Dario Bonino, Ph.D e-m@il: [hidden email] www: https://www.linkedin.com/in/dariobonino <foaf:Person> <foaf:firstName>Dario</foaf:firstName> <foaf:surname>Bonino</foaf:surname> <foaf:msnChatID>[hidden email]</foaf:msnChatID> </foaf:Person><script type="application/ld+json"> { "@context": "http://schema.org", "@type": "Person", "name": "Dario Bonino", "jobTitle": "POst-doc Researcher", "affiliation": "PHPower s.r.l.", "additionalName": "Dario", "url": "https://www.linkedin.com/in/dariobonino", } </script> |
Re: Manual creation of TypedValue instances in flink statefun
|
|
Hi Bonino, I'm glad that the issue was resolved for you. Generally there are no additional StateFun constraints in that regard, we need to complete the shading process of Protobuf to avoid situations such as these (issue tracked here: FLINK-22584). Once that will be done, you can use that artifact from within the datastream integration sdk. Cheers, Igal. On Tue, May 25, 2021 at 12:15 PM Bonino Dario <[hidden email]> wrote:
|
| Free forum by Nabble | Edit this page |