(DEPRECATED) Apache Flink User Mailing List archive.

flink sql read hive table throw java.lang.ArrayIndexOutOfBoundsException: 1024

classic Classic list List threaded Threaded
♦
♦
 2 messages Options
house－张浩
Reply | Threaded
Open this post in threaded view
♦
♦
|

flink sql read hive table throw java.lang.ArrayIndexOutOfBoundsException: 1024

house－张浩
when i use pyflink hive sql read data insert into es ,throw the follow exeception : the environment ： flink 1.11.2 flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar hive 3.1.2

2020-12-17 21:10:24,398 WARN  org.apache.flink.runtime.taskmanager.Task                    [] - Source: HiveTableSource(driver_id, driver_base_lc_p1, driver_90d_lc_p1, driver_30d_lc_p1, driver_14d_lc_p1, driver_180d_lc_p1, vehicle_base_lc_p1, driver_active_zone, is_incremental, dt) TablePath: algorithm.jiawei_oas_driver_features_for_incremental_hive2kafka, PartitionPruned: false, PartitionNums: null, ProjectedFields: [0, 8, 9] -> Calc(select=[driver_id, is_incremental, dt, () AS bdi_feature_create_time]) -> Sink: Sink(table=[default_catalog.default_database.0_demo4_903157246_tmp], fields=[driver_id, is_incremental, dt, bdi_feature_create_time]) (1/1) (98f4259c3d00fac9fc3482a4cdc8df3c) switched from RUNNING to FAILED.
    at org.apache.orc.impl.ConvertTreeReaderFactory$AnyIntegerTreeReader.nextVector(ConvertTreeReaderFactory.java:445) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1300) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.flink.connectors.hive.read.HiveVectorizedOrcSplitReader.reachedEnd(HiveVectorizedOrcSplitReader.java:99) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:90) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
    at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
    at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:213) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
2020-12-17 21:10:24,402 INFO  org.apache.flink.runtime.taskmanager.Task                    [] - Freeing task resources for Source: HiveTableSource(driver_id, driver_base_lc_p1, driver_90d_lc_p1, driver_30d_lc_p1, driver_14d_lc_p1, driver_180d_lc_p1, vehicle_base_lc_p1, driver_active_zone, is_incremental, dt) TablePath: algorithm.jiawei_oas_driver_features_for_incremental_hive2kafka, PartitionPruned: false, PartitionNums: null, ProjectedFields: [0, 8, 9] -> Calc(select=[driver_id, is_incremental, dt, () AS bdi_feature_create_time]) -> Sink: Sink(table=[default_catalog.default_database.0_demo4_903157246_tmp], fields=[driver_id, is_incremental, dt, bdi_feature_create_time]) (1/1) (98f4259c3d00fac9fc3482a4cdc8df3c).
java.lang.ArrayIndexOutOfBoundsException: 1024
    at org.apache.flink.orc.shim.OrcShimV210.nextBatch(OrcShimV210.java:35) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.flink.orc.shim.OrcShimV210.nextBatch(OrcShimV210.java:29) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.flink.orc.OrcSplitReader.ensureBatch(OrcSplitReader.java:134) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.orc.impl.TreeReaderFactory$LongTreeReader.nextVector(TreeReaderFactory.java:612) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.orc.impl.TreeReaderFactory$TreeReader.nextVector(TreeReaderFactory.java:269) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.orc.impl.ConvertTreeReaderFactory$StringGroupFromAnyIntegerTreeReader.nextVector(ConvertTreeReaderFactory.java:1477) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.orc.impl.TreeReaderFactory$StructTreeReader.nextBatch(TreeReaderFactory.java:2012) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.flink.orc.OrcSplitReader.reachedEnd(OrcSplitReader.java:101) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.flink.connectors.hive.read.HiveTableInputFormat.reachedEnd(HiveTableInputFormat.java:261) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
2020-12-17 21:10:24,406 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor     
so ,how to solve this question?
r_khachatryan
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: flink sql read hive table throw java.lang.ArrayIndexOutOfBoundsException: 1024

r_khachatryan
The stacktrace looks similar to https://issues.apache.org/jira/browse/HIVE-14483
However, it should be fixed in the version used in your setup.

Jingsong Li can you take a look at this error?

Regards,
Roman


On Thu, Dec 17, 2020 at 3:57 PM house－张浩 <[hidden email]> wrote:
when i use pyflink hive sql read data insert into es ,throw the follow exeception : the environment ： flink 1.11.2 flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar hive 3.1.2

2020-12-17 21:10:24,398 WARN  org.apache.flink.runtime.taskmanager.Task                    [] - Source: HiveTableSource(driver_id, driver_base_lc_p1, driver_90d_lc_p1, driver_30d_lc_p1, driver_14d_lc_p1, driver_180d_lc_p1, vehicle_base_lc_p1, driver_active_zone, is_incremental, dt) TablePath: algorithm.jiawei_oas_driver_features_for_incremental_hive2kafka, PartitionPruned: false, PartitionNums: null, ProjectedFields: [0, 8, 9] -> Calc(select=[driver_id, is_incremental, dt, () AS bdi_feature_create_time]) -> Sink: Sink(table=[default_catalog.default_database.0_demo4_903157246_tmp], fields=[driver_id, is_incremental, dt, bdi_feature_create_time]) (1/1) (98f4259c3d00fac9fc3482a4cdc8df3c) switched from RUNNING to FAILED.
    at org.apache.orc.impl.ConvertTreeReaderFactory$AnyIntegerTreeReader.nextVector(ConvertTreeReaderFactory.java:445) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1300) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.flink.connectors.hive.read.HiveVectorizedOrcSplitReader.reachedEnd(HiveVectorizedOrcSplitReader.java:99) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:90) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
    at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
    at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:213) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
2020-12-17 21:10:24,402 INFO  org.apache.flink.runtime.taskmanager.Task                    [] - Freeing task resources for Source: HiveTableSource(driver_id, driver_base_lc_p1, driver_90d_lc_p1, driver_30d_lc_p1, driver_14d_lc_p1, driver_180d_lc_p1, vehicle_base_lc_p1, driver_active_zone, is_incremental, dt) TablePath: algorithm.jiawei_oas_driver_features_for_incremental_hive2kafka, PartitionPruned: false, PartitionNums: null, ProjectedFields: [0, 8, 9] -> Calc(select=[driver_id, is_incremental, dt, () AS bdi_feature_create_time]) -> Sink: Sink(table=[default_catalog.default_database.0_demo4_903157246_tmp], fields=[driver_id, is_incremental, dt, bdi_feature_create_time]) (1/1) (98f4259c3d00fac9fc3482a4cdc8df3c).
java.lang.ArrayIndexOutOfBoundsException: 1024
    at org.apache.flink.orc.shim.OrcShimV210.nextBatch(OrcShimV210.java:35) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.flink.orc.shim.OrcShimV210.nextBatch(OrcShimV210.java:29) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.flink.orc.OrcSplitReader.ensureBatch(OrcSplitReader.java:134) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.orc.impl.TreeReaderFactory$LongTreeReader.nextVector(TreeReaderFactory.java:612) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.orc.impl.TreeReaderFactory$TreeReader.nextVector(TreeReaderFactory.java:269) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.orc.impl.ConvertTreeReaderFactory$StringGroupFromAnyIntegerTreeReader.nextVector(ConvertTreeReaderFactory.java:1477) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.orc.impl.TreeReaderFactory$StructTreeReader.nextBatch(TreeReaderFactory.java:2012) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.flink.orc.OrcSplitReader.reachedEnd(OrcSplitReader.java:101) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.flink.connectors.hive.read.HiveTableInputFormat.reachedEnd(HiveTableInputFormat.java:261) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2]
    at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
2020-12-17 21:10:24,406 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor     
so ,how to solve this question?
Free forum by Nabble Edit this page