Hey Hao,
1) Regarding Hadoop S3: are you using the repackaged Hadoop S3
dependency from the /opt folder of the Flink distribution? Or the
actual Hadoop implementation? If latter, would you mind also running
it with the one that comes packaged with Flink? For this you can
remove all Hadoop-related configuration in your flink-conf.yaml and
copy the Hadoop S3 dependency from /opt to /lib and configure it [1].
2) Could you please share your complete Flink configuration for when
you tried to run with Presto S3? If you don't want to share this
publicly, feel free to share it privately with me. I'm curious to see
whether we can reproduce this.
– Ufuk
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/deployment/aws.html#shaded-hadooppresto-s3-file-systems-recommended
On Sat, Nov 10, 2018 at 4:07 PM Hao Sun <
[hidden email]> wrote:
>
> Hi Ufuk, thanks for checking. I am using openJDK 1.8_171, I still have the same issue with presto.
>
> - why checkpoint is not starting from 1? old chk stored in ZK caused it, I cleaned it up, but not very helpful
> - I switched to Flink + Hadoop28, and used hadoop s3, with no other changes, check pointing is working with the hadoop flavour.
>
> On Fri, Nov 9, 2018 at 2:02 PM Ufuk Celebi <
[hidden email]> wrote:
>>
>> Hey Hao Sun,
>>
>> - Is this an intermittent failure or permanent? The logs indicate that
>> some checkpoints completed before the error occurs (e.g. checkpoint
>> numbers are greater than 1).
>>
>> - Which Java versions are you using? And which Java image? I've
>> Googled similar issues that seem to be related to the JVM, e.g. [1].
>>
>> Best,
>>
>> Ufuk
>>
>> [1]
https://dev.lucee.org/t/could-not-initialize-class-sun-security-ssl-sslcontextimp/3972
>>
>>
>> On Thu, Nov 8, 2018 at 8:55 PM Hao Sun <
[hidden email]> wrote:
>> >
>> > Thanks, any insight/help here is appreciated.
>> >
>> > On Thu, Nov 8, 2018 at 4:38 AM Dawid Wysakowicz <
[hidden email]> wrote:
>> >>
>> >> Hi Hao,
>> >>
>> >> I am not sure, what might be wrong, but I've cc'ed Gary and Kostas who were recently working with S3, maybe they will have some ideas.
>> >>
>> >> Best,
>> >>
>> >> Dawid
>> >>
>> >> On 03/11/2018 03:09, Hao Sun wrote:
>> >>
>> >> Same environment, new error.
>> >>
>> >> I can run the same docker image with my local Mac, but on K8S, this gives me this error.
>> >> I can not think of any difference between local Docker and K8S Docker.
>> >>
>> >> Any hint will be helpful. Thanks
>> >>
>> >> ============
>> >>
>> >> 2018-11-02 23:29:32,981 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job ConnectedStreams maxwell.accounts (00000000000000000000000000000000) switched from state RUNNING to FAILING.
>> >> AsynchronousException{java.lang.Exception: Could not materialize checkpoint 235 for operator Source: KafkaSource(maxwell.accounts) -> MaxwellFilter->Maxwell(maxwell.accounts) -> FixedDelayWatermark(maxwell.accounts) -> MaxwellFPSEvent->InfluxDBData(maxwell.accounts) -> Sink: influxdbSink(maxwell.accounts) (1/1).}
>> >> at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointExceptionHandler.tryHandleCheckpointException(StreamTask.java:1153)
>> >> at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:947)
>> >> at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:884)
>> >> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> >> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> >> at java.lang.Thread.run(Thread.java:748)
>> >> Caused by: java.lang.Exception: Could not materialize checkpoint 235 for operator Source: KafkaSource(maxwell.accounts) -> MaxwellFilter->Maxwell(maxwell.accounts) -> FixedDelayWatermark(maxwell.accounts) -> MaxwellFPSEvent->InfluxDBData(maxwell.accounts) -> Sink: influxdbSink(maxwell.accounts) (1/1).
>> >> at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:942)
>> >> ... 6 more
>> >> Caused by: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: Could not initialize class sun.security.ssl.SSLSessionImpl
>> >> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> >> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>> >> at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:53)
>> >>
>> >> at org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.<init>(
http://OperatorSnapshotFinalizer.java:53)
>> >>
>> >> at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:853)
>> >> ... 5 more
>> >> Caused by: java.lang.NoClassDefFoundError: Could not initialize class sun.security.ssl.SSLSessionImpl
>> >> at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:604)
>> >>
>> >> at sun.security.ssl.SSLSocketImpl.<init>(
http://SSLSocketImpl.java:572)
>> >>
>> >> at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:110)
>> >> at org.apache.flink.fs.s3presto.shaded.org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:365)
>> >> at org.apache.flink.fs.s3presto.shaded.org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
>> >> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:132)
>> >> at org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
>> >> at org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)
>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >> at java.lang.reflect.Method.invoke(Method.java:498)
>> >> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
>> >> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.$Proxy4.connect(Unknown Source)
>> >> at org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
>> >> at org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
>> >> at org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
>> >> at org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
>> >> at org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
>> >> at org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
>> >> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
>> >> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1189)
>> >> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029)
>> >> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741)
>> >> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715)
>> >> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697)
>> >> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665)
>> >> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647)
>> >> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511)
>> >> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227)
>> >> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:4988)
>> >> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4962)
>> >> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4211)
>> >> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174)
>> >> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1253)
>> >> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1228)
>> >> at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.lambda$getS3ObjectMetadata$2(PrestoS3FileSystem.java:559)
>> >> at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:138)
>> >> at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getS3ObjectMetadata(PrestoS3FileSystem.java:556)
>> >> at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getFileStatus(PrestoS3FileSystem.java:307)
>> >> at org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)
>> >> at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.create(PrestoS3FileSystem.java:352)
>> >> at org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
>> >> at org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
>> >> at org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
>> >> at org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:141)
>> >> at org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:37)
>> >> at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:126)
>> >> at org.apache.flink.core.fs.EntropyInjector.createEntropyAware(EntropyInjector.java:61)
>> >> at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:350)
>> >> at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.flush(FsCheckpointStreamFactory.java:228)
>> >> at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.write(FsCheckpointStreamFactory.java:212)
>> >> at java.io.DataOutputStream.write(DataOutputStream.java:107)
>> >> at org.apache.flink.api.common.typeutils.TypeSerializerSerializationUtil.writeSerializersAndConfigsWithResilience(TypeSerializerSerializationUtil.java:161)
>> >> at org.apache.flink.runtime.state.metainfo.StateMetaInfoSnapshotReadersWriters$CurrentWriterImpl.writeStateMetaInfoSnapshot(StateMetaInfoSnapshotReadersWriters.java:182)
>> >> at org.apache.flink.runtime.state.OperatorBackendSerializationProxy.writeStateMetaInfoSnapshots(OperatorBackendSerializationProxy.java:85)
>> >> at org.apache.flink.runtime.state.OperatorBackendSerializationProxy.write(OperatorBackendSerializationProxy.java:76)
>> >> at org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:418)
>> >> at org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:359)
>> >> at org.apache.flink.runtime.io.async.AbstractAsyncCallableWithResources.call(AbstractAsyncCallableWithResources.java:75)
>> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> >> at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:50)
>> >> ... 7 more
>> >> 2018-11-02 23:29:32,983 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or fail the job ConnectedStreams maxwell.accounts (00000000000000000000000000000000) if no longer possible.
>> >>
>> >> =========================
>> >>
>> >> On Thu, Nov 1, 2018 at 9:22 PM Hao Sun <
[hidden email]> wrote:
>> >>>
>> >>> I am on Flink 1.6.2 (no Hadoop, in docker + K8S), using rocksdb and S3 (presto)
>> >>> I got this error when flink creating a checking point
>> >>>
>> >>>
>> >>> ===========
>> >>> 2018-11-02 04:00:55,011 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job ConnectedStreams maxwell.accounts (00000000000000000000000000000000) switched from state RUNNING to FAILING.
>> >>> AsynchronousException{java.lang.Exception: Could not materialize checkpoint 130 for operator Source: KafkaSource(maxwell.accounts) -> MaxwellFilter->Maxwell(maxwell.accounts) -> FixedDelayWatermark(maxwell.accounts) -> MaxwellFPSEvent->InfluxDBData(maxwell.accounts) -> Sink: influxdbSink(maxwell.accounts) (1/1).}
>> >>> at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointExceptionHandler.tryHandleCheckpointException(StreamTask.java:1153)
>> >>> at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:947)
>> >>> at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:884)
>> >>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> >>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> >>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> >>> at java.lang.Thread.run(Thread.java:748)
>> >>> Caused by: java.lang.Exception: Could not materialize checkpoint 130 for operator Source: KafkaSource(maxwell.accounts) -> MaxwellFilter->Maxwell(maxwell.accounts) -> FixedDelayWatermark(maxwell.accounts) -> MaxwellFPSEvent->InfluxDBData(maxwell.accounts) -> Sink: influxdbSink(maxwell.accounts) (1/1).
>> >>> at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:942)
>> >>> ... 6 more
>> >>> Caused by: java.util.concurrent.ExecutionException: java.lang.ExceptionInInitializerError
>> >>> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> >>> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>> >>> at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:53)
>> >>>
>> >>> at org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.<init>(
http://OperatorSnapshotFinalizer.java:53)
>> >>>
>> >>>
>> >>> at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:853)
>> >>> ... 5 more
>> >>> Caused by: java.lang.ExceptionInInitializerError
>> >>>
>> >>> at sun.security.ssl.SSLSessionImpl.<init>(
http://SSLSessionImpl.java:183)
>> >>> at sun.security.ssl.SSLSessionImpl.<init>(
http://SSLSessionImpl.java:148)
>> >>> at sun.security.ssl.SSLSessionImpl.<clinit>(
http://SSLSessionImpl.java:79)
>> >>> at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:604)
>> >>> at sun.security.ssl.SSLSocketImpl.<init>(
http://SSLSocketImpl.java:572)
>> >>>
>> >>>
>> >>> at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:110)
>> >>> at org.apache.flink.fs.s3presto.shaded.org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:365)
>> >>> at org.apache.flink.fs.s3presto.shaded.org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:132)
>> >>> at org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
>> >>> at org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)
>> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> >>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >>> at java.lang.reflect.Method.invoke(Method.java:498)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.$Proxy4.connect(Unknown Source)
>> >>> at org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
>> >>> at org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
>> >>> at org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
>> >>> at org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
>> >>> at org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
>> >>> at org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1189)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:4988)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4962)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4211)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1253)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1228)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.lambda$getS3ObjectMetadata$2(PrestoS3FileSystem.java:559)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:138)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getS3ObjectMetadata(PrestoS3FileSystem.java:556)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getFileStatus(PrestoS3FileSystem.java:307)
>> >>> at org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)
>> >>> at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.create(PrestoS3FileSystem.java:352)
>> >>> at org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
>> >>> at org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
>> >>> at org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
>> >>> at org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:141)
>> >>> at org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:37)
>> >>> at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:126)
>> >>> at org.apache.flink.core.fs.EntropyInjector.createEntropyAware(EntropyInjector.java:61)
>> >>> at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:350)
>> >>> at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.flush(FsCheckpointStreamFactory.java:228)
>> >>> at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.write(FsCheckpointStreamFactory.java:212)
>> >>> at java.io.DataOutputStream.write(DataOutputStream.java:107)
>> >>> at org.apache.flink.api.common.typeutils.TypeSerializerSerializationUtil.writeSerializersAndConfigsWithResilience(TypeSerializerSerializationUtil.java:161)
>> >>> at org.apache.flink.runtime.state.metainfo.StateMetaInfoSnapshotReadersWriters$CurrentWriterImpl.writeStateMetaInfoSnapshot(StateMetaInfoSnapshotReadersWriters.java:182)
>> >>> at org.apache.flink.runtime.state.OperatorBackendSerializationProxy.writeStateMetaInfoSnapshots(OperatorBackendSerializationProxy.java:85)
>> >>> at org.apache.flink.runtime.state.OperatorBackendSerializationProxy.write(OperatorBackendSerializationProxy.java:76)
>> >>> at org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:418)
>> >>> at org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:359)
>> >>> at org.apache.flink.runtime.io.async.AbstractAsyncCallableWithResources.call(AbstractAsyncCallableWithResources.java:75)
>> >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> >>> at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:50)
>> >>> ... 7 more
>> >>> Caused by: java.security.ProviderException: Could not initialize NSS
>> >>>
>> >>> at sun.security.pkcs11.SunPKCS11.<init>(
http://SunPKCS11.java:223)
>> >>> at sun.security.pkcs11.SunPKCS11.<init>(
http://SunPKCS11.java:103)
>> >>>
>> >>>
>> >>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> >>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>> >>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> >>> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>> >>> at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:224)
>> >>> at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:206)
>> >>> at java.security.AccessController.doPrivileged(Native Method)
>> >>> at sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:206)
>> >>> at sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:187)
>> >>> at sun.security.jca.ProviderList.getProvider(ProviderList.java:233)
>> >>> at sun.security.jca.ProviderList.getIndex(ProviderList.java:263)
>> >>> at sun.security.jca.ProviderList.getProviderConfig(ProviderList.java:247)
>> >>> at sun.security.jca.ProviderList.getProvider(ProviderList.java:253)
>> >>> at java.security.Security.getProvider(Security.java:501)
>> >>>
>> >>> at sun.security.ssl.SignatureAndHashAlgorithm.<clinit>(
http://SignatureAndHashAlgorithm.java:415)
>> >>>
>> >>>
>> >>> ... 72 more
>> >>> Caused by: java.io.IOException: NSS is already initialized
>> >>> at sun.security.pkcs11.Secmod.initialize(Secmod.java:173)
>> >>>
>> >>> at sun.security.pkcs11.SunPKCS11.<init>(
http://SunPKCS11.java:218)
>> >>> ... 88 more
>> >>>
>> >>> ===========
Locked
signature.asc (849 bytes)