Error checkpointing to S3 like FS (EMC ECS)
Locked
Error checkpointing to S3 like FS (EMC ECS)
 
|
Hi,
I am using flink version 1.7.2 , I am trying to use S3 like object storage EMC ECS( https://www.emc.com/techpubs/ecs/ecs_s3_supported_features-1.htm). Not all S3 apis are supported by EMC ESC according to this document. Here is my config s3.endpoint: SU73ECSG1P1d.***.COM s3.access-key: vdna_np_user security.ssl.rest.enabled: false web.timeout: 10000 s3.secret-key: J*** I can access this bucket from s3cmd client. I set the state backend from my scala application env.setStateBackend(new FsStateBackend("s3://aip-featuretoolkit/checkpoints/")) However when I run my application I get this exception : ClientException: Unable to execute HTTP request: aip-featuretoolkit.SU73ECSG1P1d.VISA.COM: Unable to execute HTTP request: aip-featuretoolkit.SU73ECSG1P1d.VISA.COM 1336 at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.<init>(CheckpointCoordinator.java:255) 1337 at org.apache.flink.runtime.executiongraph.ExecutionGraph.enableCheckpointing(ExecutionGraph.java:498) 1338 at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:345) 1339 at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:100) 1340 at org.apache.flink.runtime.jobmaster.JobMaster.createExecutionGraph(JobMaster.java:1173) 1341 at org.apache.flink.runtime.jobmaster.JobMaster.createAndRestoreExecutionGraph(JobMaster.java:1153) 1342 at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:296) 1343 at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:157) 1344 ... 10 more 1345 Caused by: org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.AWSClientIOException: doesBucketExist on aip-featuretoolkit: org.apache.flink.fs.s3base.shaded.co m.amazonaws.SdkClientException: Unable to execute HTTP request: aip-featuretoolkit.SU73ECSG1P1d.VISA.COM: Unable to execute HTTP request: aip-featuretoolkit.SU73ECSG1P1 d.VISA.COM 1346 at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:177) 1347 at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111) 1348 at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:260) 1349 at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:317) 1350 at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:256) 1351 at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:231) 1352 at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:372) 1353 at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:308) 1354 at org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:125) 1355 at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:395) 1356 at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:318) 1357 at org.apache.flink.core.fs.Path.getFileSystem(Path.java:298) 1358 at org.apache.flink.runtime.state.filesystem.FsCheckpointStorage.<init>(FsCheckpointStorage.java:58) 1359 at org.apache.flink.runtime.state.filesystem.FsStateBackend.createCheckpointStorage(FsStateBackend.java:444) 1360 at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.<init>(CheckpointCoordinator.java:249) 1361 ... 17 more 1362 Caused by: org.apache.flink.fs.s3base.shaded.com.amazonaws.SdkClientException: Unable to execute HTTP request: aip-featuretoolkit.SU73ECSG1P1d.VISA.COM 1363 at org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1114) 1364 at org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1064) 1365 at org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) 1366 at org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) 1367 at org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) 1368 at org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) 1369 at org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) 1370 at org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) 1371 at org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4325) 1372 at org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4272) 1373 at org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1337) 1374 at org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1277) 1375 at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$verifyBucketExists$1(S3AFileSystem.java:373) 1376 at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109) 1377 ... 30 more 1378 Caused by: java.net.UnknownHostException: aip-featuretoolkit.SU73ECSG1P1d.VISA.COM 1379 at java.net.InetAddress.getAllByName0(InetAddress.java:1280) 1380 at java.net.InetAddress.getAllByName(InetAddress.java:1192) 1381 at java.net.InetAddress.getAllByName(InetAddress.java:1126) 1382 at org.apache.flink.fs.s3base.shaded.com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27) 1383 at org.apache.flink.fs.s3base.shaded.com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38) 1384 at org.apache.flink.fs.s3base.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112) 1385 at org.apache.flink.fs.s3base.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) 1386 at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) 1387 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 1388 at java.lang.reflect.Method.invoke(Method.java:498) 1389 at org.apache.flink.fs.s3base.shaded.com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) 1390 at org.apache.flink.fs.s3base.shaded.com.amazonaws.http.conn.$Proxy45.connect(Unknown Source) 1391 at org.apache.flink.fs.s3base.shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) 1392 at org.apache.flink.fs.s3base.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) 1393 at org.apache.flink.fs.s3base.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) 1394 at org.apache.flink.fs.s3base.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) 1395 at org.apache.flink.fs.s3base.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) 1396 at org.apache.flink.fs.s3base.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) 1397 at org.apache.flink.fs.s3base.shaded.com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) 1398 at org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1236) 1399 at org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056) Is this backend compatible with flink ? Thanks, Vishwas |
Re: Error checkpointing to S3 like FS (EMC ECS)
|
|
Hi Vishwas, Sorry for the late response. Are you still facing the issue? I have no experience with EMC ECS, but the exception suggests an issue with the host name: 1378 Caused by: java.net.UnknownHostException: aip-featuretoolkit.SU73ECSG1P1d.***.COM 1379 at java.net.InetAddress.getAllByName0(InetAddress.java:1280) 1380 at java.net.InetAddress.getAllByName(InetAddress.java:1192) 1381 at java.net.InetAddress.getAllByName(InetAddress.java:1126) 1382 at org.apache.flink.fs.s3base.shaded.com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27) Best, Fabian Am Di., 25. Juni 2019 um 03:26 Uhr schrieb Vishwas Siravara <[hidden email]>: Hi, |
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |