Hi Ragini,
Since this exception is coming from the Hbase client, I assume the issue has nothing to do with Flink directly.
I would recommend carefully studying the HBase client configuration parameters, maybe setup a simple Java application that "hammers" data into Hbase at a maximum rate to understand the impact of different combinations of configuration parameters.
Best,
Robert
On Tue, May 4, 2021 at 5:05 AM Ragini Manjaiah <
[hidden email]> wrote:
Hi ,
One of my flink applications needs to get and put records from HBASE for every event while processing in real time . When there are less events the application process without any issues. when the number of events increases we start hitting with the below mentioned exception .Can these exceptions bring down the throughput and start to build lag . What are the parameters we can tune at HBASE /flink side to overcome this exception . We are seeing 7000/sec hits as minimum hits to HBase when load is normal. The hbase table 3 region server
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: IOException: 1 time,
at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:258)
at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$2000(AsyncProcess.java:238)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1817)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:240)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:190)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1434)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1018)
at org.xxxx.xxxx.xxxx.xxxx.xxxxxxxxxxxx.xxxxxx(xxxxxxx.java:202)
at org.xxxx.xxxx.xxxx.xxxx.xxxxxxxxxxxx.xxxxxxxxxxx.xxxxxxx(xxxxxxx.java:144)
at org.xxxx.xxxx.xxxx.xxxx.xxxxxxxxxxxx.xxxxxxxxxxx.xxxxxxxx(xxxxxxxxx.java:30)
at org.apache.flink.streaming.api.operators.KeyedProcessOperator.processElement(KeyedProcessOperator.java:85)
at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:202)
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:105)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
at java.lang.Thread.run(Thread.java:745)
Locked
