(DEPRECATED) Apache Flink User Mailing List archive.

HBase on 4 machine cluster - OutOfMemoryError

classic Classic list List threaded Threaded
♦
♦
 4 messages Options
Lydia Ickler
Reply | Threaded
Open this post in threaded view
♦
♦
|

HBase on 4 machine cluster - OutOfMemoryError

Lydia Ickler
Hi all,

I am trying to read a data set from HBase within a cluster application. 
The data is about 90MB big.

When I run the program on a cluster consisting of 4 machines (8GB RAM) I get the following error on the head-node:

16:57:41,572 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host grips5
17:17:26,127 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - DataSource (at createInput(ExecutionEnvironment.java:502) (org.apache.flink.addons.hbase.HBaseR$
17:17:26,128 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Status of job b768ff76167fa3ea3e4cb3cc3481ba80 (Labeled - ML) changed to FAILING.

And within the machine grips5:
16:57:23,769 INFO  org.apache.flink.addons.hbase.TableInputFormat                - opening split [1|[grips1:16020]|LUAD+5781|LUAD+7539]
16:57:33,734 WARN  org.apache.hadoop.ipc.RpcClient                               - IPC Client (767445418) connection to grips1/130.73.20.14:16020 from hduser: unexpected exceptio$
java.lang.OutOfMemoryError: Java heap space
        at org.apache.hadoop.hbase.ipc.RpcClient$Connection.readResponse(RpcClient.java:1117)
        at org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:727)
16:57:39,969 WARN  org.apache.hadoop.ipc.RpcClient                               - IPC Client (767445418) connection to grips1/130.73.20.14:16020 from hduser: unexpected exceptio$
java.lang.OutOfMemoryError: Java heap space

and then it just closes the zookeeper…

Do you have a suggestion how to avoid this OutOfMemoryError?
Best regards,
Lydia
Stephan Ewen
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: HBase on 4 machine cluster - OutOfMemoryError

Stephan Ewen

This error is in the HBase RPC Service. Apparently the RPC message is very large.

Is the data that you request in one row?

Am 18.07.2015 00:50 schrieb "Lydia Ickler" <[hidden email]>:
Hi all,

I am trying to read a data set from HBase within a cluster application. 
The data is about 90MB big.

When I run the program on a cluster consisting of 4 machines (8GB RAM) I get the following error on the head-node:

16:57:41,572 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host grips5
17:17:26,127 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - DataSource (at createInput(ExecutionEnvironment.java:502) (org.apache.flink.addons.hbase.HBaseR$
17:17:26,128 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Status of job b768ff76167fa3ea3e4cb3cc3481ba80 (Labeled - ML) changed to FAILING.

And within the machine grips5:
16:57:23,769 INFO  org.apache.flink.addons.hbase.TableInputFormat                - opening split [1|[grips1:16020]|LUAD+5781|LUAD+7539]
16:57:33,734 WARN  org.apache.hadoop.ipc.RpcClient                               - IPC Client (767445418) connection to grips1/130.73.20.14:16020 from hduser: unexpected exceptio$
java.lang.OutOfMemoryError: Java heap space
        at org.apache.hadoop.hbase.ipc.RpcClient$Connection.readResponse(RpcClient.java:1117)
        at org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:727)
16:57:39,969 WARN  org.apache.hadoop.ipc.RpcClient                               - IPC Client (767445418) connection to grips1/130.73.20.14:16020 from hduser: unexpected exceptio$
java.lang.OutOfMemoryError: Java heap space

and then it just closes the zookeeper…

Do you have a suggestion how to avoid this OutOfMemoryError?
Best regards,
Lydia
Lydia Ickler
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: HBase on 4 machine cluster - OutOfMemoryError

Lydia Ickler
Hi,

yes, it is in one row. Each row represents a patient that has values of 20.000 different genes stored in one column family and one value of health status in a second column family.


Am 18.07.2015 um 15:38 schrieb Stephan Ewen <[hidden email]>:

This error is in the HBase RPC Service. Apparently the RPC message is very large.

Is the data that you request in one row?

Am 18.07.2015 00:50 schrieb "Lydia Ickler" <[hidden email]>:
Hi all,

I am trying to read a data set from HBase within a cluster application. 
The data is about 90MB big.

When I run the program on a cluster consisting of 4 machines (8GB RAM) I get the following error on the head-node:

16:57:41,572 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host grips5
17:17:26,127 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - DataSource (at createInput(ExecutionEnvironment.java:502) (org.apache.flink.addons.hbase.HBaseR$
17:17:26,128 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Status of job b768ff76167fa3ea3e4cb3cc3481ba80 (Labeled - ML) changed to FAILING.

And within the machine grips5:
16:57:23,769 INFO  org.apache.flink.addons.hbase.TableInputFormat                - opening split [1|[grips1:16020]|LUAD+5781|LUAD+7539]
16:57:33,734 WARN  org.apache.hadoop.ipc.RpcClient                               - IPC Client (767445418) connection to grips1/130.73.20.14:16020 from hduser: unexpected exceptio$
java.lang.OutOfMemoryError: Java heap space
        at org.apache.hadoop.hbase.ipc.RpcClient$Connection.readResponse(RpcClient.java:1117)
        at org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:727)
16:57:39,969 WARN  org.apache.hadoop.ipc.RpcClient                               - IPC Client (767445418) connection to grips1/130.73.20.14:16020 from hduser: unexpected exceptio$
java.lang.OutOfMemoryError: Java heap space

and then it just closes the zookeeper…

Do you have a suggestion how to avoid this OutOfMemoryError?
Best regards,
Lydia
Stephan Ewen
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: HBase on 4 machine cluster - OutOfMemoryError

Stephan Ewen
Okay. If you are using very big values, it often helps to tell Flink to reserve less memory for its internal processing.

Can you try and set the memory fraction lower, e.g., 0.5 lower.

Have a look at the option "taskmanager.memory.fraction" (https://ci.apache.org/projects/flink/flink-docs-release-0.9/setup/config.html)

Greetings,
Stephan


Am 18.07.2015 15:47 schrieb "Lydia Ickler" <[hidden email]>:
Hi,

yes, it is in one row. Each row represents a patient that has values of 20.000 different genes stored in one column family and one value of health status in a second column family.


Am 18.07.2015 um 15:38 schrieb Stephan Ewen <[hidden email]>:

This error is in the HBase RPC Service. Apparently the RPC message is very large.

Is the data that you request in one row?

Am 18.07.2015 00:50 schrieb "Lydia Ickler" <[hidden email]>:
Hi all,

I am trying to read a data set from HBase within a cluster application. 
The data is about 90MB big.

When I run the program on a cluster consisting of 4 machines (8GB RAM) I get the following error on the head-node:

16:57:41,572 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host grips5
17:17:26,127 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - DataSource (at createInput(ExecutionEnvironment.java:502) (org.apache.flink.addons.hbase.HBaseR$
17:17:26,128 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Status of job b768ff76167fa3ea3e4cb3cc3481ba80 (Labeled - ML) changed to FAILING.

And within the machine grips5:
16:57:23,769 INFO  org.apache.flink.addons.hbase.TableInputFormat                - opening split [1|[grips1:16020]|LUAD+5781|LUAD+7539]
16:57:33,734 WARN  org.apache.hadoop.ipc.RpcClient                               - IPC Client (767445418) connection to grips1/130.73.20.14:16020 from hduser: unexpected exceptio$
java.lang.OutOfMemoryError: Java heap space
        at org.apache.hadoop.hbase.ipc.RpcClient$Connection.readResponse(RpcClient.java:1117)
        at org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:727)
16:57:39,969 WARN  org.apache.hadoop.ipc.RpcClient                               - IPC Client (767445418) connection to grips1/130.73.20.14:16020 from hduser: unexpected exceptio$
java.lang.OutOfMemoryError: Java heap space

and then it just closes the zookeeper…

Do you have a suggestion how to avoid this OutOfMemoryError?
Best regards,
Lydia
Free forum by Nabble Edit this page