Read from HBase flink-0.9.0-milestone-1-bin-hadoop2

[Hilmi Yildirim]

Hi,
I also tried to read from HBase using the flink-0.9.0-milestone-1-bin-hadoop2 version. It works great and it uses the config declared in the hbase-site.xml.

I configured a flink cluster with 2 machines. For that I deployed the flink folder with the corresponding configurations on each machine. To start the cluster I use the start script ./start-cluster.sh. Then, I run the job and it works. Now, I configured a flink cluster with 13 machines. All machines read from hbase correctly besides 1 or 2 machines. These 1 or 2 machines get following exception
"org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name 2423, already closed?"

When I restart the job then the exception occurs for other 1 or 2 machines.

Does anyone know why this exception occurs?

Best Regards,

[Stephan Ewen]

Can you look into the Flink and HBase logs if it locks some exception there?

On Thu, May 28, 2015 at 10:24 AM, Hilmi Yildirim <[hidden email]> wrote:

Hi,
I also tried to read from HBase using the flink-0.9.0-milestone-1-bin-hadoop2 version. It works great and it uses the config declared in the hbase-site.xml.

I configured a flink cluster with 2 machines. For that I deployed the flink folder with the corresponding configurations on each machine. To start the cluster I use the start script ./start-cluster.sh. Then, I run the job and it works. Now, I configured a flink cluster with 13 machines. All machines read from hbase correctly besides 1 or 2 machines. These 1 or 2 machines get following exception
"org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name 2423, already closed?"

When I restart the job then the exception occurs for other 1 or 2 machines.

Does anyone know why this exception occurs?

Best Regards,

-- 
--
Hilmi Yildirim
Software Developer R&D

T: <a href="tel:%2B49%2030%2024627-281" value="[phone]" target="_blank">[phone]
[hidden email]

http://www.neofonie.de

Besuchen Sie den Neo Tech Blog für Anwender:
http://blog.neofonie.de/

Folgen Sie uns:
https://plus.google.com/+neofonie
http://www.linkedin.com/company/neofonie-gmbh
https://www.xing.com/companies/neofoniegmbh

Neofonie GmbH | Robert-Koch-Platz 4 | 10115 Berlin
Handelsregister Berlin-Charlottenburg: HRB 67460
Geschäftsführung: Thomas Kitlitschko

[Flavio Pompermaier]

   The unknown scanner exception is just a warning due to the fact that the table scan is not eager but lazy..so between two consecutive getNextRecord() call the Habase server timeouts the scanner resource's..In the house extension I just recreate the scanner from the last read key. This is a workaround to avoid too much tuning with caching and timeouts hbase params...

On 28 May 2015 10:24, "Hilmi Yildirim" <[hidden email]> wrote:

Hi,
I also tried to read from HBase using the flink-0.9.0-milestone-1-bin-hadoop2 version. It works great and it uses the config declared in the hbase-site.xml.

I configured a flink cluster with 2 machines. For that I deployed the flink folder with the corresponding configurations on each machine. To start the cluster I use the start script ./start-cluster.sh. Then, I run the job and it works. Now, I configured a flink cluster with 13 machines. All machines read from hbase correctly besides 1 or 2 machines. These 1 or 2 machines get following exception
"org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name 2423, already closed?"

When I restart the job then the exception occurs for other 1 or 2 machines.

Does anyone know why this exception occurs?

Best Regards,

-- 
--
Hilmi Yildirim
Software Developer R&D

T: <a href="tel:%2B49%2030%2024627-281" value="[phone]" target="_blank">[phone]
[hidden email]

http://www.neofonie.de

Besuchen Sie den Neo Tech Blog für Anwender:
http://blog.neofonie.de/

Folgen Sie uns:
https://plus.google.com/+neofonie
http://www.linkedin.com/company/neofonie-gmbh
https://www.xing.com/companies/neofoniegmbh

Neofonie GmbH | Robert-Koch-Platz 4 | 10115 Berlin
Handelsregister Berlin-Charlottenburg: HRB 67460
Geschäftsführung: Thomas Kitlitschko

[Hilmi Yildirim]

I solved the problem. The problem was that I created an HBase table only on 1 machine. 13 machines requested data from this sigle machine which lead to a bottleneck. Now, I have split the table on 13 machines and it is working now. It continues to throw UnknownScannerExceptions but it is working. Maybe it is beneficially to create a task to catch this exception or to handle it correctly.

Best Regards,
Hilmi

Am 28.05.2015 um 11:34 schrieb Flavio Pompermaier:

   The unknown scanner exception is just a warning due to the fact that the table scan is not eager but lazy..so between two consecutive getNextRecord() call the Habase server timeouts the scanner resource's..In the house extension I just recreate the scanner from the last read key. This is a workaround to avoid too much tuning with caching and timeouts hbase params...

On 28 May 2015 10:24, "Hilmi Yildirim" <[hidden email]> wrote:

Hi,
I also tried to read from HBase using the flink-0.9.0-milestone-1-bin-hadoop2 version. It works great and it uses the config declared in the hbase-site.xml.

I configured a flink cluster with 2 machines. For that I deployed the flink folder with the corresponding configurations on each machine. To start the cluster I use the start script ./start-cluster.sh. Then, I run the job and it works. Now, I configured a flink cluster with 13 machines. All machines read from hbase correctly besides 1 or 2 machines. These 1 or 2 machines get following exception
"org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name 2423, already closed?"

When I restart the job then the exception occurs for other 1 or 2 machines.

Does anyone know why this exception occurs?

Best Regards,

-- 
--
Hilmi Yildirim
Software Developer R&D

T: <a moz-do-not-send="true" href="tel:%2B49%2030%2024627-281" value="[phone]" target="_blank">[phone]
[hidden email]

http://www.neofonie.de

Besuchen Sie den Neo Tech Blog für Anwender:
http://blog.neofonie.de/

Folgen Sie uns:
https://plus.google.com/+neofonie
http://www.linkedin.com/company/neofonie-gmbh
https://www.xing.com/companies/neofoniegmbh

Neofonie GmbH | Robert-Koch-Platz 4 | 10115 Berlin
Handelsregister Berlin-Charlottenburg: HRB 67460
Geschäftsführung: Thomas Kitlitschko