Error with HBase and Solr job

[Flavio Pompermaier]

Hi to all,

I have a source HBase table and I have to write to a Solr index.

Unfortunately when I try to run the program on the cluster (Flink 1.1.1) I think I have some problem with dependencies. Can someone suggest me some fix?

This is the error I have just after launching the job:

Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:229)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:202)
    at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:314)
    at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:289)
    at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:161)
    at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:156)
    at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888)
    at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:187)
    at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:89)
    at org.apache.hadoop.hbase.client.MetaScanner.listTableRegionLocations(MetaScanner.java:334)
    at org.apache.hadoop.hbase.client.HTable.listRegionLocations(HTable.java:702)
    at org.apache.hadoop.hbase.client.HTable.getStartEndKeys(HTable.java:685)
    at org.apache.flink.addons.hbase.TableInputFormat.createInputSplits(TableInputFormat.java:170)
    at org.apache.flink.addons.hbase.TableInputFormat.createInputSplits(TableInputFormat.java:45)
    at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:156)
    ... 25 more
Caused by: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
    at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:434)
    at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:60)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1122)
    at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:299)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:155)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:63)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
    ... 38 more

Best,

Flavio

[Stephan Ewen]

You have the classical guava version conflict. Flink itself shades Guava away, but there may be multiple conflicting dependencies in your case (HBase / Solr).

I would try to see which of the tool/libraries (HBase, Solr, ...) depend on Guava and create a shaded version of one of them

On Fri, Dec 23, 2016 at 6:33 PM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I have a source HBase table and I have to write to a Solr index.

Unfortunately when I try to run the program on the cluster (Flink 1.1.1) I think I have some problem with dependencies. Can someone suggest me some fix?

This is the error I have just after launching the job:

Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:229)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:202)
    at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:314)
    at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:289)
    at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:161)
    at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:156)
    at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888)
    at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:187)
    at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:89)
    at org.apache.hadoop.hbase.client.MetaScanner.listTableRegionLocations(MetaScanner.java:334)
    at org.apache.hadoop.hbase.client.HTable.listRegionLocations(HTable.java:702)
    at org.apache.hadoop.hbase.client.HTable.getStartEndKeys(HTable.java:685)
    at org.apache.flink.addons.hbase.TableInputFormat.createInputSplits(TableInputFormat.java:170)
    at org.apache.flink.addons.hbase.TableInputFormat.createInputSplits(TableInputFormat.java:45)
    at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:156)
    ... 25 more
Caused by: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
    at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:434)
    at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:60)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1122)
    at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:299)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:155)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:63)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
    ... 38 more

Best,

Flavio

[Flavio Pompermaier]

What do you meab exactly...? do you have a snippet of how I should edit the pom?

On 23 Dec 2016 19:31, "Stephan Ewen" <[hidden email]> wrote:

You have the classical guava version conflict. Flink itself shades Guava away, but there may be multiple conflicting dependencies in your case (HBase / Solr).

I would try to see which of the tool/libraries (HBase, Solr, ...) depend on Guava and create a shaded version of one of them

On Fri, Dec 23, 2016 at 6:33 PM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I have a source HBase table and I have to write to a Solr index.

Unfortunately when I try to run the program on the cluster (Flink 1.1.1) I think I have some problem with dependencies. Can someone suggest me some fix?

This is the error I have just after launching the job:

Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:229)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:202)
    at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:314)
    at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:289)
    at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:161)
    at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:156)
    at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888)
    at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:187)
    at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:89)
    at org.apache.hadoop.hbase.client.MetaScanner.listTableRegionLocations(MetaScanner.java:334)
    at org.apache.hadoop.hbase.client.HTable.listRegionLocations(HTable.java:702)
    at org.apache.hadoop.hbase.client.HTable.getStartEndKeys(HTable.java:685)
    at org.apache.flink.addons.hbase.TableInputFormat.createInputSplits(TableInputFormat.java:170)
    at org.apache.flink.addons.hbase.TableInputFormat.createInputSplits(TableInputFormat.java:45)
    at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:156)
    ... 25 more
Caused by: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
    at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:434)
    at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:60)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1122)
    at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:299)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:155)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:63)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
    ... 38 more

Best,

Flavio

[Stephan Ewen]

I would just google for Guava conflict and "shading". Its a fairly common problem with Guava, there are quite a few guides out there how to address that.

On Fri, Dec 23, 2016 at 9:12 PM, Flavio Pompermaier <[hidden email]> wrote:

What do you meab exactly...? do you have a snippet of how I should edit the pom?

On 23 Dec 2016 19:31, "Stephan Ewen" <[hidden email]> wrote:

You have the classical guava version conflict. Flink itself shades Guava away, but there may be multiple conflicting dependencies in your case (HBase / Solr).

I would try to see which of the tool/libraries (HBase, Solr, ...) depend on Guava and create a shaded version of one of them

On Fri, Dec 23, 2016 at 6:33 PM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I have a source HBase table and I have to write to a Solr index.

Unfortunately when I try to run the program on the cluster (Flink 1.1.1) I think I have some problem with dependencies. Can someone suggest me some fix?

This is the error I have just after launching the job:

Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:229)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:202)
    at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:314)
    at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:289)
    at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:161)
    at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:156)
    at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888)
    at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:187)
    at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:89)
    at org.apache.hadoop.hbase.client.MetaScanner.listTableRegionLocations(MetaScanner.java:334)
    at org.apache.hadoop.hbase.client.HTable.listRegionLocations(HTable.java:702)
    at org.apache.hadoop.hbase.client.HTable.getStartEndKeys(HTable.java:685)
    at org.apache.flink.addons.hbase.TableInputFormat.createInputSplits(TableInputFormat.java:170)
    at org.apache.flink.addons.hbase.TableInputFormat.createInputSplits(TableInputFormat.java:45)
    at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:156)
    ... 25 more
Caused by: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
    at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:434)
    at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:60)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1122)
    at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:299)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:155)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:63)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
    ... 38 more

Best,

Flavio

[Flavio Pompermaier]

Ok thans Stephan!

On 23 Dec 2016 21:16, "Stephan Ewen" <[hidden email]> wrote:

I would just google for Guava conflict and "shading". Its a fairly common problem with Guava, there are quite a few guides out there how to address that.

On Fri, Dec 23, 2016 at 9:12 PM, Flavio Pompermaier <[hidden email]> wrote:

What do you meab exactly...? do you have a snippet of how I should edit the pom?

On 23 Dec 2016 19:31, "Stephan Ewen" <[hidden email]> wrote:

You have the classical guava version conflict. Flink itself shades Guava away, but there may be multiple conflicting dependencies in your case (HBase / Solr).

I would try to see which of the tool/libraries (HBase, Solr, ...) depend on Guava and create a shaded version of one of them

On Fri, Dec 23, 2016 at 6:33 PM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I have a source HBase table and I have to write to a Solr index.

Unfortunately when I try to run the program on the cluster (Flink 1.1.1) I think I have some problem with dependencies. Can someone suggest me some fix?

This is the error I have just after launching the job:

Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:229)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:202)
    at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:314)
    at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:289)
    at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:161)
    at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:156)
    at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888)
    at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:187)
    at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:89)
    at org.apache.hadoop.hbase.client.MetaScanner.listTableRegionLocations(MetaScanner.java:334)
    at org.apache.hadoop.hbase.client.HTable.listRegionLocations(HTable.java:702)
    at org.apache.hadoop.hbase.client.HTable.getStartEndKeys(HTable.java:685)
    at org.apache.flink.addons.hbase.TableInputFormat.createInputSplits(TableInputFormat.java:170)
    at org.apache.flink.addons.hbase.TableInputFormat.createInputSplits(TableInputFormat.java:45)
    at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:156)
    ... 25 more
Caused by: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
    at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:434)
    at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:60)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1122)
    at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:299)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:155)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:63)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
    ... 38 more

Best,

Flavio

[Flavio Pompermaier]

Found the problem...I compiled my flink dist using maven 3.3.9 but I didn't rerun install from dist directory as specified at https://ci.apache.org/projects/flink/flink-docs-release-1.1/setup/building.html#dependency-shading.

Why is it necessary to recompile flink-dist? 

Best,

Flavio

On Fri, Dec 23, 2016 at 9:24 PM, Flavio Pompermaier <[hidden email]> wrote:

Ok thans Stephan!

On 23 Dec 2016 21:16, "Stephan Ewen" <[hidden email]> wrote:

I would just google for Guava conflict and "shading". Its a fairly common problem with Guava, there are quite a few guides out there how to address that.

On Fri, Dec 23, 2016 at 9:12 PM, Flavio Pompermaier <[hidden email]> wrote:

What do you meab exactly...? do you have a snippet of how I should edit the pom?

On 23 Dec 2016 19:31, "Stephan Ewen" <[hidden email]> wrote:

You have the classical guava version conflict. Flink itself shades Guava away, but there may be multiple conflicting dependencies in your case (HBase / Solr).

I would try to see which of the tool/libraries (HBase, Solr, ...) depend on Guava and create a shaded version of one of them

On Fri, Dec 23, 2016 at 6:33 PM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I have a source HBase table and I have to write to a Solr index.

Unfortunately when I try to run the program on the cluster (Flink 1.1.1) I think I have some problem with dependencies. Can someone suggest me some fix?

This is the error I have just after launching the job:

Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:229)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:202)
    at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:314)
    at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:289)
    at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:161)
    at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:156)
    at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888)
    at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:187)
    at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:89)
    at org.apache.hadoop.hbase.client.MetaScanner.listTableRegionLocations(MetaScanner.java:334)
    at org.apache.hadoop.hbase.client.HTable.listRegionLocations(HTable.java:702)
    at org.apache.hadoop.hbase.client.HTable.getStartEndKeys(HTable.java:685)
    at org.apache.flink.addons.hbase.TableInputFormat.createInputSplits(TableInputFormat.java:170)
    at org.apache.flink.addons.hbase.TableInputFormat.createInputSplits(TableInputFormat.java:45)
    at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:156)
    ... 25 more
Caused by: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
    at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:434)
    at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:60)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1122)
    at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:299)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:155)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:63)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
    ... 38 more

Best,

Flavio

--