Uploaded image for project: 'WebDanica'
  1. WebDanica
  2. WEBDAN-111

Timeout from hbase during getSeedsCount call

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Fix
    • Minor
    • None
    • None
    • DATABASE
    • None

    Description

      [webdanica@kb-test-hadoop-01 test-hbase-0.2.0]$ bash databaseStats.
      bash: databaseStats.: Ingen sådan fil eller filkatalog
      [webdanica@kb-test-hadoop-01 test-hbase-0.2.0]$ bash databaseStats.sh 
      log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
      log4j:WARN Please initialize the log4j system properly.
      log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
      Exception in thread "main" org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
      Fri Nov 04 12:05:14 CET 2016, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60327: row '�' on table 'SEEDS_STATUS_IDX' at region=SEEDS_STATUS_IDX,,1477707730366.ca0cf2813d835a9c29781c661bd6c3bd., hostname=kb-test-hadoop-03.kb.dk,16020,1476197286897, seqNum=101868080
      
      	at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:111)
      	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:695)
      	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:638)
      	at org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:50)
      	at org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:97)
      	at org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)
      	at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:64)
      	at org.apache.phoenix.iterate.UngroupedAggregatingResultIterator.next(UngroupedAggregatingResultIterator.java:39)
      	at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:778)
      	at dk.kb.webdanica.datamodel.dao.HBasePhoenixSeedsDAO.getSeedsCount(HBasePhoenixSeedsDAO.java:166)
      	at dk.kb.webdanica.utils.DatabaseUtils.printDatabaseStats(DatabaseUtils.java:48)
      	at dk.kb.webdanica.tools.ComputeStats.main(ComputeStats.java:22)
      Caused by: java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
      Fri Nov 04 12:05:14 CET 2016, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60327: row '�' on table 'SEEDS_STATUS_IDX' at region=SEEDS_STATUS_IDX,,1477707730366.ca0cf2813d835a9c29781c661bd6c3bd., hostname=kb-test-hadoop-03.kb.dk,16020,1476197286897, seqNum=101868080
      
      	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      	at java.util.concurrent.FutureTask.get(FutureTask.java:202)
      	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:690)
      	... 10 more
      Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
      Fri Nov 04 12:05:14 CET 2016, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60327: row '�' on table 'SEEDS_STATUS_IDX' at region=SEEDS_STATUS_IDX,,1477707730366.ca0cf2813d835a9c29781c661bd6c3bd., hostname=kb-test-hadoop-03.kb.dk,16020,1476197286897, seqNum=101868080
      
      	at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:111)
      	at org.apache.phoenix.iterate.TableResultIterator.initScanner(TableResultIterator.java:125)
      	at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:108)
      	at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:103)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
      Fri Nov 04 12:05:14 CET 2016, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60327: row '�' on table 'SEEDS_STATUS_IDX' at region=SEEDS_STATUS_IDX,,1477707730366.ca0cf2813d835a9c29781c661bd6c3bd., hostname=kb-test-hadoop-03.kb.dk,16020,1476197286897, seqNum=101868080
      
      	at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:271)
      	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:199)
      	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
      	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
      	at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
      	at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
      	at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
      	at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
      	at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
      	at org.apache.phoenix.iterate.TableResultIterator.initScanner(TableResultIterator.java:121)
      	... 7 more
      Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=60327: row '�' on table 'SEEDS_STATUS_IDX' at region=SEEDS_STATUS_IDX,,1477707730366.ca0cf2813d835a9c29781c661bd6c3bd., hostname=kb-test-hadoop-03.kb.dk,16020,1476197286897, seqNum=101868080
      	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
      	at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64)
      	... 3 more
      Caused by: java.io.IOException: Call to kb-test-hadoop-03.kb.dk/130.226.228.20:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=81, waitTime=60001, operationTimeout=60000 expired.
      	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.wrapException(AbstractRpcClient.java:278)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1239)
      	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:217)
      	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:318)
      	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32831)
      	at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:373)
      	at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:200)
      	at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
      	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
      	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:350)
      	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:324)
      	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
      	... 4 more
      Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=81, waitTime=60001, operationTimeout=60000 expired.
      	at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1213)
      	... 14 more
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            svc Søren Vejrup Carlsen (Inactive)
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: