hbase client + ScannerTimeoutException

Bonjour � tous,

J'utilise une application d�velopp�e en java qui traite de gros volume de donn�es gr�ce � hbase.
Mon but est de configurer correctement hbase afin de faire fonctionner cette application chez nous.
Une nouvelle version de cette derni�re, qui optimise grandement les temps de traitements, me pose probl�me.

J'ai l'erreur suivante qui intervient au bout de 10 � 15 minutes de traitement :

Code :

S�lectionner tout - Visualiser dans une fen�tre � part

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
[2016-02-17 09:35:42.751] ERROR PolarizationMapper businessRule 306122407
org.apache.hadoop.hbase.client.ScannerTimeoutException: 74140ms passed since the last invocation, timeout is currently set to 60000
	at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:432)
	at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:364)
	at gde.sphere.snapshot.hbase.PolarizationMapper.fill(PolarizationMapper.java:231)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:229)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
	at gde.sphere.snapshot.CalculationService$_fillTempRecords_closure10_closure21.doCall(CalculationService.groovy:682)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:88)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
	at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
	at groovy.lang.Closure.call(Closure.java:276)
	at groovy.lang.Closure.call(Closure.java:289)
	at groovy.lang.Closure$call$0.call(Unknown Source)
	at groovyx.gpars.GParsPoolUtil$_eachParallel_closure3.doCall(GParsPoolUtil.groovy:123)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:88)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
	at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
	at groovy.lang.Closure.call(Closure.java:276)
	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:51)
	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:79)
	at com.sun.proxy.$Proxy25.map(Unknown Source)
	at jsr166y.forkjoin.ParallelArray$WithBoundedMapping.leafTransfer(ParallelArray.java:1960)
	at jsr166y.forkjoin.PAS$FJRMap.atLeaf(PAS.java:439)
	at jsr166y.forkjoin.PAS$FJBase.compute(PAS.java:299)
	at jsr166y.forkjoin.RecursiveAction.exec(RecursiveAction.java:247)
	at jsr166y.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:286)
Caused by: org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: 38, already closed?
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2223)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
	at java.lang.Thread.run(Thread.java:745)
 
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:322)
	at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:255)
	at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:369)
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:343)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
	at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Name: 38, already closed?
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2223)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
	at java.lang.Thread.run(Thread.java:745)
 
	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1206)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
	at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:213)
	... 9 more
[2016-02-17 09:35:42.754] ERROR PolarizationMapper org.apache.hadoop.hbase.client.ScannerTimeoutException: 74140ms passed since the last invocation, timeout is currently set to 60000

J'ai fais plusieurs recherches, et ce lien m'a fait modifier des variables dans mon hbase-site.xml :
http://stackoverflow.com/questions/1...apreduce-error

Code :

S�lectionner tout - Visualiser dans une fen�tre � part

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
 
  <property>
    <name>hbase.master.lease.period</name>
    <value>900000</value>
  </property>
  <property>
   <name>hbase.regionserver.lease.period</name>
   <value>450000</value>
  </property>
  <property>
    <name>hbase.rpc.timeout</name>
    <value>450000</value>
  </property>
  <property>
    <name>zookeeper.session.timeout</name>
    <value>80000</value>
  </property>

Ma question : est-ce que quelqu'un aurait une id�e de pourquoi mon timeout est toujours sett� � 60 secondes ?

J'ai contr�l� dans ma config hbase, hdfs et zookeeper, plus aucune trace d'une variable qui serait sett�e � 60s.

J'ai relanc� hbase, clean�, modifi� de nouveau, mais je ne trouve pas de solution � ce probl�me.

Merci d'avance pour vos �claircissements et bonne journ�e � tous.

Si jamais quelqu'un a le m�me probl�me, ce thread permet de r�pondre � la probl�matique :
https://developer.ibm.com/answers/qu...outexcept.html

En fait, les variables c�t� serveur ne sont pas prises en compte, il faut faire la modif c�t� client.

Du coup, notre partenaire nous a ajout� la variable dans un fichier de config, et le probl�me est r�solu