[NAS-2427] JmsBroker stops because of unhandled messages to XXX_THE_BAMON queue Created: 23/Jun/15  Updated: 31/Aug/15  Resolved: 29/Jun/15

Status: Resolved
Project: NetarchiveSuite
Component/s: None
Affects Version/s: 5.0-Milestone1
Fix Version/s: 5.0

Type: Task Priority: Critical
Reporter: Mikis Seth Sørensen (Inactive) Assignee: Søren Vejrup Carlsen (Inactive)
Resolution: Fixed  
Labels: None
Remaining Estimate: Not Specified
Time Spent: 10m
Original Estimate: Not Specified


 Description   

Lots of 'message count limit reached for destination XXX_THE_BAMON' errors in broker log:

com.sun.messaging.jmq.jmsserver.util.BrokerException: [B4120]: Can not add message 269807-172.17.0.53(87:f:46:55:d5:ab)-56285-1434558854196 to destination TESTNICL_KB_THE_BAMON [Queue]. The destination message count limit (maxNumMsgs) of 10000 has been reached.
com.sun.messaging.jmq.jmsserver.util.BrokerException: [B4120]: Can not add message 5035-172.17.0.55(e6:fe:8:4d:d0:51)-63414-1434558858221 to destination TEST5B_KB_THE_BAMON [Queue]. The destination message count limit (maxNumMsgs) of 10000 has been reached.
com.sun.messaging.jmq.jmsserver.util.BrokerException: [B4120]: Can not add message 5035-172.17.0.55(e6:fe:8:4d:d0:51)-63414-1434558858221 to destination TEST5B_KB_THE_BAMON [Queue]. The destination message count limit (maxNumMsgs) of 10000 has been reached.
com.sun.messaging.jmq.jmsserver.util.BrokerException: [B4120]: Can not add message 5976-172.17.0.53(ce:4f:41:45:53:e9)-56282-1434558858642 to destination TEST5B_KB_THE_BAMON [Queue]. The destination message count limit (maxNumMsgs) of 10000 has been reached.
com.sun.messaging.jmq.jmsserver.util.BrokerException: [B4120]: Can not add message 5976-172.17.0.53(ce:4f:41:45:53:e9)-56282-1434558858642 to destination TEST5B_KB_THE_BAMON [Queue]. The destination message count limit (maxNumMsgs) of 10000 has been reached.
com.sun.messaging.jmq.jmsserver.util.BrokerException: [B4120]: Can not add message 57031-172.17.0.53(d4:33:e0:c7:59:66)-56283-1434558969692 to destination SystemTest_KB_THE_BAMON [Queue]. The destination message count limit (maxNumMsgs) of 10000 has been reached.


 Comments   
Comment by Søren Vejrup Carlsen (Inactive) [ 23/Jun/15 ]

This is a symptom of KB bitapps still running when the other apps in a NAS instance has been stopped.

Comment by Søren Vejrup Carlsen (Inactive) [ 23/Jun/15 ]

The reason for this problem seems to be that the operation on Windows is not necessarily finished before the SSH connection is terminated.

TLR has added

@ping 1.1.1.0 -n 1 -w 10000

as last line in every startall.bat and killall.bat
This depends on the fact that 1.1.1.0 is not pingable.

Alternatively this could be replaced by the code

timeout /t 10 /nobreak > NUL
Comment by Søren Vejrup Carlsen (Inactive) [ 23/Jun/15 ]

This causes zombie processes to be alive on the Windows servers, while the rest of the NetarchiveSuite instance is shutdown down cleanly.

Comment by Søren Vejrup Carlsen (Inactive) [ 23/Jun/15 ]

However, use the timeout command gives me this error instead:

ERROR: Input redirection is not supported, exiting the process immediately.
Comment by Søren Vejrup Carlsen (Inactive) [ 23/Jun/15 ]

This error doesn't show when the batch file is called locally, only, when it is called remotely using a SSH command

Comment by Søren Vejrup Carlsen (Inactive) [ 24/Jun/15 ]

The solution in the end is another one. We already had a wait.vbs script, which waited 5 seconds used in the restart script on windows.
The number of seconds waited has now been changed to 10 seconds, and this script is now called as the last operation in killall.bat and startall.bat scripts.

Comment by Søren Vejrup Carlsen (Inactive) [ 24/Jun/15 ]

This has already been tested successfully in the TEST6 releasetest

Comment by Colin Rosenthal [ 28/Aug/15 ]

Is there any reason not to close this issue?

Comment by Søren Vejrup Carlsen (Inactive) [ 31/Aug/15 ]

This has already been tested successfully in our testenvironment

Generated at Tue Apr 07 09:22:58 CEST 2020 using Jira 7.13.8#713008-sha1:1606a5c1e7006e1ab135aac81f7a9566b2dbc3a6.