My favorites | Sign in
Project Home Downloads Wiki Issues Source
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 1528: significant slowdowns and eventual hang on e-mail notifications
10 people starred this issue and may be notified of changes. Back to list
Status:  New
Owner:  ----


Sign in to add a comment
 
Reported by kom...@gmail.com, Aug 16, 2012
Affected Version: 2.4.1
What steps will reproduce the problem?

There are no particular steps to reproduce.
We just use it as usual and on operations, requiring e-mail notifications, it just stops responding (infinite 'Working' in web-interface). When we try to monitor what's going on with 'gerrit ps' command, we see a plenty of threads (processes?) in waiting state with email-related operations (see attached files for detailed output).
During this time in postfix logs: postfix/smtpd[7899]: timeout after DATA (0 bytes) from localhost.localdomain[127.0.0.1]
The output of top, system configuration and gerrit.config are also attached.

Also, could you point out how to monitor java thread's (processes) callstacks? We tried to "kill -QUIT <process_id>" magic and nothing came out (do we need attach with gdb to see stdout?) How do you debug this kind of java issues (not Java programmer, sorry)? 
gerrit.config
740 bytes   View   Download
show-queue.txt
3.5 KB   View   Download
systemConfiguration.txt
326 bytes   View   Download
top.txt
2.3 KB   View   Download
Aug 24, 2012
#2 kom...@gmail.com
ps is alias for gerrit show-queue.
Sep 14, 2012
#3 virt...@gmail.com
I'm aware of that. New details though: it appears that at some moment email sending just stop working and all subsequent attempts to add reviewer result infinite "Working" and yet another task in show-queue.
Sep 23, 2012
#4 virt...@gmail.com
Can I change priority at least?
Jul 31, 2013
#5 presich....@gmail.com
Hello guys!
We have faced this issue on 2.5.1 version.

But in our case Gerrit stops to Merge changes to the repository. A lot of changes was in status: Submitted, Merge Pending.

So, I can assume this is not Minor bug..yes it is rare but not Minor)

Also cannot say what is exact steps to reproduce but I can say that there are a LOT of comments are sent in our instance of Gerrit. Also function of set-reviewer is also used heavily.

Have you found any reasons and solutions for this issue?

Many thanks.
Jun 12, 2014
#6 stayle...@gmail.com
Hello,
We faced this issue on 2.8 version recently several times.
After restart the gerrit service, it back to normally.
But after 3-4 days, it occurred again!
Anyone known what happened?
Jul 28, 2014
#7 euler.ji...@gmail.com
We also meet such problem when upgrade to 2.8 version and later 2.8.5 version.

Lots of send-email task pending in the queue.

Anyone have any suggestion?
Oct 6, 2014
#8 jeanjacq...@gmail.com
Just happened to us right now (2.8.1).
The lone worker thread was blocked trying to connect to the smtp server.
Apparently there was a temporary glitch, I closed the socket and everything dequeued properly, including further send-email tasks.
This has been handled in change #55671 which allows you to set a custom timeout instead of the default "wait forever", useful if your smtp server isn't too reliable.
Sign in to add a comment

Powered by Google Project Hosting