there is a log cache file in /var/folders/.../-Cache-/com.apple.ATS that has a mod time of each test, but nuking 
it and rebooting seems to recreate a large file anyways, so that's probably not it (and it doesn't seem to be 
growing)

a reboot seems to have dropped the cycle times on ui, but i'm not sure why yet.

(No comment was entered for this change.)

not climbing yet...

http://build.chromium.org/buildbot/waterfall/stats/Chromium%20Mac

They are all climbing again.

Can we defrag a mac?

(No comment was entered for this change.)

I looked at the machine over VNC and there are like 50+ chrome icons in the dock, and 
they are not linked to any tests in progress. I could not see those processes with the 
activity monitor though. Not sure what's up with that.

I think we should try defragging.

The FS generally defragments on the fly.  If we've got an idle bot we can swap with this that's never been used, we 
can try that.

Last time, a simple bounce seemed to bring numbers back in line.  So something builds up that a bounce clears.

Looking at the process list, all those apps in the dock aren't actually running.  It's sorta like the dock is confused.  
I've go ahead and bounced the machine (couldn't find anything else going on).

(No comment was entered for this change.)

(No comment was entered for this change.)

Mark - Have you got any ideas, I've looked at this a few times now and haven't come up with anything.

I don’t really. :(

My one fear here is this is currently the only bot running tests that is on 10.5 Server.  If we switch to more VMs 
again, we'll be using more Server installs, and does that mean this slow down will happen on those bots also?

there are two chromium icons stuck in the dock again, but nothing shows up in ps to back those.  We've seen this 
a few times before, but no idea on the cause.  I tried sending a HUP to the dock to reset it, we'll see if that has 
any relationship to the problem.

no extra swapfiles like we've seen on some bots

/var/servermgrd has a bunch of files that get written every minute, they don't seem to be growing.  mark 
fixed a config in servermgr where it had a bad host (old machine name), but not real effect was seen.

There are some directories in lib that keep showing mod times of the last test runs, but no files within them 
changed, so not sure what's triggering things.

iosnoop hasn't shown anything of interest when i've been watching it while tests run.

it would appear taking all the perf tests off have greatly stopped this issue.  ui_tests might still have a small climb going on, but until all the data from other runs ages out, it will be hard to tell for sure.

note, the compile time will keep the scale on the graphs large enough, it will be hard to really see if there is a climb.

Still slowly climbing, but punting to mstone-7 since I have yet to find a smoking gun, and we're migrating all the build bots in the near future, so it's not really worth digging for answers with the whole OS about to change.

 Issue 49140  has been merged into this issue.

Clearing mstone since it isn't happening any more (knock on wood), and we're about to go through a major bot shuffle which will change all the hardware.

Thomas, according to http://build.chromium.org/buildbot/perf/mac-release-10.5/startup/report.html?history=500&rev=58042&trace=t&trace=t_ref, the Mac10.5 Perf(2) startup test is showing up as regressing on the perf charts again.  Time for another reboot?

bounced

Misbehaving again in http://build.chromium.org/f/chromium/perf/mac-release-10.5/startup/report.html?history=100&rev=69710.  Uptime on mini117-m1 is 27 days, rebooting.

 Issue 58709  has been merged into this issue.

http://build.chromium.org/buildbot/waterfall/stats/Chromium%20Mac

if you look at some of the testers, we don't see these slowdowns, so it's something on this bot, and 
not on all bots.

Misbehaving in http://build.chromium.org/f/chromium/perf/mac-release-10.5/startup/report.html?history=1500&rev=79018.  If I get cycles I will take a closer look using Instruments before and after a reboot (the reboot should clear it).

Just rebooted mini106-m1 again today.

(No comment was entered for this change.)

(No comment was entered for this change.)

(No comment was entered for this change.)

(No comment was entered for this change.)

(No comment was entered for this change.)