Exactly the same here. Defraggler shows MASSIVE fragmentation (thousands of fragments 
per cache file). Chrome's cache folder is the most fragmented thing on my hard disk. 
After a while, startup becomes very slow.

Consider issue#10727

(No comment was entered for this change.)

The internal files already grow in discrete increments (from 32 KB to 4 MB). We could 
increase those values.

External files have some limited buffering but they tend to grow by the number of 
bytes read from the network each time (the common request size is 32 KB). We could do 
something here but we would be a) increasing the memory footprint due to extra 
buffering, or b) increasing the wasted disk space due to additional requested-but-
not-used space.

As a side note, even if we ask for 10 MB from the OS, that doesn't mean we're getting 
a single chunk of consecutive disk space.

Ricardo, do you have any ideas how we could easily measure fragmentation over time to 
try to reproduce this?

Limiting cache file truncation would help as well if it's ever done.

This is likely too complex to do in the span of a month to get ready for 4.0, let's try and 
do this in 5.0.

Replacing labels:
   Area-BrowserBackend by Area-Internals

"We could do 
something here but we would be a) increasing the memory footprint due to extra 
buffering, or b) increasing the wasted disk space due to additional requested-but-
not-used space."

Many web servers report the size of a requested file in the response headers... this 
could be used to reserve an exact chunk for the file (Firefox does this for downloads 
IIRC).  If the file ends up being smaller you could just truncate the file when the 
download finishes.

Might want to put a limit on the size you reserve in this way though, although on 
Windows NTFS has some useful functions that could remove the need for a limit... 
"sparse files" I think the feature is called?

"As a side note, even if we ask for 10 MB from the OS, that doesn't mean we're 
getting 
a single chunk of consecutive disk space."

True but in that case there's not much one can do, might as well let the OS handle 
it.  Besides getting a slightly defragmented 10MB chunk is still an improvement over 
a ton of unfragmented 32kb chunks all over the drive, imo.

(No comment was entered for this change.)

if Content-Length is known then pre-allocating will give a much better chance of getting a 
contiguous extent on disk. it's not guaranteed, this functionality is only available with 
admin rights, but it's sgnifcantly better than extending the file continuously. on 
premature EOF, the file can be truncated to save space.

if Content-Length is NOT known, i'd recommend pre-allocating significantly larger chunks on 
disk to avoid fragmentation. something on the order of 1-10MB feels about right. obviously 
truncate to length on EOF to save space.

Typical defragmentation algorithms will fit the smaller cache files into sectors that 
have lots of other small files, and then precisely because they expand in size, they 
will become fragmented when they hit the end of the sector and need to keep going. If 
the files were pre-allocated in larger sizes, they would need to grow less frequently 
throughout their lifecycle and would thus produce less overall fragmentation. I realize 
it comes at the expense of disk space, but personally I couldn't care less if Chrome 
creates large, mostly empty cache files, so long as I don't have to defragment them 
every day to keep them in one piece.

I don't know if Snow Leopard is exacerbating things, but the fragmentation is totally out of control, turning 
Chrome on a new MacBook i7 2.66 w/ 4GB ram into the slowest browser on my system and less performant than 
safari on my iphone.  The 500GB drive is less than 20% used.

This is Chrome 5.0.375.55.

I agree with ayan4m1, I don't care about disk usage.  Pre-allocate larger files when there's low disk utilization.  
Clearing the cache doesn't help because suddenly it's creating *new* highly fragmented files.

Downloaded files are also fragmented a lot.
Just imagine a 1GB file...

I recommend everyone who wants to test (and temporarily cure) this problem to download a good disk defrag, e.g. a free one from Auslogics:

http://www.auslogics.com/en/software/disk-defrag/

It shows something like 1000 fragments for dozens of the most fragmented files - even if you run defragger many times a day.

FYI, fragmentation is *really* an issue for a few of the cache files (data_), and for them it is fixed by the first de-fragmentation.

Maybe is a small problem, but is *really* only a problem of Chrome. No other program make the same fragmentation, no one.

Some fragmentation is unavoidable. However, it's disturbing to have 30-40 files that look like this: <Defraggler Data Snippet>
f_00896a	587	28630942	C:\Users\Snap\AppData\Local\Google\Chrome\User Data\Default\Cache\	
f_0088a7	281	2851891	C:\Users\Snap\AppData\Local\Google\Chrome\User Data\Default\Cache\	
f_008939	234	5220111	C:\Users\Snap\AppData\Local\Google\Chrome\User Data\Default\Cache\	</Defraggler data>
And I de-frag weekly. Granted this is the most-used program on my computer, but with fragmentation this intense it's impacting overall PC speed, and Chrome's speed.
The fragmentation from Chrome - increases fragmentation on my other programs as well.

(No comment was entered for this change.)

On a Mac, the problem is grave. Since the system does not get rid of fragmented free space and there is no free tool for defragmentation, I needed to buy a tool for almost 30$ just to be able to work properly again. Chrome is the only application I know that does this. I will now delete Chrome and switch back to Safari until that problem is fixed. I attached a Screenshot of my most fragmented files after a full defragmentation and running Chrome for only a couple of hors. 

My solution would be: create the cache file at its maximum size. If its full, create another one, and maybe free the place that's not used. Maybe its worth checking out how Safari does it, since its cache always stays in one peace. 

Mac OS X has some mechanism to defragment files on the fly under certain circumstances… Maybe its worth to take a look at the paragraph "Built-in Measures in Mac OS X Against Fragmentation" on http://www.osxbook.com/software/hfsdebug/fragmentation.html

	topfragfiles.png 284 KB   View   Download

@muellerstephan: Perhaps fragmentation has a more significant effect on OS X, but on windows I frequently see Chrome's cache files broken into 400+ fragments after 3 to 4 days of moderate use. It is around this point that I start noticing performance degradation. The # of fragments you have per cache file is quite a bit lower than what I show even after just a couple days of use, so perhaps OS X *is* trying to help reduce the fragmentation of Chrome's cache files to some degree.

Frankly I'm surprised this issue isn't being taken more seriously, considering Google's focus on performance. This was first identified in Chrome 1.0. Why does it keep getting pushed back?


FYI: Defraggler is the tool I use to view and defragment Chrome's cache files. http://www.defraggler.com

(No comment was entered for this change.)

The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=57082 

------------------------------------------------------------------------
r57082 | rvargas@google.com | 2010-08-23 11:47:25 -0700 (Mon, 23 Aug 2010) | 18 lines
Changed paths:
   M http://src.chromium.org/viewvc/chrome/trunk/src/net/disk_cache/backend_impl.cc?r1=57082&r2=57081
   M http://src.chromium.org/viewvc/chrome/trunk/src/net/disk_cache/backend_impl.h?r1=57082&r2=57081
   M http://src.chromium.org/viewvc/chrome/trunk/src/net/disk_cache/backend_unittest.cc?r1=57082&r2=57081
   M http://src.chromium.org/viewvc/chrome/trunk/src/net/disk_cache/disk_cache_test_base.cc?r1=57082&r2=57081
   M http://src.chromium.org/viewvc/chrome/trunk/src/net/disk_cache/disk_cache_test_base.h?r1=57082&r2=57081
   M http://src.chromium.org/viewvc/chrome/trunk/src/net/disk_cache/entry_impl.cc?r1=57082&r2=57081
   M http://src.chromium.org/viewvc/chrome/trunk/src/net/disk_cache/entry_impl.h?r1=57082&r2=57081
   M http://src.chromium.org/viewvc/chrome/trunk/src/net/disk_cache/entry_unittest.cc?r1=57082&r2=57081
   M http://src.chromium.org/viewvc/chrome/trunk/src/net/disk_cache/file.h?r1=57082&r2=57081
   M http://src.chromium.org/viewvc/chrome/trunk/src/net/disk_cache/file_posix.cc?r1=57082&r2=57081
   M http://src.chromium.org/viewvc/chrome/trunk/src/net/disk_cache/file_win.cc?r1=57082&r2=57081

Disk cache: Extend the internal buffering performed by each entry
to cover external files.

We now keep a variable-size buffer and use it even after we
know that the data is not going to be stored by a block-file.
The backend keeps track of the total memory used by all entries
and prevents that value from going over a max value that
depends on the total memory available.

This CL removes the tests that were checking the synchronous
operation of sparse IO because that model is no longer supported
by the public API, and this CL would add complexity to them
(they fail due to thread safety concerns).

BUG=6626
TEST=net_unittests

Review URL: http://codereview.chromium.org/3167020
------------------------------------------------------------------------

(No comment was entered for this change.)

Similar on WindowsXP. Got hundreds of fragmented Chrome files. Some of them in more than thousand fragments...
Microsoft offers free sysinternal tool Contig.exe that can be used to defragment the Chrome folder recursively like this:

cd "to\your\chrome5\folder"
"your\path\to\Contig.exe" -v -s .

I'm not sure, but I think Contig.exe source code was available before sysinternals was taken over by Microsoft. Then there is the public Windows Defragmentation API as also used by some open source tools. Perhaps Chrome can learn to deframgent itself ... ;-)

I have Chrome 6.0.482.55 beta, did a cache clear and defrag 3 days ago, and now I get this from the windows disk defragmenter.
Perhaps the fix by lvargas hasn't been pushed yet.

I guess this also reveals that the history files are subject to the same kind of fragmentation.

Fragments       File Size       Most fragmented files
554             5 MB            \Documents and Settings\Rares\Local Settings\Application Data\Google\Chrome\User Data\Default\Cache\f_000160
155             10 MB           \WINDOWS\pchealth\helpctr\DataColl\CollectedData_10814.xml
129             1 MB            \Documents and Settings\Rares\Local Settings\Application Data\Google\Chrome\User Data\Default\Cache\f_0001c1
105             1 KB            \Documents and Settings\Rares\ntuser.dat.LOG
104             21 MB           \Documents and Settings\Rares\Local Settings\Application Data\Google\Chrome\User Data\Default\History Index 2010-09
93              1 MB            \Documents and Settings\Rares\Local Settings\Application Data\Google\Chrome\User Data\Default\Cache\f_0001c3
89              1 MB            \Documents and Settings\Rares\Local Settings\Application Data\Google\Chrome\User Data\Default\Cache\f_0001c2
81              1 MB            \Documents and Settings\Rares\Local Settings\Application Data\Google\Chrome\User Data\Default\Cache\f_0000f9
80              615 KB          \Documents and Settings\Rares\Local Settings\Application Data\Google\Chrome\User Data\Default\Cache\f_00004f
77              317 KB          \Documents and Settings\Rares\Local Settings\Application Data\Google\Chrome\User Data\Default\Cache\f_000140
69              284 KB          \Documents and Settings\Rares\Local Settings\Application Data\Google\Chrome\User Data\Default\Cache\f_00000a
69              615 KB          \Documents and Settings\Rares\Local Settings\Application Data\Google\Chrome\User Data\Default\Cache\f_00004e
68              305 KB          \Documents and Settings\Rares\Local Settings\Application Data\Google\Chrome\User Data\Default\Cache\f_000159
46              221 KB          \Documents and Settings\Rares\Local Settings\Application Data\Google\Chrome\User Data\Default\Cache\f_000155
44              220 KB          \Documents and Settings\Rares\Local Settings\Application Data\Google\Chrome\User Data\Default\Cache\f_000151

@rpamfil: The change is not part of Chrome 6. It is part of Chrome 7.

It is really improved now, there are only few fragmentations left.
It would be great if it would apply to more small user data files.

Too tired to wait :  I switch to IE9. Maybe, someday, I ll'be back.

I can confirm that this eliminates all significant fragmentation of cache files on Windows (Chrome 7.0.517 beta). Like @MayhemYDG says, history and some other user data files still get fragmented, but that's not causing a significant performance impact

Would several files of multi megabyte size work (5-10MB)? The idea is to pre-allocate enough of them to account for the maximum size of the cache. In each large file you would place small objects separated by some type of marker which includes the details of the cached object that follows. In order not to fragment files internally as objects are added and invalidated (in the large files) Chrome should compact the cached objects in the large files when idle. When a cached object is found to be different then the stored object it is counted as a new object and the prior is invalidated and overwritten on the next compact. The new cached object is written in a large file as if it was new. New cached objects would only be written in larger files that would fully fit them. Ideally an index of the objects and what file they are in would be stored in ram. When Chrome closes it discards the index. When it opens it would scan the large files and rebuild the index. 

I have no idea how practical this would be to implement but it would only cause fragmentation on initial allocation. After which a single defrag would leave the cache defragmented indefinitely (assuming the size of the cache is not increased).

(No comment was entered for this change.)

P2 bugs with an owner that are not marked as started are being automatically moved to mstone:11.

My "History Index 2010-11" file was in 1077 fragments. I don't know if it impacts chrome's performances but it sure makes a lot of fragmentations.

Instead of dragging this bug for further optimizations, I created  issue 72947 .

How to speed up your pc

http://www.acebyte.com/

That's not a reputable site- listed in comment 45. Most likely spam. (If you're interested in improving your PC health see piriform's software.)
 ~ I find this issue as solved. I have seen no significant fragmentation produced by Google Chrome after the fixes were implemented. (I've been regularly checking and testing this aspect of chrome)
Thank you to the users here that helped address and solve this!

This issue has been closed for some time. No one will pay attention to new comments.
If you are seeing this bug or have new data, please click New Issue to start a new bug.

(No comment was entered for this change.)

(No comment was entered for this change.)