From derek.br...@gmail.com on December 10, 2010 17:58:11

PR 496304

for Dr. Heapstat, once a callstack is dumped to disk, we only keep the full string around for
comparison with later callstacks. we can instead store an md5 and free up
a lot of memory. with PR 493134 shrinking the snapshots, the memory used
by callstacks is now larger than the snapshot memory:

unit test run that succeeds w/ -snapshots 32 and higher hard limit:
app mallocs: 13041574, frees: 12587531
unique malloc stacks: 350665
app heap regions: 21
Heap usage:
snapshot: cur= 58123 KB, max= 58158 KB
callstack: cur= 81195 KB, max= 81197 KB
hashtable: cur= 15395 KB, max= 17023 KB

PR 496304: keep md5 and throw away full callstack to save memory

• pulled alloc_stack_table out of common/alloc.c and gave client control
  over it to enable Dr. Heapstat to use different hashes
• took md5 code from DynamoRIO and placed in separate file instead of
  utils.[ch] so I could include the DR license, even though not strictly
  necessary since VMware owns the copyright
• md5 is noticeably slow on heap-intensive apps so I put in a two-part
  crc32, the whole callstack plus the first half, as a faster option: crc32
  is on by default, but I left the md5 code in place so we can use it as a
  check since we have no other way of identifying a collison.
  numbers:
  short cfrac: tot 4.6s, crc: 4.8s, md5 5.4s
  long cfrac: tot 56s, crc 56.5s, md5 67s
  roboop: tot 62s, crc 61s, md5 71s
• since full callstacks no longer in memory, on fork have to copy the
  actual file from the parent to the child. since fds are shared across
  forks, need to dup the fd: added dr_dup_file_handle() for this.
• two new cmake options:
  option(USE_MD5 "use md5 instead of crc32 for callstack hashes for Dr. Heapstat")
  option(CHECK_WITH_MD5 "use crc32 for callstack hashes but check for collisions with md5")

Results show 8x-9x reduction in callstack memory usage

Original issue: http://code.google.com/p/drmemory/issues/detail?id=167