Kernel module + userspace utils to replay swap events in userspace.

Introduction

A kernel module dumps <sector no.> <R/W bit> <compress len> to userspace in compact form. This is parsed using a parser lib which then presents event callback interface to these swap R/W event. We use this event interface to simulate same swapping activity in userspace.

<compressed len> - On every swap-out, page is compressed and its compressed length is noted.

We do not require actual data itself, so its not dumped.

Details

compcache project compresses and stores swap pages in memory. The most challenging part of project is to design allocator than can efficiently store these variable sized compressed pages.

Simulating swapping activity allows us to see allocator behavior in userspace itself. This is much more friendly than rebooting, setting up workload to trigger swapping again and again. Also, gradual improvements to allocator will be easier to develop/debug in userspace.

Following shows various components involved:

sr_relay.ko (kernel mod): relays info to user.
  |
  |
  V
sr_collect (user): collects data and dumps to a file.
  |
  |
  V
libsr_parse.so (user lib): Parses data and provides callback interface to let client app process data.
  |
  +--> Sample client app: sr_data_histo - provides compressed page size histogram - easy to plot using gnuplot (see below).

Details for individual components:

sr_relay.ko (kernel mod)

1. Kernel mod loaded as:

insmod sr_relay.ko backing_dev=/dev/sdaXX

Where /dev/sdaXX is partition where real swapping happens. Be very careful: if you specify wrong partition, all data there will be lost - it must be a valid swap partition.

1. This creates pseudo block device /dev/sr_relay.
2. Swapon is issued on /dev/sr_relay (give this swap device highest priority):

swapon /dev/sr_relay -p 1

1. On swap-in/out request to this swap dev, note sector no. and whether its R/W.
2. For swap-out (write), also compress page (using LZO1X) and note compressed length.
3. Relay this info (<sector no.> <R/W bit> <compress len>) to userland.
4. I/O operation is forwarded to backing_dev.

sr_collect (user app)

1. Started as: sr_collect <input file> <output file>

where, <input file> is /debug/sr_relay0 (assuming debugfs is mounted on /debug)

<output file> is file where to dump data sent by sr_relay (e.g. ~/temp/sr_data)

2. Start some workload to trigger swapping. These swap events will be collected by sr_collect and dumped to <output file>.
3. To stop collecting events, terminate this app. Resulting <output file> is not directly readable - it can be parsed using parser apps (see below).
4. Here is compressed sample_sr_data file (151K). (System: 64M swap with 512M system memory. Workload: web browsers, editos, gimp, N/W apps etc.).

lib_srparse.so (user lib)

1. Parses replay data obtained from sr_collect and provides callback interface for swap events.
2. Interface provided by this lib:

<libsr_parse.h>

typedef void (swap_read_fn)(unsigned long index);
typedef void (swap_write_fn)(unsigned long index, unsigned long len);

/*
 * Arg:
 *      0: Sucess
 *      >0: Error: partial read
 *      <0: Error: file read error
 */
typedef void (replay_end_fn)(int err);

int set_sr_file(const char *);
int set_sr_callbacks(swap_read_fn, swap_write_fn, replay_end_fn);
int sr_start(void);

3. So, for client app (see e.g.: SVN/sub-projects/swap_replay/sr_relay/sr_data_histo.c)
1. Client sets replay data file (output file from sr_collect) using set_sr_file().
2. Client starts replay with sr_start()
3. For Read events, lib will call swap_read_fn() implemented in client.
4. For Write events, lib will call swap_write_fn() in client.
5. At end of replay data, lib will call replay_end_fn() in client.

sr_data_histo (user app)

1. This is a sample app that uses libsr_parse to provide compression size histogram.
2. Prints out histograms in format thats easy to plot using gnuplot (histogram for sample_sr_data show below).
3. Here is sample_sr_data_histo.txt as generated by this app. This is plotted using gnuplot as:

<commands at gnuplot prompt>

set xlabel "Compressed page size"
set ylabel "No. of pages"
set xtics 0,256
plot "sample_sr_data_histo.txt" using 1:2 w boxes t "Compressed page size histogram"

• Data summary
• Total: R=186305 W=34834
• % pages with compress ratio <= 50%: 71
• No. of pages with compress length >= 4096: 93

(data is compressed using LZO1X).

Here is MoreReplayData

We will get info about allocator when we do actual alloc()/free() for these variable size chunks. Plan is to relay this data over TLSF allocator. This will provide data regarding its fragmentation and speed.

In general, it will allow easy study/tweaking/testing of different compression algorithms and memory allocators. Collecting this replay data over variety of workloads will present more patterns which we can utilize. Resulting dump is small (just 151K for this sample data) so its easy to share too.