compcache - Project Hosting on Google Code

Last 7 days Jan 04, 2010 Performance (Performance numbers for compcache) Wiki page edited by gmatht Jan 04, 2010 Eeepc701 (Application benchmark on Eeepc 701.) Wiki page added by gmatht Jan 04, 2010 issue 24 (Scalability improvements) commented on by nitingupta910 - See: http://code.google.com/p/compcache/wiki/Scalability HTH. See: http://code.google.com/p/compcache/wiki/Scalability HTH. Jan 04, 2010 Scalability (Handling multiple cores) Wiki page edited by nitingupta910 Jan 04, 2010 xvMalloc (xvMalloc memory allocator) Wiki page edited by nitingupta910 Jan 04, 2010 Scalability (Handling multiple cores) Wiki page added by nitingupta910 Jan 04, 2010 Performance (Performance numbers for compcache) Wiki page edited by gmatht Jan 04, 2010 CompilingAndUsingNew Wiki page commented on by rvpittman - Anyone found a fedora rpm for compcache? I did get it installed manually but would like to suggest compcache´s use by the fedora lxde project to allow folks to INSTALL that have 256mb or less. Currently the spin will RUN fine on 256mb even with X but the gui install fails and reverts to text mode. Anyone found a fedora rpm for compcache? I did get it installed manually but would like to suggest compcache´s use by the fedora lxde project to allow folks to INSTALL that have 256mb or less. Currently the spin will RUN fine on 256mb even with X but the gui install fails and reverts to text mode. Jan 04, 2010 Revision 5962668703 (Added some systemtap scripts) pushed by nitingupta910 - Added some systemtap scripts Added some systemtap scripts Jan 02, 2010 2 new revisions pushed by nitingupta910 - addef49610:[PATCH] qemu-kvm: support for virtio vswap device Parse scatterlist and dump {<R/W>, <sector>} tuple. 4d2b9744a1:merge addef49610:[PATCH] qemu-kvm: support for virtio vswap device Parse scatterlist and dump {<R/W>, <sector>} tuple. 4d2b9744a1:merge Jan 02, 2010 2 new revisions pushed by nitingupta910 - fa940f005a:Construct proper scatterlist for vswap scatterlist now contains: - type (R/W) - swap sector - PAGE_SIZE buffer (swap bio page) - status bit 37b7de782e:merge fa940f005a:Construct proper scatterlist for vswap scatterlist now contains: - type (R/W) - swap sector - PAGE_SIZE buffer (swap bio page) - status bit 37b7de782e:merge Jan 02, 2010 Revision 3d77b16b29 ([PATCH] Support for virtio vswap device vswap is a virtual ...) pushed by nitingupta910 - [PATCH] Support for virtio vswap device vswap is a virtual block device (/dev/vswap) visible inside a guest. This device can be used only as a swap device. Pages swapped to this device are sent to the host (hypervisor) directly. If write to host fails, the page is swapped within guest itself. To enable this device use '-vswap virtio' as command line option for qemu-kvm. Signed-off-by: Nitin Gupta <ngupta@vflare.org> [PATCH] Support for virtio vswap device vswap is a virtual block device (/dev/vswap) visible inside a guest. This device can be used only as a swap device. Pages swapped to this device are sent to the host (hypervisor) directly. If write to host fails, the page is swapped within guest itself. To enable this device use '-vswap virtio' as command line option for qemu-kvm. Signed-off-by: Nitin Gupta <ngupta@vflare.org> Jan 02, 2010 Revision ae13bc148c (Send {<R/W>, <sector>} tuple to host.) pushed by nitingupta910 - Send {<R/W>, <sector>} tuple to host. Send {<R/W>, <sector>} tuple to host. Last 30 days Jan 01, 2010 Revision 4890b58751 (Initial import of vswap. Compiles cleanly on 2.6.30.10) pushed by nitingupta910 - Initial import of vswap. Compiles cleanly on 2.6.30.10 Initial import of vswap. Compiles cleanly on 2.6.30.10 Dec 30, 2009 StatsExplained Wiki page commented on by nitingupta910 - gmatht: the version in hg does not have notify support. I'm working on making a single patch for 2.6.33 that will have everything included. gmatht: the version in hg does not have notify support. I'm working on making a single patch for 2.6.33 that will have everything included. Dec 30, 2009 StatsExplained Wiki page commented on by gmatht - I don't have a "#define CONFIG_SWAP_FREE_NOTIFY'" line in compcache/compat.h . I am using the latest version in hg. How do I enable SWAP_FREE_NOTIFY? Also is there an easy way to tell if NOTIFY is enabled? If e.g. I swap off and --stats still reports that memory is used, this means that NOTIFY is not enabled right? I don't have a "#define CONFIG_SWAP_FREE_NOTIFY'" line in compcache/compat.h . I am using the latest version in hg. How do I enable SWAP_FREE_NOTIFY? Also is there an easy way to tell if NOTIFY is enabled? If e.g. I swap off and --stats still reports that memory is used, this means that NOTIFY is not enabled right? Dec 28, 2009 issue 50 (ramzswap not useful without userland) commented on by mwsealey - No but, I thought maybe like you can set "embedded system" options for squashfs to tweak its behaviour, maybe ramzswap could initialize a swap device without a userland tool at all, it used to be able to do this before rzscontrol (module arguments) but a default compressed swap device might be very useful. No but, I thought maybe like you can set "embedded system" options for squashfs to tweak its behaviour, maybe ramzswap could initialize a swap device without a userland tool at all, it used to be able to do this before rzscontrol (module arguments) but a default compressed swap device might be very useful. Dec 28, 2009 issue 50 (ramzswap not useful without userland) commented on by nitingupta910 - > Basically, I'm a little disappointed that we have to run a userland tool to bring up the ramzswap devices. getting rzscontrol to run on embedded system is painful? > Basically, I'm a little disappointed that we have to run a userland tool to bring up the ramzswap devices. getting rzscontrol to run on embedded system is painful? Dec 27, 2009 Revision 6990c42233 (Minor cleanups for git-diffc script.) pushed by nitingupta910 - Minor cleanups for git-diffc script. Minor cleanups for git-diffc script. Dec 27, 2009 2 new revisions pushed by nitingupta910 - 9bb596fdaf:script: comprehensive graphical git diff viewer dc83b01e45:merge 9bb596fdaf:script: comprehensive graphical git diff viewer dc83b01e45:merge Dec 26, 2009 issue 50 (ramzswap not useful without userland) reported by mwsealey - Basically, I'm a little disappointed that we have to run a userland tool to bring up the ramzswap devices. Would it be possible for ramzswap to initialize and use a standard, default ramzswap (for "embedded systems") in RAM (the same 25% of memory defaults would be fine) such that when you boot, you ALWAYS have some kind of compressed swap and at least one swap device initialized? Extra swap devices on backing storage could then be added on top using rzscontrol, or the on-boot one deconfigured and reconfigured. Basically, I'm a little disappointed that we have to run a userland tool to bring up the ramzswap devices. Would it be possible for ramzswap to initialize and use a standard, default ramzswap (for "embedded systems") in RAM (the same 25% of memory defaults would be fine) such that when you boot, you ALWAYS have some kind of compressed swap and at least one swap device initialized? Extra swap devices on backing storage could then be added on top using rzscontrol, or the on-boot one deconfigured and reconfigured. Dec 26, 2009 issue 48 (0.6 fails to build for 2.6.32_rc6) commented on by nitingupta910 - > Is the swap notify patch in 2.6.33-rc2? where can I download it? Is it safe to apply or is it alpha/beta? swap notify patch is not yet in mainline. Anyway, I will soon upload patch for upcoming 2.6.33 containing notify support and some bugfixes. > merry christmas to you and your family Nitin! Same for you and Happy new year! > Is the swap notify patch in 2.6.33-rc2? where can I download it? Is it safe to apply or is it alpha/beta? swap notify patch is not yet in mainline. Anyway, I will soon upload patch for upcoming 2.6.33 containing notify support and some bugfixes. > merry christmas to you and your family Nitin! Same for you and Happy new year! Dec 25, 2009 issue 48 (0.6 fails to build for 2.6.32_rc6) commented on by devsku - Is the swap notify patch in 2.6.33-rc2? where can I download it? Is it safe to apply or is it alpha/beta? merry christmas to you and your family Nitin! Is the swap notify patch in 2.6.33-rc2? where can I download it? Is it safe to apply or is it alpha/beta? merry christmas to you and your family Nitin! Dec 19, 2009 issue 48 (0.6 fails to build for 2.6.32_rc6) commented on by nitingupta910 - after modprobe, you need to do: rzscontrol /dev/ramzswap0 --init and then do swapon. you should use rzscontrol tool directly from repository instead of the one included with compcache-0.6. --- swap notify patch is almost ready for 2.6.33-rc1 and I will upload it soon. Please test with this notify support if possible. after modprobe, you need to do: rzscontrol /dev/ramzswap0 --init and then do swapon. you should use rzscontrol tool directly from repository instead of the one included with compcache-0.6. --- swap notify patch is almost ready for 2.6.33-rc1 and I will upload it soon. Please test with this notify support if possible. Dec 19, 2009 issue 48 (0.6 fails to build for 2.6.32_rc6) commented on by devsku - ahhh...ok. Now, another issue: I modeprobe'd the module in 2.6.33-rc1 kernel and tried to 'swapon /dev/ramzswap0' but it cribs saying that it can't find swap signature on it. Do you know if its fixed in a version greater than the one in 2.6.33-rc1? ahhh...ok. Now, another issue: I modeprobe'd the module in 2.6.33-rc1 kernel and tried to 'swapon /dev/ramzswap0' but it cribs saying that it can't find swap signature on it. Do you know if its fixed in a version greater than the one in 2.6.33-rc1? Dec 19, 2009 issue 48 (0.6 fails to build for 2.6.32_rc6) commented on by nitingupta910 - > Its not in 2.6.33-rc1. Or I did no find the config option? Device Drivers --> Staging drivers --> Compressed in-memory swap device (ramzswap) > Its not in 2.6.33-rc1. Or I did no find the config option? Device Drivers --> Staging drivers --> Compressed in-memory swap device (ramzswap) Dec 19, 2009 issue 48 (0.6 fails to build for 2.6.32_rc6) commented on by devsku - Its not in 2.6.33-rc1. Or I did no find the config option? Its not in 2.6.33-rc1. Or I did no find the config option? Dec 15, 2009 issue 49 (hg source doesn't compile) commented on by nitingupta910 - I mentioned/ "portable" in a casual sense -- i meant how to create makefile that runs on, at least slightly, different environments. However, it looks like there is zero diff here. On my system: $ make -v GNU Make 3.81 Copyright (C) 2006 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. This program built for x86_64-redhat-linux-gnu I mentioned/ "portable" in a casual sense -- i meant how to create makefile that runs on, at least slightly, different environments. However, it looks like there is zero diff here. On my system: $ make -v GNU Make 3.81 Copyright (C) 2006 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. This program built for x86_64-redhat-linux-gnu Dec 15, 2009 issue 49 (hg source doesn't compile) commented on by xmingske - I am using gmake on Linux, I suppose you do too. I don't understand what you mean by "portable", we are suppose to be on the same platform using the same make (GNU version), unless the version of gmake does matter. $ make -v GNU Make 3.81 Copyright (C) 2006 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. This program built for x86_64-pc-linux-gnu I am using gmake on Linux, I suppose you do too. I don't understand what you mean by "portable", we are suppose to be on the same platform using the same make (GNU version), unless the version of gmake does matter. $ make -v GNU Make 3.81 Copyright (C) 2006 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. This program built for x86_64-pc-linux-gnu Dec 15, 2009 issue 45 (Unable to handle kernel paging request) commented on by gmatht - I appear to still get the I/O errors. See attached dmesg.2. I appear to still get the I/O errors. See attached dmesg.2. Dec 15, 2009 issue 45 (Unable to handle kernel paging request) commented on by nitingupta910 - > presumably it only occurs if one operation happens immediately after the other. Looks like you are correct. I just committed a change that should prevent these I/O errors: now we sync disk and then set it to unintialized (we were doing the reverse earlier). Can you please test it again? I will try to get similar setup on my side too. The information you provided should be sufficient. Thanks. > presumably it only occurs if one operation happens immediately after the other. Looks like you are correct. I just committed a change that should prevent these I/O errors: now we sync disk and then set it to unintialized (we were doing the reverse earlier). Can you please test it again? I will try to get similar setup on my side too. The information you provided should be sufficient. Thanks. Dec 15, 2009 Revision 16e4131a28 (sync and then reset init flag) pushed by nitingupta910 - sync and then reset init flag sync and then reset init flag Dec 15, 2009 issue 45 (Unable to handle kernel paging request) commented on by gmatht - Not sure when it happens. I added logging to my script, but then it stopped happening, presumably it only occurs if one operation happens immediately after the other. Does this dump from dmesg help? [ 578.284149] ISO 9660 Extensions: RRIP_1991A [ 700.937862] ramzswap: Creating 1 devices ... [ 701.918331] Adding 2096440k swap on /dev/sdb1. Priority:-1 extents:1 across:2096440k [ 740.684180] ramzswap: Disk size set to 20059 kB [ 740.821390] Adding 20048k swap on /dev/ramzswap0. Priority:100 extents:1 across:20048k SS [ 740.888751] Adding 2096440k swap on /dev/sdb1. Priority:-1 extents:1 across:2096440k [ 762.255004] Buffer I/O error on device ramzswap0, logical block 0 [ 762.255036] Buffer I/O error on device ramzswap0, logical block 1 [ 762.255058] Buffer I/O error on device ramzswap0, logical block 2 [ 762.255079] Buffer I/O error on device ramzswap0, logical block 3 [ 762.255099] Buffer I/O error on device ramzswap0, logical block 4 [ 762.255119] Buffer I/O error on device ramzswap0, logical block 5 [ 762.255140] Buffer I/O error on device ramzswap0, logical block 6 [ 762.255161] Buffer I/O error on device ramzswap0, logical block 7 [ 762.255181] Buffer I/O error on device ramzswap0, logical block 8 [ 762.255201] Buffer I/O error on device ramzswap0, logical block 9 [ 770.219911] ramzswap: Disk size set to 52657 kB [ 770.341297] Adding 52648k swap on /dev/ramzswap0. Priority:100 extents:1 across:52648k SS [ 770.391994] Adding 2096440k swap on /dev/sdb1. Priority:-1 extents:1 across:2096440k I am not sure what you wanted from my setup, but I have attached some details on my VM (Ubuntu 9.10, uniprocessor, 2.6.31-14-generic). No swapfree patches have been applied. Not sure when it happens. I added logging to my script, but then it stopped happening, presumably it only occurs if one operation happens immediately after the other. Does this dump from dmesg help? [ 578.284149] ISO 9660 Extensions: RRIP_1991A [ 700.937862] ramzswap: Creating 1 devices ... [ 701.918331] Adding 2096440k swap on /dev/sdb1. Priority:-1 extents:1 across:2096440k [ 740.684180] ramzswap: Disk size set to 20059 kB [ 740.821390] Adding 20048k swap on /dev/ramzswap0. Priority:100 extents:1 across:20048k SS [ 740.888751] Adding 2096440k swap on /dev/sdb1. Priority:-1 extents:1 across:2096440k [ 762.255004] Buffer I/O error on device ramzswap0, logical block 0 [ 762.255036] Buffer I/O error on device ramzswap0, logical block 1 [ 762.255058] Buffer I/O error on device ramzswap0, logical block 2 [ 762.255079] Buffer I/O error on device ramzswap0, logical block 3 [ 762.255099] Buffer I/O error on device ramzswap0, logical block 4 [ 762.255119] Buffer I/O error on device ramzswap0, logical block 5 [ 762.255140] Buffer I/O error on device ramzswap0, logical block 6 [ 762.255161] Buffer I/O error on device ramzswap0, logical block 7 [ 762.255181] Buffer I/O error on device ramzswap0, logical block 8 [ 762.255201] Buffer I/O error on device ramzswap0, logical block 9 [ 770.219911] ramzswap: Disk size set to 52657 kB [ 770.341297] Adding 52648k swap on /dev/ramzswap0. Priority:100 extents:1 across:52648k SS [ 770.391994] Adding 2096440k swap on /dev/sdb1. Priority:-1 extents:1 across:2096440k I am not sure what you wanted from my setup, but I have attached some details on my VM (Ubuntu 9.10, uniprocessor, 2.6.31-14-generic). No swapfree patches have been applied. Dec 14, 2009 issue 45 (Unable to handle kernel paging request) commented on by nitingupta910 - yes, the fix is in hg. > I still get Buffer I/O errors on logical blocks 0-9 do you get these specifically during swapon only? These should be harmless and think they are due to disk readahead stuff. I never get these errors on my side so can you please provide details about your setup? yes, the fix is in hg. > I still get Buffer I/O errors on logical blocks 0-9 do you get these specifically during swapon only? These should be harmless and think they are due to disk readahead stuff. I never get these errors on my side so can you please provide details about your setup? Dec 14, 2009 issue 45 (Unable to handle kernel paging request) commented on by gmatht - I guess you mean the fix in hg? I still get Buffer I/O errors on logical blocks 0-9, but I am not getting any kernel bugs. I guess you mean the fix in hg? I still get Buffer I/O errors on logical blocks 0-9, but I am not getting any kernel bugs. Dec 14, 2009 issue 45 (Unable to handle kernel paging request) commented on by nitingupta910 - Can you please test this fix. Can you please test this fix. Dec 12, 2009 issue 41 (Swapoff ramzswap0 hard freezes system if utilized) Status changed by nitingupta910 - Status: Done Status: Done Dec 12, 2009 issue 41 (Swapoff ramzswap0 hard freezes system if utilized) commented on by nitingupta910 - Thanks for the update. > I think my problem may have had something to do with the swap_free_notify patch, as I've since recompiled my kernel without it and compiled Compcache from rev. 1ea72b25ba (latest revision last I pulled it), and after several stress tests I've been unable to reproduce the problem. Ok, then I will close it for now. Thanks for the update. > I think my problem may have had something to do with the swap_free_notify patch, as I've since recompiled my kernel without it and compiled Compcache from rev. 1ea72b25ba (latest revision last I pulled it), and after several stress tests I've been unable to reproduce the problem. Ok, then I will close it for now. Dec 12, 2009 issue 41 (Swapoff ramzswap0 hard freezes system if utilized) commented on by cjhard - I found it in Ubuntu's universe repo. It's described as "a tool that imposes a configurable amount of CPU, memory, I/O, or disk stress on a POSIX-compliant operating system and reports any errors it detects." I think my problem may have had something to do with the swap_free_notify patch, as I've since recompiled my kernel without it and compiled Compcache from rev. 1ea72b25ba (latest revision last I pulled it), and after several stress tests I've been unable to reproduce the problem. I found it in Ubuntu's universe repo. It's described as "a tool that imposes a configurable amount of CPU, memory, I/O, or disk stress on a POSIX-compliant operating system and reports any errors it detects." I think my problem may have had something to do with the swap_free_notify patch, as I've since recompiled my kernel without it and compiled Compcache from rev. 1ea72b25ba (latest revision last I pulled it), and after several stress tests I've been unable to reproduce the problem. Dec 11, 2009 issue 41 (Swapoff ramzswap0 hard freezes system if utilized) commented on by nitingupta910 - how can I get that 'stress' utility you mentioned in step 2: > 2. stress --vm 7 --vm-keep how can I get that 'stress' utility you mentioned in step 2: > 2. stress --vm 7 --vm-keep Dec 11, 2009 issue 38 (Backing swap with swap file does not work on G1 running Andr...) commented on by nitingupta910 - ccyrowski, MikeTaylor00: is it working for you now? ccyrowski, MikeTaylor00: is it working for you now? Dec 11, 2009 issue 49 (hg source doesn't compile) commented on by nitingupta910 - /me wonders whats a more portable way to create such Makefile. /me wonders whats a more portable way to create such Makefile. Dec 11, 2009 issue 48 (0.6 fails to build for 2.6.32_rc6) Status changed by nitingupta910 - Its now in mainline which should compile correctly. Status: Fixed Its now in mainline which should compile correctly. Status: Fixed Dec 10, 2009 StatsExplained Wiki page commented on by nitingupta910 - > Is compressing pages before sent to disk currently on the roadmap of compcache? Yes, it is on the roadmap. It is especially useful for SSD disks. > Is compressing pages before sent to disk currently on the roadmap of compcache? Yes, it is on the roadmap. It is especially useful for SSD disks. Dec 10, 2009 StatsExplained Wiki page commented on by grooby - nitingupta - Thank you. Is compressing pages before sent to disk currently on the roadmap of compcache? nitingupta - Thank you. Is compressing pages before sent to disk currently on the roadmap of compcache? Dec 09, 2009 StatsExplained Wiki page commented on by nitingupta910 - > compcache w/out backswapping: Initialize Compcache of 24MB means that kernel is aware of a 24MB "swap space". This space is now allocated yet in memory (right?!?!) As kernel writes to this swap space, the data is compressed, memory allocated, and store in the allocated memory. Yes, right. Memory is allocated/freed as required. Its not pre-allocated. > compcache w/ backswapping: Initialize compcache of 24MB + a 96MB swap partition means that kernel is aware of a 96MB "swap space". as kernel writes to this "Swap space", data is compressed and stored in memory first. If there is an overflow, the data is written to the disk. Is disk acting like a secondary caching where the lower hit rate pages are moved to the disk? also are data in the disk compressed too? With memlimit of 24M and backing swap size of 96M, kernel sees 96M "swap space". When we need to store more than 24M, we start sending any new pages to the backing swap -- there is no criteria to decide which page should go to disk. All pages that arrive after memlimit is reached are forwarded to the backing swap. Pages are not compressed when sent to disk. > compcache w/out backswapping: Initialize Compcache of 24MB means that kernel is aware of a 24MB "swap space". This space is now allocated yet in memory (right?!?!) As kernel writes to this swap space, the data is compressed, memory allocated, and store in the allocated memory. Yes, right. Memory is allocated/freed as required. Its not pre-allocated. > compcache w/ backswapping: Initialize compcache of 24MB + a 96MB swap partition means that kernel is aware of a 96MB "swap space". as kernel writes to this "Swap space", data is compressed and stored in memory first. If there is an overflow, the data is written to the disk. Is disk acting like a secondary caching where the lower hit rate pages are moved to the disk? also are data in the disk compressed too? With memlimit of 24M and backing swap size of 96M, kernel sees 96M "swap space". When we need to store more than 24M, we start sending any new pages to the backing swap -- there is no criteria to decide which page should go to disk. All pages that arrive after memlimit is reached are forwarded to the backing swap. Pages are not compressed when sent to disk. Dec 09, 2009 issue 45 (Unable to handle kernel paging request) commented on by nitingupta910 - This bug is happening due to a race between read and reset device. When reset is issued, we can still have some in-flight I/Os. However, we are not making sure that the device is quiesced before freeing all the data structures. This is causing use-after-free bug for I/Os that arrive while we are freeing data in parallel. A fix for this has been checked-in but it still needs to be refined a bit. Some more testing is needed too! ==================== Debugging this issue is a bit involved so I'm dumping all the details. This should help us debug similar problems in future. Some steps could be skipped but I will go extra verbose so others can also do this debugging. --- Compiling the module with optimizations disabled (-O0) reveals location of BUG: Dec 9 18:27:07 localhost kernel: BUG: unable to handle kernel NULL pointer dereference at 00000217 Dec 9 18:27:07 localhost kernel: IP: [<d0dc32d0>] rzs_test_flag+0x24/0x44 [ramzswap] <--------- Dec 9 18:27:07 localhost kernel: pdpt = 000000000e472001 pde = 000000000e47f067 pte = 0000000000000000 Dec 9 18:27:07 localhost kernel: Oops: 0000 [#1] SMP Dec 9 18:27:07 localhost kernel: last sysfs file: /sys/devices/virtual/block/ramzswap0/removable Dec 9 18:27:07 localhost kernel: Modules linked in: ramzswap lzo_decompress lzo_compress sit tunnel4 sunrpc ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 dm_multipath vboxvfs snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device ppdev snd_pcm parport_pc snd_timer snd parport pcnet32 soundcore snd_page_alloc i2c_piix4 i2c_core vboxadd mii [last unloaded: ramzswap] Dec 9 18:27:07 localhost kernel: Dec 9 18:27:07 localhost kernel: Pid: 2649, comm: blkid Not tainted (2.6.31.6-162.fc12.i686.PAE #1) VirtualBox Dec 9 18:27:07 localhost kernel: EIP: 0060:[<d0dc32d0>] EFLAGS: 00010202 CPU: 0 Dec 9 18:27:07 localhost kernel: EIP is at rzs_test_flag+0x24/0x44 [ramzswap] Dec 9 18:27:07 localhost kernel: EAX: 00000210 EBX: ce420780 ECX: 00000001 EDX: 00000210 Dec 9 18:27:07 localhost kernel: ESI: cf880e70 EDI: 000000ff EBP: ce51bc60 ESP: ce51bc4c Dec 9 18:27:07 localhost kernel: DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 Dec 9 18:27:07 localhost kernel: Process blkid (pid: 2649, ti=ce51a000 task=ce430cc0 task.ti=ce51a000) Dec 9 18:27:07 localhost kernel: Stack: Dec 9 18:27:07 localhost kernel: 00000001 00000042 cd4c5600 ce420780 cf880e70 ce51bca4 d0dc5080 ce51bc84 Dec 9 18:27:07 localhost kernel: <0> c0588dbe ce51bc88 ce420780 cd4c5600 cf58e898 00001000 00000210 c058f74f Dec 9 18:27:07 localhost kernel: <0> 00000000 c11cb2e0 cd4c5600 00000042 cf880e70 ce420780 ce51bcbc d0dc588c Dec 9 18:27:07 localhost kernel: Call Trace: Dec 9 18:27:07 localhost kernel: [<d0dc5080>] ? ramzswap_read+0x57/0x1cd [ramzswap] Dec 9 18:27:07 localhost kernel: [<c0588dbe>] ? __elv_add_request+0x92/0x99 Dec 9 18:27:07 localhost kernel: [<c058f74f>] ? __make_request+0x2d2/0x345 Dec 9 18:27:07 localhost kernel: [<d0dc588c>] ? ramzswap_make_request+0xa4/0xbc [ramzswap] Dec 9 18:27:07 localhost kernel: [<c058d664>] ? generic_make_request+0x214/0x261 Dec 9 18:27:07 localhost kernel: [<c049d379>] ? mempool_alloc_slab+0x13/0x15 Dec 9 18:27:07 localhost kernel: [<c049d586>] ? mempool_alloc+0x50/0xe9 <snip> cat /proc/modules \| grep ramzswap ramzswap 28644 2 - Live 0xd0dc3000 <---- ramzswap module base address lzo_decompress 2248 1 ramzswap, Live 0xd0d11000 lzo_compress 1944 1 ramzswap, Live 0xd0d0b000 disassemble: objdump -S -D --adjust-vma=0xd0dc3000 ramzswap.ko > ramzswap.S BUG was caused at EIP 0xd0dc32d0 (from BUG report). Dump nearby code area: static int rzs_test_flag(struct ramzswap rzs, u32 index, enum rzs_pageflags flag) { d0dc32ac: 55 push %ebp d0dc32ad: 89 e5 mov %esp,%ebp d0dc32af: 56 push %esi d0dc32b0: 53 push %ebx d0dc32b1: 83 ec 0c sub $0xc,%esp d0dc32b4: e8 fc ff ff ff call d0dc32b5 <rzs_test_flag+0x9> d0dc32b9: 89 45 f4 mov %eax,-0xc(%ebp) <--- arg1: rzs d0dc32bc: 89 55 f0 mov %edx,-0x10(%ebp) <--- arg2: index d0dc32bf: 89 4d ec mov %ecx,-0x14(%ebp) <--- arg3: flag return rzs->table[index].flags & BIT(flag); d0dc32c2: 8b 45 f4 mov -0xc(%ebp),%eax // EAX = rzs d0dc32c5: 8b 40 0c mov 0xc(%eax),%eax // EAX = rzs->table ('table' is at 12th offset of struct ramzswap) d0dc32c8: 8b 55 f0 mov -0x10(%ebp),%edx // EDX = index = 0x42 (decoded below) d0dc32cb: c1 e2 03 shl $0x3,%edx // EDX = index * (1 << 3) (each table entry has size of 8b) = 0x42 * 8 = 0x210 d0dc32ce: 01 d0 add %edx,%eax // EAX = EDX + EAX (rzs->table + index8 = 0x210) d0dc32d0: 0f b6 40 07 movzbl 0x7(%eax),%eax // EAX = (EAX + 0x7) ('flags' is at 7th offset of struct table) ^ BUG here! = (rzs->table + index8 + 0x7) = (rzs->table + 0x210) d0dc32d4: 0f b6 d0 movzbl %al,%edx d0dc32d7: 8b 45 ec mov -0x14(%ebp),%eax Getting values for ramzswap_test_flag() args: EBP=ce51bc60, ESP=0xce51bc4c (from BUG report) -0x14(%ebp) == 0xce420780 - 0x14 == 0xce51bc4c = ESP So, Stack dump in the BUG report gives us the args (in reverse order): flag = 0x1 (RZS_ZERO) index = 0x42 rzs = 0xcd4c5600 From disasm above, we can see that rzs->table must be 0 to make final address in EAX (rzs->table + index8 + 0x7) equal to 0x217 which the BUG report shows. This confirms our suspicion that its a race between read and reset device (which frees this table). --- The solution is to: - reset init_done to 0 before freeing all the data structures. - wait for all pending I/O on this ramzswap device to finish. - After this wait, if any further I/O is queued, the rzs device will simply reject it since init_done check will now fail in ramzswap_make_request() -- till the next init. This bug is happening due to a race between read and reset device. When reset is issued, we can still have some in-flight I/Os. However, we are not making sure that the device is quiesced before freeing all the data structures. This is causing use-after-free bug for I/Os that arrive while we are freeing data in parallel. A fix for this has been checked-in but it still needs to be refined a bit. Some more testing is needed too! ==================== Debugging this issue is a bit involved so I'm dumping all the details. This should help us debug similar problems in future. Some steps could be skipped but I will go extra verbose so others can also do this debugging. --- Compiling the module with optimizations disabled (-O0) reveals location of BUG: Dec 9 18:27:07 localhost kernel: BUG: unable to handle kernel NULL pointer dereference at 00000217 Dec 9 18:27:07 localhost kernel: IP: [<d0dc32d0>] rzs_test_flag+0x24/0x44 [ramzswap] <--------- Dec 9 18:27:07 localhost kernel: pdpt = 000000000e472001 pde = 000000000e47f067 pte = 0000000000000000 Dec 9 18:27:07 localhost kernel: Oops: 0000 [#1] SMP Dec 9 18:27:07 localhost kernel: last sysfs file: /sys/devices/virtual/block/ramzswap0/removable Dec 9 18:27:07 localhost kernel: Modules linked in: ramzswap lzo_decompress lzo_compress sit tunnel4 sunrpc ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 dm_multipath vboxvfs snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device ppdev snd_pcm parport_pc snd_timer snd parport pcnet32 soundcore snd_page_alloc i2c_piix4 i2c_core vboxadd mii [last unloaded: ramzswap] Dec 9 18:27:07 localhost kernel: Dec 9 18:27:07 localhost kernel: Pid: 2649, comm: blkid Not tainted (2.6.31.6-162.fc12.i686.PAE #1) VirtualBox Dec 9 18:27:07 localhost kernel: EIP: 0060:[<d0dc32d0>] EFLAGS: 00010202 CPU: 0 Dec 9 18:27:07 localhost kernel: EIP is at rzs_test_flag+0x24/0x44 [ramzswap] Dec 9 18:27:07 localhost kernel: EAX: 00000210 EBX: ce420780 ECX: 00000001 EDX: 00000210 Dec 9 18:27:07 localhost kernel: ESI: cf880e70 EDI: 000000ff EBP: ce51bc60 ESP: ce51bc4c Dec 9 18:27:07 localhost kernel: DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 Dec 9 18:27:07 localhost kernel: Process blkid (pid: 2649, ti=ce51a000 task=ce430cc0 task.ti=ce51a000) Dec 9 18:27:07 localhost kernel: Stack: Dec 9 18:27:07 localhost kernel: 00000001 00000042 cd4c5600 ce420780 cf880e70 ce51bca4 d0dc5080 ce51bc84 Dec 9 18:27:07 localhost kernel: <0> c0588dbe ce51bc88 ce420780 cd4c5600 cf58e898 00001000 00000210 c058f74f Dec 9 18:27:07 localhost kernel: <0> 00000000 c11cb2e0 cd4c5600 00000042 cf880e70 ce420780 ce51bcbc d0dc588c Dec 9 18:27:07 localhost kernel: Call Trace: Dec 9 18:27:07 localhost kernel: [<d0dc5080>] ? ramzswap_read+0x57/0x1cd [ramzswap] Dec 9 18:27:07 localhost kernel: [<c0588dbe>] ? __elv_add_request+0x92/0x99 Dec 9 18:27:07 localhost kernel: [<c058f74f>] ? __make_request+0x2d2/0x345 Dec 9 18:27:07 localhost kernel: [<d0dc588c>] ? ramzswap_make_request+0xa4/0xbc [ramzswap] Dec 9 18:27:07 localhost kernel: [<c058d664>] ? generic_make_request+0x214/0x261 Dec 9 18:27:07 localhost kernel: [<c049d379>] ? mempool_alloc_slab+0x13/0x15 Dec 9 18:27:07 localhost kernel: [<c049d586>] ? mempool_alloc+0x50/0xe9 <snip> cat /proc/modules \| grep ramzswap ramzswap 28644 2 - Live 0xd0dc3000 <---- ramzswap module base address lzo_decompress 2248 1 ramzswap, Live 0xd0d11000 lzo_compress 1944 1 ramzswap, Live 0xd0d0b000 disassemble: objdump -S -D --adjust-vma=0xd0dc3000 ramzswap.ko > ramzswap.S BUG was caused at EIP 0xd0dc32d0 (from BUG report). Dump nearby code area: static int rzs_test_flag(struct ramzswap rzs, u32 index, enum rzs_pageflags flag) { d0dc32ac: 55 push %ebp d0dc32ad: 89 e5 mov %esp,%ebp d0dc32af: 56 push %esi d0dc32b0: 53 push %ebx d0dc32b1: 83 ec 0c sub $0xc,%esp d0dc32b4: e8 fc ff ff ff call d0dc32b5 <rzs_test_flag+0x9> d0dc32b9: 89 45 f4 mov %eax,-0xc(%ebp) <--- arg1: rzs d0dc32bc: 89 55 f0 mov %edx,-0x10(%ebp) <--- arg2: index d0dc32bf: 89 4d ec mov %ecx,-0x14(%ebp) <--- arg3: flag return rzs->table[index].flags & BIT(flag); d0dc32c2: 8b 45 f4 mov -0xc(%ebp),%eax // EAX = rzs d0dc32c5: 8b 40 0c mov 0xc(%eax),%eax // EAX = rzs->table ('table' is at 12th offset of struct ramzswap) d0dc32c8: 8b 55 f0 mov -0x10(%ebp),%edx // EDX = index = 0x42 (decoded below) d0dc32cb: c1 e2 03 shl $0x3,%edx // EDX = index * (1 << 3) (each table entry has size of 8b) = 0x42 * 8 = 0x210 d0dc32ce: 01 d0 add %edx,%eax // EAX = EDX + EAX (rzs->table + index8 = 0x210) d0dc32d0: 0f b6 40 07 movzbl 0x7(%eax),%eax // EAX = (EAX + 0x7) ('flags' is at 7th offset of struct table) ^ BUG here! = (rzs->table + index8 + 0x7) = (rzs->table + 0x210) d0dc32d4: 0f b6 d0 movzbl %al,%edx d0dc32d7: 8b 45 ec mov -0x14(%ebp),%eax Getting values for ramzswap_test_flag() args: EBP=ce51bc60, ESP=0xce51bc4c (from BUG report) -0x14(%ebp) == 0xce420780 - 0x14 == 0xce51bc4c = ESP So, Stack dump in the BUG report gives us the args (in reverse order): flag = 0x1 (RZS_ZERO) index = 0x42 rzs = 0xcd4c5600 From disasm above, we can see that rzs->table must be 0 to make final address in EAX (rzs->table + index8 + 0x7) equal to 0x217 which the BUG report shows. This confirms our suspicion that its a race between read and reset device (which frees this table). --- The solution is to: - reset init_done to 0 before freeing all the data structures. - wait for all pending I/O on this ramzswap device to finish. - After this wait, if any further I/O is queued, the rzs device will simply reject it since init_done check will now fail in ramzswap_make_request() -- till the next init. Older Dec 09, 2009 StatsExplained Wiki page commented on by grooby - just trying to confirm what I read here to make sure I understand how Compcache and backswap works. (there are also questions in there too.) compcache w/out backswapping: Initialize Compcache of 24MB means that kernel is aware of a 24MB "swap space". This space is now allocated yet in memory (right?!?!) As kernel writes to this swap space, the data is compressed, memory allocated, and store in the allocated memory. compcache w/ backswapping: Initialize compcache of 24MB + a 96MB swap partition means that kernel is aware of a 96MB "swap space". as kernel writes to this "Swap space", data is compressed and stored in memory first. If there is an overflow, the data is written to the disk. Is disk acting like a secondary caching where the lower hit rate pages are moved to the disk? also are data in the disk compressed too? just trying to confirm what I read here to make sure I understand how Compcache and backswap works. (there are also questions in there too.) compcache w/out backswapping: Initialize Compcache of 24MB means that kernel is aware of a 24MB "swap space". This space is now allocated yet in memory (right?!?!) As kernel writes to this swap space, the data is compressed, memory allocated, and store in the allocated memory. compcache w/ backswapping: Initialize compcache of 24MB + a 96MB swap partition means that kernel is aware of a 96MB "swap space". as kernel writes to this "Swap space", data is compressed and stored in memory first. If there is an overflow, the data is written to the disk. Is disk acting like a secondary caching where the lower hit rate pages are moved to the disk? also are data in the disk compressed too? Dec 09, 2009 Revision 6a70aa1093 (Fix race between reset_device and ramzswap_make_request) pushed by nitingupta910 - Fix race between reset_device and ramzswap_make_request Fix race between reset_device and ramzswap_make_request Dec 04, 2009 issue 49 (hg source doesn't compile) commented on by xmingske - ok I see that now. My problem seem to be with $(PWD), after make -C $(KERNEL_BUILD_PATH), the $(PWD) is obviously the kernel dir and not compcache dir. It seems that my make is not expanding $(PWD) before (see -I/usr/src/linux-2.6.32/sub-projects/allocators/xvmalloc-kmod below). I have to hardcode it to get compcache to compile. make -f /usr/src/linux-2.6.32/scripts/Makefile.build obj=/tmp/compcache gcc -Wp,-MD,/tmp/compcache/.ramzswap_drv.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/include -Iinclude -Iinclude2 -I/usr/src/linux-2.6.32/include -I/usr/src/linux-2.6.32/arch/x86/include -include include/linux/autoconf.h -I/tmp/compcache -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -fno-delete-null-pointer-checks -Os -m64 -march=k8 -mno-red-zone -mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -fno-stack-protector -fomit-frame-pointer -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -DCONFIG_RAMZSWAP_STATS -I/usr/src/linux-2.6.32/sub-projects/allocators/xvmalloc-kmod -g -Wall -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ramzswap_drv)" -D"KBUILD_MODNAME=KBUILD_STR(ramzswap)" -c -o /tmp/compcache/ramzswap_drv.o /tmp/compcache/ramzswap_drv.c ok I see that now. My problem seem to be with $(PWD), after make -C $(KERNEL_BUILD_PATH), the $(PWD) is obviously the kernel dir and not compcache dir. It seems that my make is not expanding $(PWD) before (see -I/usr/src/linux-2.6.32/sub-projects/allocators/xvmalloc-kmod below). I have to hardcode it to get compcache to compile. make -f /usr/src/linux-2.6.32/scripts/Makefile.build obj=/tmp/compcache gcc -Wp,-MD,/tmp/compcache/.ramzswap_drv.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/include -Iinclude -Iinclude2 -I/usr/src/linux-2.6.32/include -I/usr/src/linux-2.6.32/arch/x86/include -include include/linux/autoconf.h -I/tmp/compcache -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -fno-delete-null-pointer-checks -Os -m64 -march=k8 -mno-red-zone -mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -fno-stack-protector -fomit-frame-pointer -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -DCONFIG_RAMZSWAP_STATS -I/usr/src/linux-2.6.32/sub-projects/allocators/xvmalloc-kmod -g -Wall -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ramzswap_drv)" -D"KBUILD_MODNAME=KBUILD_STR(ramzswap)" -c -o /tmp/compcache/ramzswap_drv.o /tmp/compcache/ramzswap_drv.c

issue 45 (Unable to handle kernel paging request) commented on by nitingupta910 - This bug is happening due to a race between read and reset device. When reset is issued, we can still have some in-flight I/Os. However, we are not making sure that the device is quiesced before freeing all the data structures. This is causing use-after-free bug for I/Os that arrive while we are freeing data in parallel. A fix for this has been checked-in but it still needs to be refined a bit. Some more testing is needed too! ==================== Debugging this issue is a bit involved so I'm dumping all the details. This should help us debug similar problems in future. Some steps could be skipped but I will go extra verbose so others can also do this debugging. --- Compiling the module with optimizations disabled (-O0) reveals location of BUG: Dec 9 18:27:07 localhost kernel: BUG: unable to handle kernel NULL pointer dereference at 00000217 Dec 9 18:27:07 localhost kernel: IP: [<d0dc32d0>] rzs_test_flag+0x24/0x44 [ramzswap] <--------- Dec 9 18:27:07 localhost kernel: *pdpt = 000000000e472001 *pde = 000000000e47f067 *pte = 0000000000000000 Dec 9 18:27:07 localhost kernel: Oops: 0000 [#1] SMP Dec 9 18:27:07 localhost kernel: last sysfs file: /sys/devices/virtual/block/ramzswap0/removable Dec 9 18:27:07 localhost kernel: Modules linked in: ramzswap lzo_decompress lzo_compress sit tunnel4 sunrpc ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 dm_multipath vboxvfs snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device ppdev snd_pcm parport_pc snd_timer snd parport pcnet32 soundcore snd_page_alloc i2c_piix4 i2c_core vboxadd mii [last unloaded: ramzswap] Dec 9 18:27:07 localhost kernel: Dec 9 18:27:07 localhost kernel: Pid: 2649, comm: blkid Not tainted (2.6.31.6-162.fc12.i686.PAE #1) VirtualBox Dec 9 18:27:07 localhost kernel: EIP: 0060:[<d0dc32d0>] EFLAGS: 00010202 CPU: 0 Dec 9 18:27:07 localhost kernel: EIP is at rzs_test_flag+0x24/0x44 [ramzswap] Dec 9 18:27:07 localhost kernel: EAX: 00000210 EBX: ce420780 ECX: 00000001 EDX: 00000210 Dec 9 18:27:07 localhost kernel: ESI: cf880e70 EDI: 000000ff EBP: ce51bc60 ESP: ce51bc4c Dec 9 18:27:07 localhost kernel: DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 Dec 9 18:27:07 localhost kernel: Process blkid (pid: 2649, ti=ce51a000 task=ce430cc0 task.ti=ce51a000) Dec 9 18:27:07 localhost kernel: Stack: Dec 9 18:27:07 localhost kernel: 00000001 00000042 cd4c5600 ce420780 cf880e70 ce51bca4 d0dc5080 ce51bc84 Dec 9 18:27:07 localhost kernel: <0> c0588dbe ce51bc88 ce420780 cd4c5600 cf58e898 00001000 00000210 c058f74f Dec 9 18:27:07 localhost kernel: <0> 00000000 c11cb2e0 cd4c5600 00000042 cf880e70 ce420780 ce51bcbc d0dc588c Dec 9 18:27:07 localhost kernel: Call Trace: Dec 9 18:27:07 localhost kernel: [<d0dc5080>] ? ramzswap_read+0x57/0x1cd [ramzswap] Dec 9 18:27:07 localhost kernel: [<c0588dbe>] ? __elv_add_request+0x92/0x99 Dec 9 18:27:07 localhost kernel: [<c058f74f>] ? __make_request+0x2d2/0x345 Dec 9 18:27:07 localhost kernel: [<d0dc588c>] ? ramzswap_make_request+0xa4/0xbc [ramzswap] Dec 9 18:27:07 localhost kernel: [<c058d664>] ? generic_make_request+0x214/0x261 Dec 9 18:27:07 localhost kernel: [<c049d379>] ? mempool_alloc_slab+0x13/0x15 Dec 9 18:27:07 localhost kernel: [<c049d586>] ? mempool_alloc+0x50/0xe9 <snip> cat /proc/modules | grep ramzswap ramzswap 28644 2 - Live 0xd0dc3000 <---- ramzswap module base address lzo_decompress 2248 1 ramzswap, Live 0xd0d11000 lzo_compress 1944 1 ramzswap, Live 0xd0d0b000 disassemble: objdump -S -D --adjust-vma=0xd0dc3000 ramzswap.ko > ramzswap.S BUG was caused at EIP 0xd0dc32d0 (from BUG report). Dump nearby code area: static int rzs_test_flag(struct ramzswap *rzs, u32 index, enum rzs_pageflags flag) { d0dc32ac: 55 push %ebp d0dc32ad: 89 e5 mov %esp,%ebp d0dc32af: 56 push %esi d0dc32b0: 53 push %ebx d0dc32b1: 83 ec 0c sub $0xc,%esp d0dc32b4: e8 fc ff ff ff call d0dc32b5 <rzs_test_flag+0x9> d0dc32b9: 89 45 f4 mov %eax,-0xc(%ebp) <--- arg1: rzs d0dc32bc: 89 55 f0 mov %edx,-0x10(%ebp) <--- arg2: index d0dc32bf: 89 4d ec mov %ecx,-0x14(%ebp) <--- arg3: flag return rzs->table[index].flags & BIT(flag); d0dc32c2: 8b 45 f4 mov -0xc(%ebp),%eax // EAX = rzs d0dc32c5: 8b 40 0c mov 0xc(%eax),%eax // EAX = rzs->table ('table' is at 12th offset of struct ramzswap) d0dc32c8: 8b 55 f0 mov -0x10(%ebp),%edx // EDX = index = 0x42 (decoded below) d0dc32cb: c1 e2 03 shl $0x3,%edx // EDX = index * (1 << 3) (each table entry has size of 8b) = 0x42 * 8 = 0x210 d0dc32ce: 01 d0 add %edx,%eax // EAX = EDX + EAX (rzs->table + index*8 = 0x210) d0dc32d0: 0f b6 40 07 movzbl 0x7(%eax),%eax // EAX = *(EAX + 0x7) ('flags' is at 7th offset of struct table) ^ BUG here! = *(rzs->table + index*8 + 0x7) = *(rzs->table + 0x210) d0dc32d4: 0f b6 d0 movzbl %al,%edx d0dc32d7: 8b 45 ec mov -0x14(%ebp),%eax Getting values for ramzswap_test_flag() args: EBP=ce51bc60, ESP=0xce51bc4c (from BUG report) -0x14(%ebp) == 0xce420780 - 0x14 == 0xce51bc4c = ESP So, Stack dump in the BUG report gives us the args (in reverse order): flag = 0x1 (RZS_ZERO) index = 0x42 rzs = 0xcd4c5600 From disasm above, we can see that rzs->table must be 0 to make final address in EAX (rzs->table + index*8 + 0x7) equal to 0x217 which the BUG report shows. This confirms our suspicion that its a race between read and reset device (which frees this table). --- The solution is to: - reset init_done to 0 *before* freeing all the data structures. - wait for all pending I/O on this ramzswap device to finish. - After this wait, if any further I/O is queued, the rzs device will simply reject it since init_done check will now fail in ramzswap_make_request() -- till the next init.

This bug is happening due to a race between read and reset device. When reset is issued, we can still have some in-flight I/Os. However, we are not making sure that the device is quiesced before freeing all the data structures. This is causing use-after-free bug for I/Os that arrive while we are freeing data in parallel. A fix for this has been checked-in but it still needs to be refined a bit. Some more testing is needed too! ==================== Debugging this issue is a bit involved so I'm dumping all the details. This should help us debug similar problems in future. Some steps could be skipped but I will go extra verbose so others can also do this debugging. --- Compiling the module with optimizations disabled (-O0) reveals location of BUG: Dec 9 18:27:07 localhost kernel: BUG: unable to handle kernel NULL pointer dereference at 00000217 Dec 9 18:27:07 localhost kernel: IP: [<d0dc32d0>] rzs_test_flag+0x24/0x44 [ramzswap] <--------- Dec 9 18:27:07 localhost kernel: *pdpt = 000000000e472001 *pde = 000000000e47f067 *pte = 0000000000000000 Dec 9 18:27:07 localhost kernel: Oops: 0000 [#1] SMP Dec 9 18:27:07 localhost kernel: last sysfs file: /sys/devices/virtual/block/ramzswap0/removable Dec 9 18:27:07 localhost kernel: Modules linked in: ramzswap lzo_decompress lzo_compress sit tunnel4 sunrpc ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 dm_multipath vboxvfs snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device ppdev snd_pcm parport_pc snd_timer snd parport pcnet32 soundcore snd_page_alloc i2c_piix4 i2c_core vboxadd mii [last unloaded: ramzswap] Dec 9 18:27:07 localhost kernel: Dec 9 18:27:07 localhost kernel: Pid: 2649, comm: blkid Not tainted (2.6.31.6-162.fc12.i686.PAE #1) VirtualBox Dec 9 18:27:07 localhost kernel: EIP: 0060:[<d0dc32d0>] EFLAGS: 00010202 CPU: 0 Dec 9 18:27:07 localhost kernel: EIP is at rzs_test_flag+0x24/0x44 [ramzswap] Dec 9 18:27:07 localhost kernel: EAX: 00000210 EBX: ce420780 ECX: 00000001 EDX: 00000210 Dec 9 18:27:07 localhost kernel: ESI: cf880e70 EDI: 000000ff EBP: ce51bc60 ESP: ce51bc4c Dec 9 18:27:07 localhost kernel: DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 Dec 9 18:27:07 localhost kernel: Process blkid (pid: 2649, ti=ce51a000 task=ce430cc0 task.ti=ce51a000) Dec 9 18:27:07 localhost kernel: Stack: Dec 9 18:27:07 localhost kernel: 00000001 00000042 cd4c5600 ce420780 cf880e70 ce51bca4 d0dc5080 ce51bc84 Dec 9 18:27:07 localhost kernel: <0> c0588dbe ce51bc88 ce420780 cd4c5600 cf58e898 00001000 00000210 c058f74f Dec 9 18:27:07 localhost kernel: <0> 00000000 c11cb2e0 cd4c5600 00000042 cf880e70 ce420780 ce51bcbc d0dc588c Dec 9 18:27:07 localhost kernel: Call Trace: Dec 9 18:27:07 localhost kernel: [<d0dc5080>] ? ramzswap_read+0x57/0x1cd [ramzswap] Dec 9 18:27:07 localhost kernel: [<c0588dbe>] ? __elv_add_request+0x92/0x99 Dec 9 18:27:07 localhost kernel: [<c058f74f>] ? __make_request+0x2d2/0x345 Dec 9 18:27:07 localhost kernel: [<d0dc588c>] ? ramzswap_make_request+0xa4/0xbc [ramzswap] Dec 9 18:27:07 localhost kernel: [<c058d664>] ? generic_make_request+0x214/0x261 Dec 9 18:27:07 localhost kernel: [<c049d379>] ? mempool_alloc_slab+0x13/0x15 Dec 9 18:27:07 localhost kernel: [<c049d586>] ? mempool_alloc+0x50/0xe9 <snip> cat /proc/modules | grep ramzswap ramzswap 28644 2 - Live 0xd0dc3000 <---- ramzswap module base address lzo_decompress 2248 1 ramzswap, Live 0xd0d11000 lzo_compress 1944 1 ramzswap, Live 0xd0d0b000 disassemble: objdump -S -D --adjust-vma=0xd0dc3000 ramzswap.ko > ramzswap.S BUG was caused at EIP 0xd0dc32d0 (from BUG report). Dump nearby code area: static int rzs_test_flag(struct ramzswap *rzs, u32 index, enum rzs_pageflags flag) { d0dc32ac: 55 push %ebp d0dc32ad: 89 e5 mov %esp,%ebp d0dc32af: 56 push %esi d0dc32b0: 53 push %ebx d0dc32b1: 83 ec 0c sub $0xc,%esp d0dc32b4: e8 fc ff ff ff call d0dc32b5 <rzs_test_flag+0x9> d0dc32b9: 89 45 f4 mov %eax,-0xc(%ebp) <--- arg1: rzs d0dc32bc: 89 55 f0 mov %edx,-0x10(%ebp) <--- arg2: index d0dc32bf: 89 4d ec mov %ecx,-0x14(%ebp) <--- arg3: flag return rzs->table[index].flags & BIT(flag); d0dc32c2: 8b 45 f4 mov -0xc(%ebp),%eax // EAX = rzs d0dc32c5: 8b 40 0c mov 0xc(%eax),%eax // EAX = rzs->table ('table' is at 12th offset of struct ramzswap) d0dc32c8: 8b 55 f0 mov -0x10(%ebp),%edx // EDX = index = 0x42 (decoded below) d0dc32cb: c1 e2 03 shl $0x3,%edx // EDX = index * (1 << 3) (each table entry has size of 8b) = 0x42 * 8 = 0x210 d0dc32ce: 01 d0 add %edx,%eax // EAX = EDX + EAX (rzs->table + index*8 = 0x210) d0dc32d0: 0f b6 40 07 movzbl 0x7(%eax),%eax // EAX = *(EAX + 0x7) ('flags' is at 7th offset of struct table) ^ BUG here! = *(rzs->table + index*8 + 0x7) = *(rzs->table + 0x210) d0dc32d4: 0f b6 d0 movzbl %al,%edx d0dc32d7: 8b 45 ec mov -0x14(%ebp),%eax Getting values for ramzswap_test_flag() args: EBP=ce51bc60, ESP=0xce51bc4c (from BUG report) -0x14(%ebp) == 0xce420780 - 0x14 == 0xce51bc4c = ESP So, Stack dump in the BUG report gives us the args (in reverse order): flag = 0x1 (RZS_ZERO) index = 0x42 rzs = 0xcd4c5600 From disasm above, we can see that rzs->table must be 0 to make final address in EAX (rzs->table + index*8 + 0x7) equal to 0x217 which the BUG report shows. This confirms our suspicion that its a race between read and reset device (which frees this table). --- The solution is to: - reset init_done to 0 *before* freeing all the data structures. - wait for all pending I/O on this ramzswap device to finish. - After this wait, if any further I/O is queued, the rzs device will simply reject it since init_done check will now fail in ramzswap_make_request() -- till the next init.

Last 7 days

Last 30 days

Older