I think this struct has been created badly.

'_RTL_ATOM_TABLE' : [ 0x44, { 'Signature' : [ 0x0, ['unsigned long']], 'CriticalSection' : [ 0x4, ['_RTL_CRITICAL_SECTION']], 'RtlHandleTable' : [ 0x1c, ['_RTL_HANDLE_TABLE']], 'NumberOfBuckets' : [ 0x3c, ['unsigned long']], 'Buckets' : [ 0x40, ['array', 1, ['pointer', ['_RTL_ATOM_TABLE_ENTRY']]]],

http://doxygen.reactos.org/d5/df7/ndk_2rtltypes_8h_source.html

Link above shows the correct version. When I tried to create my atomscanner for the first time, this bug stopped me first.

What were the changes that you made to the struct to get it to work? I'm wondering if the preprocessor directives might have had something to do with how that structure was autogenerated. Can you tell if NTOS_MODE_USER is enabled on the system you are analyzing?

Hey Marko.

You are right, glad you brought this up.

So the _RTL_ATOM_TABLE you pasted is auto-generated from the PDB of ntoskrnl.exe. As Gleeda stated, the preprocessor directives may have something to do with the inconsistency. For example, if we used the PDB for an NT module with some members excluded due to #ifdef, and then you analyzed memory of a machine whose NT module included those members, some things may be off.

However...

I can confirm that Windbg on a given Win7 system shows:

0: kd> dt nt!_RTL_ATOM_TABLE nt!_RTL_ATOM_TABLE +0x000 Signature : Uint4B +0x004 CriticalSection : _RTL_CRITICAL_SECTION +0x01c RtlHandleTable : _RTL_HANDLE_TABLE +0x03c NumberOfBuckets : Uint4B +0x040 Buckets : [1] Ptr32 _RTL_ATOM_TABLE_ENTRY

Analyzing memory of that same system with the shown _RTL_ATOM_TABLE will not work (as you stated). This indicates that we're not dealing with a preprocessor directive thing.

Instead, in my plugin I re-define the structure this way:

address_space.profile.add_types({ '_RTL_ATOM_TABLE': [ None, { 'Signature': [ 0x0, ['unsigned long']], 'NumBuckets': [ 0xC, ['unsigned long']], 'Buckets': [ 0x10, ['array', 1, ['pointer', ['_RTL_ATOM_TABLE_ENTRY']]]], }]})

That seems to work for all OS that Volatility supports. As for why its different compared to how Microsoft defines it in the PDB, I really don't know.

Hi,

ah crap, forgot to mention the changes I made. I think it is not NTOS_MODE_USER.

'Signature' : [ 0x0, ['unsigned long']],
'CriticalSection' : [ 0x4, ['_RTL_CRITICAL_SECTION']],
'RtlHandleTable' : [ 0x8, ['_RTL_HANDLE_TABLE']],
'NumberOfBuckets' : [ 0xc, ['unsigned long']],
'Buckets' : [ 0x10, ['array', 1, ['pointer', ['_RTL_ATOM_TABLE_ENTRY']]]], 

Above is shown how I changed it. It's completely wrong but that way I managed to get correct amount of buckets and back then I was a bit clueless about the struct.

I think the way Michael showed the struct would be the most proper since unions cannot be used.

When the volatility structure types were designed, were there any thoughts about using ctypes.Structures?

I am not sure if you know, but unions can be used in Volatiltiy. Just go ahead and add another entry for the same offset. Then depending on how you access the member it will be interpreted the correct way.

'RtlHandleTable' : [ 0x8, ['_RTL_HANDLE_TABLE']], 'ExHandleTable' : [ 0x8, ['pointer', 'HANDLE_TABLE']],

etc.

Also I am not sure if this helps but you can use lambdas to specify the length of arrays. E.g.

'Buckets' : [ 0x10, ['array', lambda x: x.NumberOfBuckets, '_RTL_ATOM_TABLE_ENTRY']]

Comment deleted

There is still a tiny prob. Without real union, it would look like below :(

'Signature'       : [ 0x0, ['unsigned long']],
'CriticalSection' : [ 0x4, ['_RTL_CRITICAL_SECTION']],
'FastMutex'       : [ 0x4, ['pointer', ['_FAST_MUTEX']]],
'RtlHandleTable'  : [ 0x1c, ['_RTL_HANDLE_TABLE']],
'ExHandleTable'   : [ 0x8, ['pointer', 'HANDLE_TABLE']]],
'NumberOfBuckets' : [ 0xc, ['unsigned long']],
'NumberOfBuckets' : [ 0x3c, ['unsigned long']],
'Buckets' : [ 0x10, ['array', 1, ['pointer', ['_RTL_ATOM_TABLE_ENTRY']]]], 
'Buckets' : [ 0x40, ['array', 1, ['pointer', ['_RTL_ATOM_TABLE_ENTRY']]]], 

Yeah, that's why I sort of overwrote the few fields I needed to adjust using add_types from within a plugin. Also that keeps us from having to modify the auto-generated vtypes files.

I am not sure what you want to happen here? Do you want to specify a flag for the ifdef and have different offsets used? In that case use this lambda for the offset

lambda x: 0xc if config.SOMEFLAGSVALUE else 0x3c

Another option is to access both settings at the same time: 'NumberOfBucketsWITHIFDEF1' : [ 0xc, ['unsigned long']], 'NumberOfBucketsWITHIFDEF2' : [ 0x3c, ['unsigned long']],

Also its possible to set all these in the overlay rather than the generated vtype: 'InterpNumberOfBuckets': [lambda x: x.NumberOfBuckets.offset, None]

I don't think we really need to do anything at the moment. _RTL_ATOM_TABLE is the only instance of this issue that I've run into. I will try to hunt down the reason, so give me a few days and I'll report back. However, we shouldn't modify the auto-generated vtype...or it may get overwritten the next time we auto-generate.

We should definitely keep in mind how ifdef preprocessors can affect our profiles though (although its probably not related to this issue). Hopefully we use a mainstream build of the OS to generate the profile...one that the majority of the world uses...and that Microsoft won't be changing often by adding/removing ifdef statements.

This is really weird. Although I know I wrote the structure as: '_RTL_ATOM_TABLE': [ 0x14, { 'Signature': [ 0x0, ['unsigned long']], 'Lock': [ 0x4, ['pointer', ['void']]], 'HandleTable': [ 0x8, ['pointer', ['_HANDLE_TABLE']]], 'NumBuckets': [ 0xC, ['unsigned long']], 'Buckets': [ 0x10, ['array', 1, ['pointer', ['_RTL_ATOM_TABLE_ENTRY']]]], } ],

...but I cannot for the life of me remember how I arrived at that structure. It clearly wasn't from using the public PDB types. I believe I was stepping through these functions at the time: http://msdn.microsoft.com/en-us/library/ms649053(v=vs.85).aspx

One guess -- maybe the _RTL_ATOM_TABLE found in ntoskrnl.pdb is for local atom tables only, and the structure I reverse engineered (and then confusingly named RTL_ATOM_TABLE) is for the global atom table?

Ah, I got it!

There are two version RtlCreateAtomTable. One is exported by ntdll.dll (usermode) and the other is exported by ntoskrnl.exe (kernelmode). Note the RtlCreateAtomTable is fully implemented in ntdll.dll - it is not just a dispatcher to the kernel mode version, as with other service functions.

The version of RtlCreateAtomTable implemented in ntdll.dll uses the _RTL_ATOM_TABLE that WinDbg shows and that is in our auto-gen'd vtypes. It is strictly used for atom tables in user memory (process-specific ones). The same Windows header file that defines _RTL_ATOM_TABLE is used by ntoskrnl.exe to represent atom tables in kernel memory, except the NTOS_MODE_USER preprocessor is reversed.

So the real question is - why does the PDB for ntoskrnl.exe include the wrong version of _RTL_ATOM_TABLE. It shows the version of the structure used by ntdll.dll.

Strange...I figure we can close this issue on Volatility Friday if nothing else comes up.

I am not sure what you want to happen here?

Nothing special beside what just happened. Since this struct brought some trouble I decided to bring it up to see if we could clear it and we did.

On my behalf, this is issue is over.

Glad you brought it up, thanks. I'm still going to try and find out why the PDB has the wrong structure...but closing the issue for now.