What is a simple way to set all bits to True (or False) given a BitString? instance?

Also, re: 2.4 support. Maybe in future versions you can include a bistring24.py file with tests (all the 1.0.0 version stuff), and in setup.py, just check sys.version to determine which one to install. For those of us stuck with 2.4, this is the best bitstring library we have!

@gregg: One simple way to set the bits is to use the integer interpretation: s.int = 0 will set all the bits to zero, s.int = -1 will set them all to one. To create BitStrings like this you can use something like zeros = BitString(length=55) or ones = ~BitString(length=55).

In terms of 2.4 support, I've not looked at conditional installation, although just packaging the 1.0.0 version with newer versions isn't going to help current users. It will still be the same version so you might as well not bother! I can see that it could be simpler for new users though so it's a good suggestion.

would be good if we can append variable directly to BitString?

a = 0b001110 b = 0x4664 c = BitString?(uint=b,length=16) c.append(a,6)

currently we have to do c.append(Bits(uint=a,6)) to append any variable

@supercoolman: Thanks for the comment. Not sure I like the suggested syntax - the variable a is just the number 14, and it's not obvious to me that c.append(14, 6) should mean add the six bits 001110 to the bitstring...

One thing to remember is that you can append anything that can auto-initialise a bitstring. So for example:

c = BitString('0x4664')
c.append('0b001110')

should work fine. As should c += '0b001110' or even c += 'uint:6=14'.

the reason I am using bitstring is that I want to append unsigned integer variables of different sizes, which I am unable to do with python's built-in modules. for network packet, I want to send fix size 100bit packets with different predefined fields all packed in side that 100 bit. I'm using bitstring to pack all fields into 100 bits and also save myself from doing lots of bit shifting. Now I'm not sure if that's the purpose of bitstring?

@supercoolman: That seems a very reasonable use of bitstring. The reason you can't append in the way you want is that there can be many different interpretations of binary data so you have to specify which you want (base-2 integers may seem the only obvious choice but there are issues of signedness, endianness and even other integer representations like exponential Golomb codes supported).

The module is meant to be flexible and my comment was just giving some other ways that might be useful to you. Another strategy could be to pre-allocate the data and then set slices of it:

s = BitString(100)   # 100 zero bits
s[0:16] = 0x4664
s[16:22] = 0b001110  # or just = 14

thanks for the clarification.

what is the difference between BitArray? and BitStream??

@xiaohan2012: A BitStream? is a BitArray? with additional methods for treating it like a stream of bits. So it has a current bit position which you can read from for example. The BitArray? is just a mutable container of bits. See the manual for more details.

The most common use case in which I want to use bitstring involves analysis of subfield(s) of a NumPy? memmapped array. So I'm looking for a way to create some kind of bitstring which provides read/write mmap access with a fixed length. It would be nice to reuse the mmap object of the array, but not necessary. Is it possible to obtain this with the current (3.0.2) architecture, or will I have to subclass (Bits, BitArray?), (MmapByteArray?)?

@00ai99: Interesting question, and I'm not sure of the best solution. As far as I know, the numpy memmap won't be compatible with the mmap used for const file-based bitstrings. The simplest way to proceed would be to extract your subfield as a bytearray and create a new bitstring from it for analysis. If that isn't possible due to memory constraints then work out the file offset and byte length of your subfield and create a new !Bits or ConstBitStream object from the file (it's not going to hurt to have two mmaps on the file). I don't think that subclassing is the best way to go, and you shouldn't use MmapByteArray as it's not part of the public interface. Hope that helps a little.

Thanks for the reply, actually you may be interested to know that, in Python3 at least, you can pass a NumPy? array's 'data' attribute directly to Bits/BitArray?, as in 'bytes=myarray.data'.

So are you saying that rather than getting a read/write mmap, I'll have to settle for read-only for the bitstring side of things, and write it back as bytes afterwards? :/ (there's no particular memory constraints -- it's just that that read, copy, write approach is ugly.)

I was hoping that, given the constraints of the bitstring being a multiple of 8 in length and fixed-length, mmapped writing would also be possible. I'm not prepared to give up on that just yet, so I'm working on a subclass (in which I'll have to override init, which is why MmapByteArray? enters the picture; if I shouldn't subclass that, it's copy+paste time.)

(the underscores on "init" were stripped in the above message, FWIW. and maybe also in this one.)

@00ai99: Yeah the enforced wiki markup in these comments really doesn't work well when talking about Python. I think the read, modify, write approach is the simplest, even if it is a little ugly. I did consider allowing writeable mmap'ed bitstrings a while back but decided against it as there would be too much fragmentation of the basic types. There are currently immutable and mutable classes, but this would require a mutable fixed-length type too and there was no elegant way to do it. But it probably isn't too difficult to do in a subclass - override __init__, disable __delitem__ and put some bit-length preserving safeguards in __setitem__ would be my first stab at it.

FWIW I now have a basic implementation -- http://paste.pocoo.org/show/549540/

I have a question regarding how bitstring handles endianness.

Say I have a Big Endian 32bit integer encoded as follows in hex... "FFFF716E"

Interactive session...

If I try this using the struct module without specifying endianness I get the following...

struct.unpack("i", "FFFF716E".decode("hex")) (1852964863,)

My Wintel machine must be little endian...

struct.unpack("<i", "FFFF716E".decode("hex")) (1852964863,)

If I specify big endian the result is correct...

struct.unpack(">i", "FFFF716E".decode("hex")) (-36498,)

bitstring appears to be correct without specifying endianness.

bitstring.BitString?("0xFFFF716E").int -36498L

also with endianness specified...

bitstring.BitString?("0xFFFF716E").intbe -36498L

for completeness...

bitstring.BitString?(bytes="FFFF716E".decode("hex")).int -36498L

@Apexi.200sx: Bitstrings are bit-wise big endian. Note that endianness almost always refers to byte-wise big or little endianness, which makes very little sense if your bitstring isn't a whole number of bytes. So the ordinary 'int' interpretation is big endian in a bit-wise way (so for example '100' represents 4, not 1), which for whole byte bitstrings is the same as big endian byte-wise. That's why 'int' and 'intbe' give the same results. The native and little endian interpretations can be got through 'intne' and 'intle' and will agree with the struct module.

I think the point to understand is that byte-wise endianness only makes sense if you have a whole number of bytes, and when thinking bit-wise then big-endian is the most natural. Hence the plain 'int' has been chosen to be big endian irrespective of the underlying hardware endianness.

OK, thanks for the response. I was just making sure nothing was going to bite me later on because of making assumptions that it is just doing the right thing.

I like how bitstring shines over the plain struct module when dealing with things that aren't the usual 8bit 16bit 32bit aligned integers etc. like the following...

Big Endian integer encoded in 3 bytes in hex... "503661"

tmp = "503661".decode("hex")

t = struct.unpack("3b", tmp)

(t0? << 16) + (t1? << 8) + t2? 5256801

Now bitstring :)

bitstring.BitStream?("0x503661").int 5256801

Cheers !