Hei,

Although my script fails in the h5py module (and, yes, I already contacted the h5py team), I suspect there is a compatibility issue between python-netCDF4 and h5py in the way they open/close access to the HDF5 library.

I am basically reading a h5 file (with h5py), process the data, and write it back to netCDF3 (using python netCDF4). This works fine for the first h5 file, but crashes for the subsequent files (I am looping through a list of h5 files).

My configuration:

Ubuntu Lucid python 2.6 h5py 2.2.0 (latest) python-netCDF4 1.0.7 (latest) libnetcdf4 4.1.1

libhdf5 1.8.5-patch1

I build python-netCDF4 and h5py from source, and in both cases use appropriate environment variables to python setup.py so that both modules will find the libraries (that are not in default system directories). I have long experience using python-netCDF4, but it is the first time I try to mix h5py and netCDF.

Here is a small script to make it happen:

<code> import h5py import netCDF4

g = netCDF.Dataset('test.nc','w') g.close()

f = h5py.File('file.h5','r')

AttributeError Traceback (most recent call last)

/home/thomasl/software/pmw_decode/my_satdata_polar/binsrc/pmw_decode/src/<ipython console> in <module>()

/home/thomasl/software/pmw_decode/my_satdata_polar/binsrc/pmw_decode/mypython/lib/python2.6/site-packages/h5py/_hl/files.pyc in init(self, name, mode, driver, libver, userblock_size, **kwds) 204 pass 205 --> 206 fapl = make_fapl(driver, libver, **kwds) 207 fid = make_fid(name, mode, userblock_size, fapl) 208

/home/thomasl/software/pmw_decode/my_satdata_polar/binsrc/pmw_decode/mypython/lib/python2.6/site-packages/h5py/_hl/files.pyc in make_fapl(driver, libver, **kwds) 30 """ Set up a file access property list """ 31 plist = h5p.create(h5p.FILE_ACCESS) ---> 32 plist.set_fclose_degree(h5f.CLOSE_STRONG) 33 34 if libver is not None:

AttributeError: 'h5py.h5p.PropLAID' object has no attribute 'set_fclose_degree' </code>

Do you python-netCDF4 folk have a hint on why this happens? For reference, here is a link to the issue I reported to the h5py team: https://groups.google.com/forum/#!topic/h5py/AZQ30pSy-RI

Cheers, Thomas

Unfortunately, I have no idea. The close method simply calls nc_close in the netcdf-c library. I don't know how nc_close interacts with the HDF5 library, but it's possible not everything is cleaned up properly. I think the thing to do is to try to replicate this in a simple C program, then report this to both the Unidata netcdf team and the HDF5 development team.

It also looks like you're using rather old versions of the netcdf and HDF5 libs. netcdf is now at 4.3.1rc4 and hdf5 at 1.8.12. Any chance you could upgrade to see if this is a problem that has been fixed at the C level?

This script works for me with netcdf 4.3.0 and hdf5 1.8.11

import netCDF4, h5py for i in range(5): g = netCDF4.Dataset('test%s.nc' % i,'w') g.close() f = h5py.File('test%s.nc' % i,'r') f.close()

It differs from yours in that it look like you are reading a separate hdf5 file (file.h5) presumably created by some other application. I don't know if that's import though. Can you try this version and let us know what happens? If it doesn't work for you, that would confirm that it's an issue that's been fixed in the C libs.

Thanks Jeff,

I will look at it first thing tomorrow morning. A colleague of mine reported that he was not experiencing such issues on our Ubuntu Precise distributions, that have newer versions of the netCDF and hdf5 libs.

I'll let you know asap. Hopefully this is solved by the C libraries.

Thomas

Jeff,

Good news. The issue seems to disappear when upgrading to our Ubuntu Precise distro (python-2.7, hdf5-1.8.4, netcdf-4.1.1, h5py-2.2.0, python-netCDF4-1.0.7).

So it seems this is fixed somewhere in the C library.

Nothing to worry about, then ;)

Cheers, and thanks for your help. Thomas

Glad to hear it. Closing...