NumPy-Discussion

Download

[email protected]

July 2010

129 participants
172 discussions

numpy.load raising IOError but EOFError expected
by Ruben Salvador July 1, 2010

July 1, 2010

Hi there, I have a .npy file built by succesively adding results from different test runs of an algorithm. Each time it's run, I save a numpy.array using numpy.save as follows: fn = 'file.npy' f = open(fn, 'a+b') np.save(f, arr) f.close() When I try to read the file with the following code, for a file containing 3 array saves (a is there to show when the error arises): f = open(fn, 'rb') arr = np.load(f) a = 0 try: while True: print a a += 1 arr = np.vstack((arr, np.load(f))) except EOFError: pass f.close() I get the following output: 0 1 2 Traceback (most recent call last): File "./proc_stat.py", line 32, in <module> arr = np.vstack((arr, np.load(f))) File "/usr/lib/python2.5/site-packages/numpy/lib/io.py", line 201, in load "Failed to interpret file %s as a pickle" % repr(file) IOError: Failed to interpret file <open file '/home/rsalvador/trabajo/research/phd/devel/evowv/retest/avgfit_random_sigma_normal_Flp_20G.npy', mode 'rb' at 0x174ca08> as a pickle Using IOError in the except makes the code work, but this way I am masking other possible sources of error. I have tried with a file containing 3 dumps from numpy.save (this is, 3 arrays saved). As shown, the error is raised when trying to read a fourth time (since EOFError is not raised). Therefore, is this a bug? Shouldn't EOFError be raised instead of IOError? Or am I missunderstanding something? If this is not a bug, how can I detect the EOF to stop reading (I expect a way for this to work without tweaking the code with saving first in the file the number of dumps done)? Thanks in advance! -- Rubén Salvador PhD student @ Centro de Electrónica Industrial (CEI) http://www.cei.upm.es Blog: http://aesatcei.wordpress.com

6 11

building numpy against Cray xt-libsci
by Amir July 1, 2010

July 1, 2010

I am trying to build numpy against Cray's xt-libsci library on a Cray XT5. I am getting an error I am hoping for hints on how to resolve: In [1]: import numpy <snip> 20 isfinite, size 21 from numpy.lib import triu ---> 22 from numpy.linalg import lapack_lite 23 from numpy.matrixlib.defmatrix import matrix_power 24 ImportError: /opt/xt-libsci/10.4.0/gnu/lib/libsci.so: undefined symbol: fftw_version These are the symbols in libsci: % nm /opt/xt-libsci/10.4.0/gnu/lib/libsci.so | grep fftw_version 00000000010f9a30 B __crafft_internal__crafft_fftw_version_num U fftw_version 00000000005aa8a4 T get_fftw_version I first built numpy with no custom site.cfg file. It built correctly and all tests ran. But it was too slow. Then I tried building numpy against libsci, which has BLAS, LAPACK, FFTW3 among other things. I had to build a libcblas.a from the netlib src as libsci does not have cblas (using gcc, gfortran 4.3.3). Here is my site.cfg, accumulated from several nice tutorials on how to build numpy on these machines, which for some reason don't work for me. The instructions were based on numpy 1.2. [blas] blas_libs = cblas library_dirs = /global/homes/amir/local/lib [lapack] lapack_libs = sci library_dirs = /opt/xt-libsci/10.4.0/gnu/lib [blas_opt] blas_libs = cblas, sci libraries = cblas, sci [lapack_opt] libraries = sci [fftw] libraries = fftw3 Here is what is linked to lapack_lite.so: % ldd ./numpy/linalg/lapack_lite.so libsci.so => /opt/xt-libsci/10.4.0/gnu/lib/libsci.so (0x00002b4493325000) libgfortran.so.3 => /opt/gcc/4.3.3/snos/lib64/libgfortran.so.3 (0x00002b44a4579000) libm.so.6 => /lib64/libm.so.6 (0x00002b44a4770000) libgcc_s.so.1 => /opt/gcc/4.3.3/snos/lib64/libgcc_s.so.1 (0x00002b44a48c6000) libc.so.6 => /lib64/libc.so.6 (0x00002b44a49dd000) /lib64/ld-linux-x86-64.so.2 (0x0000555555554000) Thanks, Amir.

3 2

Jump to page: