Mailman 3 December 2012 - NumPy-Discussion

Proposal to drop python 2.4 support in numpy 1.8
by Charles R Harris Dec. 15, 2012

Dec. 15, 2012

Time to raise this topic again. Opinions welcome. Chuck

14 16

Failure in test_iterator.py at Travis
by Ondřej Čertík Dec. 14, 2012

Dec. 14, 2012

Hi, Another weird bug sometimes happen in numpy/core/tests/test_iterator.py, it looks like this: ====================================================================== FAIL: test_iterator.test_iter_array_cast ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/travis/virtualenv/python2.5/lib/python2.5/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/travis/virtualenv/python2.5/lib/python2.5/site-packages/numpy/core/tests/test_iterator.py", line 836, in test_iter_array_cast assert_equal(i.operands[0].strides, (-96,8,-32)) File "/home/travis/virtualenv/python2.5/lib/python2.5/site-packages/numpy/testing/utils.py", line 252, in assert_equal assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k,err_msg), verbose) File "/home/travis/virtualenv/python2.5/lib/python2.5/site-packages/numpy/testing/utils.py", line 314, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: item=0 ACTUAL: 96 DESIRED: -96 ---------------------------------------------------------------------- But the problem is that there is no numpy/core/tests/test_iterator.py file in current branches.... This error was triggered for example by these PRs: https://github.com/numpy/numpy/pull/2765 https://github.com/numpy/numpy/pull/2815 and here are links to the failing Travis tests: https://travis-ci.org/certik/numpy/builds/3656959 https://travis-ci.org/numpy/numpy/builds/3330234 Any idea what is happening here and how to fix it? Ondrej

3 7

Building numpy with OpenBLAS
by Sergey Bartunov Dec. 14, 2012

Dec. 14, 2012

Hi. I'm trying to build numpy (1.6.2 and master from git) with OpenBLAS on Ubuntu server 11.10. I succeed with this just once and performance boost was really big for me, but unfortunately something went wrong with my application and I had to reinstall numpy. After that I couldn't reproduce this result and even just perform faster than default numpy installation with no external libraries anyhow. Now things went even worse. I assume that numpy built with BLAS and LAPACK should do dot operation faster than "clean" installation on relatively large matirces (say 2000 x 2000). Here I don't use OpenBLAS anyway. I install libblas-dev and liblapack-dev by apt-get and after that build numpy from sources / by pip (that doesn't matter for the result). Building tool reports that BLAS and LAPACK are detected on my system, so says "numpy.distutils.system_info" after installation. But matrix multiplication by dot takes the same time as clean installation (12 s vs 0.16 s with OpenBLAS). That's the first thing I'm wondering about. Nevertheless I tried to compile numpy with OpenBLAS only. I have forced this by setting ATLAS="" BLAS=/usr/lib/libopenblas.a LAPACK=/usr/lib/libopenblas.a as I saw somewhere in the internet. I had installed numpy exactly this way at the first time when I was lucky. But now it doesn't work for me. I tryied installing OpenBLAS from sources and as libopenblas-dev ubuntu package. So how can I fix this? Many thanks in advance.

3 2

np.seterr doesn't work for masked array?
by Chao YUE Dec. 14, 2012

Dec. 14, 2012

Dear all, I tried to capture the zero divide error when I divide a masked array by another. It seems that np.seterr is not working for masked array? when I do np.divide on two masked array, it directly put the zero divides part as being masked. The np.seterr works if the two arrays for dividing are not masked arrays. could anyone explain? thanks!! np.__version__ = 1.6.2 In [87]: np.seterr(all='print') Out[87]: {'divide': 'print', 'invalid': 'print', 'over': 'print', 'under': 'print'} In [88]: a = np.arange(8,dtype=float).reshape(2,4) In [89]: b = np.ma.masked_less(a,4) In [90]: b[1,-2:] = 0. In [91]: b Out[91]: masked_array(data = [[-- -- -- --] [4.0 5.0 0.0 0.0]], mask = [[ True True True True] [False False False False]], fill_value = 1e+20) In [92]: c = a.copy() In [93]: c[1,-2:] = 0. In [94]: c Out[94]: array([[ 0., 1., 2., 3.], [ 4., 5., 0., 0.]]) In [95]: np.divide(a,b) Warning: divide by zero encountered in divide Out[95]: masked_array(data = [[-- -- -- --] [1.0 1.0 -- --]], mask = [[ True True True True] [False False True True]], fill_value = 1e+20) In [96]: np.divide(a,c) Warning: divide by zero encountered in divide Out[96]: array([[ nan, 1., 1., 1.], [ 1., 1., inf, inf]]) Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

2 3

an easy way to know if a functions works or not for masked array?
by Chao YUE Dec. 14, 2012

Dec. 14, 2012

Dear numpy users, I think since long I am confused by if a function works as expected or not for masked array. like np.reshape works for masked array, but not np.sum (I mean, I expect np.sum to drop the masked elements when do summing, of course we have np.ma.sum). So I always use fuctions preceding by np.ma to make sure there is nothing going woring if I expect there will be masked array participating in the calculation in the data process chain. When I handle masked array, Before I use a function, normally I check if there is an equivalent np.ma function and if there is, I use np.mafunction; Then I check how the documentation says about np.func and np.ma.func and see there is anything mentioned explicitly on the handling of masked array. Howevery, In most cases I will try with simple arrays to test both np.ma.func and np.func before I use some function. but this is sometimes time consuming. Does anyone have similar situation? thanks! Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

1 0

Attaching metadata to dtypes: what's the best way?
by Andrew Collette Dec. 14, 2012

Dec. 14, 2012

Hi all, I have a question for the list sparked by this discussion of a bug in NumPy 1.6.2 and 1.7: http://mail.scipy.org/pipermail/numpy-discussion/2012-December/064682.html and this open issue in h5py: https://code.google.com/p/h5py/issues/detail?id=217 In h5py we need to represent variable length strings and HDF5 object references within the existing NumPy dtype system. The way this is handled at the moment is with object (type "O") dtypes with a small amount of metadata attached; in other words, an "O" array could have a dtype marked as representing variable-length strings, and HDF5 would convert the Python string objects into the corresponding type in the HDF5 file. Likewise, an "O" dtype marked as containing HDF5 object references (h5py.Reference instances) would be converted to native HDF5 references when written. The trouble I'm having is trying to attach metadata to a dtype in such a way that it is preserved in NumPy. Right now I create an "O" dtype with a single field and store the information in the field "description", e.g.: dtype(('O', [( ({'type': bytes},'vlen'), 'O' )] )) This works (it's how special types have worked in h5py for years) but is quite unwieldy, and leads to interesting side effects. For example, because of the single field used, array[index] returns a 1-element NumPy array containing a Python object, instead of the Python object itself. Worse, our fix for this behavior (remove the field when returning data from h5py) triggered the above bug in NumPy. Is there a better way to add metadata to dtypes I'm not aware of? Note I'm *not* interested in creating a custom type; one of the advantages of the current system is that people deal with the resulting "O" object arrays like any other object array in NumPy. Andrew Collette

1 0

ValueError: low level cast function is for unequal type numbers
by Christoph Gohlke Dec. 13, 2012

Dec. 13, 2012

Hello, the following code using np.object_ data types works with numpy 1.5.1 but fails with 1.6.2. Is this intended or a regression? Other data types, np.float64 for example, seem to work. In [1]: import numpy as np In [2]: np.array(['a'], dtype='O').astype(('O', [('name', 'O')])) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-3-f4a62241f13c> in <module>() ----> 1 np.array(['a'], dtype='O').astype(('O', [('name', 'O')])) ValueError: low level cast function is for unequal type numbers In [3]: np.array([16], dtype='d').astype(('d', [('name', 'd')])) Out[3]: array([ 16.]) These downstream issues could be related: http://code.google.com/p/h5py/issues/detail?id=217 https://github.com/CellProfiler/CellProfiler/issues/421 Thank you, Christoph

2 2

Calling LAPACK function dbdsqr()?
by Philip Semanchuk Dec. 13, 2012

Dec. 13, 2012

Hi all, I'm porting some Fortran code that makes use of a number of BLAS and LAPACK functions, including dbdsqr(). I've found all of the functions I need via scipy.linalg.lapack.get_lapack_funcs/get_blas_funcs() except for dbdsqr(). I see that the numpy source code (I looked at numpy-1.6.0b2) contains dbdsqr() in numpy/linalg/dlapack_lite.c, but grep can't find that string in the binary distribution on my Mac nor on Linux. If it's buried in a numpy binary somewhere, I'm comfortable with using ctypes to call it, but I suspect it isn't. Can anyone point me to a cross-platform (OS X, Linux & Windows) way I can call this function? I'm unfortunately quite naïve about the math in the code I'm porting, so I'm porting the code blindly -- if you ask me what problem I'm trying to solve with dbdsqr(), I won't be able to explain. Thanks in advance for any suggestions, Philip

3 4

site.cfg: Custom BLAS / LAPACK configuration
by Bradley M. Froehle Dec. 13, 2012

Dec. 13, 2012

I recently installed NumPy 1.6.2 on a new computer and wanted to use ACML as the BLAS/LAPACK library. [I'm aware that ACML doesn't provide CBLAS, but that is easy to work around by compiling it yourself to produce libcblas.a or libcblas.so]. I experienced a great bit of difficulty in getting NumPy to use ACML (-lcblas -lacml), primarily stemming from the fact that there was a working ATLAS installation already in /usr/lib64. As far as I can tell, it's IMPOSSIBLE to create a site.cfg which will link to ACML when a system installed ATLAS is present. The detection routine for blas_opt (and similarly for lapack_opt) seem to operate as: * Is MKL present? If so, use it. * Is ATLAS present? If so, use it. * Use [blas] section from site.cfg. Instead I would have expected the detection routine to be more like: * Is [blas_opt] present in site.cfg? If so, use it. * Is MKL present? ... * Is ATLAS present? ... * Use [blas] section from site.cfg. This is not just a problem with ACML. I've also experienced this when using NumPy on some cray supercomputers where the default C compiler automatically links a preferred BLAS/LAPACK. I created a GitHub issue for this: https://github.com/numpy/numpy/issues/2728. In addition, I created a pull request with a "works for me" solution, but which should have needs some wider visibility https://github.com/numpy/numpy/pull/2751. I'd appreciate any reviews, workarounds, or other general feedback. If you want to test our the library detection mechanism you can run the following from within the NumPy source directory:: import __builtin__ as builtins builtins.__NUMPY_SETUP__ = True import numpy.distutils.system_info as si print si.get_info('blas_opt') Thanks, Brad

4 10

non-integer index misfeature?
by Neal Becker Dec. 13, 2012

Dec. 13, 2012

I think it's a misfeature that a floating point is silently accepted as an index. I would prefer a warning for: bins = np.arange (...) for b in bins: ... w[b] = blah when I meant: for ib,b in enumerate (bins): w[ib] = blah

5 6