Mailman 3 November 2012 - NumPy-Discussion

Hierarchical vs non-hierarchical ndarray.base and __array_interface__
by Gamblin, Todd Dec. 8, 2012

Dec. 8, 2012

Hi all, I posted on the change in semantics of ndarray.base here: https://github.com/numpy/numpy/commit/6c0ad59#commitcomment-2153047 And some folks asked me to post my question to the numpy mailing list. I've implemented a tool for mapping processes in parallel applications to nodes in cartesian networks. It uses hierarchies of numpy arrays to represent the domain decomposition of the application, as well as corresponding groups of processes on the network. You can "map" an application to the network using assignment of through views. The tool is here if anyone is curious: https://github.com/tgamblin/rubik. I used numpy to implement this because I wanted to be able to do mappings for arbitrary-dimensional networks. Blue Gene/Q, for example, has a 5-D network. The reason I bring this up is because I rely on the ndarray.base pointer and some of the semantics in __array_interface__ to translate indices within my hierarchy of views. e.g., if a value is at (0,0) in a view I want to know that it's actually at (4,4) in its immediate parent array. After looking over the commit I linked to above, I realized I'm actually relying on a lot of stuff that's not guaranteed by numpy. I rely on .base pointing to its closest parent, and I rely on __array_interface__.data containing the address of the array's memory and its strides. None of these is guaranteed by the API docs: http://docs.scipy.org/doc/numpy/reference/arrays.interface.html So I guess I have a few questions: 1. Is translating indices between base arrays and views something that would be useful to other people? 2. Is there some better way to do this than using ndarray.base and __array_interface__? 3. What's the numpy philosophy on this? Should views know about their parents or not? They obviously have to know a little bit about their memory, but whether or not they know how they were derived from their owning array is a different question. There was some discussion on the vagueness of .base here: http://thread.gmane.org/gmane.comp.python.numeric.general/51688/focus=51703 But it doesn't look like you're deprecating .base in 1.7, only changing its behavior, which I tend to agree is worse than deprecating it. After thinking about all this, I'm not sure what I would like to happen. I can see the value of not keeping extra references around within numpy, and my domain is pretty different from the ways that I imagine people use numpy. I wouldn't have to change my code much to make it work without .base, but I do rely on __array_interface__. If that doesn't include the address and strides, t think I'm screwed as far as translating indices go. Any suggestions? Thanks! -Todd ______________________________________________________________________ Todd Gamblin, tgamblin(a)llnl.gov, http://people.llnl.gov/gamblin2 CASC @ Lawrence Livermore National Laboratory, Livermore, CA, USA

2 1

allclose changed behaviour in 1.6.2 ?
by Martin Raspaud Dec. 2, 2012

Dec. 2, 2012

Hi, We noticed that comparing arrays of different shapes with allclose doesn't work anymore in numpy 1.6.2. Is this a feature or a bug ? :) See the output in both 1.6.1 and 1.6.2 at the end of this mail. Best regards, Martin 1.6.1:: In [1]: import numpy as np In [2]: np.__version__ Out[2]: '1.6.1' In [3]: a = np.array([1, 2, 3]) In [4]: b = np.array([1, 2, 3, 4]) In [5]: np.allclose(a, b) Out[5]: False 1.6.2:: In[1]: import numpy as np In[2]: np.__version__ Out[2]: '1.6.2' In [3]: a = np.array([1, 2, 3]) In[4]: b = np.array([1, 2, 3, 4]) In[5]: np.allclose(a, b) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/maarten/pytroll/local/lib/python2.7/site-packages/numpy-1.6.2-py2.7-linux-x86_64.egg/numpy/core/numeric.py", line 1936, in allclose return all(less_equal(abs(x-y), atol + rtol * abs(y))) ValueError: operands could not be broadcast together with shapes (3) (4)

3 2

install numpy 1.6.2 .dmg on macosx 10.7, check for python 2.7
by denis Nov. 29, 2012

Nov. 29, 2012

Trying to install numpy 1.6.2 on a mac osx 10.7.4 from this .dmg 9323135 numpy-1.6.2-py2.7-python.org-macosx10.3.dmg gives "numpy 1.6.2 can't be installed on this disk. numpy requires python.org Python 2.7 to install." But python 2.7.3 *is* installed from python.org 18761950 python-2.7.3-macosx10.6.dmg and /usr/bin/python is linked as described in http://wolfpaulus.com/journal/mac/installing_python_osx python -c 'import sys; print sys.version' 2.7.3 (v2.7.3:70274d53c1dd, Apr 9 2012, 20:52:43) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] How is the code in the .dmg checking for python.org python ? Thanks, cheers -- denis

4 4

Truth value of ndarray not Pythonic
by Jim Kitchen Nov. 29, 2012

Nov. 29, 2012

I understand the historical reason for the "Truth value of an array" error, avoiding the pitfalls of: >>> a = np.arange(10) >>> if a==0: # this is ambiguous, use any() or all() I also understand the issues with logical and/or: >>> if (a<10 and a > 5): # this will break due to "and" vs "&" However the main point in this thread from 3 years ago<http://thread.gmane.org/gmane.comp.python.numeric.general/32653>is very valid. If I write code that uses lists and then convert that to an array for efficiency or more powerful computation, I have my own pitfall trying to do: >>> if a: # why doesn't this just check for size? My Proposal ------------------ It seems to me that an elegant solution to this dilemma is to separate the behavior of ndarrays of type bool from all other ndarrays. Keep the current behavior for ndarrays of type bool, but let the __nonzero__ for all other ndarrays be based on size. >>> if a==0: # still raises Truth error because it's of dtype bool >>> if (a<10 and a>5): # still raises Truth error because it's of dtype bool >>> if a: # works fine because dtype is int64 This solution avoids all the primary pitfalls of ambiguity where someone would need any() or all() because they're working with bools at the element level. But for cases where a function may return data or None, I really like to use the normal Python truth test for that instead of: >>> if a is not None and len(a) > 0: # that was a chore to find out The only problem I see with this solution is with the case of the single-element array. >>> s = np.array([[0]]) >>> if s: # works today, returns False With my proposal, >>> if s: # still works, but returns True because array is not empty It's a wart to be sure, but it would make ndarrays much easier to work with when converting from standard Python containers. Maybe we need something like this (probably not possible): >>> from numpy.__future__ import truthiness I've especially found this Truth error a challenge converting from dictionaries of lists to pandas DataFrames. It raises the same error, tracing back to this ambiguity in ndarrays. If it's too big of a change to make for ndarrays, maybe the same proposal could be implemented in pandas. Jim

2 1

Simple Loadtxt question
by Robert Love Nov. 29, 2012

Nov. 29, 2012

I have a file with thousands of lines like this: Signal was returned in 204 microseconds Signal was returned in 184 microseconds Signal was returned in 199 microseconds Signal was returned in 4274 microseconds Signal was returned in 202 microseconds Signal was returned in 189 microseconds I try to read it like this: data = np.loadtxt('dummy.data', dtype={'names':('label','times','musec'), 'fmts':('|S23','i8','|S13')}) It fails, I think, because it wants a string format and field for each of the words 'Signal' 'was' 'returned' etc. Can I make it treat that whole string before the number as one string, one field? All I really care about is the numbers anyway. Any advice appreciated.

2 1

Windows installation problem
by Jim O'Brien Nov. 28, 2012

Nov. 28, 2012

I have tried to install the 1.6.2 win32 superpack on my Windows 7 Pro (64 bit) system which has ActiveState ActivePython 2.7.2.5 (64 bit) installed. However, I get an error that Python 2.7 is required and can't be found in the Registry. I only need numpy as it is a pre-requisite for another package and numpy is the only pre-requisite that won't install. Other are specific to Python 2.7 and they install. Some are Win64 and some are Win32. Is there a work around for this? I have no facilities available to build numpy. Regards, Jim

2 3

result shape from dot for 0d, 1d, 2d scalar
by Skipper Seabold Nov. 28, 2012

Nov. 28, 2012

I discovered this because scipy.optimize.fmin_powell appears to squeeze 1d argmin to 0d unlike the other optimizers, but that's a different story. I would expect the 0d array to behave like the 1d array not the 2d as it does below. Thoughts? Maybe too big of a pain to change this behavior if indeed it's not desired, but I found it to be unexpected. [255]: np.version.full_version # same on 1.5.1 [255]: '1.8.0.dev-8e0a542' [262]: arr = np.random.random((25,1)) [~/] [263]: np.dot(arr, np.array([1.])).shape [263]: (25,) [~/] [264]: np.dot(arr, np.array([[1.]])).shape [264]: (25, 1) [~/] [265]: np.dot(arr, np.array(1.)).shape [265]: (25, 1) [~/] [271]: np.dot(arr.squeeze(), np.array(1.)).shape [271]: (25,) Huh? 0d arrays broadcast with dot? [~] [279]: arr = np.random.random((25,2)) [~/] [280]: np.dot(arr.squeeze(), np.array(2.)).shape [280]: (25, 2) Skipper

2 4

Conditional update of recarray field
by Bartosz Nov. 28, 2012

Nov. 28, 2012

Hi, I try to update values in a single field of numpy record array based on a condition defined in another array. I found that that the result depends on the order in which I apply the boolean indices/field names. For example: cond = np.zeros(5, dtype=np.bool) cond[2:] = True X = np.rec.fromarrays([np.arange(5)], names='a') X[cond]['a'] = -1 print X returns: [(0,) (1,) (2,) (3,) (4,)] (the values were not updated) X['a'][cond] = -1 print X returns: [(0,) (1,) (-1,) (-1,) (-1,)] (it worked this time). I find this behaviour very confusing. Is it expected? Would it be possible to emit a warning message in the case of "faulty" assignments? Bartosz

2 4

the difference between "+" and np.add?
by Chao YUE Nov. 28, 2012

Nov. 28, 2012

Dear all, if I have two ndarray arr1 and arr2 (with the same shape), is there some difference when I do: arr = arr1 + arr2 and arr = np.add(arr1, arr2), and then if I have more than 2 arrays: arr1, arr2, arr3, arr4, arr5, then I cannot use np.add anymore as it only recieves 2 arguments. then what's the best practice to add these arrays? should I do arr = arr1 + arr2 + arr3 + arr4 + arr5 or I do arr = np.sum(np.array([arr1, arr2, arr3, arr4, arr5]), axis=0)? because I just noticed recently that there are functions like np.add, np.divide, np.substract... before I am using all like directly arr1/arr2, rather than np.divide(arr1,arr2). best regards, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

4 5

In case anybody wants them: py4science.* domains...
by Fernando Perez Nov. 27, 2012

Nov. 27, 2012

Hi folks, years ago, John Hunter and I bought the py4science.{com, org, info} domains thinking they might be useful. We never did anything with them, and with his passing I realized I'm not really in the mood to keep renewing them without a clear goal in mind. Does anybody here want to do anything with these? They expire December 3, 2012. I can just let them lapse, but I figured I'd give a heads-up in case anybody has a concrete use for them I'd rather transfer them than let them go to a domain squatter. Basically, if you want them, they're yours to have as of 12/3/12. Cheers, f

3 5