Mailman 3 January 2014 - NumPy-Discussion

Creating an ndarray from an iterable over sequences
by Dr. Leo Jan. 21, 2014

Jan. 21, 2014

Hi, I would like to write something like: In [25]: iterable=((i, i**2) for i in range(10)) In [26]: a=np.fromiter(iterable, int32) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-26-5bcc2e94dbca> in <module>() ----> 1 a=np.fromiter(iterable, int32) ValueError: setting an array element with a sequence. Is there an efficient way to do this? Creating two 1-dimensional arrays first is costly as one has to iterate twice over the data. So the only way I see is creating an empty [10,2] array and filling it row by row. This is memory-efficient but slow. List comprehension is vice versa. If there is no solution, wouldn't it be possible to rewrite fromiter so as to accept sequences? Leo

3 2

Re: [Numpy-discussion] A one-byte string dtype? (Charles R Harris)
by David Goldsmith Jan. 20, 2014

Jan. 20, 2014

On Mon, Jan 20, 2014 at 9:11 AM, <numpy-discussion-request(a)scipy.org> wrote: > I think that is right. Not having an effective way to handle these common > scientific data sets will block acceptance of Python 3. But we do need to > figure out the best way to add this functionality. > > Chuck > Sounds like it might be time for some formal data collection, e.g., a wiki-poll of users' use-cases. (I know this wouldn't be exhaustive, but at least it will provide guidance and a "checklist" of situations we should be sure our solution covers.) DG

1 0

(no subject)
by jennifer stone Jan. 18, 2014

Jan. 18, 2014

Hello, This is Jennifer Stupensky. I would like to contribute to NumPy this GSoC. What are the potential projects that can be taken up within the scope of GSoC? Thanks a lot in anticipation Regards Jennifer

2 1

Allowing slices as arguments for ndarray.take
by Stephan Hoyer Jan. 17, 2014

Jan. 17, 2014

There was a discussion last year about slicing along specified axes in numpy arrays: http://mail.scipy.org/pipermail/numpy-discussion/2012-April/061632.html I'm finding that slicing along specified axes is a common task for me when writing code to manipulate N-D arrays. The method ndarray.take basically does what I would like, except it cannot take slice objects as argument. In the mean-time, I've written a little helper function: def take(a, indices, axis): index = [slice(None)] * a.ndim index[axis] = indices return a[tuple(index)] Is there support for allowing the `indices` argument to `take` to take Python slice objects as well as arrays? That would alleviate the need for my helper function. Cheers, Stephan

2 1

Re: [Numpy-discussion] using loadtxt to load a text file in to a numpy array (Charles R Harris)
by David Goldsmith Jan. 15, 2014

Jan. 15, 2014

On Wed, Jan 15, 2014 at 9:52 AM, <numpy-discussion-request(a)scipy.org> wrote: > Date: Wed, 15 Jan 2014 10:57:51 -0700 > From: Charles R Harris <charlesr.harris(a)gmail.com> > Subject: Re: [Numpy-discussion] using loadtxt to load a text file in > to a numpy array > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Message-ID: > < > CAB6mnxJpvJbsoZzY0Ctk1bk+kDCUDivC9KrzYt1johU33bZOLw(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > On Wed, Jan 15, 2014 at 10:27 AM, Chris Barker <chris.barker(a)noaa.gov > >wrote: > > There was a discussion of this long ago and UCS-4 was chosen as the numpy > standard. There are just too many complications that arise in supporting > both. > > Chuck > In that case, perhaps another function altogether is called for. DG

1 0

Memory allocation cleanup
by Charles R Harris Jan. 10, 2014

Jan. 10, 2014

Apropos Julian's changes <https://github.com/numpy/numpy/pull/4177> to use the PyObject_* allocation suite for some parts of numpy, I posted the following I think numpy memory management is due a cleanup. Currently we have PyDataMem_* PyDimMem_* PyArray_* Plus the malloc, PyMem_*, and PyObject_* interfaces. That is six ways to manage heap allocations. As far as I can tell, PyArray_* is always PyMem_*in practice. We probably need to keep the PyDataMem family as it has a memory tracking option, but PyDimMem just confuses things, I'd rather just use PyMem_* with explicit size. Curiously, the PyObject_Malloc family is not documented apart from some release notes. We should also check for the macro versions of PyMem_* as they are deprecated for extension modules. Nathaniel then suggested that we consider going all Python allocators, especially as new memory tracing tools are coming online in 3.4. Given that these changes could have some impact on current extension writers I thought I'd bring this up on the list to gather opinions. Thoughts?

4 6

Bug in resize of structured array (with initial size = 0)
by Nicolas Rougier Jan. 10, 2014

Jan. 10, 2014

Hi, I've tried to resize a record array that was first empty (on purpose, I need it) and I got the following error (while it's working for regular array). Traceback (most recent call last): File "test_resize.py", line 10, in <module> print np.resize(V,2) File "/usr/locaL/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/fromnumeric.py", line 1053, in resize if not Na: return mu.zeros(new_shape, a.dtype.char) TypeError: Empty data-type I'm using numpy 1.8.0, python 2.7.6, osx 10.9.1. Can anyone confirm before I submit an issue ? Here is the script: V = np.zeros(0, dtype=np.float32) print V.dtype print np.resize(V,2) V = np.zeros(0, dtype=[('a', np.float32, 1)]) print V.dtype print np.resize(V,2)

1 0

Why do weights in np.polyfit have to be 1D?
by Andreas Hilboll Jan. 10, 2014

Jan. 10, 2014

Hi, in using np.polyfit (in version 1.7.1), I ran accross TypeError: expected a 1-d array for weights when trying to fit k polynomials at once (x.shape = (4, ), y.shape = (4, 136), w.shape = (4, 136)). Is there any specific reason why this is not supported? -- Andreas.

2 1

adding fused multiply and add to numpy
by Julian Taylor Jan. 9, 2014

Jan. 9, 2014

Hi, Since AMDs bulldozer and Intels Haswell x86 cpus now also support the fused-multiply-and-add operation in hardware. http://en.wikipedia.org/wiki/Multiply–accumulate_operation This operation is interesting for numpy for two reasons: - Only one rounding is involved in two arithmetic operations, this is good reducing rounding errors. - Two operations are done in one memory pass, so it improves the performance if ones operations are bound by the memory bandwidth which is very common in numpy. I have done some experiments using a custom ufunc: https://github.com/juliantaylor/npufuncs See the README.md on how to try it out. It requires a recent GCC compiler, at least 4.7 I think. It contains SSE2, AVX, FMA3 (AVX2), FMA4 and software emulation variants. Edit the file to select which one to use. Note if the cpu does not support the instruction it will just crash. Only the latter three are real FMA operations, the SSE2 and AVX variants just perform two regular operations in one loop. My current machine only supports SSE2, so here are the timings for it: In [25]: a = np.arange(500000.) In [26]: b = np.arange(500000.) In [27]: c = np.arange(500000.) In [28]: %timeit npfma.fma(a, b, c) 100 loops, best of 3: 2.49 ms per loop In [30]: def pure_numpy_fma(a,b,c): ....: return a * b + c In [31]: %timeit pure_numpy_fma(a, b, c) 100 loops, best of 3: 7.36 ms per loop In [32]: def pure_numpy_fma2(a,b,c): ....: tmp = a *b ....: tmp += c ....: return tmp In [33]: %timeit pure_numpy_fma2(a, b, c) 100 loops, best of 3: 3.47 ms per loop As you can see even without real hardware support it is about 30% faster than inplace unblocked numpy due better use of memory bandwidth. Its even more than two times faster than unoptimized numpy. If you have a machine capable of fma instructions give it a spin to see if you get similar or better results. Please verify the assembly (objdump -d fma-<suffix>.o) to check if the compiler properly used the machine fma. An issue is software emulation of real fma. This can be enabled in the test ufunc with npfma.set_type("libc"). This is unfortunately incredibly slow about a factor 300 on my machine without hardware fma. This means we either have a function that is fast on some platforms and slow on others but always gives the same result or we have a fast function that gives better results on some platforms. Given that we are not worth that what numpy currently provides I favor the latter. Any opinions on whether this should go into numpy or maybe stay a third party ufunc? Concerning the interface one should probably add several variants mirroring the FMA3 instruction set: http://en.wikipedia.org/wiki/Multiply–accumulate_operation additionally there is fmaddsub (a0 * b0 + c0, a1 *b1 - c1) which can be used for complex numbers, but they probably don't need an explicit numpy interface.

7 10

ENH: add a 'return_counts=' keyword argument to `np.unique`
by Jaime Fernández del Río Jan. 9, 2014

Jan. 9, 2014

Hi, I have just sent a PR, adding a `return_counts` keyword argument to `np.unique` that does exactly what the name suggests: counting the number of times each unique time comes up in the array. It reuses the `flag` array that is constructed whenever any optional index is requested, extracts the indices of the `True`s in it, and returns their diff. You can check it here: https://github.com/numpy/numpy/pull/4180 Regards, Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

1 0