Mailman 3 August 2007 - NumPy-Discussion

comparing arrays with NaN in them.
by Christopher Barker Aug. 24, 2007

Aug. 24, 2007

Hi all, I was just trying to write a unit test for something where I was expecting to get some NaN's in the array. However, since NaN == NaN returns false, the simple test: assert(alltrue(a == b)) >>> a = N.array((1,2,3,N.nan)) >>> b = N.array((1,2,3,N.nan)) >>> a == b array([ True, True, True, False], dtype=bool) >>> assert(N.alltrue(a == b)) Traceback (most recent call last): File "<stdin>", line 1, in <module> AssertionError >>> So is there any way to test is two arrays are the same, when there may be a NaN or two mixed in??? With a bit of thought -- this works: >>> N.alltrue(a[~N.isnan(a)] == b[~N.isnan(b)]) True but that feels like a kludge. maybe some sort of "TheseArrays are binary equal" would be useful. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

9 12

Reference counter of builtin descriptor objects
by Matthias Höffken Aug. 24, 2007

Aug. 24, 2007

Greetings, I struggling with the numpy C-API (version 1.0.3). Now I have obscurities concerning the reference counter of builtin descriptor objects. In some situation, when running my own code, the reference counter fall to zero an I get warning messages. In some other samples the reference counter increases more and more while the program is running and the average number of used object is keeping constant. Now I would like to know when I have to take care about the reference counter of builtin descriptor objects. Especially when using "PyArray_SimpleNew", "PyArray_SimpleNewFromData", "PyArray_NewFromDescr" and "PyArray_SimpleNewFromDescr". Up to now I never touched the counters in my code after using this functions resulting in the described problems. Another case concerns parsing the arguments of functions. I often use such kind of expressions: PyArg_ParseTupleAndKeywords(args, kwds, "O!O!", kwlist, &PyArray_Type, &array1, &PyArray_Type, &array1)) Normally I would expect, that no reference counter is changed. Is that really true? Many thanks in advance! Matthias

1 0

Branch and Tag Maintenance
by Jarrod Millman Aug. 22, 2007

Aug. 22, 2007

Hello, I deleted any old (2+ years since modified) branches and tags. Nothing is actually deleted so if you need to access the old code simply use the relevant revision number with svn checkout, svn switch, or svn list. It is also very easy to restore if you are planning to continue working on some of the code. For example if you need to restore the numarray branch just use: svn copy -r 3988 http://svn.scipy.org/svn/numpy/branches/numarray \ http://svn.scipy.org/svn/numpy/branches/numarray I have attached a text file with the deletions I made and what revision they were committed on. You can also view the changes using the trac site: http://projects.scipy.org/scipy/numpy/timeline Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/

1 0

memory error caused by astype()
by Gong, Shawn (Contractor) Aug. 22, 2007

Aug. 22, 2007

Hi list, When I do large array manipulations, I get out-of-memory errors. If the array size is 5000 by 6000, the following codes use nearly 1G. Then my PC displays a Python error box. The try/except won't catch it if the memory error happens in "astype" instead of "array1* array2" try: if ( array1.typecode() in cplx_types ): array1 = abs(array1.astype(Numeric.Complex32)) else: array1 = array1.astype(Numeric.Float32) if ( array2.typecode() in cplx_types ): array2 = abs(array2.astype(Numeric.Complex32)) else: array2 = array2.astype(Numeric.Float32) array1 = Numeric.sqrt(array1) * Numeric.sqrt(array2) return array1 except: gvutils.error("Memory error occurred\nPlease select a smaller array") return None My questions are: 1) Is there a more memory efficient way of doing this? 2) How do I deal with exception if astype is the only way to go 3) Is there a way in Python that detects the available RAM and limits the array size before he/she can go ahead with the array multiplications? i.e. detects the available RAM, say 800K Assume worst case - Complex32 Figure out how many temp_arrays used by numpy Calculate array size limit = ?? 4) If there is no 3) Is there something in Python that monitors memory and warns the user. I have these "astype" at a number functions. Do I have to put try/except at each location? Thanks, Shaw Gong

1 1

Non-contiguous array from newaxis indexing
by Jens Jørgen Mortensen Aug. 22, 2007

Aug. 22, 2007

I would like all these arrays to be contiguous: >>> import numpy as npy >>> npy.__version__ '1.0.4.dev3967' >>> x = npy.arange(4) >>> y = x[npy.newaxis, :] >>> z = x.reshape((1, 4)) >>> for a in [x, y, z]: ... print a.shape, a.strides, a.flags.contiguous ... (4,) (4,) True (1, 4) (0, 4) False (1, 4) (16, 4) True But y is not contiguous according to y.flags.contiguous - why not and why does y and z not have the same strides? I found this comment just before the _IsContiguous function in arrayobject.c: /* 0-strided arrays are not contiguous (even if dimension == 1) */ Is this correct? Jens Jørgen Mortensen

2 2

Setting numpy record array elements
by Sameer DCosta Aug. 21, 2007

Aug. 21, 2007

Hi, In the example below I have a record array *a* that has a column *col1". I am trying to set the first element of a.col1 to zero in two different ways. 1. a[0].col1 = 0 (This fails silently) 2. a.col1[0] = 0 (This works fine) I am using the latest svn version of numpy. Is this a bug? or is the first method supposed to fail? If it is supposed to fail should it fail silently? Thanks in advance for any replies. Cheers, Sameer %%%% Example Code %%% import numpy print numpy.__version__ a = numpy.rec.fromrecords( [ (1,2,3), (4,5,6)], dtype=[("col1", "<i4"), ("col2", "<i4"), ("col3", "<i4")]) print "a = %s" % a.__repr__() print "First element of col1 is %ld" % a.col1[0] print "Setting first element of col1 to zero using a[0].col1 = 0" a[0].col1 = 0 print "Result: a.col1[0] = %ld" % a[0].col1 print "Setting first element of col1 to zero using a.col1[0] = 0" a.col1[0] = 0 print "Result: a.col1[0] = %ld" % a[0].col1 %%%% Example Code Output %%% 1.0.4.dev3975 a = recarray([(1, 2, 3), (4, 5, 6)], dtype=[('col1', '<i4'), ('col2', '<i4'), ('col3', '<i4')]) First element of col1 is 1 Setting first element of col1 to zero using a[0].col1 = 0 Result: a.col1[0] = 1 Setting first element of col1 to zero using a.col1[0] = 0 Result: a.col1[0] = 0

2 3

NumPy 1.0.3.1 released
by Jarrod Millman Aug. 21, 2007

Aug. 21, 2007

I'm pleased to announce the release of NumPy 1.0.3.1 This a minor bug fix release, which enables the latest release of SciPy to build. Bug-fixes =============== * Add back get_path to numpy.distutils.misc_utils * Fix 64-bit zgeqrf * Add parenthesis around GETPTR macros Thank you to everybody who contributed to the recent release. Best regards, NumPy Developers http://numpy.scipy.org

1 0

Error in allclose with inf values
by Matthew Brett Aug. 20, 2007

Aug. 20, 2007

Hi, I noticed that allclose does not always behave correctly for arrays with infs. I've attached a test script for allclose, and here's an alternative implementation that I believe behaves correctly. Obviously the test script could be a test case in core/tests/test_numeric.py I wonder if we should allow nans in the test arrays - possibly with an optional keyword arg like allownan. After all inf-inf is nan - but we allow that in the test. Best, Matthew

2 2

fast putmask implementation
by Eric Firing Aug. 20, 2007

Aug. 20, 2007

In looking at maskedarray performance, I found that the filled() function or method is a bottleneck. I think it can be sped up by using putmask instead of indexed assignment, but I found that putmask itself is slower than it needs to be. So I followed David Cournapeau's example of fastclip and made a similar fastputmask. The diff relative to current svn (3967) is attached. The faster version makes a factor-of-ten or larger improvement in putmask speed. numpy.test() still passes. With 10000-element integer arrays the new version reduces the times from 136 to 15 usec for 1000 masked elements, and 445 to 18 usec for 5000 masked elements, with a scalar value argument. It is only slightly slower with an array value argument. (Times are for Intel Core2, 2 GH, linux.) I hope someone will take a look and either tell me what I need to fix or commit it as-is. Thanks. Eric

6 8

selecting part of an array like a[ a<5 ]
by mark Aug. 20, 2007

Aug. 20, 2007

Hello - I am wondering what the better way is to select part of an array. Say I have an array a: a = arange(10) Now I want to select the values larger than 5 a[ a>5 ] and later I need the values smaller or equal to 5 a[ a<=5 ] It seems that doing the comparison twice is extra work (especially if the array is large). So I thought I store the comparison b = a>5 Now I can do a[b] But how do I get the others? a[not b] or a[!b] don't work. So it's gotta be something different. Besides, is it a good idea to store b like I suggest? Thanks for the help, Mark

2 1