Mailman 3 July 2010 - NumPy-Discussion

Help w/ indexing, please
by David Goldsmith July 27, 2010

July 27, 2010

Hi! I have a large M x K, M, K ~ 1e3 array L of indices - non-negative integers in the range 0 to N-1 - and an N x 3 array C (a matplotlib colormap). I need to create an M x K x 3 array R such that R[m,k,j] = C[L[m,k], j], j = 0,1,2. I want to do so w/out having to loop through all the (m,k) index pairs, but I'm having trouble wrapping my brain around how to do it - please help. Thanks. DG

2 2

atleast_3d() strips subclass info
by Benjamin Root July 27, 2010

July 27, 2010

I have filed ticket #1560: http://projects.scipy.org/numpy/ticket/1560 The .atleast_3d() function can not be used with arrays that are subclassed from ndarray(). Because the function uses .asarray() instead of .asanyarray(), subclass info such as masks for masked arrays are stripped from the results. See the following example: >>> a = numpy.ma.masked_array([0.0, 1.2, 3.5], mask=[False, True, False]) >>> b = numpy.atleast_3d(a) >>> b array([[[ 0. ], [ 1.2], [ 3.5]]]) >>> a masked_array(data = [0.0 -- 3.5], mask = [False True False], fill_value = 1e+20) >>> Note that the .atleast_1d() and .atleast_2d() functions are safe for subclassed arrays. I believe that simply replacing .asarray() with .asanyarray() should fix the issue. This ticket has a proposed patch that replaces .asarray() with .asanyarray(). Ben Root

1 0

atleast_1d and atleast2d breaks masked arrays
by Benjamin Root July 27, 2010

July 27, 2010

I have filed ticket #1559: http://projects.scipy.org/numpy/ticket/1559 Calling a function like .atleast_2d() to change the number of dimensions an array has can break the original masked array object. See the following example using a 1d masked array: >>> import numpy >>> a = numpy.ma.masked_array([0.0, 1.2, 3.5], mask=[False, True, False]) >>> b = numpy.atleast_2d(a) >>> b masked_array(data = [[0.0 -- 3.5]], mask = [[False True False]], fill_value = 1e+20) >>> a Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/bvr/Programs/numpy/numpy/ma/core.py", line 3570, in __repr__ data=str(self), mask=str(self._mask), File "/home/bvr/Programs/numpy/numpy/ma/core.py", line 3554, in __str__ res[m] = f ValueError: boolean index array should have 1 dimension >>> The problem does not occur if there is no change to the number of dimensions. Also note that this does not appear to occur with atleast_3d(), although it does have a different problem (covered in a separate email and ticket #1560). I have included a patch, but I merely imitated atleast_3d()'s approach, and I wonder if there is a better way to go about this. Ben Root

1 0

loadtxt() behavior on single-line files
by Benjamin Root July 27, 2010

July 27, 2010

Hi, I was having the hardest time trying to figure out an intermittent bug in one of my programs. Essentially, in some situations, it was throwing an error saying that the array object was not an array. It took me a while, but then I figured out that my program was assuming that the object returned from a loadtxt() call was always a structured array (I was using dtypes). However, if the data file being loaded only had one data record, then all you get back is a structured record. import numpy as np from StringIO import StringIO strData = StringIO("89.23 47.2\n13.2 42.2") a = np.loadtxt(strData, dtype=[('x', float), ('y', float)]) print "Length Two" print a print a.shape print len(a) strData = StringIO("53.2 49.2") a = np.loadtxt(strData, dtype=[('x', float), ('y', float)]) print "\n\nLength One" print a print a.shape try : print len(a) except TypeError as err print "ERROR:", err Which gets me this output: Length Two [(89.230000000000004, 47.200000000000003) (13.199999999999999, 42.200000000000003)] (2,) 2 Length One (53.200000000000003, 49.200000000000003) () ERROR: len() of unsized object Note that this isn't restricted to structured arrays. For regular ndarrays, loadtxt() appears to mimic the behavior of np.squeeze(): >>> a = np.ones((1, 1, 1)) >>> np.squeeze(a)[0] IndexError: 0-d arrays can't be indexed >>> strData = StringIO("53.2") >>> a = np.loadtxt(strData) >>> a[0] IndexError: 0-d arrays can't be indexed So, if you have multiple lines with multiple columns, you get a 2-D array, as expected. if you have a single line of data with multiple columns, you get a 1-D array. If you have a single column with many lines, you also get a 1-D array (which is probably expected, I guess). If you have a single column with a single line, you get a scalar (actually, a 0-D array). Is this a bug or a feature? I can see the advantages of having loadtxt() returning the lowest # of dimensions that can hold the given data, but it leaves the code vulnerable to certain edge cases. Maybe there is a different way I should be doing this, but I feel that this behavior at the very least should be included in the loadtxt documentation. Ben Root

3 4

ANN: SciPy 0.8.0
by Ralf Gommers July 27, 2010

July 27, 2010

I'm pleased to announce the release of SciPy 0.8.0. SciPy is a package of tools for science and engineering for Python. It includes modules for statistics, optimization, integration, linear algebra, Fourier transforms, signal and image processing, ODE solvers, and more. This release comes one and a half year after the 0.7.0 release and contains many new features, numerous bug-fixes, improved test coverage, and better documentation. Please note that SciPy 0.8.0 requires Python 2.4-2.6 and NumPy 1.4.1 or greater. For more information, please see the release notes at the end of this email. You can download the release from here: https://sourceforge.net/projects/scipy/ Python 2.5/2.6 binaries for Windows and OS X are available, as well as source tarballs for other platforms and the documentation in pdf form. Thank you to everybody who contributed to this release. Enjoy, The SciPy developers ========================= SciPy 0.8.0 Release Notes ========================= .. contents:: SciPy 0.8.0 is the culmination of 17 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. There have been a number of deprecations and API changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on the 0.8.x branch, and on adding new features on the development trunk. This release requires Python 2.4 - 2.6 and NumPy 1.4.1 or greater. Please note that SciPy is still considered to have "Beta" status, as we work toward a SciPy 1.0.0 release. The 1.0.0 release will mark a major milestone in the development of SciPy, after which changing the package structure or API will be much more difficult. Whilst these pre-1.0 releases are considered to have "Beta" status, we are committed to making them as bug-free as possible. However, until the 1.0 release, we are aggressively reviewing and refining the functionality, organization, and interface. This is being done in an effort to make the package as coherent, intuitive, and useful as possible. To achieve this, we need help from the community of users. Specifically, we need feedback regarding all aspects of the project - everything - from which algorithms we implement, to details about our function's call signatures. Python 3 ======== Python 3 compatibility is planned and is currently technically feasible, since Numpy has been ported. However, since the Python 3 compatible Numpy 1.5 has not been released yet, support for Python 3 in Scipy is not yet included in Scipy 0.8. SciPy 0.9, planned for fall 2010, will very likely include experimental support for Python 3. Major documentation improvements ================================ SciPy documentation is greatly improved. Deprecated features =================== Swapping inputs for correlation functions (scipy.signal) -------------------------------------------------------- Concern correlate, correlate2d, convolve and convolve2d. If the second input is larger than the first input, the inputs are swapped before calling the underlying computation routine. This behavior is deprecated, and will be removed in scipy 0.9.0. Obsolete code deprecated (scipy.misc) ------------------------------------- The modules `helpmod`, `ppimport` and `pexec` from `scipy.misc` are deprecated. They will be removed from SciPy in version 0.9. Additional deprecations ----------------------- * linalg: The function `solveh_banded` currently returns a tuple containing the Cholesky factorization and the solution to the linear system. In SciPy 0.9, the return value will be just the solution. * The function `constants.codata.find` will generate a DeprecationWarning. In Scipy version 0.8.0, the keyword argument 'disp' was added to the function, with the default value 'True'. In 0.9.0, the default will be 'False'. * The `qshape` keyword argument of `signal.chirp` is deprecated. Use the argument `vertex_zero` instead. * Passing the coefficients of a polynomial as the argument `f0` to `signal.chirp` is deprecated. Use the function `signal.sweep_poly` instead. * The `io.recaster` module has been deprecated and will be removed in 0.9.0. New features ============ DCT support (scipy.fftpack) --------------------------- New realtransforms have been added, namely dct and idct for Discrete Cosine Transform; type I, II and III are available. Single precision support for fft functions (scipy.fftpack) ---------------------------------------------------------- fft functions can now handle single precision inputs as well: fft(x) will return a single precision array if x is single precision. At the moment, for FFT sizes that are not composites of 2, 3, and 5, the transform is computed internally in double precision to avoid rounding error in FFTPACK. Correlation functions now implement the usual definition (scipy.signal) ----------------------------------------------------------------------- The outputs should now correspond to their matlab and R counterparts, and do what most people expect if the old_behavior=False argument is passed: * correlate, convolve and their 2d counterparts do not swap their inputs depending on their relative shape anymore; * correlation functions now conjugate their second argument while computing the slided sum-products, which correspond to the usual definition of correlation. Additions and modification to LTI functions (scipy.signal) ---------------------------------------------------------- * The functions `impulse2` and `step2` were added to `scipy.signal`. They use the function `scipy.signal.lsim2` to compute the impulse and step response of a system, respectively. * The function `scipy.signal.lsim2` was changed to pass any additional keyword arguments to the ODE solver. Improved waveform generators (scipy.signal) ------------------------------------------- Several improvements to the `chirp` function in `scipy.signal` were made: * The waveform generated when `method="logarithmic"` was corrected; it now generates a waveform that is also known as an "exponential" or "geometric" chirp. (See http://en.wikipedia.org/wiki/Chirp.) * A new `chirp` method, "hyperbolic", was added. * Instead of the keyword `qshape`, `chirp` now uses the keyword `vertex_zero`, a boolean. * `chirp` no longer handles an arbitrary polynomial. This functionality has been moved to a new function, `sweep_poly`. A new function, `sweep_poly`, was added. New functions and other changes in scipy.linalg ----------------------------------------------- The functions `cho_solve_banded`, `circulant`, `companion`, `hadamard` and `leslie` were added to `scipy.linalg`. The function `block_diag` was enhanced to accept scalar and 1D arguments, along with the usual 2D arguments. New function and changes in scipy.optimize ------------------------------------------ The `curve_fit` function has been added; it takes a function and uses non-linear least squares to fit that to the provided data. The `leastsq` and `fsolve` functions now return an array of size one instead of a scalar when solving for a single parameter. New sparse least squares solver ------------------------------- The `lsqr` function was added to `scipy.sparse`. `This routine <http://www.stanford.edu/group/SOL/software/lsqr.html>`_ finds a least-squares solution to a large, sparse, linear system of equations. ARPACK-based sparse SVD ----------------------- A naive implementation of SVD for sparse matrices is available in scipy.sparse.linalg.eigen.arpack. It is based on using an symmetric solver on <A, A>, and as such may not be very precise. Alternative behavior available for `scipy.constants.find` --------------------------------------------------------- The keyword argument `disp` was added to the function `scipy.constants.find`, with the default value `True`. When `disp` is `True`, the behavior is the same as in Scipy version 0.7. When `False`, the function returns the list of keys instead of printing them. (In SciPy version 0.9, the default will be reversed.) Incomplete sparse LU decompositions ----------------------------------- Scipy now wraps SuperLU version 4.0, which supports incomplete sparse LU decompositions. These can be accessed via `scipy.sparse.linalg.spilu`. Upgrade to SuperLU 4.0 also fixes some known bugs. Faster matlab file reader and default behavior change ------------------------------------------------------ We've rewritten the matlab file reader in Cython and it should now read matlab files at around the same speed that Matlab does. The reader reads matlab named and anonymous functions, but it can't write them. Until scipy 0.8.0 we have returned arrays of matlab structs as numpy object arrays, where the objects have attributes named for the struct fields. As of 0.8.0, we return matlab structs as numpy structured arrays. You can get the older behavior by using the optional ``struct_as_record=False`` keyword argument to `scipy.io.loadmat` and friends. There is an inconsistency in the matlab file writer, in that it writes numpy 1D arrays as column vectors in matlab 5 files, and row vectors in matlab 4 files. We will change this in the next version, so both write row vectors. There is a `FutureWarning` when calling the writer to warn of this change; for now we suggest using the ``oned_as='row'`` keyword argument to `scipy.io.savemat` and friends. Faster evaluation of orthogonal polynomials ------------------------------------------- Values of orthogonal polynomials can be evaluated with new vectorized functions in `scipy.special`: `eval_legendre`, `eval_chebyt`, `eval_chebyu`, `eval_chebyc`, `eval_chebys`, `eval_jacobi`, `eval_laguerre`, `eval_genlaguerre`, `eval_hermite`, `eval_hermitenorm`, `eval_gegenbauer`, `eval_sh_legendre`, `eval_sh_chebyt`, `eval_sh_chebyu`, `eval_sh_jacobi`. This is faster than constructing the full coefficient representation of the polynomials, which was previously the only available way. Note that the previous orthogonal polynomial routines will now also invoke this feature, when possible. Lambert W function ------------------ `scipy.special.lambertw` can now be used for evaluating the Lambert W function. Improved hypergeometric 2F1 function ------------------------------------ Implementation of `scipy.special.hyp2f1` for real parameters was revised. The new version should produce accurate values for all real parameters. More flexible interface for Radial basis function interpolation --------------------------------------------------------------- The `scipy.interpolate.Rbf` class now accepts a callable as input for the "function" argument, in addition to the built-in radial basis functions which can be selected with a string argument. Removed features ================ scipy.stsci: the package was removed The module `scipy.misc.limits` was removed. scipy.io -------- The IO code in both NumPy and SciPy is being extensively reworked. NumPy will be where basic code for reading and writing NumPy arrays is located, while SciPy will house file readers and writers for various data formats (data, audio, video, images, matlab, etc.). Several functions in `scipy.io` are removed in the 0.8.0 release including: `npfile`, `save`, `load`, `create_module`, `create_shelf`, `objload`, `objsave`, `fopen`, `read_array`, `write_array`, `fread`, `fwrite`, `bswap`, `packbits`, `unpackbits`, and `convert_objectarray`. Some of these functions have been replaced by NumPy's raw reading and writing capabilities, memory-mapping capabilities, or array methods. Others have been moved from SciPy to NumPy, since basic array reading and writing capability is now handled by NumPy.

1 0

determinant of a scalar not handled
by Skipper Seabold July 27, 2010

July 27, 2010

Right now np.linalg.det does not handle scalars or 1d (scalar) arrays. It first tests that an array is 2d, then if it is square. This seems redundant to me. This currently works In [20]: np.linalg.det([[1]]) Out[20]: 1.0 but In [21]: np.linalg.det([1]) <snip> LinAlgError: 1-dimensional array given. Array must be two-dimensional The diff attached to this ticket <http://projects.scipy.org/numpy/ticket/1556> gives In [3]: np.linalg.det([[1]]) Out[3]: 1.0 In [4]: np.linalg.det([1]) Out[4]: 1.0 In [5]: np.linalg.det(1) Out[5]: 1.0 Skipper

5 15

numpydoc broken with sphinx 1.0
by Gael Varoquaux July 27, 2010

July 27, 2010

Hi there, I just wanted to mention that numpydoc is broken with the latest sphinx release. I had a quick look, but could find how to solve the problem, so I am just pointing it out here: The trace is: # Sphinx version: 1.0b2+/3fed33465259 # Docutils version: 0.6 release # Jinja2 version: 2.3.1 Traceback (most recent call last): File "/home/varoquau/dev/sphinx/sphinx/cmdline.py", line 172, in main warningiserror, tags) File "/home/varoquau/dev/sphinx/sphinx/application.py", line 116, in __init__ self.setup_extension(extension) File "/home/varoquau/dev/sphinx/sphinx/application.py", line 253, in setup_extension mod.setup(self) File "/home/varoquau/dev/scikit-learn/doc/sphinxext/numpydoc.py", line 120, in setup app.add_directive('np-' + name, wrap_mangling_directive(name, objtype)) File "/home/varoquau/dev/scikit-learn/doc/sphinxext/numpydoc.py", line 140, in wrap_mangling_directive base_directive = get_directive(base_directive_name) File "/home/varoquau/dev/scikit-learn/doc/sphinxext/numpydoc.py", line 137, in get_directive raise RuntimeError("No directive named '%s' found" % name) RuntimeError: No directive named 'function' found Indeed, there is no 'function' in the directives._directives dictionnary.. Gaël

2 3

size_t or npy_intp?
by Francesc Alted July 27, 2010

July 27, 2010

Hi, I'm a bit confused on which datatype should I use when referring to NumPy ndarray lengths. In one hand I'd use `size_t` that is the canonical way to refer to lengths of memory blocks. In the other hand, `npy_intp` seems the standard data type used in NumPy for this. Which one would you recommend to use in NumPy extensions? -- Francesc Alted

5 8

np.fromstring and Python 3
by Thomas Robitaille July 26, 2010

July 26, 2010

Hi, The following example illustrates a problem I'm encountering a problem with the np.fromstring function in Python 3: Python 3.1.2 (r312:79360M, Mar 24 2010, 01:33:18) [GCC 4.0.1 (Apple Inc. build 5493)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> string = "".join(chr(i) for i in range(256)) >>> a = np.fromstring(string, dtype=np.int8) >>> print(len(string)) 256 >>> print(len(a)) 384 The array 'a' should have the same size as 'string' since I'm using a 1-byte datatype. Is this a bug, or do I need to change the way I use this function in Python 3? I am using Numpy r8523 Cheers, Tom

2 2

Could numpy.fromfile() work with gzip-compressed files ?
by bruno Piguet July 26, 2010

July 26, 2010

Hi all, I've got some code which basically does : f = open (path, 'rb') header = f.read (some_length) data = np.fromfile (f, dtype= some_type, count=-1) In order to process compressed files, I switched the open sequence to : if (plain): f = open (path, 'rb') else: f = gzip.open (path, 'rb') Then, the header reading is OK, since the gzip.file class has a "read " method : hdr = f.read (some_length) But I've got problems with np.fromfile, which says : IOError: first argument must be an open file I guess that fromfile() uses methods other than file.read(), while GzipFile does not simulates all the methods of a file object (readinto() and truncate() are listed the doc), so making fromfile() work with compressed file may not be straightforward. I had a try at StringIO() : data = np.fromfile (StringIO.StringIO(f.read()), dtype= some_type, count=-1) but it doesn't work better. So, I've got two questions : - Could numpy.fromfile() work with gzip-compressed files, as some other file functions do ? - I'm thinking of using something like np.array(f.read(), dtype=some_type). Any better solution ? With such a simple solution, maybe my error was to try to use fromfile() at the beginning ? Bruno.

1 1