Mailman 3 December 2009 - NumPy-Discussion

[ANN] numpy 1.4.0 rc2
by David Cournapeau Dec. 29, 2009

Dec. 29, 2009

Hi, I have just released the 2nd release candidate for numpy 1.4.0, which fixes a few critical bugs founds since the RC1. Tarballs and binary installers for numpy/scipy may be found on https://sourceforge.net/projects/numpy. cheers, David

6 25

Re: [Numpy-discussion] [matplotlib-devel] [SciPy-dev] Announcing toydist, improving distribution and packaging situation
by Gael Varoquaux Dec. 29, 2009

Dec. 29, 2009

On Mon, Dec 28, 2009 at 02:29:24PM -0500, Neal Becker wrote: > Perhaps this could be useful: > http://checkinstall.izto.org/ Yes, checkinstall is really cool. However, I tend to prefer things with no magic that I don't have to sandbox to know what they are doing. This is why I am also happy to hear about toydist. Gaël

2 1

[ANN] PyDSTool 0.88 -- dynamical systems modeling tools
by Rob Clewley Dec. 28, 2009

Dec. 28, 2009

A new release of the dynamical systems modeling toolbox PyDSTool is available from Sourceforge: http://www.sourceforge.net/projects/pydstool/ Highlights from the release notes: * Cleanup of global imports, especially: entire numpy.random and linalg namespaces no longer imported by default * Added support for 'min' and 'max' keywords in functional specifications (for ODE right-hand sides, for instance) * Optimization tools from third-party genericOpt (included with permission) and improved parameter estimation examples making use of this code * Numerical phase-response calculations now possible in PRC toolbox * Fully-fledged DSSRT toolbox for neural modeling (see wiki page) * New tests/demonstrations in PyDSTool/tests * Major improvements to intelligent expr2func (symbolic -> python function conversion) * Improved compatibility with cross-platform use and with recent python versions and associated libraries * Added many minor features (see timeline on Trac http://jay.cam.cornell.edu/pydstool/timeline) * Fixed many bugs and quirks (see timeline on Trac http://jay.cam.cornell.edu/pydstool/timeline) This is mainly a bugfix release in preparation for a substantial upgrade at version 0.90, which will have a proper installer, unit testing, symbolic expression support via SymPy, and greatly improved interfacing to legacy ODE integrators. These features are being actively developed in 2009/2010. For installation and setting up, please carefully read the GettingStarted page at our wiki for platform-specific details: http://pydstool.sourceforge.net Please use the bug tracker and user discussion list at Sourceforge to report bugs or provide feedback. Code and documentation contributions are always welcome. Regards, Rob Clewley

1 0

[ANN] Numpy 1.4.0 release
by David Cournapeau Dec. 28, 2009

Dec. 28, 2009

Hi, I am pleased to announce the release of numpy 1.4.0. The highlights of this release are: - Faster import time - Extended array wrapping mechanism for ufuncs - New Neighborhood iterator (C-level only) - C99-like complex functions in npymath, and a lot of portability fixes for basic floating point math functions The full release notes are at the end of the email. The sources are uploaded on Pypi, and the binary installers will soon come on the sourceforge page: https://sourceforge.net/projects/numpy/ Thank you to everyone involved in this release, developers, users who reported bugs, fix documentation, etc... enjoy, the numpy developers. ========================= NumPy 1.4.0 Release Notes ========================= This minor includes numerous bug fixes, as well as a few new features. It is backward compatible with 1.3.0 release. Highlights ========== * Faster import time * Extended array wrapping mechanism for ufuncs * New Neighborhood iterator (C-level only) * C99-like complex functions in npymath New features ============ Extended array wrapping mechanism for ufuncs ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ An __array_prepare__ method has been added to ndarray to provide subclasses greater flexibility to interact with ufuncs and ufunc-like functions. ndarray already provided __array_wrap__, which allowed subclasses to set the array type for the result and populate metadata on the way out of the ufunc (as seen in the implementation of MaskedArray). For some applications it is necessary to provide checks and populate metadata *on the way in*. __array_prepare__ is therefore called just after the ufunc has initialized the output array but before computing the results and populating it. This way, checks can be made and errors raised before operations which may modify data in place. Automatic detection of forward incompatibilities ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Previously, if an extension was built against a version N of NumPy, and used on a system with NumPy M < N, the import_array was successfull, which could cause crashes because the version M does not have a function in N. Starting from NumPy 1.4.0, this will cause a failure in import_array, so the error will be catched early on. New iterators ~~~~~~~~~~~~~ A new neighborhood iterator has been added to the C API. It can be used to iterate over the items in a neighborhood of an array, and can handle boundaries conditions automatically. Zero and one padding are available, as well as arbitrary constant value, mirror and circular padding. New polynomial support ~~~~~~~~~~~~~~~~~~~~~~ New modules chebyshev and polynomial have been added. The new polynomial module is not compatible with the current polynomial support in numpy, but is much like the new chebyshev module. The most noticeable difference to most will be that coefficients are specified from low to high power, that the low level functions do *not* work with the Chebyshev and Polynomial classes as arguements, and that the Chebyshev and Polynomial classes include a domain. Mapping between domains is a linear substitution and the two classes can be converted one to the other, allowing, for instance, a Chebyshev series in one domain to be expanded as a polynomial in another domain. The new classes should generally be used instead of the low level functions, the latter are provided for those who wish to build their own classes. The new modules are not automatically imported into the numpy namespace, they must be explicitly brought in with an "import numpy.polynomial" statement. New C API ~~~~~~~~~ The following C functions have been added to the C API: #. PyArray_GetNDArrayCFeatureVersion: return the *API* version of the loaded numpy. #. PyArray_Correlate2 - like PyArray_Correlate, but implements the usual definition of correlation. Inputs are not swapped, and conjugate is taken for complex arrays. #. PyArray_NeighborhoodIterNew - a new iterator to iterate over a neighborhood of a point, with automatic boundaries handling. It is documented in the iterators section of the C-API reference, and you can find some examples in the multiarray_test.c.src file in numpy.core. New ufuncs ~~~~~~~~~~ The following ufuncs have been added to the C API: #. copysign - return the value of the first argument with the sign copied from the second argument. #. nextafter - return the next representable floating point value of the first argument toward the second argument. New defines ~~~~~~~~~~~ The alpha processor is now defined and available in numpy/npy_cpu.h. The failed detection of the PARISC processor has been fixed. The defines are: #. NPY_CPU_HPPA: PARISC #. NPY_CPU_ALPHA: Alpha Testing ~~~~~~~ #. deprecated decorator: this decorator may be used to avoid cluttering testing output while testing DeprecationWarning is effectively raised by the decorated test. #. assert_array_almost_equal_nulps: new method to compare two arrays of floating point values. With this function, two values are considered close if there are not many representable floating point values in between, thus being more robust than assert_array_almost_equal when the values fluctuate a lot. #. assert_array_max_ulp: raise an assertion if there are more than N representable numbers between two floating point values. #. assert_warns: raise an AssertionError if a callable does not generate a warning of the appropriate class, without altering the warning state. Reusing npymath ~~~~~~~~~~~~~~~ In 1.3.0, we started putting portable C math routines in npymath library, so that people can use those to write portable extensions. Unfortunately, it was not possible to easily link against this library: in 1.4.0, support has been added to numpy.distutils so that 3rd party can reuse this library. See coremath documentation for more information. Improved set operations ~~~~~~~~~~~~~~~~~~~~~~~ In previous versions of NumPy some set functions (intersect1d, setxor1d, setdiff1d and setmember1d) could return incorrect results if the input arrays contained duplicate items. These now work correctly for input arrays with duplicates. setmember1d has been renamed to in1d, as with the change to accept arrays with duplicates it is no longer a set operation, and is conceptually similar to an elementwise version of the Python operator 'in'. All of these functions now accept the boolean keyword assume_unique. This is False by default, but can be set True if the input arrays are known not to contain duplicates, which can increase the functions' execution speed. Improvements ============ #. numpy import is noticeably faster (from 20 to 30 % depending on the platform and computer) #. The sort functions now sort nans to the end. * Real sort order is [R, nan] * Complex sort order is [R + Rj, R + nanj, nan + Rj, nan + nanj] Complex numbers with the same nan placements are sorted according to the non-nan part if it exists. #. The type comparison functions have been made consistent with the new sort order of nans. Searchsorted now works with sorted arrays containing nan values. #. Complex division has been made more resistent to overflow. #. Complex floor division has been made more resistent to overflow. Deprecations ============ The following functions are deprecated: #. correlate: it takes a new keyword argument old_behavior. When True (the default), it returns the same result as before. When False, compute the conventional correlation, and take the conjugate for complex arrays. The old behavior will be removed in NumPy 1.5, and raises a DeprecationWarning in 1.4. #. unique1d: use unique instead. unique1d raises a deprecation warning in 1.4, and will be removed in 1.5. #. intersect1d_nu: use intersect1d instead. intersect1d_nu raises a deprecation warning in 1.4, and will be removed in 1.5. #. setmember1d: use in1d instead. setmember1d raises a deprecation warning in 1.4, and will be removed in 1.5. The following raise errors: #. When operating on 0-d arrays, ``numpy.max`` and other functions accept only ``axis=0``, ``axis=-1`` and ``axis=None``. Using an out-of-bounds axes is an indication of a bug, so Numpy raises an error for these cases now. #. Specifying ``axis > MAX_DIMS`` is no longer allowed; Numpy raises now an error instead of behaving similarly as for ``axis=None``. Internal changes ================ Use C99 complex functions when available ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The numpy complex types are now guaranteed to be ABI compatible with C99 complex type, if availble on the platform. Moreoever, the complex ufunc now use the platform C99 functions intead of our own. split multiarray and umath source code ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The source code of multiarray and umath has been split into separate logic compilation units. This should make the source code more amenable for newcomers. Separate compilation ~~~~~~~~~~~~~~~~~~~~ By default, every file of multiarray (and umath) is merged into one for compilation as was the case before, but if NPY_SEPARATE_COMPILATION env variable is set to a non-negative value, experimental individual compilation of each file is enabled. This makes the compile/debug cycle much faster when working on core numpy. Separate core math library ~~~~~~~~~~~~~~~~~~~~~~~~~~ New functions which have been added: * npy_copysign * npy_nextafter * npy_cpack * npy_creal * npy_cimag * npy_cabs * npy_cexp * npy_clog * npy_cpow * npy_csqr * npy_ccos * npy_csin

2 3

Re: [Numpy-discussion] [SciPy-dev] [matplotlib-devel] Announcing toydist, improving distribution and packaging situation
by David Cournapeau Dec. 28, 2009

Dec. 28, 2009

On Tue, Dec 29, 2009 at 3:03 AM, Neal Becker <ndbecker2(a)gmail.com> wrote: > David Cournapeau wrote: > >> On Mon, Dec 28, 2009 at 11:47 PM, Stefan Schwarzburg >> <stefan.schwarzburg(a)googlemail.com> wrote: >>> Hi, >>> I would like to add a comment from the user perspective: >>> >>> - the main reason why I'm not satisfied with pypi/distutils/etc. and why >>> I will not be satisfied with toydist (with the features you listed), is >>> that they break my installation (debian/ubuntu). >> >> Toydist (or distutils) does not break anything as is. It would be like >> saying make breaks debian - it does not make much sense. As stated, >> one of the goal of giving up distutils is to make packaging by os >> vendors easier. In particular, by allowing to follow the FHS, and >> making things more consistent. It should be possible to automatically >> convert most packages to .deb (or .rpm) relatively easily. When you >> look at the numpy .deb package, most of the issues are distutils >> issues, and almost everything else can be done automatically. >> >> Note that even ignoring the windows problem, there are systems to do >> the kind of things I am talking about for linux-only systems (the >> opensuse build service), because distributions are not always really >> good at tracking fast changing softwares. IOW, traditional linux >> packaging has some issues as well. And anyway, nothing prevents debian >> or other OS vendors to package things as they want (as they do for R >> packages). >> >> David > > I think the breakage that is referred to I can describe on my favorite > system, fedora. > > I can install the fedora numpy rpm using yum. I could also use > easy_install. Unfortunately: > 1) Each one knows nothing about the other > 2) They may install things into conflicting paths. In particular, on fedora > arch-dependent things go in /usr/lib64/python<version>/site-packages while > arch-independent goes into /usr/lib/python<version>... If you mix yum with > easy_install (or setuptools), you many times wind up with 2 versions and a > lot of confusion. > > This is NOT unusual. Let's say I have numpy-1.3.0 installed from rpms. I > see the announcement of numpy-1.4.0, and decide I want it, before the rpm is > available, so I use easy_install. Now numpy-1.4.0 shows up as a standard > rpm, and a subsequent update (which could be automatic!) could produce a > broken system. Several points: - First, this is caused by distutils misfeature of defaulting to /usr. This is a mistake. It should default to /usr/local, as does every other install method from sources. - A lot of instructions start by sudo easy_install... This is a very bad advice, especially given the previous issue. > I don't really know what could be done about it. Perhaps a design that > attempts to use native backends for installation where available? The idea would be that for a few major distributions at least, you would have .rpm available on the repository. If you install from sources, there would be a few mechanisms to avoid your exact issue (like maybe defaulting to --user kind of installs). Of course, it can only be dealt up to a point. David

1 0

Re: [Numpy-discussion] [matplotlib-devel] Announcing toydist, improving distribution and packaging situation
by David Cournapeau Dec. 28, 2009

Dec. 28, 2009

On Mon, Dec 28, 2009 at 11:47 PM, Stefan Schwarzburg <stefan.schwarzburg(a)googlemail.com> wrote: > Hi, > I would like to add a comment from the user perspective: > > - the main reason why I'm not satisfied with pypi/distutils/etc. and why I > will not be satisfied with toydist (with the features you listed), is that > they break my installation (debian/ubuntu). Toydist (or distutils) does not break anything as is. It would be like saying make breaks debian - it does not make much sense. As stated, one of the goal of giving up distutils is to make packaging by os vendors easier. In particular, by allowing to follow the FHS, and making things more consistent. It should be possible to automatically convert most packages to .deb (or .rpm) relatively easily. When you look at the numpy .deb package, most of the issues are distutils issues, and almost everything else can be done automatically. Note that even ignoring the windows problem, there are systems to do the kind of things I am talking about for linux-only systems (the opensuse build service), because distributions are not always really good at tracking fast changing softwares. IOW, traditional linux packaging has some issues as well. And anyway, nothing prevents debian or other OS vendors to package things as they want (as they do for R packages). David

1 0

Error for numpy.test() with doctest environment configs
by Matthew Brett Dec. 28, 2009

Dec. 28, 2009

Hi, Because I like doctests, I have the following set in my .noserc file: with-doctest=1 This setting breaks numpy.test() like this: In [2]: numpy.test() Running unit tests for numpy NumPy version 1.5.0.dev8029 NumPy is installed in /Users/mb312/usr/local/lib/python2.6/site-packages/numpy Python version 2.6.4 (r264:75706, Dec 22 2009, 14:55:30) [GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] nose version 0.11.1 ------------------------------------------------------------ Traceback (most recent call last): File "<ipython console>", line 1, in <module> File "/Users/mb312/usr/local/lib/python2.6/site-packages/numpy/testing/nosetester.py", line 335, in test t = NumpyTestProgram(argv=argv, exit=False, plugins=plugins) File "/Users/mb312/usr/local/lib/python2.6/site-packages/nose-0.11.1-py2.6.egg/nose/core.py", line 113, in __init__ argv=argv, testRunner=testRunner, testLoader=testLoader) File "/Users/mb312/usr/local/lib/python2.6/unittest.py", line 816, in __init__ self.parseArgs(argv) File "/Users/mb312/usr/local/lib/python2.6/site-packages/nose-0.11.1-py2.6.egg/nose/core.py", line 130, in parseArgs self.config.configure(argv, doc=self.usage()) File "/Users/mb312/usr/local/lib/python2.6/site-packages/nose-0.11.1-py2.6.egg/nose/config.py", line 249, in configure options, args = self._parseArgs(argv, cfg_files) File "/Users/mb312/usr/local/lib/python2.6/site-packages/nose-0.11.1-py2.6.egg/nose/config.py", line 237, in _parseArgs return parser.parseArgsAndConfigFiles(argv[1:], cfg_files) File "/Users/mb312/usr/local/lib/python2.6/site-packages/nose-0.11.1-py2.6.egg/nose/config.py", line 132, in parseArgsAndConfigFiles self._applyConfigurationToValues(self._parser, config, values) File "/Users/mb312/usr/local/lib/python2.6/site-packages/nose-0.11.1-py2.6.egg/nose/config.py", line 118, in _applyConfigurationToValues name=name, filename=filename) File "/Users/mb312/usr/local/lib/python2.6/site-packages/nose-0.11.1-py2.6.egg/nose/config.py", line 234, in warn_sometimes raise ConfigError(msg) ConfigError: Error reading config file '/Users/mb312/.noserc': no such option 'with-doctest' The reason for this is that, in the numpy testing machinery (nostester.py, around line 249, the 'doctest' plugin is pulled out of the plugins list, because we prefer our own numpy doctest tester. Accordingly, when we initialize our noseclasses.NumpyTestProgram, the 'doctest' plugin is not present, and therefore cannot parse any of its configuration, and hence the error. An obvious idea is to allow the testing machinery to parse all the configs, including the doctest configs, then throw away the native (non-numpy) doctest plugin before we get to collecting and running the tests. I've attached a patch that does this; it's a little bit magic because of the class structure of nose, but I hope it makes sense. I'd be very grateful for a review, Thanks a lot, Matthew

1 1

[ANN] pandas 0.1, a new NumPy-based data analysis library
by Wes McKinney Dec. 26, 2009

Dec. 26, 2009

Hello all, (resending, as this didn't make it through to the ML the first time) I'm very happy to announce the release of a new data analysis library that many of you will hopefully find useful. This release is the product of a long period of development and use; hence, despite the low version number, it is quite suitable for general use. The documentation is still a bit sparse but will become much more complete in the coming weeks and months. Info / Documentation: http://pandas.sourceforge.net/ Overview slides: http://pandas.googlecode.com/files/nyfpug.pdf What it is ========== pandas is a library for pan-el da-ta analysis, i.e. multidimensional time series and cross-sectional data sets commonly found in statistics, econometrics, or finance. It provides convenient and easy-to-understand NumPy-based data structures for generic labeled data, with focus on automatically aligning data based on its label(s) and handling missing observations. One major goal of the library is to simplify the implementation of statistical models on unreliable data. Main Features ============= * Data structures: for 1, 2, and 3 dimensional labeled data sets. Some of their main features include: * Automatically aligning data * Handling missing observations in calculations * Convenient slicing and reshaping ("reindexing") functions * Provide 'group by' aggregation or transformation functionality * Tools for merging / joining together data sets * Simple matplotlib integration for plotting * Date tools: objects for expressing date offsets or generating date ranges; some functionality similar to scikits.timeseries * Statistical models: convenient ordinary least squares and panel OLS implementations for in-sample or rolling time series / cross-sectional regressions. These will hopefully be the starting point for implementing other models pandas is not necessarily intended as a standalone library but rather as something which can be used in tandem with other NumPy-based packages like scikits.statsmodels. Where possible wheel-reinvention has largely been avoided. Also, its time series manipulation capability is not as extensive as scikits.timeseries; pandas does have its own time series object which fits into the unified data model. Some other useful tools for time series data (moving average, standard deviation, etc.) are available in the codebase but do not yet have a convenient interface. These will be highlighted in a future release. Where to get it =============== The source code is currently hosted on googlecode at: http://pandas.googlecode.com Releases can be downloaded currently on the Python package index or using easy_install PyPi: http://pypi.python.org/pypi/pandas/ License ======= BSD Documentation ============= The official documentation is hosted on SourceForge. http://pandas.sourceforge.net/ The sphinx documentation is still in an incomplete state, but it should provide a good starting point for learning how to use the library. Expect the docs to continue to expand as time goes on. Background ========== Work on pandas started at AQR (a quantitative hedge fund) in 2008 and has been under active development since then. Discussion and Development ========================== Since pandas development is related to a number of other scientific Python projects, questions are welcome on the scipy-user mailing list. Specialized discussions or design issues should take place on the pystatsmodels mailing list / google group, where scikits.statsmodels and other libraries will also be discussed: http://groups.google.com/group/pystatsmodels Best regards, Wes McKinney

1 0

[ANN] pandas 0.1, a new NumPy-based data analysis library
by Wes McKinney Dec. 25, 2009

Dec. 25, 2009

Hello all, I'm very happy to announce the release of a new data analysis library that many of you will hopefully find useful. This release is the product of a long period of development and use; hence, despite the low version number, it is quite suitable for general use. The documentation is still a bit sparse but will become much more complete in the coming weeks and months. Info / Documentation: http://pandas.sourceforge.net/ Overview slides: http://pandas.googlecode.com/files/nyfpug.pdf What it is ========== pandas is a library for pan-el da-ta analysis, i.e. multidimensional time series and cross-sectional data sets commonly found in statistics, econometrics, or finance. It provides convenient and easy-to-understand NumPy-based data structures for generic labeled data, with focus on automatically aligning data based on its label(s) and handling missing observations. One major goal of the library is to simplify the implementation of statistical models on unreliable data. Main Features ============= * Data structures: for 1, 2, and 3 dimensional labeled data sets. Some of their main features include: * Automatically aligning data * Handling missing observations in calculations * Convenient slicing and reshaping ("reindexing") functions * Provide 'group by' aggregation or transformation functionality * Tools for merging / joining together data sets * Simple matplotlib integration for plotting * Date tools: objects for expressing date offsets or generating date ranges; some functionality similar to scikits.timeseries * Statistical models: convenient ordinary least squares and panel OLS implementations for in-sample or rolling time series / cross-sectional regressions. These will hopefully be the starting point for implementing other models pandas is not necessarily intended as a standalone library but rather as something which can be used in tandem with other NumPy-based packages like scikits.statsmodels. Where possible wheel-reinvention has largely been avoided. Also, its time series manipulation capability is not as extensive as scikits.timeseries; pandas does have its own time series object which fits into the unified data model. Some other useful tools for time series data (moving average, standard deviation, etc.) are available in the codebase but do not yet have a convenient interface. These will be highlighted in a future release. Where to get it =============== The source code is currently hosted on googlecode at: http://pandas.googlecode.com Releases can be downloaded currently on the Python package index or using easy_install PyPi: http://pypi.python.org/pypi/pandas/ License ======= BSD Documentation ============= The official documentation is hosted on SourceForge. http://pandas.sourceforge.net/ The sphinx documentation is still in an incomplete state, but it should provide a good starting point for learning how to use the library. Expect the docs to continue to expand as time goes on. Background ========== Work on pandas started at AQR (a quantitative hedge fund) in 2008 and has been under active development since then. Discussion and Development ========================== Since pandas development is related to a number of other scientific Python projects, questions are welcome on the scipy-user mailing list. Specialized discussions or design issues should take place on the pystatsmodels mailing list / google group, where scikits.statsmodels and other libraries will also be discussed: http://groups.google.com/group/pystatsmodels Best regards, Wes McKinney

1 0

Ironclad v2.6.0rc1 released
by William Reade Dec. 23, 2009

Dec. 23, 2009

Hi all I'm very happy to announce the latest release (candidate) of Ironclad, the 120-proof home-brewed CPython compatibility layer, now available for IronPython 2.6! No longer need .NET pythonistas toil thanklessly without the benefits of bz2, csv, numpy and scipy: with a simple 'import ironclad', (most parts of) the above packages -- and many more -- will transparently Just Work. For reference: over 1500 tests pass in numpy 1.3.0; over 1900 in 1.4.0RC1; and over 2300 in scipy 0.7.1. Get the package from: http://code.google.com/p/ironclad/ ...and get support from: http://groups.google.com/group/c-extensions-for-ironpython ...or just ask me directly. I'm very keen to hear your experiences, both positive and negative; I haven't been able to test it on as many machines as I have in the past, so your feedback is especially important this time round*. Cheers William * I'd be especially grateful if someone with a newish multicore machine would run the numpy and scipy test scripts (included in the source distrbution) a few times to check for consistent results and absence of weird crashes; if someone volunteers, I'll help however I can.

1 0