Mailman 3 November 2015 - NumPy-Discussion

ANN: pandas v0.17.1 Released
by Jeff Reback Nov. 21, 2015

Nov. 21, 2015

Hi, We are proud to announce that *pandas* has become a sponsored project of the NUMFocus organization <http://numfocus.org/news/2015/10/09/numfocus-announces-new-fiscally-sponsor…> This will help ensure the success of development of *pandas* as a world-class open-source project. This is a minor bug-fix release from 0.17.0 and includes a large number of bug fixes along several new features, enhancements, and performance improvements. We recommend that all users upgrade to this version. This was a release of 5 weeks with 176 commits by 61 authors encompassing 84 issues and 128 pull-requests. *What is it:* *pandas* is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language. *Highlights*: - Support for Conditional HTML Formatting, see here <http://pandas.pydata.org/pandas-docs/version/0.17.1/whatsnew.html#whatsnew-…> - Releasing the GIL on the csv reader & other ops, see here <http://pandas.pydata.org/pandas-docs/version/0.17.1/whatsnew.html#whatsnew-…> - Fixed regression in DataFrame.drop_duplicates from 0.16.2, causing incorrect results on integer values see Issue 11376 See the Whatsnew <http://pandas.pydata.org/pandas-docs/version/0.17.1/whatsnew.html> for much more information and the full Documentation <http://pandas.pydata.org/pandas-docs/stable/> link. *How to get it:* Source tarballs, windows wheels, and macosx wheels are available on PyPI <https://pypi.python.org/pypi/pandas> Installation via conda is: - conda install pandas windows wheels are courtesy of Christoph Gohlke and are built on Numpy 1.9 macosx wheels are courtesy of Matthew Brett *Issues:* Please report any issues on our issue tracker <https://github.com/pydata/pandas/issues>: Jeff *Thanks to all of the contributors* * - Aleksandr Drozd - Alex Chase - Anthonios Partheniou - BrenBarn - Brian J. McGuirk - Chris - Christian Berendt - Christian Perez - Cody Piersall - Data & Code Expert Experimenting with Code on Data - DrIrv - Evan Wright - Guillaume Gay - Hamed Saljooghinejad - Iblis Lin - Jake VanderPlas - Jan Schulz - Jean-Mathieu Deschenes - Jeff Reback - Jimmy Callin - Joris Van den Bossche - K.-Michael Aye - Ka Wo Chen - Loïc Séguin-C - Luo Yicheng - Magnus Jöud - Manuel Leonhardt - Matthew Gilbert - Maximilian Roos - Michael - Nicholas Stahl - Nicolas Bonnotte - Pastafarianist - Petra Chong - Phil Schaf - Philipp A - Rob deCarvalho - Roman Khomenko - Rémy Léone - Sebastian Bank - Thierry Moisan - Tom Augspurger - Tux1 - Varun - Wieland Hoffmann - Winterflower - Yoav Ram - Younggun Kim - Zeke - ajcr - azuranski - behzad nouri - cel4 - emilydolson - hironow - lexual - llllllllll - rockg - silentquasar - sinhrks - taeold *

2 1

asarray(sparse) -> object
by josef.pktd＠gmail.com Nov. 20, 2015

Nov. 20, 2015

Is this intentional? >>> exog <50x5 sparse matrix of type '<class 'numpy.float64'>' with 50 stored elements in Compressed Sparse Column format> >>> np.asarray(exog) array(<50x5 sparse matrix of type '<class 'numpy.float64'>' with 50 stored elements in Compressed Sparse Column format>, dtype=object) I'm just a newbie who thought to use the usual pattern. .... >>> np.asarray(exog).dot(beta) array([ <50x5 sparse matrix of type '<class 'numpy.float64'>' with 50 stored elements in Compressed Sparse Column format>, <50x5 sparse matrix of type '<class 'numpy.float64'>' with 50 stored elements in Compressed Sparse Column format>, <50x5 sparse matrix of type '<class 'numpy.float64'>' with 50 stored elements in Compressed Sparse Column format>, <50x5 sparse matrix of type '<class 'numpy.float64'>' with 50 stored elements in Compressed Sparse Column format>, <50x5 sparse matrix of type '<class 'numpy.float64'>' with 50 stored elements in Compressed Sparse Column format>], dtype=object) C:\programs\WinPython-64bit-3.4.3.1\python-3.4.3.amd64\lib\site-packages\scipy\sparse\compressed.py:306: SparseEfficiencyWarning: Comparing sparse matrices using >= and <= is inefficient, using <, >, or !=, instead. "using <, >, or !=, instead.", SparseEfficiencyWarning) seems to warn only once >>> y = np.asarray(exog).dot(beta) >>> y.shape (5,) >>> np.__version__ '1.9.2rc1' >>> scipy.__version__ '0.15.1' Josef

2 2

Numpy 1.10.2rc1
by Charles R Harris Nov. 20, 2015

Nov. 20, 2015

Hi All, I am pleased to announce the release of Numpy 1.10.2rc1. This release should fix the problems exposed in 1.10.1, which is not to say there are no remaining problems. Please test this thoroughly, exspecially if you experienced problems with 1.10.1. Julian Taylor has opened an issue relating to cblas detection on Debian (and probably Debian derived distributions) that is not dealt with in this release. Hopefully a solution will be available before the final. To all who reported issues with 1.10.1 and to those who helped close them, a big thank you. Source and binary files may be found on Sourceforge <https://sourceforge.net/projects/numpy/files/NumPy/1.10.1/>. Chuck

2 3

[JOB] Project Jupyter is hiring two postdoctoral fellows @ UC Berkeley
by Fernando Perez Nov. 19, 2015

Nov. 19, 2015

Hi all, We are delighted to announce today that Project Jupyter/IPython has two postdoctoral fellowships open at UC Berkeley, open immediately. Interested candidates can apply here: https://aprecruit.berkeley.edu/apply/JPF00899 We hope to find candidates who will work on a number of challenging questions over the next few years, as described in our grant proposal here: http://blog.jupyter.org/2015/07/07/project-jupyter-computational-narratives… Interested candidates should carefully read that proposal before applying to familiarize themselves with the full scope of the questions we intend to tackle. We'd like to thank the support of the Helmsley Trust, the Gordon and Betty Moore Foundation and the Alfred P. Sloan Foundation. Cheers, Brian Granger and Fernando Perez. -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail

1 0

reshaping array question
by Neal Becker Nov. 17, 2015

Nov. 17, 2015

I have an array of shape (7, 24, 2, 1024) I'd like an array of (7, 24, 2048) such that the elements on the last dimension are interleaving the elements from the 3rd dimension [0,0,0,0] -> [0,0,0] [0,0,1,0] -> [0,0,1] [0,0,0,1] -> [0,0,2] [0,0,1,1] -> [0,0,3] ... What might be the simplest way to do this? ------------ A different question, suppose I just want to stack them [0,0,0,0] -> [0,0,0] [0,0,0,1] -> [0,0,1] [0,0,0,2] -> [0,0,2] ... [0,0,1,0] -> [0,0,1024] [0,0,1,1] -> [0,0,1025] [0,0,1,2] -> [0,0,1026] ...

3 5

ANN: bcolz 0.12.0 released
by Francesc Alted Nov. 16, 2015

Nov. 16, 2015

======================= Announcing bcolz 0.12.0 ======================= What's new ========== This release copes with some compatibility issues with NumPy 1.10. Also, several improvements have happened in the installation procedure, allowing for a smoother process. Last but not least, the tutorials haven been migrated to the IPython notebook format (a huge thank you to Francesc Elies for this!). This will hopefully will allow users to better exercise the different features of bcolz. For a more detailed change log, see: https://github.com/Blosc/bcolz/blob/master/RELEASE_NOTES.rst What it is ========== *bcolz* provides columnar and compressed data containers that can live either on-disk or in-memory. Column storage allows for efficiently querying tables with a large number of columns. It also allows for cheap addition and removal of column. In addition, bcolz objects are compressed by default for reducing memory/disk I/O needs. The compression process is carried out internally by Blosc, an extremely fast meta-compressor that is optimized for binary data. Lastly, high-performance iterators (like ``iter()``, ``where()``) for querying the objects are provided. bcolz can use numexpr internally so as to accelerate many vector and query operations (although it can use pure NumPy for doing so too). numexpr optimizes the memory usage and use several cores for doing the computations, so it is blazing fast. Moreover, since the carray/ctable containers can be disk-based, and it is possible to use them for seamlessly performing out-of-memory computations. bcolz has minimal dependencies (NumPy), comes with an exhaustive test suite and fully supports both 32-bit and 64-bit platforms. Also, it is typically tested on both UNIX and Windows operating systems. Together, bcolz and the Blosc compressor, are finally fulfilling the promise of accelerating memory I/O, at least for some real scenarios: http://nbviewer.ipython.org/github/Blosc/movielens-bench/blob/master/queryi… Other users of bcolz are Visualfabriq (http://www.visualfabriq.com/) the Blaze project (http://blaze.pydata.org/), Quantopian (https://www.quantopian.com/) and Scikit-Allel (https://github.com/cggh/scikit-allel) which you can read more about by pointing your browser at the links below. * Visualfabriq: * *bquery*, A query and aggregation framework for Bcolz: * https://github.com/visualfabriq/bquery * Blaze: * Notebooks showing Blaze + Pandas + BColz interaction: * http://nbviewer.ipython.org/url/blaze.pydata.org/notebooks/timings-csv.ipynb * http://nbviewer.ipython.org/url/blaze.pydata.org/notebooks/timings-bcolz.ip… * Quantopian: * Using compressed data containers for faster backtesting at scale: * https://quantopian.github.io/talks/NeedForSpeed/slides.html * Scikit-Allel * Provides an alternative backend to work with compressed arrays * https://scikit-allel.readthedocs.org/en/latest/model/bcolz.html Installing ========== bcolz is in the PyPI repository, so installing it is easy:: $ pip install -U bcolz Resources ========= Visit the main bcolz site repository at: http://github.com/Blosc/bcolz Manual: http://bcolz.blosc.org Home of Blosc compressor: http://blosc.org User's mail list: bcolz(a)googlegroups.com http://groups.google.com/group/bcolz License is the new BSD: https://github.com/Blosc/bcolz/blob/master/LICENSES/BCOLZ.txt Release notes can be found in the Git repository: https://github.com/Blosc/bcolz/blob/master/RELEASE_NOTES.rst ---- **Enjoy data!** -- Francesc Alted

1 0

Removiing 1.10.0 and 1.10.1 from sourceforge and pypi
by Charles R Harris Nov. 14, 2015

Nov. 14, 2015

Hi All, I think 1.10.0 and 1.10.1 are sufficiently buggy that they should be removed from circulation as soon as 1.10.2 comes out. The inner product bug for non contiguous arrays is particularly egregious. It is not customary to remove outdated Numpy releases from sourceforge and pypi, but I'd like to make an exception for those two. Thoughts? Chuck

6 8

loadtxt and usecols
by Irvin Probst Nov. 13, 2015

Nov. 13, 2015

Hi, I've recently seen many students, coming from Matlab, struggling against the usecols argument of loadtxt. Most of them tried something like: loadtxt("foo.bar", usecols=2) or the ones with better documentation reading skills tried loadtxt("foo.bar", usecols=(2)) but none of them understood they had to write usecols=[2] or usecols=(2,). Is there a policy in numpy stating that this kind of arguments must be sequences ? I think that being able to an int or a sequence when a single column is needed would make this function a bit more user friendly for beginners. I would gladly submit a PR if noone disagrees. Regards. -- Irvin

5 12

Question about structure arrays
by aerojockey Nov. 11, 2015

Nov. 11, 2015

Hello, Recently I made some changes to a program I'm working on, and found that the changes made it four times slower than before. After some digging, I found out that one of the new costs was that I added structure arrays. Inside a low-level loop, I create a structure array, populate it Python, then turn it over to some handwritten C code for processing. It turned out that, when passed a structure array as a dtype, numpy has to parse the dtype, which included calls to re.match and eval. Now, this is not a big deal for me to work around by using ordinary slicing and such, and also I can improve things by reusing arrays. Since this is inner loop stuff, sacrificing readability for speed is an appropriate tradeoff. Nevertheless, I was curious if there was a way (or any plans for there to be a way) to compile a struture array dtype. I realize it's not the bread-and-butter of numpy, but it turned out to be a very convenient feature for my use case (populating an array of structures to pass off to C). Thanks -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/Question-about-structure-arrays… Sent from the Numpy-discussion mailing list archive at Nabble.com.

4 4

Feedback on new argument positions for ma.dot and MaskedArray.dot
by Charles R Harris Nov. 10, 2015

Nov. 10, 2015

Hi All, I'd like some feedback for the position of the `strict` and `out` arguments for masked arrays. See gh-6653 <https://github.com/numpy/numpy/pull/6653> for the PR in question. Current status without #6652 1. ma.dot(a, b, strict=False) -- established 2. a.dot(b, out=None) -- new in 1.10 Note that 1. requires adding `out` to the end for backward compatibility. OTOH, 2. is new(ish). We can either keep it compatible with ndarray.dot and add `strict` to the end and have it incompatible with 1., or, slightly changing it in 1.10.2, make it compatible with with 1. but incompatible with ndarray. We will face the same sort of problem with adding newer ndarray arguments other existing ma functions that have their own specialized arguments, so having a policy up front will be helpful. My own inclination here is to keep 1. and 2. compatible, and then perhaps at some point following a future warning, make both `strict` and `out` keyword arguments only. Another possiblitly is to make that transition immediate for the method. Thoughts? Chuck

3 4