Mailman 3 February 2012 - NumPy-Discussion

test errors on deprecation/runtime warnings
by Ralf Gommers March 4, 2012

March 4, 2012

Hi, Last week we merged https://github.com/numpy/numpy/pull/201, which causes DeprecationWarning's and RuntimeWarning's to be converted to errors if they occur when running the test suite. The purpose of that is to make sure that code that still uses other deprecated code (or code that for some reason generates warnings) to be cleaned up. In principle this is a good idea IMHO, but after merging we quickly found some problems with failing tests in scipy. Because this potentially affects any other projects or users that use the numpy NoseTester test runner, here's a proposal to deal with this issue: - make this behavior configurable by a keyword in `NoseTester.__init__()` - default to raising an error in numpy master - when making a branch for release, immediately set the default to not raise. Do this not only for the 1.7 release, but for any future release (at least until the oldest numpy version still in use has the keyword). The reason for this is that otherwise a new numpy release will trigger test failures in older releases of scipy or other packages. The pros: - We find issues that otherwise get ignored (see the issues Christoph just found when compiling with MSVC). - We're forced to clean up code when submitting PRs, instead of letting the warnings accumulate and having to deal with them just before release time. The con: - You may see test errors if you run a released version of scipy with a development version of numpy. Opinions? Concerns? Ralf

2 3

Possible roadmap addendum: building better text file readers
by Wes McKinney March 2, 2012

March 2, 2012

dear all, I haven't read all 180 e-mails, but I didn't see this on Travis's initial list. All of the existing flat file reading solutions I have seen are not suitable for many applications, and they compare very unfavorably to tools present in other languages, like R. Here are some of the main issues I see: - Memory usage: creating millions of Python objects when reading a large file results in horrendously bad memory utilization, which the Python interpreter is loathe to return to the operating system. Any solution using the CSV module (like pandas's parsers-- which are a lot faster than anything else I know of in Python) suffers from this problem because the data come out boxed in tuples of PyObjects. Try loading a 1,000,000 x 20 CSV file into a structured array using np.genfromtxt or into a DataFrame using pandas.read_csv and you will immediately see the problem. R, by contrast, uses very little memory. - Performance: post-processing of Python objects results in poor performance. Also, for the actual parsing, anything regular expression based (like the loadtable effort over the summer, all apologies to those who worked on it), is doomed to failure. I think having a tool with a high degree of compatibility and intelligence for parsing unruly small files does make sense though, but it's not appropriate for large, well-behaved files. - Need to "factorize": as soon as there is an enum dtype in NumPy, we will want to enable the file parsers for structured arrays and DataFrame to be able to "factorize" / convert to enum certain columns (for example, all string columns) during the parsing process, and not afterward. This is very important for enabling fast groupby on large datasets and reducing unnecessary memory usage up front (imagine a column with a million values, with only 10 unique values occurring). This would be trivial to implement using a C hash table implementation like khash.h To be clear: I'm going to do this eventually whether or not it happens in NumPy because it's an existing problem for heavy pandas users. I see no reason why the code can't emit structured arrays, too, so we might as well have a common library component that I can use in pandas and specialize to the DataFrame internal structure. It seems clear to me that this work needs to be done at the lowest level possible, probably all in C (or C++?) or maybe Cython plus C utilities. If anyone wants to get involved in this particular problem right now, let me know! best, Wes

21 57

[Numpy] quadruple precision
by Paweł Biernat March 2, 2012

March 2, 2012

I am completely new to Numpy and I know only the basics of Python, to this point I was using Fortran 03/08 to write numerical code. However, I am starting a new large project of mine and I am looking forward to using Python to call some low level Fortran code responsible for most of the intensive number crunching. In this context I stumbled into f2py and it looks just like what I need, but before I start writing an app in mixture of Python and Fortran I have a question about numerical precision of variables used in numpy and f2py. Is there any way to interact with Fortran's real(16) (supported by gcc and Intel's ifort) data type from numpy? By real(16) I mean the binary128 type as in IEEE 754. (In C this data type is experimentally supported as __float128 (gcc) and _Quad (Intel's icc).) I have investigated the float128 data type, but it seems to work as binary64 or binary80 depending on the architecture. If there is currently no way to interact with binary128, how hard would it be to patch the sources of numpy to add such data type? I am interested only in basic stuff, comparable in functionality to libmath. As said before, I have little knowledge of Python, Numpy and f2py, I am however, interested in investing some time in learing it and implementing the mentioned features, but only if there is any hope of succeeding.

8 10

Special matrices with structure?
by Jaakko Luttinen March 1, 2012

March 1, 2012

Hi! I was wondering whether it would be easy/possible/reasonable to have classes for arrays that have special structure in order to use less memory and speed up some computations? For instance: - symmetric matrix could be stored in almost half the memory required by a non-symmetric matrix - diagonal matrix only needs to store the diagonal vector - Toeplitz matrix only needs to store one or two vectors - sparse matrix only needs to store non-zero elements (some implementations in scipy.sparse) - and so on If such classes were implemented, it would be nice if they worked with numpy functions (dot, diag, ...) and operations (+, *, +=, ...) easily. I believe this has been discussed before but google didn't help a lot.. Regards, Jaakko

2 3

Proposed Roadmap Overview
by Travis Oliphant March 1, 2012

March 1, 2012

Mark Wiebe and I have been discussing off and on (as well as talking with Charles) a good way forward to balance two competing desires: * addition of new features that are needed in NumPy * improving the code-base generally and moving towards a more maintainable NumPy I know there are load voices for just focusing on the second of these and avoiding the first until we have finished that. I recognize the need to improve the code base, but I will also be pushing for improvements to the feature-set and user experience in the process. As a result, I am proposing a rough outline for releases over the next year: * NumPy 1.7 to come out as soon as the serious bugs can be eliminated. Bryan, Francesc, Mark, and I are able to help triage some of those. * NumPy 1.8 to come out in July which will have as many ABI-compatible feature enhancements as we can add while improving test coverage and code cleanup. I will post to this list more details of what we plan to address with it later. Included for possible inclusion are: * resolving the NA/missing-data issues * finishing group-by * incorporating the start of label arrays * incorporating a meta-object * a few new dtypes (variable-length string, varialbe-length unicode and an enum type) * adding ufunc support for flexible dtypes and possibly structured arrays * allowing generalized ufuncs to work on more kinds of arrays besides just contiguous * improving the ability for NumPy to receive JIT-generated function pointers for ufuncs and other calculation opportunities * adding "filters" to Input and Output * simple computed fields for dtypes * accepting a Data-Type specification as a class or JSON file * work towards improving the dtype-addition mechanism * re-factoring of code so that it can compile with a C++ compiler and be minimally dependent on Python data-structures. * NumPy 2.0 to come out in January of 2013. Mark Wiebe and I will post to this list a document that explains some of it's proposed features and enhancements. I won't steal his thunder for some of the things he is working on. If there are code issues people would like to see addressed, it would be a great time to speak up and/or propose something that you would like to see. In general NumPy 1.8 will have new features that need to be explored in order that NumPy 2.0 has enough code "experience" in order to be as useful as possible. I recognize that NumPy 1.8 has quite a few proposed features. These have been building up and are the big reason I've committed so many resources to NumPy. The feature-list did not just come out of my head. They are the result of talking and interacting with many NumPy users and watching the code get used (and not used) in the real world. This will be a faster pace of development. But, all of this will be in the open. If the NumPy 2.0 schedule is too aggressive, then we will have a NumPy 1.9 release in order to allow features to come out. Thanks, -Travis

37 196

Re: [Numpy-discussion] Cygwin compile: fatal error: fenv/fenv.c
by Matt Miller Feb. 29, 2012

Feb. 29, 2012

That fixed changed my error message to this: numpy/core/src/private/lowlevel_strided_loops.h:404:1: warning: ‘PyArray_PrepareThreeRawArrayIter’ declared ‘static’ but never defined numpy/core/src/private/lowlevel_strided_loops.h:430:1: warning: ‘PyArray_PrepareFourRawArrayIter’ declared ‘static’ but never defined gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src/umath/umathmodule_onefile.o -L/usr/lib/python2.6/config -Lbuild/temp.cygwin-1.7.11-i686-2.6 -lnpymath -lpython2.6 -o build/lib.cygwin-1.7.11-i686-2.6/numpy/core/umath.dll build/temp.cygwin-1.7.11-i686-2.6/libnpymath.a(ieee754.o):ieee754.c:(.rdata+0x0): multiple definition of `_npy__fe_dfl_env' build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src/umath/umathmodule_onefile.o:umathmodule_onefile.c:(.rdata+0x13158): first defined here collect2: ld returned 1 exit status build/temp.cygwin-1.7.11-i686-2.6/libnpymath.a(ieee754.o):ieee754.c:(.rdata+0x0): multiple definition of `_npy__fe_dfl_env' build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src/umath/umathmodule_onefile.o:umathmodule_onefile.c:(.rdata+0x13158): first defined here collect2: ld returned 1 exit status error: Command "gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src/umath/umathmodule_onefile.o -L/usr/lib/python2.6/config -Lbuild/temp.cygwin-1.7.11-i686-2.6 -lnpymath -lpython2.6 -o build/lib.cygwin-1.7.11-i686-2.6/numpy/core/umath.dll" failed with exit status 1 Thanks for the quick reply! > > > Hi all, > > > > I am getting the following error when running `python setup.py install` > > for Numpy in Cygwin. This error happens on the latest as well as > > the maintenance branched for 1.5 and 1.6. > > > > This should fix it: http://projects.scipy.org/numpy/ticket/1944. > > Can you confirm that that works? Then I'll make the change in master. > > Ralf > > > > ... > > creating build/temp.cygwin-1.7.11-i686-2.6 > > creating build/temp.cygwin-1.7.11-i686-2.6/build > > creating > build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6 > > creating > > build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy > > creating > > > build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core > > creating > > > build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core/src > > creating > > > build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath > > creating build/temp.cygwin-1.7.11-i686-2.6/numpy > > creating build/temp.cygwin-1.7.11-i686-2.6/numpy/core > > creating build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src > > creating build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath > > compile options: '-Inumpy/core/include > > -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/include/numpy > > -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core > > -Inumpy/core/src/npymath -Inumpy/core/src/multiarray > -Inumpy/core/src/umath > > -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.6 > > -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/multiarray > > -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/umath -c' > > gcc: build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/npy_math.c > > gcc: build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/ieee754.c > > numpy/core/src/npymath/ieee754.c.src:590:25: fatal error: fenv/fenv.c: No > > such file or directory > > compilation terminated. > > numpy/core/src/npymath/ieee754.c.src:590:25: fatal error: fenv/fenv.c: No > > such file or directory > > compilation terminated. > > error: Command "gcc -fno-strict-aliasing -g -O2 -pipe -DNDEBUG -g -fwrapv > > -O3 -Wall -Wstrict-prototypes -Inumpy/core/include > > -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/include/numpy > > -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core > > -Inumpy/core/src/npymath -Inumpy/core/src/multiarray > -Inumpy/core/src/umath > > -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.6 > > -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/multiarray > > -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/umath -c > > build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/ieee754.c -o > > > build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/ieee754.o" > > failed with exit status 1 > > > > > > Thanks > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion(a)scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > >

2 2

YouTrack License
by Travis Oliphant Feb. 29, 2012

Feb. 29, 2012

I just received word that NumPy has a license to use TeamCity and YouTrack for NumPy development. YouTrack is a really nice issue tracker: http://www.jetbrains.com/youtrack/ TeamCity is a really nice Continuous Integration system: http://www.jetbrains.com/teamcity/ I'm planning to set this up to try out. Is anyone interested in helping out? Thanks, -Travis

4 3

Cygwin compile: fatal error: fenv/fenv.c:
by Matt Miller Feb. 29, 2012

Feb. 29, 2012

Hi all, I am getting the following error when running `python setup.py install` for Numpy in Cygwin. This error happens on the latest as well as the maintenance branched for 1.5 and 1.6. ... creating build/temp.cygwin-1.7.11-i686-2.6 creating build/temp.cygwin-1.7.11-i686-2.6/build creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6 creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core/src creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath creating build/temp.cygwin-1.7.11-i686-2.6/numpy creating build/temp.cygwin-1.7.11-i686-2.6/numpy/core creating build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src creating build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath compile options: '-Inumpy/core/include -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.6 -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/multiarray -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/umath -c' gcc: build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/npy_math.c gcc: build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/ieee754.c numpy/core/src/npymath/ieee754.c.src:590:25: fatal error: fenv/fenv.c: No such file or directory compilation terminated. numpy/core/src/npymath/ieee754.c.src:590:25: fatal error: fenv/fenv.c: No such file or directory compilation terminated. error: Command "gcc -fno-strict-aliasing -g -O2 -pipe -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Inumpy/core/include -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.6 -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/multiarray -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/umath -c build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/ieee754.c -o build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/ieee754.o" failed with exit status 1 Thanks

2 1

Determining the 'viewness' of an array, and flags.owndata confusion
by Kurt Smith Feb. 29, 2012

Feb. 29, 2012

For an arbitrary numpy array 'a', what does 'a.flags.owndata' indicate? I originally thought that owndata is False iff 'a' is a view. But that is incorrect. Consider the following: In [119]: a = np.zeros((3,3)) In [120]: a.flags.owndata # should be True; zeros() creates and returns a non-view array. Out[120]: True In [121]: a_view1 = a[2:, :] In [122]: a_view1.flags.owndata # expected to be False Out[122]: False In [123]: a_fancy1 = a[[0,1], :] In [124]: a_fancy1.flags.owndata # expected to be True, a_fancy1 is a fancy-indexed array. Out[124]: True In [125]: a_fancy2 = a[:, [0,1]] In [126]: a_fancy2.flags.owndata # expected to be True, a_fancy2 is a fancy-indexed array. Out[126]: False So when I query an array's flags.owndata, what is it telling me? What I want to know is whether an array is a view or not. If flags.owndata has nothing to do with the 'viewness' of an array, how would I determine if an array is a view? In the previous example, a_fancy2 does not own its data, as indicated by 'owndata' being False. But when I modify a_fancy2, 'a' is not modified, as expected, but contrary to what 'owndata' would seem to indicate. The numpybook's documentation of owndata could perhaps be a bit clearer on the subject: """ OWNDATA (O) the array owns the memory it uses or if it borrows it from another object (if this is False, the base attribute retrieves a reference to the object this array obtained its data from) """ >From my reading, owndata has nothing to do with the 'viewness' of an array. Is this correct? What is the intent of this flag, then? Its wording could perhaps be improved: owndata is True iff (1) the array owns the memory it uses or (2) the array borrows it from another object. The second clause seems to indicate that the array **does not** own its data since it is borrowing it from another object. However, flags.owndata will be true in this case. If I cannot use flags.owndata, what is a reliable way to determine whether or not an array is a view?

6 6

Problem Building Numpy with Python 2.7.1 and OS X 10.7.3
by Patrick Armstrong Feb. 28, 2012

Feb. 28, 2012

Hi there, I'm having a problem building NumPy on Python 2.7.1 and OS X 10.7.3. Here is my build log: https://gist.github.com/1895377 Does anyone have any idea what might be happening? I get a very similar error when compiling with clang. Installing a binary really isn't an option for me due to some specifics of my project. Does anyone have an idea what might be wrong? Thanks. --patrick

3 5