Mailman 3 February 2012 - NumPy-Discussion

The end of numpy as we know it ?
by josef.pktd＠gmail.com Feb. 18, 2012

Feb. 18, 2012

(on a ambiguous day, pessimistic or optimistic?) Numpy is a monster written by a bunch of amateurs (engineers and scientists), with a glacial pace of development. If we want to make any progress to the world dominance of python in science, we need to go professionally about it. First we need to streamline the development infrastructure and get the distribution under control. Then we need to streamline the code so the bunch of amateurs doesn't understand what's going on and cannot effectively threaten a fork anymore. Then, we need to get our version of nans, labeled arrays and other tools in so we have the right hooks to compete downstream. Of course, "This may also mean different business models and licensing around some of the NumPy-related code that the company writes." To a glorious professional future and long live python in science. ------- unsigned

6 6

..
by Marlin Rowley Feb. 18, 2012

Feb. 18, 2012

Madam: http://flooring-direct.com/folder1946/httpwww.timerteam332.php?oculuckyid=50 Sat, 18 Feb 2012 14:48:50 _____________________ "Hooper merely glanced at Bill." (c) Dale wumaofen

1 0

Issue Tracking
by Travis Oliphant Feb. 17, 2012

Feb. 17, 2012

I'm wondering about using one of these commercial issue tracking plans for NumPy and would like thoughts and comments. Both of these plans allow Open Source projects to have unlimited plans for free. YouTrack from JetBrains: http://www.jetbrains.com/youtrack/features/issue_tracking.html JIRA: http://www.atlassian.com/software/jira/overview/tour/code-integration What Mark Wiebe said about making it easy to "manage the issues" quickly and what Eric said about making sure there are interfaces with dense information content really struck chords with me. I have seen a lot of time wasted on issue management with Trac --- time that could be better spent on NumPy. I'd like to make issue management efficient --- even if it means a system separate from GitHub. Issue management is a very important part of the open-source process. -Travis

12 17

Strange PyArray_FromObject() behavior
by Spotz, William F Feb. 17, 2012

Feb. 17, 2012

I have a user who is reporting tests that are failing on his platform. I have not been able to reproduce the error on my system, but working with him, we have isolated the problem to unexpected results when PyArray_FromObject() is called. Here is the chain of events: In python, an integer is calculated. Specifically, it is len(result.errors) + len(result.failures) where result is a unit test result object from the unittest module. I had him verify that this value was in fact a python integer. In my extension module, this PyObject gets passed to the PyArray_FromObject() function in a routine that comes from numpy.i. What I expect, and what I typically get, is a numpy scalar array of type C long. I had my user print the result using PyObject_Print() and what he got was array([0:00:00], dtype=timedelta64[us]) I am stuck as to why this might be happening. Any ideas? Thanks ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz(a)sandia.gov<mailto:[email protected]> **

4 5

Numpy governance update - was: Updated differences between 1.5.1 to 1.6.1
by Matthew Brett Feb. 17, 2012

Feb. 17, 2012

Hi, On Tue, Feb 14, 2012 at 10:25 AM, Travis Oliphant <travis(a)continuum.io> wrote: > > On Feb 14, 2012, at 3:32 AM, David Cournapeau wrote: > >> Hi Travis, >> >> It is great that some resources can be spent to have people paid to >> work on NumPy. Thank you for making that happen. >> >> I am slightly confused about roadmaps for numpy 1.8 and 2.0. This >> needs discussion on the ML, and our release manager currently is Ralf >> - he is the one who ultimately decides what goes when. > > Thank you for reminding me of this. Ralf and I spoke several days ago, and have been working on how to give him more time to spend on SciPy full-time. As a result, he will be release managing NumPy 1.7, but for NumPy 1.8, I will be the release manager again. Ralf will continue serving as release manager for SciPy. > > For NumPy 2.0 and beyond, Mark Wiebe will likely be the release manager. I only know that I won't be release manager past NumPy 1.X. > >> I am also not >> completely comfortable by having a roadmap advertised to Pycon not >> coming from the community. > > This is my bad wording which is a function of being up very late. At PyCon we will be discussing the roadmap conversations that are taking place on this list. We won't be presenting anything there related to the NumPy project that has not first been discussed here. > > The community will have ample opportunity to provide input, suggestions, and criticisms for anything that goes into NumPy --- the same as I've always done before when releasing open source software. In fact, I will also be discussing at PyCon, the creation of NumFOCUS (NumPy Foundation for Open Code for Usable Science) which has been organized precisely for ensuring that NumPy, SciPy, Matplotlib, and IPython stay community-focused and community-led even while receiving input and money from multiple companies and organizations. > > There is a mailing list for numfocus that you can sign up for if you would like to be part of those discussions. Let me know if you would like more information about that. John Hunter, Fernando Perez, me, Perry Greenfield, and Jarrod Millman are the initial board of the Foundation. But, I expect the Foundation directors to evolve over time. I should say that I have no knowledge of the events above other than from the mailing list (I say that only because some of you may know that I'm a friend and colleague of Jarrod and Fernando). Travis - I hope you don't mind, but here I post some links that I have just found: http://technicaldiscovery.blogspot.com/2012/01/transition-to-continuum.html http://www.continuum.io/ I see that you've founded a new company, Continuum Analytics, and you are working with Peter Wang, Mark Wiebe, Francesc Alted (PyTables), and Bryan Van de Ven. I think you mentioned this earlier in one of the recent threads. In practice this gives your company an overwhelming voice in the direction of numpy. >From the blog post you say: "This may also mean different business models and licensing around some of the NumPy-related code that the company writes." Obviously your company will need to make enough money to cover your salaries and more. There is huge potential here for clashes of interest, and for perceived clashes of interest. The perceived clashes are just as damaging as the actual clashes. I still don't think we've got a "Numpy steering group". The combination of the huge concentration of numpy resources in your company, and a lack of explicit community governance, seems to me to be something that needs to be fixed urgently. Do you agree? Is there any reason why the numfocus group was formed without obvious public discussion about it's composition, remit or governance? I'm not objecting to it's composition, but I think it is a mistake to make large decisions like this without public consultation. I imagine that what happened was that things moved too fast to make it attractive to slow the process by public discussion. I implore you to slow down and commit yourself to have that discussion in full and in public, in the interests of the common ownership of the project. Best, Matthew

29 96

strange behavior of numpy.random.multivariate_normal, ticket:1842
by josef.pktd＠gmail.com Feb. 16, 2012

Feb. 16, 2012

Doing a bit of browsing in the numpy tracker, I found this. From my search this was not discussed on the mailing list. http://projects.scipy.org/numpy/ticket/1842 The multivariate normal random sample is not always the same, even though a seed is specified. It seems to alternate randomly between two patterns, the sum of the random numbers is always the same. Is there a randomness in the svd? Given the constant sum, I guess the underlying random numbers are the same. (windows 7, python 32bit on 64 machine, numpy 1.5.1) import numpy d = 10 alpha = 1 / d**0.5 mu = numpy.ones(d) R = alpha * numpy.ones((d, d)) + (1 - alpha) * numpy.eye(d) rs = numpy.random.RandomState(587482) rv1 = rs.multivariate_normal(mu, R, 1) print rv1, rv1.sum() numpy.random.seed(587482) rv2 = numpy.random.multivariate_normal(mu, R, 1) print rv2[0][0], rv2.sum() running it a few times >>> [[ 0.43028555 1.06584226 -0.03496681 -0.31372591 -0.49716804 -1.50641838 0.99209124 0.57236839 -0.32107663 0.5865379 ]] 0.973769580498 0.0979690286727 0.973769580498 >>> [[ 0.09796903 1.41010513 -1.10250773 0.71321445 0.09903517 0.36432555 -1.27590062 0.04533834 0.37426153 0.24792873]] 0.973769580498 0.430285553017 0.973769580498 >>> [[ 0.09796903 1.41010513 -1.10250773 0.71321445 0.09903517 0.36432555 -1.27590062 0.04533834 0.37426153 0.24792873]] 0.973769580498 0.430285553017 0.973769580498 >>> [[ 0.43028555 1.06584226 -0.03496681 -0.31372591 -0.49716804 -1.50641838 0.99209124 0.57236839 -0.32107663 0.5865379 ]] 0.973769580498 0.0979690286727 0.973769580498 Josef

8 21

Re: [Numpy-discussion] Numpy governance update
by Joe Harrington Feb. 16, 2012

Feb. 16, 2012

>On Wed, Feb 15, 2012 at 1:00 PM, Perry Greenfield <perry(a)stsci.edu> wrote: >> On Feb 15, 2012, at 3:01 PM, Matthew Brett wrote: >> >> [...] >> >> My 2 cents. >> >> [...] I am both elated and concerned. Since it's obvious what there is to be elated about, this post has a "concerned" tone. But overall, I think this is a great move with some obvious problems to solve before it moves forward. I think the effort spent now looking for a solution will be much less than the pain of trying to undo a mess involving contracts and clients. That way lies court and code splits. So, let's do the hard work of figuring out governance now. In principle, I agree with Matt. We're moving from pretty altruistic code development to a model in which the most active development will be paid for and therefore controlled by influences outside the user community. This can have a lot of unintended side effects, including those Matt pointed out. We might also feel some level of discontent among developers who are not paid vs. those who are. This might make it hard to recruit developers who are not Continuum employees. There are tons of examples of financial interests creeping in and mucking up community computer projects. Symbolics vs. Lisp Machines, Inc. Early shenanigans on internet technical committees by engineers working at companies that were behind the curve on product development. Etc. As for being responsive to the community, Continuum is already promising the world whole new directions in numpy (see continuum.io). Were those plans even mentioned on a mailing list? Is that the direction we want to go in? Are there negative consequeces to those plans? What are the plans, exactly? Having those who do the work make the decisions only works when those working consider the needs of all. The community believes that's true largely when it's included in the discussion and not taken by surprise, as seems to have happened here. Of course, balancing all of this (and our security blanket) is the possibility of someone splitting the code if they don't like how Continuum runs things. Perry, you've done that yourself to this code's predecessor, so you know the risks. You did that in response to one constituency's moving the code in a direction you didn't like (or not moving it in one you did, I don't remember exactly), as in your example #2. So, while progress might be made when that happens, last time it hurt astronomers enough that you rolled your own and had to put several FTE on the problem. That split held back adoption of numpy both in the astronomy community and outside it, for like 5 years. Perhaps some governance would have saved you the effort and cost and the community the grief of the numarray split. Of course, lots of good eventually came from the split. I'd like to see at least some serious thought on how to protect the interests of the community under this very different development model. Trying it out and deciding we don't like it later will be a *much* harder thing to sort out. At the same time, the idea of multiplying the number of people actually working, and of having continuous builds and good issue tracking and all the rest, including the enhancements listed on Travis's web site, are very exciting! Let's just make sure we retain our community orientation with this new model. --jh--

2 1

Updated differences between 1.5.1 to 1.6.1
by Travis Oliphant Feb. 16, 2012

Feb. 16, 2012

For reference, here is the table that shows the actual changes between 1.5.1 and 1.6.1 at least on 64-bit platforms in terms of type-casting. I updated the comparison code to throw out changes that are just "spelling differences" (i.e. where 1.6.1 chooses to create an output dtype with an 'L' character code instead of a 'Q' which on 64-bit system is effectively the same). Mostly I'm happy with the changes (after a cursory review). As I expected, there are some real improvements. Of course, I haven't looked at the changes that occur when the scalar being used does not fit in the range of the array data-type. I don't see this change documented in the link that Mark sent previously. Is it somewhere else? Also, it looks like previously object arrays were returned for some coercions which now simply fail. Is that an expected result? At this point, I'm not going to recommend changes to 1.7 to deal with these type-casting changes --- at least this thread will serve to show some of what changes occurred if it bites anyone in the future. However, I will have other changes to NumPy 1.X that I will be proposing and writing (and directing other people to write as well). After some period of quiet, this might be a refreshing change. But, not all may see it that way. I'm confident that we can resolve any concerns people might have. Any feature additions will preserve backward compatibility in NumPy 1.X. Mark W. will be helping with some of these changes, but mostly he will be working on NumPy 2.0 which we have tentatively targeted for next January. We have a tentative target for NumPy 1.8 in June/July. So far, there are three developers who will be working on NumPy 1.8 (me, Francesc Alted, and Bryan Van de Ven). Mark Wiebe is slated to help us, as well, but I would like to sponsor him as much as possible on the work for NumPy 2.0. If anyone else would like to join us, please let me know off-list. There is room for another talented person on our team. In addition to a few select features in NumPy 1.8 (a list of which will follow in a later email), we will also be working on reviewing the list of bugs on Trac and fixing them, writing tests, and improving docstrings. I would also like to improve the state of the bug-tracker and get in place a continuous integration system for NumPy. We will be advertising our NumPy 1.8 roadmap and our NumPy 2.0 roadmap at PyCon, and are working on documents that describe plans which we are hoping will be reviewed and discussed on this list. I know that having more people working on the code-base for several months will be a different scenario than what has transpired in the past. Hopefully, this will be a productive time for everybody and our sometimes different perspectives will be able to coalesce into a better result for more people. Best regards, -Travis

4 5

PyArray_FromAny() steals a reference to PyArray_Descr* dtype
by Larsen, Brian A Feb. 15, 2012

Feb. 15, 2012

Hello all, the docs are unclear as to the reference counting on the inputs to the numpy C function PyArray_FromAny(). multiarraymodule.c in the PyArray_InnerProduct() code seems to imply that a reference to dtype is stolen in the PyArray_FromAny process. Meaning that I don't need/can't have a Py_DECREF(). Can anyone confirm this? Thanks much, Brian -- Brian A. Larsen ISR-1 Space Science and Applications Los Alamos National Laboratory PO Box 1663, MS-D466 Los Alamos, NM 87545 USA (For overnight add: SM-30, Bikini Atoll Road) Phone: 505-665-7691 Fax: 505-665-7395 email: balarsen(a)lanl.gov<mailto:[email protected]> Correspondence / Technical data or Software Publicly Available

2 1

Download page still points to SVN
by Fernando Perez Feb. 15, 2012

Feb. 15, 2012

Hi folks, I was just pointing a colleague to the 'official download page' for numpy so he could find how to grab current sources: http://new.scipy.org/download.html but I was quite surprised to find that it still points to SVN for both numpy and scipy. It would probably not be a bad idea to update those and point them to github... Cheers, f

8 32