Mailman 3 February 2005 - NumPy-Discussion

NumPy on 64-bit Linux (Opteron)
by konrad.hinsen＠laposte.net Feb. 9, 2005

Feb. 9, 2005

Is anyone here using NumPy on Opteron machines running Linux in 64 bit mode? I am running into problems such as: Python 2.4 (#4, Jan 18 2005, 18:06:45) [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from Numeric import * >>> a = arange(4) >>> a array([0, 1, 2, 3]) >>> a.shape = (2, 2) Traceback (most recent call last): File "<stdin>", line 1, in ? ValueError: total size of new array must be unchanged This is with Python 2.4 and Numeric 23.7. Before looking into this myself, I thought I'd ask here, perhaps someone has already found a fix. Konrad. -- ------------------------------------------------------------------------ ------- Konrad Hinsen Laboratoire Leon Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen(a)cea.fr ------------------------------------------------------------------------ -------

6 8

Proposal for making of Numarray a real Numeric 'NG'
by Francesc Altet Feb. 9, 2005

Feb. 9, 2005

Hi List, I would like to make a formal proposal regarding with the subject of previous discussions in that list. This message is a bit long, but I've tried my best to expose my thoughts as clearly as possible. Is Numarray a good replacement of Numeric? ========================================== It has been a debate lately with regard to the convinience of claiming numarray to be a replacement of Numeric. Perhaps the main source for this claim has been the home page of the Numeric project [1]: """ If you are new to Numerical Python, please use Numarray. The older module, Numeric, is unsupported. At this writing Numarray is slower for very small arrays but faster for large ones. Numarray contains facilities to help you convert older code to use it. Some parts of the community have not made the switch yet but the Numarray libraries have been carefully named differently so that Numeric and Numarray can coexist in one application. """ So the paragraph is giving the impression that Numeric was going to be deprecated. While I recognize that I was between those that this statement lent us to think about numarray as a kind of 'Next Generation of Numeric', it seems now (from the previous discussions) that this was sort of unfortunate/misleading observation. In fact, Perry Greenfield, one of the main authors of numarray will be taking some steps in order to correct that observation in the near future [2]. However, I'd like to believe (and with me, quite a few more people for sure) that the mentioned statement, apart of creating some confusion, would eventually easy the long term convergence of both packages. This would be great not only to unify efforts, but also to allow the inclusion of Numeric/Numarray in the Python Standard Library, which would be a Good Thing. Numarray vs Numeric: Pros and Cons ================================== It's worth remembering that Numeric has been a major breakthrough in introducing the capability to deal with large (homogeneous) datasets in Python in a very efficient mannner. In my opinion Numarray is, generally speaking, a very good package as well with many interesting new features that lack Numeric. Between the main advantages of Numarray vs Numeric I can list the next (although I can be a bit misleaded here because of my own user cases of both libraries): - Memory-mapped objects: Allow working with on-disk numarray objects like if they were in-memory. - RecArrays: Objects that allow to deal with heterogeneous datasets (tables) in an efficient manner. This ought to be very beneficial in many fields. - CharArrays: Allow to work with large amounts of fixed and variable length strings. I see this implementation much more powerful that Numeric. - Index arrays within subscripts: e.g. if ind = array([4, 4, 0, 2]) and x = 2*arange(6), x[inx] results in array([8, 8, 0, 4]) - New design interface: We should not forget that numarray has been designed from the ground with Python Library integration in mind (or at least, this is my impression). So, it should have more chances (if there is some hope) to enter in the Standard Library than Numeric. [See [3] for a more acurate description of differences] In this point, it would be also fair to recognize the important effort that has been done by the Numarray crew (and others) to create a fairly good replacement for Numeric: the API is getting closer bit a bit, the numerix module makes easier to support both Numeric and numarray by an application (see [5] for a concrete case of switching between Numeric and Numarray in SciPy or [6] for matplotlib), the current effort to support Numarray in SciPy, and last but not least, their good responsiveness to enhancements in that respect. The real problem for Numarray: Object Creation Time =================================================== On the other hand, the main drawback of Numarray vs Numeric is, in my opinion, its poor performance regarding object creation. This might look like a banal thing at first glance, but it is not in many cases. One example recently reported in this list is: >>> from timeit import Timer >>> setup = 'import Numeric; a = Numeric.arange(2000);a.shape=(1000,2)' >>> Timer('for i in range(len(a)): row=a[i]', setup).timeit(number=100) 0.12782907485961914 >>> setup = 'import numarray; a = numarray.arange(2000);a.shape=(1000,2)' >>> Timer('for i in range(len(a)): row=a[i]', setup).timeit(number=100) 1.2013700008392334 So, numarray performs 10 times slower than Numeric not because its indexing access code would be 10 times slower, but mainly due to the fact that object creation is between 5 and 10 times slower, and the loop above implies an object creation on each iteration. Other case of use where object creation time is important can be seen in [4]. Proposal for making of Numarray a real Numeric 'NG' (Next Generation) ===================================================================== Provided that the most important reason (IMO) to not consider Numarray to be a good replacement of Numeric is object creation time, I would like to propose a coordinated effort to solve this precise issue. First of all, it would be nice if the most experienced people with Numarray (i.e. the Numarray crew) would give a deep analysis to that, and end with a series of small, autocontained benchmarks files that clearly exposes the possible bottlenecks. This maybe hard to do, but this is crucial. Once the problem has been reduced to optimize these small, auto-contained benchmarks, they can be made publicly accessible together with an explanation of what the problem is and what the benchmarks are intended for. After this, I suggest a call for contributions (in this list and scipy list, for example) on optimizing this code and spark discussions on that (a Wiki can work great here). I'm pretty sure that there is enough brain and challenge-hungry people in these lists to contribute solving the problem. If after these efforts, there are issues that can't be solved yet, at least the problem would be much more centered, and much more people can think on that (hopefully, the solution may not depend on the intricacies of Numeric/Numarray), so it maybe possible to sent it to the general Python list and hope that some guru would be willing to help us on that. Well, this is my proposal. Uh, sorry for the length of the message. Perhaps you may think that I've smoked too much and maybe you are right. However, I'm so convinced that such a Numeric/Numarray unification is going to be a Very Good Thing that I unrecklessly spend some time making this proposal (and look forward contributing in some way or another if this is going to be done). Cheers, [1] http://www.pfdubois.com/numpy/ [2] http://sourceforge.net/mailarchive/message.php?msg_id=10608642 [3] http://stsdas.stsci.edu/numarray/numarray-1.1.html/node18.html [4] http://sourceforge.net/mailarchive/message.php?msg_id=10582525 [5] http://aspn.activestate.com/ASPN/Mail/Message/scipy-dev/2299767 [6] http://matplotlib.sourceforge.net/matplotlib.numerix.html -- >qo< Francesc Altet http://www.carabos.com/ V V Cárabos Coop. V. Enjoy Data ""

6 9

Efficiently converting a numarray object to Numeric
by Francesc Altet Feb. 9, 2005

Feb. 9, 2005

Hi, I'm wondering if there is a more efficient way to convert a numarray object into a Numeric one than: >>> import Numeric >>> import numarray >>> from time import time >>> a = numarray.arange(100*1000) >>> t1=time();b=Numeric.array(a);time()-t1 0.43448090553283691 Any suggestions? -- >qo< Francesc Altet http://www.carabos.com/ V V Cárabos Coop. V. Enjoy Data ""

2 2

numarray.codegenerator.UfuncModule nary ufuncs
by Simon Burton Feb. 9, 2005

Feb. 9, 2005

It looks like I may need to make n-ary (multi argument) ufuncs. The use case is something like this: some tricky function operates on single values but has many parameters (eg. offset and multiply). So the ufunc is called with an array as the first argument, but just scalars in the other arguments. I'm wondering how far is numarray.codegenerator.UfuncModule from handling n-ary functions ? If it's just not on the radar, i'll need to rethink my approach... bye for now, Simon.

2 1

numarray: "concatenate along new axis" returns 'unexpected' type
by Sebastian Haase Feb. 8, 2005

Feb. 8, 2005

Hi, from the numarray documentation: <doc> concatenate(arrs, axis=0) Returns a new array containing copies of the data contained in all arrays # of arrs= (a0, a1, ... an). The arrays ai will be concatenated along the specified axis (default=0). All arrays ai must have the same shape along every axis except for the one specified in axis. To concatenate arrays along a newly created axis, you can use array((a0, ..., an)), as long as all arrays have the same shape. </doc> Is it a bug that array((a0,a1)) returns different type that a0 ? (type(a0)==type(a1)) >>> a = na.zeros((3,3),na.UInt16) >>> b = na.zeros((3,3),na.UInt16) >>> na.concatenate((a,b)) [[0 0 0] [0 0 0] [0 0 0] [0 0 0] [0 0 0] [0 0 0]] >>> _.type() UInt16 >>> na.array((a,b)) [[[0 0 0] [0 0 0] [0 0 0]] [[0 0 0] [0 0 0] [0 0 0]]] >>> _.type() Int32 Thanks, Sebastian Haase

3 5

Re: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)
by Bruce Southey Feb. 8, 2005

Feb. 8, 2005

Hi, I should also add that on a P3 (same info as below), there is no problem as the correct dimensions are returned. Regards Bruce ---- Original message ---- >Date: Tue, 8 Feb 2005 09:06:04 -0600 >From: Bruce Southey <southey(a)uiuc.edu> >Subject: Re: [Numpy-discussion] NumPy on 64-bit Linux (Opteron) >To: konrad.hinsen(a)laposte.net, numpy-discussion <numpy-discussion(a)lists.sourceforge.net> > >Hi, >Yes (Python 2.3.3 [GCC 3.3.3 (SuSE Linux)]) with Numeric 23.7 (clean install). > >It is to do with the function: >extern PyObject * PyArray_Reshape(PyArrayObject *self, PyObject *shape) > >the variable s_known is not being assigned correctly (gets set to zero). I >think that stems from this call that doesn't set the dimensions correctly: > >PyArray_As1D(&shape, (char **)&dimensions, &n, PyArray_LONG) > >dimensions in the second loop has value 0. > >Regards >Bruce > >---- Original message ---- >>Date: Tue, 8 Feb 2005 12:44:18 +0100 >>From: konrad.hinsen(a)laposte.net >>Subject: [Numpy-discussion] NumPy on 64-bit Linux (Opteron) >>To: numpy-discussion <numpy-discussion(a)lists.sourceforge.net> >> >>Is anyone here using NumPy on Opteron machines running Linux in 64 bit >>mode? I am running into problems such as: >> >>Python 2.4 (#4, Jan 18 2005, 18:06:45) >>[GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2 >>Type "help", "copyright", "credits" or "license" for more information. >> >>> from Numeric import * >> >>> a = arange(4) >> >>> a >>array([0, 1, 2, 3]) >> >>> a.shape = (2, 2) >>Traceback (most recent call last): >> File "<stdin>", line 1, in ? >>ValueError: total size of new array must be unchanged >> >>This is with Python 2.4 and Numeric 23.7. Before looking into this >>myself, I thought I'd ask here, perhaps someone has already found a >>fix. >> >>Konrad. >>-- >>------------------------------------------------------------------------ >>------- >>Konrad Hinsen >>Laboratoire Leon Brillouin, CEA Saclay, >>91191 Gif-sur-Yvette Cedex, France >>Tel.: +33-1 69 08 79 25 >>Fax: +33-1 69 08 82 61 >>E-Mail: khinsen(a)cea.fr >>------------------------------------------------------------------------ >>------- >> >> >> >>------------------------------------------------------- >>SF email is sponsored by - The IT Product Guide >>Read honest & candid reviews on hundreds of IT Products from real users. >>Discover which products truly live up to the hype. Start reading now. >>http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click >>_______________________________________________ >>Numpy-discussion mailing list >>Numpy-discussion(a)lists.sourceforge.net >>https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > >------------------------------------------------------- >SF email is sponsored by - The IT Product Guide >Read honest & candid reviews on hundreds of IT Products from real users. >Discover which products truly live up to the hype. Start reading now. >http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion(a)lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion

1 0

Re: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)
by Bruce Southey Feb. 8, 2005

Feb. 8, 2005

Hi, Yes (Python 2.3.3 [GCC 3.3.3 (SuSE Linux)]) with Numeric 23.7 (clean install). It is to do with the function: extern PyObject * PyArray_Reshape(PyArrayObject *self, PyObject *shape) the variable s_known is not being assigned correctly (gets set to zero). I think that stems from this call that doesn't set the dimensions correctly: PyArray_As1D(&shape, (char **)&dimensions, &n, PyArray_LONG) dimensions in the second loop has value 0. Regards Bruce ---- Original message ---- >Date: Tue, 8 Feb 2005 12:44:18 +0100 >From: konrad.hinsen(a)laposte.net >Subject: [Numpy-discussion] NumPy on 64-bit Linux (Opteron) >To: numpy-discussion <numpy-discussion(a)lists.sourceforge.net> > >Is anyone here using NumPy on Opteron machines running Linux in 64 bit >mode? I am running into problems such as: > >Python 2.4 (#4, Jan 18 2005, 18:06:45) >[GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2 >Type "help", "copyright", "credits" or "license" for more information. > >>> from Numeric import * > >>> a = arange(4) > >>> a >array([0, 1, 2, 3]) > >>> a.shape = (2, 2) >Traceback (most recent call last): > File "<stdin>", line 1, in ? >ValueError: total size of new array must be unchanged > >This is with Python 2.4 and Numeric 23.7. Before looking into this >myself, I thought I'd ask here, perhaps someone has already found a >fix. > >Konrad. >-- >------------------------------------------------------------------------ >------- >Konrad Hinsen >Laboratoire Leon Brillouin, CEA Saclay, >91191 Gif-sur-Yvette Cedex, France >Tel.: +33-1 69 08 79 25 >Fax: +33-1 69 08 82 61 >E-Mail: khinsen(a)cea.fr >------------------------------------------------------------------------ >------- > > > >------------------------------------------------------- >SF email is sponsored by - The IT Product Guide >Read honest & candid reviews on hundreds of IT Products from real users. >Discover which products truly live up to the hype. Start reading now. >http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion(a)lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion

1 0

MA and iterators
by Andrew McNamara Feb. 7, 2005

Feb. 7, 2005

I did some benchmarking today and discovered that iterating over MA arrays is 14 times slower than iterating over a Numeric array (or a python list). One simple solution to this is to add iterator support to the MaskedArray class. To prove the principle, I patched MaskedArray on the fly: import MA def ma__iter__(self): "Get an MA iterator." def scaler_iter(d, m): for i, v in enumerate(d): if m[i]: v = masked yield v def array_iter(d, m, fill_value): ss = d.spacesaver() tc = d.typecode() for i, v in enumerate(d): yield MA.array(v, typecode=tc, copy=1, savespace=ss, mask=m[i], fill_value=fill_value) if self._mask is None: return iter(self._data) if len(self.shape) > 1: return array_iter(self._data, self._mask) else: return scalar_iter(self._data, self._mask, self.fill_value()) MA.MaskedArray.__iter__ = ma__iter__ Note that this implementation requires generator support. -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/

1 0

Re: [Numpy-discussion] Numeric3
by Bruce Southey Feb. 6, 2005

Feb. 6, 2005

Hi, I am not the developer of any of Numpy (in any form) so take my thoughts 'with a grain of salt' because I am not doing the coding. I am really very thankful that people have developed Numeric and numarray for Python! Many of the comments I fully agree with especially on installation. Large packages, like SciPy, require time and effort to ensure every thing works. Also, various platforms lack compilers (i.e. MS Windows are not by default and free Fortran 90 compilers for Linux). A user without 'system admin' experience or access can not be expected get these compilers installed especially if the dependencies don't compile or are incorrectly compiled. With all the arguments that have been presented so far, I fail to see any need for at least a third Python numerical option, especially one that may not support many needs. This discussion shows that there doesn't seem to be the 'maturity' of the code base to justify a push towards the Python Standard Library. By maturity, I mean that there are still too many differences between Numeric and numarray that, while appear to be solvable, are not trivial to address at the present time. (This is also one of the main reasons why many different versions of the same thing exist.) I would like to see a single NumPy option that everyone can use. If numarray or Numeric3 does that then great as I will support either. However, I do fall in the numarray camp as I do tend to work with large matrices (plus features like Masked Arrays are a big plus). Regards Bruce Southey

3 2

Optimization question for ufuncs
by Travis Oliphant Feb. 4, 2005

Feb. 4, 2005

I've been thinking lately about ufuncs and I would love to hear the opinion of others. I like what numarray has done with the temporary buffer ideas so that full copies are never made if they are just going to be thrown away. This has led to other thoughts about possible improvements to the ufunc object to support "ufunc chaining" so that array operations on expressions don't have to create any temporary copies (using buffers instead) --- I think I remember the numarray guys thinking along these lines as well. Regardless, there is always an inner for loop (for each type) that performs the requested operation. The question I have is whether to assume unit strides for the inner loop. The current Numeric ufunc inner loops allow for discontiguous memory to be accessed during the loop (non-unit strides). I'm not sure what numarray does, I think it only allows for unit strides and uses temporary buffers to support discontiguous arrays. Is this requirement for unit-strides on the inner loop a good one? Does it allow faster code to be compiled? Is it part of the reason that numarray is a little faster on large arrays? I am not an optimization expert, though I've read a bit as of late. I'm just wondering what the experts on this list think about unit-strides versus non unit-strides on the inner loop? Thanks, Travis O.

1 0