Mailman 3 January 2014 - NumPy-Discussion

cannot decode 'S'
by josef.pktd＠gmail.com Jan. 23, 2014

Jan. 23, 2014

truncating null bytes in 'S' breaks decoding that needs them >>> a = np.array([si.encode('utf-16LE') for si in ['Õsc', 'zxc']], dtype='S') >>> a array([b'\xd5\x00s\x00c', b'z\x00x\x00c'], dtype='|S6') >>> [ai.decode('utf-16LE') for ai in a] Traceback (most recent call last): File "<pyshell#118>", line 1, in <module> [ai.decode('utf-16LE') for ai in a] File "<pyshell#118>", line 1, in <listcomp> [ai.decode('utf-16LE') for ai in a] File "C:\Programs\Python33\lib\encodings\utf_16_le.py", line 16, in decode return codecs.utf_16_le_decode(input, errors, True) UnicodeDecodeError: 'utf16' codec can't decode byte 0x63 in position 4: truncated data messy workaround (arrays in contrast to scalars are not truncated in `tostring`) >>> [a[i:i+1].tostring().decode('utf-16LE') for i in range(len(a))] ['Õsc', 'zxc'] Found while playing with examples in the other thread. Josef

2 1

IRR
by Nathaniel Smith Jan. 23, 2014

Jan. 23, 2014

Hey all, We have a PR languishing that fixes np.irr to handle negative rate-of-returns: https://github.com/numpy/numpy/pull/4210 I don't even know what "IRR" stands for, and it seems rather confusing from the discussion there. Anyone who knows something about the issues is invited to speak up... -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org

1 0

Re: [Numpy-discussion] (no subject)
by jennifer stone Jan. 22, 2014

Jan. 22, 2014

> >What are your interests and experience? If you use numpy, are there things > >you would like to fix, or enhancements you would like to see? > > Chuck > > I am an undergraduate student with CS as major and have interest in Math and Physics. This has led me to use NumPy and SciPy to work on innumerable cases involving special polynomial functions and polynomials like Legendre polynomials, Bessel Functions and so on. So, The packages are closer known to me from this point of view. I have a* few proposals* in mind. But I don't have any idea if they are acceptable within the scope of GSoC 1. Many special functions and polynomials are neither included in NumPy nor on SciPy.. These include Ellipsoidal Harmonic Functions (lames function), Cylindrical Harmonic function. Scipy at present supports only spherical Harmonic function. Further, why cant we extend SciPy to incorporate* Inverse Laplace Transforms*? At present Matlab has this amazing function *ilaplace* and SymPy does have *Inverse_Laplace_transform* but it would be better to incorporate all in one package. I mean SciPy does have function to evaluate laplace transform After having written this, I feel that this post should have been sent to SciPy but as a majority of contributors are the same I proceed. Please suggest any other possible projects, as I would like to continue with SciPy or NumPy, preferably NumPy as I have been fiddling with its source code for a month now and so am pretty comfortable with it. As for my experience, I have known C for past 4 years and have been a python lover for past 1 year. I am pretty new to open source communities, started before a manth and a half. regards Jennifer

4 4

accumulation operation
by Ralf Juengling Jan. 22, 2014

Jan. 22, 2014

Executing the following code, >>> import numpy as np >>> a = np.zeros((3,)) >>> w = np.array([0, 1, 0, 1, 2]) >>> v = np.array([10.0, 1, 10.0, 2, 9]) >>> a[w] += v I was expecting 'a' to be array([20., 3., 9.]. Instead I get >>> a array([ 10., 2., 9.]) This with numpy version 1.6.1. Is there another way to do the accumulation I want? Thanks, Ralf

3 2

fromiter cannot create array of object - was: Creating an ndarray from an iterable, over sequences
by Dr. Leo Jan. 22, 2014

Jan. 22, 2014

Hi, thanks. Both recarray and itertools.chain work just fine in the example case. However, the real purpose of this is to read strings from a large xml file into a pandas DataFrame. But fromiter cannot create arrays of dtype 'object'. Fixed length strings may be worth trying. But as the xml schema does not guarantee a max. length, and pandas generally uses 'object' arrays for strings, I see no better way than creating the array through list comprehensions and turn it into a DataFrame. Maybe a variable length string/unicode type would help in the long term. Leo > > I would like to write something like: > > In [25]: iterable=((i, i**2) for i in range(10)) > > In [26]: a=np.fromiter(iterable, int32) > --------------------------------------------------------------------------- > ValueError Traceback (most recent call > last) > <ipython-input-26-5bcc2e94dbca> in <module>() > ----> 1 a=np.fromiter(iterable, int32) > > ValueError: setting an array element with a sequence. > > > Is there an efficient way to do this? > Perhaps you could just utilize structured arrays ( http://docs.scipy.org/doc/numpy/user/basics.rec.html), like: iterable= ((i, i**2) for i in range(10)) a= np.fromiter(iterable, [('a', int32), ('b', int32)], 10) a.view(int32).reshape(-1, 2) You could use itertools: >>> from itertools import chain >>> g = ((i, i**2) for i in range(10)) >>> import numpy >>> numpy.fromiter(chain.from_iterable(g), numpy.int32).reshape(-1, 2)

2 1

Re: [Numpy-discussion] A one-byte string dtype?
by David Goldsmith Jan. 22, 2014

Jan. 22, 2014

Date: Tue, 21 Jan 2014 19:20:12 +0000 > From: Robert Kern <robert.kern(a)gmail.com> > Subject: Re: [Numpy-discussion] A one-byte string dtype? > > The wiki is frozen. Please do not add anything to it. It plays no role in > our current development workflow. Drafting a NEP or two and iterating on > them would be the next step. > > -- > Robert Kern > OK, well that's definitely beyond my level of expertise. DG

2 1

Re: [Numpy-discussion] A one-byte string dtype?
by David Goldsmith Jan. 21, 2014

Jan. 21, 2014

On Tue, Jan 21, 2014 at 10:00 AM, <numpy-discussion-request(a)scipy.org>wrote: > Date: Tue, 21 Jan 2014 09:53:25 -0800 > From: David Goldsmith <d.l.goldsmith(a)gmail.com> > Subject: Re: [Numpy-discussion] A one-byte string dtype? > To: numpy-discussion(a)scipy.org > Message-ID: > <CAFtPsZqRrDxrshBMVyS+Z= > 7aLTPXMrz4MiujY2XEbyi_fY5WWw(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > > Date: Tue, 21 Jan 2014 17:35:26 +0000 > > From: Nathaniel Smith <njs(a)pobox.com> > > Subject: Re: [Numpy-discussion] A one-byte string dtype? > > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > > Message-ID: > > <CAPJVwB=+47ofYvnvN76= > > KE3xLGa2+gz+Qd4F0xS2UBoEYsgedA(a)mail.gmail.com> > > Content-Type: text/plain; charset="utf-8" > > > > On 21 Jan 2014 17:28, "David Goldsmith" <d.l.goldsmith(a)gmail.com> wrote: > > > > > > > > > Am I the only one who feels that this (very important--I'm being > sincere, > > not sarcastic) thread has matured and specialized enough to warrant it's > > own home on the Wiki? > > > > Sounds plausible, perhaps you could write up such a page? > > > > -n > > > > I can certainly get one started (but I don't think I can faithfully > summarize all this thread's current content, so I apologize in advance for > leaving that undone). > > DG > OK, I'm "lost" already: is there general agreement that this should "jump" straight to one or more NEP's? If not (or if there should be a Wiki page for it additionally), should such become part of the NumPy Wiki @ Sourceforge or the SciPy Wiki at the scipy.org site? If the latter, is one's SciPy Wiki login the same as one's mailing list subscriber maintenance login? I guess starting such a page is not as trivial as I had assumed. DG

2 1

Re: [Numpy-discussion] A one-byte string dtype?
by David Goldsmith Jan. 21, 2014

Jan. 21, 2014

Am I the only one who feels that this (very important--I'm being sincere, not sarcastic) thread has matured and specialized enough to warrant it's own home on the Wiki? DG

4 4

Re: [Numpy-discussion] A one-byte string dtype?
by David Goldsmith Jan. 21, 2014

Jan. 21, 2014

> Date: Tue, 21 Jan 2014 17:35:26 +0000 > From: Nathaniel Smith <njs(a)pobox.com> > Subject: Re: [Numpy-discussion] A one-byte string dtype? > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Message-ID: > <CAPJVwB=+47ofYvnvN76= > KE3xLGa2+gz+Qd4F0xS2UBoEYsgedA(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > On 21 Jan 2014 17:28, "David Goldsmith" <d.l.goldsmith(a)gmail.com> wrote: > > > > > > Am I the only one who feels that this (very important--I'm being sincere, > not sarcastic) thread has matured and specialized enough to warrant it's > own home on the Wiki? > > Sounds plausible, perhaps you could write up such a page? > > -n > I can certainly get one started (but I don't think I can faithfully summarize all this thread's current content, so I apologize in advance for leaving that undone). DG

1 0

A one-byte string dtype?
by Chris Barker Jan. 21, 2014

Jan. 21, 2014

Folks, I've been blathering away on the related threads a lot -- sorry if it's too much. It's gotten a bit tangled up, so I thought I'd start a new one to address this one question (i.e. dont bring up genfromtext here): Would it be a good thing for numpy to have a one-byte--per-character string type? We did have that with the 'S' type in py2, but the changes in py3 have made it not quite the right thing. And it appears that enough people use 'S' in py3 to mean 'bytes', so that we can't change that now. The only difference may be that 'S' currently auto translates to a bytes object, resulting in things like: np.array(['some text',], dtype='S')[0] == 'some text' yielding False on Py3. And you can't do all the usual text stuff with the resulting bytes object, either. (and it probably used the default encoding to generate the bytes, so will barf on some inputs, though that may be unavoidable.) So you need to decode the bytes that are given back, and now that I think about it, I have no idea what encoding you'd need to use in the general case. So the correct solution is (particularly on py3) to use the 'U' (unicode) dtype for text in numpy arrays. However, the 'U' dtype is 4 bytes per character, and that may be "too big" for some use-cases. And there is a lot of text in scientific data sets that are pure ascii, or at least some 1-byte-per-character encoding. So, in the spirit of having multiple numeric types that use different amounts of memory, and can hold different ranges of values, a one-byte-per character dtype would be nice: (note, this opens the door for a 2-byte per (UCS-2) dtype too, I personally don't think that's worth it, but maybe that's because I'm an english speaker...) It could use the 's' (lower-case s) type identifier. For passing to/from python built-in objects, it would * Allow either Python bytes objects or Python unicode objects as input a) bytes objects would be passed through as-is b) unicode objects would be encoded as latin-1 [note: I'm not entirely sure that bytes objects should be allowed, but it would provide an nice efficiency in a fairly common case] * It would create python unicode text objects, decoded as latin-1. Could we have a way to specify another encoding? I'm not sure how that would fit into the dtype system. I've explained the latin-1 thing on other threads, but the short version is: - It will work perfectly for ascii text - It will work perfectly for latin-1 text (natch) - It will never give you an UnicodeEncodeError regardless of what arbitrary bytes you pass in. - It will preserve those arbitrary bytes through a encoding/decoding operation. (it still wouldn't allow you to store arbitrary unicode -- but that's the limitation of one-byte per character...) So: Bad idea all around: shut up already! or Fine idea, but who's going to write the code? not me! or We really should do this. (of course, with the options of amending the above not-very-fleshed out proposal) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

7 24