Mailman 3 September 2019 - NumPy-Discussion

Low-level API for Random
by Kevin Sheppard Sept. 25, 2019

Sept. 25, 2019

There are some users of the NumPy C code in randomkit. This was never officially supported. There has been a long open issue to provide this officially. When I wrote randomgen I supplied .pdx files that make it simpler to write Cython code that uses the components. The lower-level API has not had much scrutiny and is in need of a clean-up. I thought this would also encourage users to extend the random machinery themselves as part of their project or code so as to minimize the requests for new (exotic) distributions to be included in Generator. Most of the generator functions follow a pattern random_DISTRIBUTION. Some have a bit more name mangling which can easily be cleaned up (like ranomd_gauss_zig, which should become PREFIX_standard_normal). Ralf Gommers suggested unprefixed names. I tried this in a local branch and it was a bit ugly since some of the distributions have common math names (e.g., gamma) and others are very short (e.g., t or f). I think a prefix is needed, and after looking through the C API docs npy_random_ seemed like a reasonable choice (since these live in numpy.random). Any thoughts on the following questions are welcome (others too): 1. Should there be a prefix on the C functions? 2. If so, what should the prefix be? 3. Should the legacy C functions be part of the API -- these are mostly the ones that produce or depend on polar transform normals (Box-Muller). I have a feeling no, but there may be reasons to prefer BM since they do not depend on rejection sampling. 4. Should low-level API be consumable like any other numpy C API by including the usual header locations and library locations? Right now, the pxd simplifies writing Cython but users have sp specify the location of the headers and source manually An alternative would be to provide a function like np.get_include() -> np.random.get_include() that would specialize in random. Kevin

8 19

NumPy Community Meeting Wednesday, Sep. 25
by Sebastian Berg Sept. 24, 2019

Sept. 24, 2019

Hi all, There will be a NumPy Community meeting Wednesday September 25 at 11 am Pacific Time. Everyone is invited to join in and edit the work-in- progress meeting topics and notes: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both Best wishes Sebastian

1 0

ANN: SfePy 2019.3
by Robert Cimrman Sept. 24, 2019

Sept. 24, 2019

I am pleased to announce release 2019.3 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method or by the isogeometric analysis (limited support). It is distributed under the new BSD license. Home page: http://sfepy.org Mailing list: https://mail.python.org/mm3/mailman3/lists/sfepy.python.org/ Git (source) repository, issue tracker: https://github.com/sfepy/sfepy Highlights of this release -------------------------- - interface to eigenvalue problem solvers in SLEPc - new Python 3 enabled Timer class and other Python 3 compatibility fixes For full release notes see [1]. Cheers, Robert Cimrman [1] http://docs.sfepy.org/doc/release_notes.html#id1 --- Contributors to this release in alphabetical order: Robert Cimrman Vladimir Lukes

1 0

Data filtering with np.genfromtxt
by Éric Depagne Sept. 24, 2019

Sept. 24, 2019

Hi all, I am reading large csv file, that has 8.5 million lines and 216 columns using genfromtxt. I'm not interested in all of the 216 columns, so I filter them out using the "usecols" and "converters" parameters. That works very well, but in my original large file, all the columns I extract are not filled with values. As expected in these cases, genfromtxt replaces them by nan and thus, in the final array, there are rows that contain these nans. I'd like to know if there is a way to filterout at the genfromtxt level the lines that do contain these nans, so that they do not appear in my final array. I'd like to have something like: genfromtxt extracts the line using the parameters I need. If the extracted line contains a NaN, do nothing and process the next line. If it has no NaNs, add it to the output array as usual. I could of course remove in the array created by genfromtxt() all the rows that contain nans (and x[~np.isnan(x).any(axis=1)] does it nicely), but I'd like to be able to get a given size of the output array. The idea is that I can get, for instance, the first 10000 (or any number) lines of the input file that contain all the columns I need not just the first 10000. I've found a few examples on SO that do some filtering, but the ones I've found do not process the extracted lines. Any help appreciated. Éric. -- Un clavier azerty en vaut deux ---------------------------------------------------------- Éric Depagne

1 0

DType Roadmap/NEP Discussion
by Sebastian Berg Sept. 23, 2019

Sept. 23, 2019

Hi all, to try and make some progress towards a decision since the broad design is pretty much settling from my side. I am thinking about making a meeting, and suggest Monday at 11am Pacific Time (I am open to other times though). My hope is to get everyone interested on board, so that we can make an informed decision about the general direction very soon. So just reach out, or discuss on the mailing list as well. The current draft for an NEP is here: https://hackmd.io/kxuh15QGSjueEKft5SaMug?both There are some design goals that I would like to clear up. I would prefer to avoid deep discussions of some specific issues, since I think the important decision right now is that my general start is in the right direction. It is not an easy topic, so my plan would be try and briefly summarize that and then hopefully clarify any questions and then we can discuss why alternatives are rejected. The most important thing is maybe gathering concerns which need to be clarified before we can go towards accepting the general design ideas. The main point of the NEP draft is actually captured by the picture in the linked document: DTypes are classes (such as Float64) and what is attached to the array is an instance of that class "<float64" or ">float64". Additionally, we would have AbstractDType classes which cannot be instantiated but define a type hierarchy. To list the main points: * DTypes are classes (corresponding to the current type number) * `arr.dtype` is an instances of its class, allowing to store additional information such as a physical unit, the string length. * Most things are defined in special dtype slots similar to Pythons type and number slots. They will be hidden and can be set through an init function similar to `PyType_FromSpec` [1]. * Promotion is defined primarily on the DType classes * Casting from one DType to another DType is defined by a new CastingImpl object (should become a special ufunc) - e.g. for strings, the CastingImpl is in charge of finding the correct string length * The AbstractDType hierarchy will be used to decide the signature when calling UFuncs. The main iffier points I can think of are: * NumPy currently uses value based promotion in some cases, which requires special AbstractDTypes to describe (and some legacy paths). (They are used use more like instances than typical classes) * Casting between flexible dtypes (such as strings) is a multi-step process to figure out the actual output dtype. - An example is: `np.can_cast("float64", "S3")` first finding that `Float64->String` is possible in principle and then asking the CastingImpl to find that `float64->S3` is not. * We have to break ABI compatibility in very minor, back-portable way. More smaller incompatibilities are likely [2]. * Since it is a major redesign, a lot of code has to be added/touched, although it is possible to channel much of it back into the old machinery. * A largish amount of new API around new DType type objects and also DTypeMeta type objects, which users can (although usually do not have to) subclass. However, most other designs will have similar issues. Basically, I currently really think this is "right", even if some details may end up a tricky. Best, Sebastian PS: The one thing outside the more general list above that I may want to discuss is how acceptable a global dict/mapping for dtype discovery during `np.array` coercion is (mapping python type -> dtype)... [1] https://docs.python.org/3/c-api/type.html#c.PyType_FromSpec [2] One possible issue may be "S0" which is normally used to denote what in the new API would be the `String` DType class.

5 9

ANN: mplcairo 0.2 release
by Antony Lee Sept. 23, 2019

Sept. 23, 2019

Dear all, I am pleased to announce the release of mplcairo 0.2. mplcairo is a Matplotlib backend based on the well-known cairo library, supporting output to both raster (including interactively) and vector formats. In other words, it provides the functionality of Matplotlib's {,qt5,gtk3,wx,tk,macos}{agg,cairo}, pdf, ps, and svg backends. Per Matplotlib's standard API, the backend can be selected by calling matplotlib.use("module://mplcairo.qt") or setting your MPLBACKEND environment variable to `module://mplcairo.qt` for Qt5, and similarly for other toolkits. mplcairo 0.2 adds support for cairo 1.17.2's high-precision floating point surfaces, simplifies the use of custom compositing operators (see `examples/operators.py`), a few other features listed in the changelog, as well as the usual bugfixes over 0.1. Enjoy, Antony Lee

1 0

Re: [Numpy-discussion] [pydata] NumPy proposal to remove the financial functions.
by Warren Weckesser Sept. 23, 2019

Sept. 23, 2019

On 9/21/19, Brendan Barnwell <brenbarn(a)gmail.com> wrote: > Hi Warren, > > I'm somewhat late to this discussion but I too have used the financial > functions. I looked at the discussion and the NEP and one thing I don't > understand is how the maintenance burden is alleviated if the functions are > > moved to a separate library. Is the intent of the Numpy devs to just > "dump" these functions into numpy_financial and then not maintain them? If > > not, what is achieved by moving them out of numpy? Brendan, There have been some more recent comments on the github issue that are relevant; take a look: https://github.com/numpy/numpy/issues/2880 It is true that when the functions are moved to numpy_financial, they will receive less attention from the core NumPy developers. Indeed, that is the point of the move. As you can see from the comments in the github issue and those quoted in the NEP, there is no interest among the current developers in maintaining these functions in NumPy. By having a smaller and more focused library that is explicitly for financial functions, it is possible that new developers with greater interest and expertise in that domain will be motivated to contribute. See, for example, Graham Duncan's recent comments in the github issue. It remains to be seen whether we'll end up with a significantly *better* library for financial calculations once the transition is complete. For the most visibility among the NumPy developers, it would be best to continue the conversation in a NumPy venue, either the github issue or the NumPy mailing list. I've cc'ed this email to the NumPy mailing list. Warren > > On Thursday, September 19, 2019 at 8:25:52 AM UTC-7, Warren Weckesser > wrote: >> >> On 9/8/19, Warren Weckesser <warren....(a)gmail.com <javascript:>> wrote: >> > NumPy is considering a NEP (NumPy Enhancement Proposal) that proposes >> the >> > deprecation and ultimate removal of the financial functions from NumPy. >> > >> > The functions would be moved to an independent library. The mailing >> list >> > discussion of this proposal is at >> > >> > >> > >> http://numpy-discussion.10968.n7.nabble.com/NEP-32-Remove-the-financial-fun… >> >> > >> > or >> > >> > >> > >> https://mail.python.org/pipermail/numpy-discussion/2019-September/079965.ht… >> >> > >> > The first message in that thread includes the proposed NEP. >> > >> > There have been a couple suggestions to ask about this on the Pandas >> > mailing list. Contributions to the thread in the numpy-discussion >> mailing >> > list would be appreciated! >> >> >> FYI: The proposal to accept the NEP to remove the financial functions >> has been made on the NumPy-Discussion mailing list: >> >> https://mail.python.org/pipermail/numpy-discussion/2019-September/080074.ht… >> >> >> Warren >> >> > >> > Thanks, >> > >> > Warren >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups >> > "PyData" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an >> > email to pyd...(a)googlegroups.com <javascript:>. >> > To view this discussion on the web visit >> > >> https://groups.google.com/d/msgid/pydata/8458a06f-ead7-4f7e-b288-4a15a60024…. >> >> >> > >> > > -- > You received this message because you are subscribed to the Google Groups > "PyData" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to pydata+unsubscribe(a)googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/pydata/8983e694-7067-44cb-a35f-5c173d44c1…. >

1 0

User Stories for https://numpy.org
by Inessa Pawson Sept. 21, 2019

Sept. 21, 2019

The NumPy web team has begun redesigning https://numpy.org determined to transform the website into a welcoming and useful digital hub of all things NumPy. We are inviting all members of our large and diverse community to submit their user stories to help us fulfill our mission. *What are we looking for?* In simple, concise terms, a user story describes what a user needs to accomplish while visiting a website. Anyone who reads the user story must be able to understand why the user needs the functionality, and what is required to implement the story. User stories must have acceptance criteria. The shorter the story the better. *Examples of good user stories* 1. Lotte is a library author that depends on NumPy. She is looking for information about major changes and a release date of the next version of NumPy. She would like to easily find it on the website instead of contacting the core team. 2. Yu Yan was introduced to NumPy in her first week of the Foundations of Data Science class. She is looking for a NumPy tutorial for absolute beginners in Mandarin. 3. Tiago is a software developer. By day, he builds enterprise applications for a Fortune 100 company. By night, he cultivates his academic interests in statistics and computer science using various Python libraries. Tiago has an idea for a new NumPy feature and would like to implement it. He is looking for information on how to contact the person(s) in charge of such decisions. *Please note* that at this stage of the numpy.org redesign our focus is not on expanding or improving the documentation but, rather, developing high-level content to provide information about the project to a multitude of stakeholders. -- Every good wish, *Inessa Pawson* NumPy Web Team

2 1

Proposal to accept NEP 32: Remove the financial functions from NumPy
by Warren Weckesser Sept. 19, 2019

Sept. 19, 2019

NEP 32 is available at https://numpy.org/neps/nep-0032-remove-financial-functions.html Recent timeline: - 30-Aug-2019 - A pull request with NEP 32 submitted. - 03-Sep-2019 - Announcement of the NEP 32 pull request on the NumPy-Discussion mailing list, with the text of the NEP included in the email. - 08-Sep-2019 - NEP 32 announced on the PyData mailing list (not standard procedure, but suggested in a response to the email in NumPy-Discussion). - 09-Sep-2019 - NEP 32 pull request merged. - 11-Sep-2019 - Emails sent to the NumPy-Discussion and PyData mailing lists with links to the online version of the NEP. Only one user (speaking for a group of 12 or so) expressed a preference for keeping the functions in NumPy, and that user acknowledged "Probably not a huge inconvenience if we would have to use another library". (The NEP includes a plan to provide an alternative package for the functions.) Several other users were in favor of removing them. Among the current NumPy developers who have expressed an opinion, all are in favor of removing the functions. There have been no additional email responses since the reminder was sent on September 11. In accordance with NEP 0, I propose that the status of NEP 32 be changed to *Accepted*. If there are no substantive objections within 7 days from this email, then the NEP will be accepted; see NEP 0 for more details ( https://numpy.org/neps/nep-0000.html#how-a-nep-becomes-accepted). Warren

1 0

keeping all meeting notes and docs in new repo?
by Ralf Gommers Sept. 17, 2019

Sept. 17, 2019

Hi all, We have had community calls for quite a while, the minutes of which are kept in https://github.com/BIDS-numpy/docs. That's quite hard to discover, it would be better if those lived under the NumPy GitHub org. Also, we have minutes from Season of Docs and website redesign calls, plus occasionally some other docs (e.g. the roadmap drafts were on hackmd.io). Would it make sense to add a new repo to contain all such meeting minutes and docs? Presentations and proposals may make sense to add as well - several people have given presentations or submitted proposals on behalf of the project. Inessa also suggested to enable HackMD Hub (see https://hackmd.io/c/tutorials/%2Fs%2Flink-with-github) so we get automatic versioning for some HackMD documents. I haven't used it before, but it looks good. Thoughts? Cheers, Ralf

3 2