Mailman 3 January 2024 - NumPy-Discussion

New Ruff rule for migrating to NumPy 2.0
by Mateusz Sokol Jan. 11, 2024

Jan. 11, 2024

Hi all! Some time ago we added a new rule to Ruff linter, "NPY201", which updates the codebase to a NumPy 2.0 compatible version. You can read about it in the migration guide: https://numpy.org/devdocs/numpy_2_0_migration_guide.html#ruff-plugin And on the Ruff docs website: https://docs.astral.sh/ruff/rules/numpy2-deprecation/ (it's still in a "preview" mode but available since 0.1.4 release). Best regards, Mateusz

2 1

[ANN] sparse 0.15.1 released
by Hameer Abbasi Jan. 10, 2024

Jan. 10, 2024

Hi all, On behalf of the sparse developers, I’m pleased to announce the release of sparse 0.15.1. This is a significant release with Array API support, pyproject.toml packaging, einsum support and other miscellaneous improvements. Source Code: https://github.com/pydata/sparse Documentation: https://sparse.pydata.org/en/stable/ Discussions: https://github.com/pydata/sparse/discussions Best regards, Hameer Abbasi

1 0

PyData yerevan chapter - sprint
by Habet Madoyan Jan. 9, 2024

Jan. 9, 2024

Dear community members, I am Habet, the co-founder and organizer of the PyData Yerevan chapter. We are planning to host a contributors' sprint for NumPy in Yerevan, Armenia, during one of our upcoming monthly meetups in either February or March. We have allocated a modest budget to facilitate the participation of individuals from Europe and Asia who can assist in organizing this event. If you have relevant experience and are interested in contributing, please feel free to reach out to me at *hmadoyan@aua*. thanks habet -- Habet Madoyan, PhD | Data Science Program Chair +374 60 612 632, hmadoyan(a)aua.am <http://www.aua.am> _________________________________________ 40 Baghramyan Avenue, Yerevan 0019, Republic of Armenia http://cse.aua.am/ds To know wisdom and instruction; to perceive the words of understanding;

1 0

argmin of empty masked sequence not correctly handled (returns 0)
by Neal Becker Jan. 5, 2024

Jan. 5, 2024

argmin of an empty sequence throws an exception, which I think is a good thing: np.argmin([]) --------------------------------------------------------------------------- ValueError But not for a masked array: n [24]: u = np.arange (10) In [25]: v = np.ma.array (u, mask=np.ones(10)) In [26]: v Out[26]: masked_array(data=[--, --, --, --, --, --, --, --, --, --], mask=[ True, True, True, True, True, True, True, True, True, True], fill_value=999999, dtype=int64) In [28]: np.argmin(v) Out[28]: 0 This is error prone and inconsistent.

1 0

PR to allow weights that align with specified axis in np.average()
by James Hamp Jan. 5, 2024

Jan. 5, 2024

https://github.com/numpy/numpy/pull/25541/ np.average() and weightsSummary This PR intends to address the limitation that the weights supplied as an optional argument in np.average() can only be the same shape as the input array, or one-dimensional. With these changes, weights that have the same shape as the input array over the specified axis can be supplied. The PR also makes some minor improvements to the sanity checks on the weights. We have implemented the extension for both np.average() and np.ma.average(), and added tests for the new functionality (all preexisting tests pass). Details Some functionalities of the numpy.average() function can be improved, and the treatment of weights in particular. Currently, np.average() is restricted to weights that are either 1. the same shape as the input array, or 2. one-dimensional. This restriction is not necessary and is disadvantageous when averaging over subdimensions the input array via supplying the axis argument. If it's possible to average over subdimensions of the input array via axis, then it is useful to be able to supply weights in accordance, checking that they have the required dimensionality and shape. Consider, for example, y = np.arange(12).reshape(2, 2, 3)w = np.array([0., 0., 1., .5, .5, 0., 0., .5, .5, 1., 0., 0.]).reshape(2, 2, 3) both the input array y and the weights w have shape (2, 2, 3). If I want to average over the whole array, that is OK: np.average(y, weights=w) 5.5 or over a one-dimensional subset (axis=0): np.average(y, axis=0 weights=w[:, 0, 1]) array([[ 6., 7., 8.], [ 9., 10., 11.]]) If, however, I want to average a subdimensions of y that are not one-dimensional, and supply weights with the same shape as the subdimensions, that is not possible: np.average(y, axis=(0, 1) weights=w[:, :, 1]) TypeError: 1D weights expected when shapes of a and weights differ. Of course, I can add an extra dimension to my weights to make them the same shape of y and get around this limitation, however, depending on the shape ofa in the dimension that is *not* being averaged over, that might be very expensive in terms of memory and perhaps prohibitively so (it was for me in one case). This PR addresses this limitation. Moreover, some of the checks on the weights could be improved, for example with the previous implementation if I supplied 1d weights but a tuple for the axis: np.average(y, axis=(0, 1), weights=w[0, 0, :]) TypeError: tuple indices must be integers or slices, not tuple This issue is not caught by one of the sanity checks and the error message does not clearly tell the user what has gone wrong. This PR improves that as well. The main idea was to generalise the function np.average(), but I ended up doing the same for np.ma.average() too, for completeness, even though I have relatively little experience with masked arrays. The implementations have been tested in lib/tests/test_function_base.py and ma/tests/test_extras.py, respectively. The two sets of tests mirror one another. All preexisting tests pass. I'd appreciate if people could kindly take a look at this PR and let know any feedback they have. Thank you James

1 0

Change definition of complex sign (and use it in copysign)
by mhvk＠astro.utoronto.ca Jan. 4, 2024

Jan. 4, 2024

Hi All, A long-standing, small wart in numpy has been that the definition of sign for complex numbers is really useless (`np.sign(z)` gives the sign of the real component, unless that is zero, in which case it gives the sign of the imaginary component, in both cases as a complex number with zero imaginary part). Useless enough, in fact, that in the Array API it is suggested [1] that sign should return `z / |z|` (a definition consistent with those of reals, giving the direction in the complex plane). The question then becomes what to do. My suggestion - see https://github.com/numpy/numpy/pull/25441 - is to adapt the Array API definition for numpy 2.0, with the logic that if we don't change it in 2.0, when will we? Implementing it, I found no real test failures except one for `np.geomspace`, where it turned out that to correct the failure, the new definition substantially simplified the implementation. Furthermore, with the redefinition, it has become possible to extend ``np.copysign(x1, x2)`` to complex numbers, since it can now generally return ``|x1| * sign(x2)`` with the sign as defined above (with no special treatment for zero). Anyway, to me the main question would be whether this would break any workflows (though it is hard to see how it could, given that the previous definition was really rather useless...). Thanks, https://github.com/data-apis/array-api/pull/556, Marten [1] https://data-apis.org/array-api/latest/API_specification/generated/array_ap… (and https://github.com/data-apis/array-api/pull/556, which has links to previous discussion)

7 7

Plans for new sparse compilation backend for PyData/Sparse
by Hameer Abbasi Jan. 3, 2024

Jan. 3, 2024

Hello everyone, The stated goal for sparse is to provide a NumPy-like API with a sparse representation of arrays. To this end, Quansight and I have been collaborating with researchers at MIT CSAIL <https://www.csail.mit.edu/> - in particular Prof. Amarasinge's group <https://www.csail.mit.edu/research/commit-group> and the TACO <https://github.com/tensor-compiler/taco/> team - to develop a performant and production-ready package for N-dimensional sparse arrays. There were several attempts made to explore this over the last couple of years, including a LLVM back-end <https://github.com/Quansight-Labs/taco/pulls?q=is%3Apr+llvm> for TACO <https://github.com/tensor-compiler/taco/>, and a pure-C++ template-metaprogramming approach called XSparse <https://github.com/hameerabbasi/xsparse>. To this end, we, at Quansight, are happy to announce that we have received funding from DARPA, together with our partners from MIT, under their Small Business Innovation Research (SBIR) program <https://www.darpa.mil/work-with-us/for-small-businesses/HR0011SB20234-06> to build out sparse using state-of-the-art just-in-time compilation strategies to boost performance for users. Additionally, as an interface, we'll adopt the Array API standard <https://data-apis.org/array-api/latest/> which was championed by major libraries like NumPy, PyTorch and CuPy. More details about the plan are posted on GitHub <https://github.com/pydata/sparse/discussions/618> — please join in the discussion there, to keep it all in one place. Best Regards, Hameer Abbasi

1 0

NumPy 1.26.3 released
by Charles R Harris Jan. 2, 2024

Jan. 2, 2024

Hi All, On behalf of the NumPy team, I'm pleased to announce the release of NumPy 1.26.3. NumPy 1.26.3 is a maintenance release that fixes bugs and regressions discovered after the 1.26.2 release. The most notable fixes are the f2py bug fixes. The Python versions supported by this release are 3.9-3.12. Wheels can be downloaded from PyPI <https://pypi.org/project/numpy/1.26.3/>; source archives, release notes, and wheel hashes are available on Github <https://github.com/numpy/numpy/releases/tag/v1.26.3>. *Contributors* A total of 18 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - @DWesl - @Illviljan - Alexander Grund - Andrea Bianchi + - Charles Harris - Daniel Vanzo - Johann Rohwer + - Matti Picus - Nathan Goldbaum - Peter Hawkins - Raghuveer Devulapalli - Ralf Gommers - Rohit Goswami - Sayed Adel - Sebastian Berg - Stefano Rivera + - Thomas A Caswell - matoro *Pull requests merged* A total of 42 pull requests were merged for this release. - #25130: MAINT: prepare 1.26.x for further development - #25188: TYP: add None to ``__getitem__`` in ``numpy.array_api`` - #25189: BLD,BUG: quadmath required where available [f2py] - #25190: BUG: alpha doesn't use REAL(10) - #25191: BUG: Fix FP overflow error in division when the divisor is scalar - #25192: MAINT: Pin scipy-openblas version. - #25201: BUG: Fix f2py to enable use of string optional inout argument - #25202: BUG: Fix -fsanitize=alignment issue in numpy/_core/src/multiarray/arraytypes.c.src - #25203: TST: Explicitly pass NumPy path to cython during tests (also... - #25204: BUG: fix issues with ``newaxis`` and ``linalg.solve`` in ``numpy.array_api`` - #25205: BUG: Disallow shadowed modulenames - #25217: BUG: Handle common blocks with kind specifications from modules - #25218: BUG: Fix moving compiled executable to root with f2py -c on Windows - #25219: BUG: Fix single to half-precision conversion on PPC64/VSX3 - #25227: TST: f2py: fix issue in test skip condition - #25240: Revert "MAINT: Pin scipy-openblas version." - #25249: MAINT: do not use ``long`` type - #25377: TST: PyPy needs another gc.collect on latest versions - #25378: CI: Install Lapack runtime on Cygwin. - #25379: MAINT: Bump conda-incubator/setup-miniconda from 2.2.0 to 3.0.1 - #25380: BLD: update vendored Meson for AIX shared library fix - #25419: MAINT: Init ``base`` in cpu_avx512_kn - #25420: BUG: Fix failing test_features on SapphireRapids - #25422: BUG: Fix non-contiguous memory load when ARM/Neon is enabled - #25428: MAINT,BUG: Never import distutils above 3.12 [f2py] - #25452: MAINT: make the import-time check for old Accelerate more specific - #25458: BUG: fix macOS version checks for Accelerate support - #25465: MAINT: Bump actions/setup-node and larsoner/circleci-artifacts-redirector-action - #25466: BUG: avoid seg fault from OOB access in RandomState.set_state() - #25467: BUG: Fix two errors related to not checking for failed allocations - #25468: BUG: Fix regression with ``f2py`` wrappers when modules and subroutines... - #25475: BUG: Fix build issues on SPR - #25478: BLD: fix uninitialized variable warnings from simd/neon/memory.h - #25480: BUG: Handle ``iso_c_type`` mappings more consistently - #25481: BUG: Fix module name bug in signature files [urgent] [f2py] - #25482: BUG: Handle .pyf.src and fix SciPy [urgent] - #25483: DOC: ``f2py`` rewrite with ``meson`` details - #25485: BUG: Add external library handling for meson [f2py] - #25486: MAINT: Run f2py's meson backend with the same python that ran... - #25489: MAINT: Update ``numpy/f2py/_backends`` from main. - #25490: MAINT: Easy updates of ``f2py/*.py`` from main. - #25491: MAINT: Update crackfortran.py and f2py2e.py from main Cheers, Charles Harris

1 0

Re: incomplete BLAS/CBLAS linking (Telling meson build which CBLAS/LAPACK (LAPACKE?) to use via pkgconfig module)
by Ralf Gommers Jan. 1, 2024

Jan. 1, 2024

On Sat, Dec 30, 2023 at 1:57 PM Dr. Thomas Orgis < thomas.orgis(a)uni-hamburg.de> wrote: > > Am Fri, 29 Dec 2023 11:34:04 +0100 > schrieb Ralf Gommers <ralf.gommers(a)gmail.com>: > > > If the library name is libcblas.so it will still be found. If it's also a > > nonstandard name, then yes it's going to fail. I'd say though that (a) > this > > isn't a real-world situation as far as we know, > > It can be more funny. I just notied on an Ubuntu system (following > Debian for sure, here) that there are both > > /usr/lib/x86_64-linux-gnu/libblas.so.3 > /usr/lib/x86_64-linux-gnu/libcblas.so.3 > > but those belong to different packages. The first contains BLAS and > CBLAS API and is installed from netlib code. > > $ readelf -d -s /usr/lib/x86_64-linux-gnu/libblas.so.3 | grep cblas_ | wc > -l > 184 > > The second is installed alongside ATLAS. > > $ readelf -d -s /usr/lib/x86_64-linux-gnu/libcblas.so.3 | grep cblas_ | > wc -l > 154 > > The symbols lists differ in that there are both functions unique to both. > > $ ldd /usr/lib/x86_64-linux-gnu/libcblas.so.3 > linux-vdso.so.1 (0x00007ffcb5720000) > libatlas.so.3 => /lib/x86_64-linux-gnu/libatlas.so.3 > (0x00007fd9b27ee000) > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd9b25c6000) > libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd9b24df000) > /lib64/ld-linux-x86-64.so.2 (0x00007fd9b2bae000) > > I _guess_ this situation would be mostly fine since libblas has enough > of the CBLAS symbols to prevent location of the wrong libcblas next to > it by the meson search. > > Quick followup regarding netlib splits. Debian only recently folded > libcblas into libblas, as > > https://lists.debian.org/debian-devel/2019/10/msg00273.html > > notes. Not that long ago … esp. considering stable debian. Not sure > when this appeared. And of course numpy is the point where things were > broken: > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=913567 > > I'm now looking into how Debian actually produces a combined BLAS+CBLAS > from netlib, as we're using the CMake build system and I do not see an > option to do that. The upstream build produces separate libraries, so I > assumed that is a case that one should handle. Yes, Debian made quite a mess there. We do have a CI job for Netlib on Debain though in NumPy, and indeed it works fine because of the CBLAS symbols already being found inside libblas.so > But it is a demonstration that any guess that libcblas belongs to > libblas just from the name may be wrong in real-world installations. > Letting this sink in some more, I realized the more fundamental reason for treating them together: when we express dependencies, we do so for a *package* (i.e., a packaged version of some project), not for a specific build output like a shared library or a header file. In this case it's a little obscured by BLAS being an interface and the libblas/libcblas mix, but it's still the case that we're looking for multiple installed things from a single package. So we want "MKL" or "Netlib BLAS", where MKL is not only a shared library (or set of them), but for example also the corresponding header file (mkl_cblas.h rather than cblas.h). The situation you are worrying about is basically that of an unknown package with a set of shared libraries and headers that have non-standard names. I'd say that that's then simply a non-supported package, until someone comes to report the situation and we can add support for it (or file a bug against that package and convince the authors not to make such a mess). I think this point is actually important, and I hope you can appreciate it as a packager - we need to depend on packages (things that have URLs to source repos, maintainers, etc.), not random library names. > > Here, it might be a strange installation remnant. > > $ dpkg -L libatlas3-base > /. > /usr > /usr/lib > /usr/lib/x86_64-linux-gnu > /usr/lib/x86_64-linux-gnu/atlas > /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3.10.3 > /usr/lib/x86_64-linux-gnu/atlas/liblapack.so.3.10.3 > /usr/lib/x86_64-linux-gnu/libatlas.so.3.10.3 > /usr/lib/x86_64-linux-gnu/libcblas.so.3.10.3 > /usr/lib/x86_64-linux-gnu/libf77blas.so.3.10.3 > /usr/lib/x86_64-linux-gnu/liblapack_atlas.so.3.10.3 > /usr/share > /usr/share/doc > /usr/share/doc/libatlas3-base > /usr/share/doc/libatlas3-base/README.Debian > /usr/share/doc/libatlas3-base/changelog.Debian.gz > /usr/share/doc/libatlas3-base/copyright > /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3 > /usr/lib/x86_64-linux-gnu/atlas/liblapack.so.3 > /usr/lib/x86_64-linux-gnu/libatlas.so.3 > /usr/lib/x86_64-linux-gnu/libcblas.so.3 > /usr/lib/x86_64-linux-gnu/libf77blas.so.3 > /usr/lib/x86_64-linux-gnu/liblapack_atlas.so.3 > > An eclectic list of redundant libraries. But as it is, this is a case where > > 1. libatlas has BLAS, libcblas has corresponding CBLAS (only > referencing ATLAS-specific ABI). > > 2. libcblas does _not_ work together with libblas (uses ATL_ symbols). > I'll note that the ATL_ symbols prefixes are an internal implementation detail; we don't prefix our own symbols when calling BLAS APIs. And there is no way Debian lets you install both of Netlib BLAS and ATLAS (it's one or the other, through `update-alternatives`), so this can't really go wrong in the way you imagine I think. > > You might consider this full of packaging errors/weirdness, but it's there. > > Also, I note that > > 1. Netlib libblas.so does contain BLAS+CBLAS symbols. > 2. Netlib liblapack.so does only ontain LAPACK symbols. > 3. Netlib liblapacke.so exists and provides LAPACKE symbols on top. > > So from my full scenario, only the separate libcblas is missing. This > hybrid is somewhat unfortunate … but, well, this liblapacke, you > actually could combine with different liblapack implementations. > > In any case, I would feel more comfortable if I knew that the build is > not locating a library that is there but that it should not use. > We actually do have a CI job for ATLAS on Debian too: https://github.com/numpy/numpy/blob/36eefeabadeb4ebe07b4be3704da93a4c126b6c…. Note that Debian in addition decided to use custom pkg-config names, so it works with `-Dblas=blas-atlas -Dlapack=lapack-atlas`. > (b) just don't do this as a > > packager, and (c) if you really must, you can still make it work by > > providing a custom `cblas.pc` > > Well, we can also hack around in your meson build files;-) The idea is > to prepare also custom .pc files for various cases and be able to tell > builds to use them. It feels like a unnecessary having to prepare > another directory with custom cblas.pc for the build that pkg-config > then can find there compared to just telling it to use cblas-foobarbaz > as name for the file installed in the prefix alongside others. > > > (see > > > http://scipy.github.io/devdocs/building/blas_lapack.html#using-pkg-config-t… > > ). > > I'm confused … this page doesn't mention blas vs cblas. You mean in > addition to the blas.pc used there, I need to but a cblas.pc (or > libblas.so) beside it. This background magic is bad. There should be no > implicit guessing, ever, for a controlled packager build. > Yes, it's not ideal. But again, we're talking about a hypothetical situation, where you already did something bad as a packager, and I'm pointing out a workaround. If you're renaming build outputs from an upstream package from their defaults (e.g. libcblas.so -> libcblas-foobar.so), you are breaking things and get to keep the pieces. The correct name for Netlib BLAS is `libcblas.so` and `cblas.pc`. So provide `cblas.pc`, and you unbreak yourself. > > We don't use LAPACKE, so that one can be ignored. > > I had the impression that you are upstreaming something to meson about > BLAS (and LAPACK) detection. So I am thinking about the full picture > for all projects using meson with BLAS dependencies. > Sure, I was talking about NumPy/SciPy. In general, the LAPACKE situation is pretty much identical to the CBLAS one. > I see that combining any subset of BLAS, CBLAS, LAPACK, LAPACKE into a > common library is a thing in practice. So I see how you'd want some > special detection code to check if just loading a dependency called > 'blas' gives you all APIs you need or of you need to add dependencies > 'cblas', 'lapack', or 'lapacke', even. > > > > So scipy locates cblas based on the name blas, but doesn't really use > > > cblas. > > > > > > It does in a few places, like SuperLU. > > You are confusing me. I see no cblas usage anywhere in SciPy 1.11.4, > did that come later? > ... > > No cblas in sight. Also, the SuperLU part specifically was what > prompted this whole renewed discussion. We noticed breakage since scipy > with -Dblas=cblas failed to link _superlu.so to libblas. This only hit > on an depending package trying to use it. > I'm sorry, I was just wrong here. Looking closer, the C code in SuperLU that I thought was using CBLAS is actually manually handling Fortran compiler symbol handling and calling the mangled Fortran symbols (it's probably not very robust to unknown compilers): https://github.com/scipy/scipy/blob/main/scipy/sparse/linalg/_dsolve/SuperL… . > The above command returns _no_ output when I build with -Dblas=cblas. > The build works fine, but the resulting shared objects are _all_ left > wanting an actual BLAS linkage. I provided a package that offers CBLAS > and would pull in BLAS indirectly. Scipy build proceeds with that, but > it tricks me with defunct algebra all around, including _superlu.so, > where no object makes use of libcblas. > > You said scipy isn't using the custom meson stuff yet. I see two > occasions of BLAS use in numpy. > > $ find > /data/projekte/pkgsrc/work/math/py-numpy/work/.destdir/data/pkg/lib/python3.11/site-packages/numpy/ > -name '*.so'|while read f; do readelf -d $f|grep blas && echo "^^^^ $f > ^^^^"; done > 0x0000000000000001 (NEEDED) Biblioteca compartida: > [libcblas.so.3] > ^^^^ > /data/projekte/pkgsrc/work/math/py-numpy/work/.destdir/data/pkg/lib/python3.11/site-packages/numpy/core/_multiarray_umath.so > ^^^^ > 0x0000000000000001 (NEEDED) Biblioteca compartida: > [libblas.so.3] > ^^^^ > /data/projekte/pkgsrc/work/math/py-numpy/work/.destdir/data/pkg/lib/python3.11/site-packages/numpy/linalg/_umath_linalg.so > ^^^^ > > So it needs both CBLAS and BLAS. (Btw. I rather like how I can tell > that apart just from looking at the used libs, not the symbols > themselves …;-) > > Building again with -Dblas=cblas … FRICK! > > $ find > /data/projekte/pkgsrc/work/math/py-numpy/work/.destdir/data/pkg/lib/python3.11/site-packages/numpy/ > -name '*.so'|while read f; do readelf -d $f|grep blas && echo "^^^^ $f > ^^^^"; done > 0x0000000000000001 (NEEDED) Biblioteca compartida: > [libcblas.so.3] > ^^^^ > /data/projekte/pkgsrc/work/math/py-numpy/work/.destdir/data/pkg/lib/python3.11/site-packages/numpy/core/_multiarray_umath.so > ^^^^ > > Now _umath_linalg.so is left with undefined BLAS symbols, just like > scipy before. I need to hot-fix that in pkgsrc, just after releasing > the stable branch. Bad, bad, bad. I really took the message from you > before that -Dblas=cblas is what I need to do. Now we ship broken numpy. > > Got to fix that right away. > > (update from second email:) > > Luckily, it isn't. It links to liblapack, which brings in libblas. So > that's fine, I suppose. The test output looks the same foor both after > installing and then testing. So this was some interference. > > I hope > > =========================== short test summary info > ============================ > FAILED > .destdir/data/pkg/lib/python3.11/site-packages/numpy/distutils/tests/test_log.py::test_log_prefix[info] > FAILED > .destdir/data/pkg/lib/python3.11/site-packages/numpy/distutils/tests/test_log.py::test_log_prefix[debug] > 2 failed, 36807 passed, 1603 skipped, 1303 deselected, 33 xfailed, 1 > xpassed, 62 warnings in 410.43s (0:06:50) > > > is OK. Those failures do sound like something basic or easy to fix, > though. Log prefixing shouldn't be so complicated. Yep, that should be okay - the two test failures should be pretty harmless. > > > Numpy is happy with libcblas bringing libblas in and calls it > > > blas, but really uses the cblas interface. This looks a bit confusing. > > > > > It is happy, but not for a good result. The build just claims that it > works, but produces defunct _umath_linalg. Or is this hidden in > practical use by loading _multiarray_umath.so before? I'm no python > user myself. What would be a minimal test for this? > Running `python -c "import numpy.linalg; numpy.test()" should do it and is pretty fast (needs `pytest` and `hypothesis` installed). If you need to avoid those test packages, maybe a simple `python -c "import numpy"` would already be enough, since that does call at least one linear algebra routine. > I may be able to add something to the docs, but there should be no > > confusion. We need "BLAS with CBLAS symbols". CBLAS should simply not be > > considered as a completely separate dependency (it's one library with two > > interfaces). I can't think of a reason to do so, nor of a build system > that > > does it like that. There is no FindCBLAS in CMake for example, it handles > > it transparently within FindBLAS. > > And it's unknown to me right now how CMake really handles CBLAS, as I > cannot recall a CMake-using project right now that relies on CBLAS. > > The asymmetry with LAPACKE in Debian confuses me, too. What do you > consider standard packaging? BLAS+CBLAS together in libblas, but > LAPACKE as a separate thing? Inconsistent. I guess I could reduce > pkgsrc-specifc breakage by merging libcblas into libblas. But keep > liblapacke separate? Would that confuse more or less? Should it be all > in one or all separate? > Ideally, each packager for any BLAS library should run the default upstream build commands, install into a prefix and bundle up the result in a package. So what is standard == whatever configure/make produces for Netlib BLAS or OpenBLAS, without renaming or deleting anything. > Do you consider BLAS+CBLAS to be one thing, LAPACK+LAPACKE the other? > All or nothing? > Yes indeed. At least, I cannot think of any package/project that doesn't provide CBLAS with BLAS. While BLAS and LAPACK clearly are separate from each other, and there are projects that provide only one (e.g., BLIS is BLAS-only). > Wouldn't the world be much simpler with > > dependency(blas) > dependency(cblas) > dependency(lapack) > dependency(lapacke) > No, I don't think so. > as requests for the specific API mentioned there, coming as separate > pkg-config modules by meson default behaviour if they're present, but > some logic that checks first if whatever provided blas (-Dblas=foobar) > also provides lapack and hence dependency(lapack) is fulfilled already? > Otherwise, look for lapack or whatever -Dlapack=baz was set to. Or is > this already what is happening? I didn't have time to read up on > everything, perhaps. > Yes, pretty much. E.g., for MKL and Accelerate we know for sure that they contain LAPACK symbols, so we don't even check. For OpenBLAS we know it's the default to include LAPACK but it can be built without, so we need to check symbols, and if they're missing we look for another library. > If you do dependency(blas with cblas) with modified/future meson … can > I override what you try if my -Dblas does not contain cblas, but > another library I can provide would? Yes, the search order and which packages are searched for is fully customizable. But you should be good with the default `-Dblas=auto`. If not, I'd like to understand why. > > > And I need to ponder if I leave it at -Dblas=$CBLAS_PC for pkgsrc now. > > > It's somewhat wrong, but also more correct, as NumPy _really_ means to > > > use CBLAS API, not BLAS. > > > > > > > My opinion is that this is not right. It may work with pkg-config in your > > case, but I don't think it'll be robust for the fallback detection that > is > > used when pkg-config is not installed. > > See above. It's broken. I assume that it actually was fine, given your follow-up email about the test suite having only 2 failures. So I'll skip replying to this part. > [...] Meson > using CMake behind the scenes is also rather surprising. CMake configs > are a nuisance when they take the focus away from pkg-config files … > and a package that doesn't actually use cmake for building is normally > not getting cmake in the build sandbox in pkgsrc. Maybe that has to be > investigated if some packages really require meson to use cmake. > It's a fallback detection method to use CMake; it can be disabled indeed. > The attached output of numpy.test() has 2 failures. I don't know if > they are related to BLAS. Re-doing the test with a properly linked > numpy now… if they persist — how bad is that? Are those known failures? > They're not a problem, it's a minor thing that's not used at runtime. > OK … lots of errors regarding f2py in the test environment. Summary: > > 8 failed, 36022 passed, 1603 skipped, 1303 deselected, 33 xfailed, 1 > xpassed, 62 warnings, 779 errors in 441.09s (0:07:21) > > With the bad linkage, it is: > > 2 failed, 36807 passed, 1603 skipped, 1303 deselected, 33 xfailed, 1 > xpassed, 62 warnings in 404.95s (0:06:44) > > Maybe there is interference with the installed numpy. I cannot say if > there is anything releated to the missing libblas linkage. I'd presume > lots of actual tests would fail if that had an effect? How many tests > need working _umath_linalg.so? > If something was really wrong, you'd get 100s to 1000s of failures/errors. I think you're fine here. Cheers, Ralf

2 1