Improving slicing (or viewing?)

[grothesque]

Here is a NumPy demonstration of what I mean with slicing:

>>> import numpy as np
>>> a = np.arange(16).reshape((2,)*4)
>>> a[1, :, :, :]
array([[[ 8,  9],
        [10, 11]],

       [[12, 13],
        [14, 15]]])
>>> a[1, ...]
array([[[ 8,  9],
        [10, 11]],

       [[12, 13],
        [14, 15]]])
>>> a[:, 1, ...]
array([[[ 4,  5],
        [ 6,  7]],

       [[12, 13],
        [14, 15]]])
>>> a[:, 1, ..., np.newaxis]
array([[[[ 4],
         [ 5]],

        [[ 6],
         [ 7]]],


       [[[12],
         [13]],

        [[14],
         [15]]]])

Mdarray provides something similar through the methods tensor, view, and view_mut of Slice. There is an equivalent of Python's : (the various ranges that implement DimIndex), but as far as I can see there is no equivalent of .../ellipsis or np.newaxis (which is actually just an alternative name for None).

One conceptual difference that I see compared to NumPy (and also Rust's ndarray: https://docs.rs/ndarray/latest/ndarray/macro.s.html) is that the view operations never take the index as a single argument, but always as a bunch of individual arguments. An appropriate trait exists (ViewIndex), but it is only used internally. This approach simplifies the syntax (are there other reasons for it?), but I note the following limitations:

• There seems to be no easy way (without rather arcane macro use) to wrap view and friends inside an API. Let's say that I'd like to write a function that internally calls view in some customizable way. I can pass in an argument that implements ViewIndex, but there is no way to "apply" this argument (in the functional programming sense) to the view method. (This is not a hypothetical problem. I'm working on a library where I'd like to be able to deal with such ViewIndex implementing objects.)

• view does not work for dynamic rank arrays (and is limited to six dimensions).

• Having an ellipsis equivalent would be nice, especially for dynamic rank.

• Having a newaxis equivalent might be nice, if it doesn't complicate things too much.

Does the author of mdarray think that these issues are worth addressing? Perhaps a possible solution could use the following elements? (This is not a complete design, just some ideas.)

• Modify view and friends to take a single argument of a type that implements ViewIndex. Not sure if the trait ViewIndex is (or can be made) general enough so that static rank

• Perhaps provide a macro (in the spirit of ndarray's) if the syntax would be unwieldy.

• Add an ellipsis type (or reuse something else as it) and implement DimIndex for it.

As far as I can tell currently there is no way to further slice a Slice if the number of dimensions is dynamic. I think that this would be necessary for dynamic dimensions to become truly useful.

One possible way would be to introduce a structure similar to ndarray's SliceInfo along with "dynamic" variants of the methods tensor, view, and view_mut.

[fre-hu]

Thanks for the suggestions. Yes, the main reason for having multiple arguments in slicing is to simplify the API. The idea is also that once variadics is available, it should be possible to have multiple arguments for any input shape type including dynamic rank.

One more limitation is that ViewIndex itself has static rank for the input and output. To support dynamic rank fully, one idea is to add a "slice axes" function that takes the axes, the new shape, and start and step values as input. That would support any slicing operation including going from and to dynamic rank. It would be similar to SliceInfo in ndarray as you suggested but somewhat different format by having the new shape explicitly.

Adding ellipsis for a number of axes and a new axis placeholder are interesting ideas. It would add some complexity but should be doable I think. They would require either variadics or switching from multiple arguments to a tuple in the slicing. The new placeholder dimension could be implemented as having constant size of 1 but broadcast when merged with other "normal" dimensions.

[grothesque]

Sounds good!

With regard to variadics, do you mean variadic generics, variadic functions, or yet something else? Is any of these efforts active and advanced enough such that one can hope to be able to use it within a couple of years?

[fre-hu]

It is variadic functions I mean, hopefully it will happen in not too long time. I found some more discussion here:

https://hackmd.io/@Jules-Bertholet/HJFy6uzDh

[grothesque]

This also looks relevant: rust-lang/rust#95174

I found this through a topic on internals: Arrays as const generics