Print bug

[jonalm]

There seems to be a glitch in the printed rows when the number of rows are 20 or greater. Notice the gap between 9-11 in the printout below.

This glitch prevents TerminalPager.jl to work nicely with the KeyedArray ronisbr/TerminalPager.jl#75

julia> N=20; k = KeyedArray(randn(N,N), x=1:N, y=1:N)
julia> sprint(show, MIME"text/plain"(), k) |> println
2-dimensional KeyedArray(NamedDimsArray(...)) with keys:
↓   x ∈ 20-element UnitRange{Int64}
→   y ∈ 20-element UnitRange{Int64}
And data, 20×20 Matrix{Float64}:
       (1)          (2)           (3)          (4)          (5)          (6)          (7)          (8)         (9)          (10)           (11)          (12)         (13)          (14)          (15)          (16)           (17)         (18)         (19)          (20)
  (1)    1.12257      0.490402      0.487161     1.3417      -0.61508      1.00027      0.75978     -1.01997    -0.507342      2.17005       -0.171787     -0.379487    -0.990666     -0.84708       0.0829884     0.911194       0.142935    -2.65874     -0.527115      0.245108
  (2)    2.4968      -0.93532      -0.397074    -0.110128     0.925698    -0.319974    -0.824232    -0.774187    0.163896      0.129388       2.79476      -0.907305     0.539264     -1.3593        0.121594      0.924924       0.619213     0.613901    -0.723006     -0.188586
  (3)    0.0208007   -0.286999      0.593305     1.27435      0.712491     0.409889     0.61506      0.766101   -2.15692      -0.354933       1.62464      -0.622595     0.75396      -1.06038       0.300836     -1.09592        0.684804    -0.462004     0.0621972    -0.111631
  (4)    0.808812     0.446653     -0.2332       0.76007     -1.09043     -0.0431402   -0.927826    -0.628285    0.405432     -0.29322       -1.26878      -0.236054     1.62768       0.918532     -0.0981469     1.54129        1.26642      0.977583     1.83501       0.0320194
  (5)   -0.273673    -0.617164      0.78719      0.693395     1.06863     -0.763251    -0.949814     0.342036   -1.32664       2.20419        0.230068      1.50768     -2.28972       0.0611954    -0.517557      0.261454       0.940659    -0.540622     0.842569      1.76228
  (6)    0.477936     1.53226       1.12786      0.502254     1.3192      -0.275781     0.495464     0.10667    -0.27726      -0.227957      -0.949501     -1.05338      0.611621      0.625151      1.19027      -0.294704       0.368311    -1.1258       0.950899     -1.68034
  (7)    0.570887    -0.586242      0.0604913    1.80167     -0.0126626   -0.726972     0.445168     0.569077    1.54544       0.340749       0.238241      0.807731    -0.728055     -0.166784     -1.81308       0.935487      -1.36998      2.00573      0.499047     -0.250083
  (8)   -0.23281      0.00270028    0.621672     2.33665      0.388242    -0.451793    -0.511436    -1.21374     0.292649     -1.44022       -1.50705      -0.381982     0.934232      0.567087     -0.0416158    -0.373854       1.26558     -0.855522    -0.24484      -1.05762
  (9)   -1.09506     -2.82651      -0.23011      0.169101     0.707845     1.87551      0.326911    -0.79621     0.0405532    -0.108328      -0.458916     -0.49695     -1.21077       1.40631      -0.383105     -0.00819703    -2.0816       0.479381    -1.8039        0.687658
 (11)    0.402395     0.067904      0.505419     1.26774     -0.128122    -0.305771     0.518124     0.320105    0.437461      0.5606         0.142811     -0.187419     1.10411      -1.11426      -1.92751      -0.0533123     -0.71343     -0.695421     0.65695       1.796
 (12)    2.75515      0.838098      0.260278     0.50885     -0.485894     3.41045     -0.189964    -1.64383    -1.23911       0.876434      -2.89447       0.89373     -1.82964       0.64817       0.669421     -1.61013        0.142432     1.05083      1.69808       1.38645
 (13)    0.881214    -1.3055       -0.79923     -0.254773     0.783476     0.342084     0.390894    -0.446912    0.368134      0.441044      -0.450815      0.470341     1.6416       -1.87338       0.0531701     0.123984      -1.54473     -0.780281    -0.36103       1.97403
 (14)    1.57546     -0.356563      0.224456     0.465481    -1.54371      0.034207    -2.17408      0.246078   -1.06823       1.54273       -0.432513      0.541496    -0.481721     -0.536497     -0.934225      0.391173      -0.213335     0.56148     -0.471436     -0.047096
 (15)    1.19577     -0.263914      1.48525      0.801261     1.37071     -0.0483607   -0.193038    -1.10729    -0.362711     -0.290259      -0.621544     -0.024465     0.104764      0.209293      1.79168       1.12739        1.43481     -0.305706     0.544616      1.21018
 (16)   -0.740074     0.486808     -1.21628      1.47814     -1.4511      -0.904593    -0.374184     0.111976    0.0153674     0.0282585     -0.805838     -1.30165     -0.117036      0.585149      0.682102      1.06212       -0.498174    -0.937592    -0.235581     -0.829192
 (17)   -0.486374    -0.0826544    -1.88369     -0.182889    -1.04927      0.568706    -0.982996    -0.405444    0.14416       1.44949        0.939727      0.820213    -0.355105     -1.2758        0.731964     -2.28131        1.18909      0.527651     0.321574      0.131979
 (18)    1.32437     -0.849305      0.354739     0.7182       1.07191     -2.73456      1.14495      0.561979   -1.1518       -1.38035        0.201228     -0.663178    -0.735969     -0.350276     -1.05175      -1.43845        0.154855     1.35661     -1.29657       0.157937
 (19)   -0.0537634   -0.485977     -0.483744    -0.0424855    0.206698    -1.99924      0.274828    -0.368498   -1.87434      -0.00671213     0.0724161     0.575972     0.116639     -0.619535     -0.391665      1.12729        0.934895    -0.810897    -1.77572       0.11368
 (20)    0.166995    -0.390374      1.23389      0.0981188   -0.764152    -0.403233    -0.0290021    1.84925     0.136199      0.321659      -0.513166      0.505695     0.0299045    -1.22091      -1.90267      -0.233389       1.73001      0.293789     0.457946      1.93672

[aplavin]

Seems like it just cuts the middle part on purpose, to avoid dump infinitely large arrays into terminal.
Would you expect show to print the whole array no matter what, even if it's multiple gigabytes?

[mcabbott]

It's a little odd that it skips 10 silently, in the above example.

What happens at the ordinary REPL printing (where stdout::IO contains the window size) is that either everything fits, or you get ⋮. With 25x80 window:

julia> N=15; k = KeyedArray(randn(N,N), x=1:N, y=1:N)
2-dimensional KeyedArray(NamedDimsArray(...)) with keys:
↓   x ∈ 15-element UnitRange{Int64}
→   y ∈ 15-element UnitRange{Int64}
And data, 15×15 Matrix{Float64}:
       (1)         (2)           (3)          …  (14)          (15)
  (1)    1.47697     0.510451      0.705914         0.20488      -1.28736
  (2)    1.30226    -1.46408      -1.41179          0.302384      0.23286
  (3)    0.899942   -0.707045      1.81067         -1.30824      -0.393464
  (4)    2.62507    -1.47469       0.428942        -0.919739     -1.11851
  (5)   -0.72418     0.901133      0.145809   …    -1.3734       -0.210647
  (6)    1.4137      0.0948597     1.98676         -0.671418     -1.86468
  (7)   -0.228045    0.00302542    1.57991          0.531497     -1.26064
  (8)    0.860517   -0.233845     -2.18334         -0.923497      0.744353
  (9)   -1.01547    -2.16297      -0.0995179       -1.31932      -0.444524
 (10)    0.594162   -0.959207     -0.0347569  …     0.0193206    -0.300771
 (11)   -1.45855    -0.779344      0.908983         0.454173     -0.0160269
 (12)    1.06171     0.400011      2.17124          1.47727       0.343414
 (13)   -0.016208   -0.732777     -0.533414         0.310175      0.382326
 (14)   -1.29887    -0.0594261    -0.527276        -0.30708       1.43401
 (15)    1.02568    -0.209493      0.283405   …    -0.59324       0.734355

julia> N=16; k = KeyedArray(randn(N,N), x=1:N, y=1:N)
2-dimensional KeyedArray(NamedDimsArray(...)) with keys:
↓   x ∈ 16-element UnitRange{Int64}
→   y ∈ 16-element UnitRange{Int64}
And data, 16×16 Matrix{Float64}:
       (1)          (2)          (3)          …  (15)          (16)
  (1)   -1.17311      0.742278    -0.925829        -2.02316      -1.41721
  (2)    0.650443     1.08115     -1.60876         -0.365797     -0.247055
  (3)   -1.27014      0.0617723   -1.61681         -0.624693      0.416926
  (4)   -0.694252     0.54337      0.160558        -2.40424      -1.0393
  (5)   -0.0329912   -0.140955    -0.570876   …    -1.0687        0.556031
  (6)    0.980979    -1.00215      0.0261444       -1.43145      -0.757553
  (7)    0.179439     0.54834      1.49001         -1.34443      -1.28616
    ⋮                                         ⋱     ⋮          
 (10)   -0.605109    -0.925501    -0.0789025  …    -0.342627     -1.30461
 (11)    2.33197      0.415978     0.228837        -0.0022514    -1.30805
 (12)   -1.47114     -0.494195     2.14628         -1.05713      -0.378415
 (13)   -0.0889765   -0.0289388   -0.200903        -0.175079     -0.42374
 (14)    1.42185      1.10661      0.551851         0.393345      0.0778288
 (15)   -0.0976808    0.107977     1.52832    …    -0.241538      0.311516
 (16)    0.291658     0.937386     2.23261          1.04164      -0.761929

julia> 

Ordinary arrays are similarly truncated, but repr(MIME"text/plain"(), arr) does not know the size and hence does not truncate:

julia> collect(1:30.0)
30-element Vector{Float64}:
  1.0
  2.0
  3.0
  4.0
  5.0
  6.0
  7.0
  8.0
  9.0
 10.0
  ⋮
 21.0
 22.0
 23.0
 24.0
 25.0
 26.0
 27.0
 28.0
 29.0
 30.0

julia> repr(MIME"text/plain"(), ans)
"30-element Vector{Float64}:\n  1.0\n  2.0\n  3.0\n  4.0\n  5.0\n  6.0\n  7.0\n  8.0\n  9.0\n 10.0\n 11.0\n 12.0\n 13.0\n 14.0\n 15.0\n 16.0\n 17.0\n 18.0\n 19.0\n 20.0\n 21.0\n 22.0\n 23.0\n 24.0\n 25.0\n 26.0\n 27.0\n 28.0\n 29.0\n 30.0"

So maybe it's a bug that this package does truncate in the string, and doesn't put ⋮ in there?

[jonalm]

@aplavin

Would you expect show to print the whole array no matter what, even if it's multiple gigabytes?

No, but I did expect KeyedArray to work with TerminalPager (see link to issue in OP). I'm not on top of the details here, but
consider how sprint behaves on a Vector:

> sprint(show, MIME"text/plain"(), collect(1:10000), context=:limit => true) |> println # does truncate
> sprint(show, MIME"text/plain"(), collect(1:10000), context=:limit => false) |> println # does not truncate

Whereas

> N=200; k = KeyedArray(randn(N,N), x=1:N, y=1:N)
> sprint(show, MIME"text/plain"(), k, context=:limit => true) |> println # behaves as expected
> sprint(show, MIME"text/plain"(), k, context=:limit => false) |> println # weirdly also truncates

Could there be an issue with how the IOContext :limit propriety is used in here?

[mcabbott]

Ok now I remember. This package tries to re-use a lot of Base's printing logic. So the way it shows "keys" next to a vector is to make a matrix containing these and the date, fakearray here:

AxisKeys.jl/src/show.jl

Lines 84 to 94 in c513117

	h, w = displaysize(io)
	wn = w÷3 # integers take 3 columns each when printed, floats more
	fakearray = if ndims(A) == 1
	top = size(A,1) < h ? Colon() : (1:(h÷2))
	bottom = size(A,1) < h ? (1:0) : size(A,1)-(h÷2):size(A,1)
	hcat(vcat(ShowWith.(no_offset(axiskeys(A,1))[top]; color=colour(A,1)),
	ShowWith.(no_offset(axiskeys(A,1))[bottom]; color=colour(A,1))),
	vcat(getindex(no_offset(unname(keyless(A))), top),
	getindex(no_offset(unname(keyless(A))), bottom)) |> full)
	else

At first it did this for the full size. But when someone tried it on long vector, the millions of strings turned out to be a problem. So now, it tries to make this fakearray just big enough to trigger the "..." style printing. How big depends on what displaysize returns.

And when you call sprint, Julia supplies a default size:

julia> displaysize(stdout)  # my window
(33, 100)

julia> displaysize(IOContext(IOBuffer()))  # the default
(24, 80)

Maybe you can tell from the IOContext that this isn't a real window size, the thing has many confusing properties. :limit does not distinguish these two, but what happens inside sprint / repr I'm not really sure:

julia> get(stdout, :limit, false)
false

julia> get(IOContext(IOBuffer()), :limit, false)
false

Although I wonder what TerminalPager.jl does about huge arrays. Converting the entire thing to a string seems like it will often be pretty expensive... and this intermediate fakearray might be worse as it's Matrix{Any} IIRC.

julia> @time randn(10^7);
  0.041015 seconds (3 allocations: 76.297 MiB, 35.31% gc time)

julia> @time string(randn(10^7));
  4.536410 seconds (40.00 M allocations: 5.043 GiB, 18.03% gc time)

[adranka]

#176 should fix this, if it's merged @jonalm do you mind testing in TerminalPager.jl? As noted by @mcabbott, internally a Matrix{Any} is constructed and converted to a string, which could have some performance implications if dealing with large arrays.