Benchmark more scenarios

[tomkins]

After adding improvements in #10 and #11 - let's see how a few other workloads might go.

[tomkins]

Linode 4 core shared CPU (AMD EPYC 7642)

uWSGI, 1 process, 1 editor

Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
GET      /admin/api/main/pages/                                                             1     0(0.00%) |    104     104     104    104 |    0.02        0.00
GET      /admin/login/                                                                      1     0(0.00%) |     17      17      17     17 |    0.02        0.00
POST     /admin/login/                                                                      1     0(0.00%) |      8       8       8      8 |    0.02        0.00
GET      /admin/pages/[base.HomePage]/edit/                                                 5     0(0.00%) |    124     115     143    120 |    0.08        0.00
POST     /admin/pages/[base.HomePage]/edit/                                                 5     0(0.00%) |     92      82     109     92 |    0.08        0.00
GET      /admin/pages/[breads.BreadPage]/edit/                                            187     0(0.00%) |    101      91     194     98 |    3.14        0.00
POST     /admin/pages/[breads.BreadPage]/edit/                                            187     0(0.00%) |     74      68      96     73 |    3.14        0.00
GET      /admin/pages/[breads.BreadsIndexPage]/edit/                                       11     0(0.00%) |     80      74     112     75 |    0.18        0.00
POST     /admin/pages/[breads.BreadsIndexPage]/edit/                                       11     0(0.00%) |     66      61      90     64 |    0.18        0.00
GET      /admin/pages/[locations.LocationPage]/edit/                                       73     0(0.00%) |    151     137     241    150 |    1.23        0.00
POST     /admin/pages/[locations.LocationPage]/edit/                                       73     0(0.00%) |    108      99     131    110 |    1.23        0.00
GET      /admin/pages/[locations.LocationsIndexPage]/edit/                                 17     0(0.00%) |     80      75     122     78 |    0.29        0.00
POST     /admin/pages/[locations.LocationsIndexPage]/edit/                                 17     0(0.00%) |     67      62      92     66 |    0.29        0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
         Aggregated                                                                       589     0(0.00%) |     97       8     241     95 |    9.90        0.00

Response time percentiles (approximated)
Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
GET      /admin/api/main/pages/                                                                100    100    100    100    100    100    100    100    100    100    100      1
GET      /admin/login/                                                                          17     17     17     17     17     17     17     17     17     17     17      1
POST     /admin/login/                                                                           8      8      8      8      8      8      8      8      8      8      8      1
GET      /admin/pages/[base.HomePage]/edit/                                                    120    120    120    140    140    140    140    140    140    140    140      5
POST     /admin/pages/[base.HomePage]/edit/                                                     92     93     93    110    110    110    110    110    110    110    110      5
GET      /admin/pages/[breads.BreadPage]/edit/                                                  98    100    100    100    110    130    160    180    190    190    190    187
POST     /admin/pages/[breads.BreadPage]/edit/                                                  73     75     77     78     82     84     90     93     96     96     96    187
GET      /admin/pages/[breads.BreadsIndexPage]/edit/                                            75     78     79     79     88    110    110    110    110    110    110     11
POST     /admin/pages/[breads.BreadsIndexPage]/edit/                                            64     66     69     69     70     90     90     90     90     90     90     11
GET      /admin/pages/[locations.LocationPage]/edit/                                           150    150    150    160    170    190    240    240    240    240    240     73
POST     /admin/pages/[locations.LocationPage]/edit/                                           110    110    110    110    120    120    120    130    130    130    130     73
GET      /admin/pages/[locations.LocationsIndexPage]/edit/                                      78     79     79     80     82    120    120    120    120    120    120     17
POST     /admin/pages/[locations.LocationsIndexPage]/edit/                                      66     67     67     69     73     92     92     92     92     92     92     17
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
         Aggregated                                                                             95    100    110    110    140    150    160    180    240    240    240    589

[tomkins]

Linode 4 core shared CPU (AMD EPYC 7642)

uWSGI, 4 processes, 4 editors

Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
GET      /admin/api/main/pages/                                                             4     0(0.00%) |    119     107     135    110 |    0.07        0.00
GET      /admin/login/                                                                      4     0(0.00%) |     12      10      16     11 |    0.07        0.00
POST     /admin/login/                                                                      4     0(0.00%) |     12       8      21      9 |    0.07        0.00
GET      /admin/pages/[base.HomePage]/edit/                                                44     0(0.00%) |    123     112     146    120 |    0.74        0.00
POST     /admin/pages/[base.HomePage]/edit/                                                44     0(0.00%) |    122      84     205    110 |    0.74        0.00
GET      /admin/pages/[breads.BreadPage]/edit/                                            521     0(0.00%) |    105      91     199    100 |    8.75        0.00
POST     /admin/pages/[breads.BreadPage]/edit/                                            519     0(0.00%) |    112      70     270    100 |    8.72        0.00
GET      /admin/pages/[breads.BreadsIndexPage]/edit/                                       40     0(0.00%) |     87      77     168     82 |    0.67        0.00
POST     /admin/pages/[breads.BreadsIndexPage]/edit/                                       40     0(0.00%) |    105      65     193     96 |    0.67        0.00
GET      /admin/pages/[locations.LocationPage]/edit/                                      301     0(0.00%) |    154     134     265    150 |    5.05        0.00
POST     /admin/pages/[locations.LocationPage]/edit/                                      300     0(0.00%) |    144     101     346    140 |    5.04        0.00
GET      /admin/pages/[locations.LocationsIndexPage]/edit/                                 41     0(0.00%) |     85      77     121     84 |    0.69        0.00
POST     /admin/pages/[locations.LocationsIndexPage]/edit/                                 41     0(0.00%) |     99      65     156     96 |    0.69        0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
         Aggregated                                                                      1903     0(0.00%) |    121       8     346    110 |   31.96        0.00

Response time percentiles (approximated)
Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
GET      /admin/api/main/pages/                                                                120    120    140    140    140    140    140    140    140    140    140      4
GET      /admin/login/                                                                          14     14     16     16     16     16     16     16     16     16     16      4
POST     /admin/login/                                                                          12     12     21     21     21     21     21     21     21     21     21      4
GET      /admin/pages/[base.HomePage]/edit/                                                    120    120    130    130    130    140    150    150    150    150    150     44
POST     /admin/pages/[base.HomePage]/edit/                                                    110    120    150    160    180    190    200    200    200    200    200     44
GET      /admin/pages/[breads.BreadPage]/edit/                                                 100    110    110    110    120    120    140    150    200    200    200    521
POST     /admin/pages/[breads.BreadPage]/edit/                                                 100    120    130    130    160    190    210    220    270    270    270    519
GET      /admin/pages/[breads.BreadsIndexPage]/edit/                                            82     86     92     95    100    100    170    170    170    170    170     40
POST     /admin/pages/[breads.BreadsIndexPage]/edit/                                           100    110    120    140    160    180    190    190    190    190    190     40
GET      /admin/pages/[locations.LocationPage]/edit/                                           150    150    160    160    170    180    210    230    260    260    260    301
POST     /admin/pages/[locations.LocationPage]/edit/                                           140    150    160    160    200    230    260    300    350    350    350    300
GET      /admin/pages/[locations.LocationsIndexPage]/edit/                                      84     87     88     89     94     97    120    120    120    120    120     41
POST     /admin/pages/[locations.LocationsIndexPage]/edit/                                      96    100    120    120    130    140    160    160    160    160    160     41
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
         Aggregated                                                                            110    130    140    150    160    180    210    230    300    350    350   1903

[tomkins]

AWS c7g.16xlarge (Graviton3, 64 cores)

uWSGI, 1 editor

Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
GET      /admin/api/main/pages/                                                             1     0(0.00%) |     69      69      69     69 |    0.02        0.00
GET      /admin/login/                                                                      1     0(0.00%) |    316     316     316    316 |    0.02        0.00
POST     /admin/login/                                                                      1     0(0.00%) |     45      45      45     45 |    0.02        0.00
GET      /admin/pages/[base.HomePage]/edit/                                                15     0(0.00%) |     87      78     205     79 |    0.25        0.00
POST     /admin/pages/[base.HomePage]/edit/                                                15     0(0.00%) |     69      67      86     68 |    0.25        0.00
GET      /admin/pages/[breads.BreadPage]/edit/                                            220     0(0.00%) |     67      65     124     66 |    3.69        0.00
POST     /admin/pages/[breads.BreadPage]/edit/                                            219     0(0.00%) |     59      57     103     59 |    3.67        0.00
GET      /admin/pages/[breads.BreadsIndexPage]/edit/                                       26     0(0.00%) |     53      51      55     53 |    0.44        0.00
POST     /admin/pages/[breads.BreadsIndexPage]/edit/                                       26     0(0.00%) |     52      51      57     53 |    0.44        0.00
GET      /admin/pages/[locations.LocationPage]/edit/                                      123     0(0.00%) |     96      93     140     94 |    2.06        0.00
POST     /admin/pages/[locations.LocationPage]/edit/                                      123     0(0.00%) |     84      81     156     83 |    2.06        0.00
GET      /admin/pages/[locations.LocationsIndexPage]/edit/                                 15     0(0.00%) |     58      53      98     53 |    0.25        0.00
POST     /admin/pages/[locations.LocationsIndexPage]/edit/                                 15     0(0.00%) |     56      52      96     53 |    0.25        0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
         Aggregated                                                                       800     0(0.00%) |     71      45     316     66 |   13.41        0.00

Response time percentiles (approximated)
Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
GET      /admin/api/main/pages/                                                                 69     69     69     69     69     69     69     69     69     69     69      1
GET      /admin/login/                                                                         320    320    320    320    320    320    320    320    320    320    320      1
POST     /admin/login/                                                                          45     45     45     45     45     45     45     45     45     45     45      1
GET      /admin/pages/[base.HomePage]/edit/                                                     79     80     80     80     80    200    200    200    200    200    200     15
POST     /admin/pages/[base.HomePage]/edit/                                                     68     68     69     69     74     86     86     86     86     86     86     15
GET      /admin/pages/[breads.BreadPage]/edit/                                                  66     66     66     67     67     78     86     90    120    120    120    220
POST     /admin/pages/[breads.BreadPage]/edit/                                                  59     59     60     60     64     66     71     75    100    100    100    219
GET      /admin/pages/[breads.BreadsIndexPage]/edit/                                            53     53     53     53     55     55     55     55     55     55     55     26
POST     /admin/pages/[breads.BreadsIndexPage]/edit/                                            53     53     53     53     54     55     57     57     57     57     57     26
GET      /admin/pages/[locations.LocationPage]/edit/                                            94     95     95     96     96    120    120    130    140    140    140    123
POST     /admin/pages/[locations.LocationPage]/edit/                                            83     84     84     84     86     89     95    100    160    160    160    123
GET      /admin/pages/[locations.LocationsIndexPage]/edit/                                      53     55     55     55     78     98     98     98     98     98     98     15
POST     /admin/pages/[locations.LocationsIndexPage]/edit/                                      53     53     54     60     61     96     96     96     96     96     96     15
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
         Aggregated                                                                             66     79     83     84     94     95     97    120    320    320    320    800

[tomkins]

AWS c7g.16xlarge (Graviton3, 64 cores)

uWSGI, 8 editors

Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
GET      /admin/api/main/pages/                                                             8     0(0.00%) |     76      71      83     74 |    0.13        0.00
GET      /admin/login/                                                                      8     0(0.00%) |    166       7     324     14 |    0.13        0.00
POST     /admin/login/                                                                      8     0(0.00%) |     27       4      53      7 |    0.13        0.00
GET      /admin/pages/[base.HomePage]/edit/                                                85     0(0.00%) |     87      81     158     83 |    1.42        0.00
POST     /admin/pages/[base.HomePage]/edit/                                                85     0(0.00%) |    196      68    1212    150 |    1.42        0.00
GET      /admin/pages/[breads.BreadPage]/edit/                                            921     0(0.00%) |     72      67     144     69 |   15.44        0.00
POST     /admin/pages/[breads.BreadPage]/edit/                                            916     0(0.00%) |    196      57    1457    140 |   15.35        0.00
GET      /admin/pages/[breads.BreadsIndexPage]/edit/                                       73     0(0.00%) |     58      55      79     57 |    1.22        0.00
POST     /admin/pages/[breads.BreadsIndexPage]/edit/                                       73     0(0.00%) |    196      52     745    160 |    1.22        0.00
GET      /admin/pages/[locations.LocationPage]/edit/                                      520     0(0.00%) |    102      94     172     99 |    8.72        0.00
POST     /admin/pages/[locations.LocationPage]/edit/                                      520     0(0.00%) |    200      81    1522    160 |    8.72        0.00
GET      /admin/pages/[locations.LocationsIndexPage]/edit/                                 71     0(0.00%) |     59      55      96     57 |    1.19        0.00
POST     /admin/pages/[locations.LocationsIndexPage]/edit/                                 70     0(0.00%) |    172      56     461    140 |    1.17        0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
         Aggregated                                                                      3358     0(0.00%) |    138       4    1522     98 |   56.29        0.00

Response time percentiles (approximated)
Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
GET      /admin/api/main/pages/                                                                 81     81     82     82     83     83     83     83     83     83     83      8
GET      /admin/login/                                                                         320    320    320    320    320    320    320    320    320    320    320      8
POST     /admin/login/                                                                          46     49     52     52     53     53     53     53     53     53     53      8
GET      /admin/pages/[base.HomePage]/edit/                                                     83     84     85     85    100    110    160    160    160    160    160     85
POST     /admin/pages/[base.HomePage]/edit/                                                    150    180    240    250    310    380   1000   1200   1200   1200   1200     85
GET      /admin/pages/[breads.BreadPage]/edit/                                                  69     70     71     72     88     90     92     94    140    140    140    921
POST     /admin/pages/[breads.BreadPage]/edit/                                                 150    190    220    250    330    550    830   1000   1500   1500   1500    916
GET      /admin/pages/[breads.BreadsIndexPage]/edit/                                            57     57     58     58     63     77     78     79     79     79     79     73
POST     /admin/pages/[breads.BreadsIndexPage]/edit/                                           160    190    230    260    400    560    700    740    740    740    740     73
GET      /admin/pages/[locations.LocationPage]/edit/                                            99     99    100    100    120    120    130    150    170    170    170    520
POST     /admin/pages/[locations.LocationPage]/edit/                                           160    190    220    230    320    560    760    870   1500   1500   1500    520
GET      /admin/pages/[locations.LocationsIndexPage]/edit/                                      57     58     59     59     75     77     79     96     96     96     96     71
POST     /admin/pages/[locations.LocationsIndexPage]/edit/                                     140    200    220    240    320    370    430    460    460    460    460     70
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
         Aggregated                                                                             98    120    150    170    240    330    600    800   1200   1500   1500   3358

There was a slight increase in requests per second with 16 editors, but the latency increases quite a bit. Beyond that it also hits the 5 second timeout quite a bit.

[tomkins]

AWS c7g.16xlarge (Graviton3, 64 cores)

uWSGI, 1 viewer

Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
GET      /[base.HomePage]                                                                 109     0(0.00%) |     46      46      47     46 |    1.83        0.00
GET      /[breads.BreadPage]                                                             1373     0(0.00%) |     21      21      22     21 |   23.02        0.00
GET      /[breads.BreadsIndexPage]                                                        114     0(0.00%) |     42      42      78     42 |    1.91        0.00
GET      /[locations.LocationPage]                                                        725     0(0.00%) |     22      22      23     22 |   12.16        0.00
GET      /[locations.LocationsIndexPage]                                                  120     0(0.00%) |     26      26      27     26 |    2.01        0.00
GET      /admin/api/main/pages/                                                             1     0(0.00%) |     73      73      73     73 |    0.02        0.00
GET      /admin/login/                                                                      1     0(0.00%) |     10      10      10     10 |    0.02        0.00
POST     /admin/login/                                                                      1     0(0.00%) |      4       4       4      4 |    0.02        0.00
GET      /admin/logout/                                                                     1     0(0.00%) |      9       9       9      9 |    0.02        0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
         Aggregated                                                                      2445     0(0.00%) |     23       4      78     22 |   40.99        0.00

Response time percentiles (approximated)
Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
GET      /[base.HomePage]                                                                       46     46     46     46     46     47     47     47     47     47     47    109
GET      /[breads.BreadPage]                                                                    21     21     21     21     22     22     22     22     22     22     22   1373
GET      /[breads.BreadsIndexPage]                                                              42     42     42     42     43     43     43     43     78     78     78    114
GET      /[locations.LocationPage]                                                              22     22     22     22     22     22     23     23     23     23     23    725
GET      /[locations.LocationsIndexPage]                                                        26     27     27     27     27     27     27     27     27     27     27    120
GET      /admin/api/main/pages/                                                                 73     73     73     73     73     73     73     73     73     73     73      1
GET      /admin/login/                                                                          10     10     10     10     10     10     10     10     10     10     10      1
POST     /admin/login/                                                                           4      4      4      4      4      4      4      4      4      4      4      1
GET      /admin/logout/                                                                          9      9      9      9      9      9      9      9      9      9      9      1
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
         Aggregated                                                                             22     22     22     22     27     42     46     46     47     78     78   2445

[tomkins]

AWS c7g.16xlarge (Graviton3, 64 cores)

uWSGI, 64 viewers

Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
GET      /[base.HomePage]                                                                5935     0(0.00%) |     60      57     501     60 |   99.49        0.00
GET      /[breads.BreadPage]                                                            65501     0(0.00%) |     27      25     108     28 | 1097.97        0.00
GET      /[breads.BreadsIndexPage]                                                       6044     0(0.00%) |     54      52     241     54 |  101.31        0.00
GET      /[locations.LocationPage]                                                      35657     0(0.00%) |     28      26     144     28 |  597.70        0.00
GET      /[locations.LocationsIndexPage]                                                 5937     0(0.00%) |     34      33     282     35 |   99.52        0.00
GET      /admin/api/main/pages/                                                            64     0(0.00%) |    107      99     115    110 |    1.07        0.00
GET      /admin/login/                                                                     64     0(0.00%) |    213      23     409     55 |    1.07        0.00
POST     /admin/login/                                                                     64     0(0.00%) |     67       7     264     68 |    1.07        0.00
GET      /admin/logout/                                                                    64     0(0.00%) |     32      12     191     16 |    1.07        0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
         Aggregated                                                                    119330     0(0.00%) |     31       7     501     28 | 2000.28        0.00

Response time percentiles (approximated)
Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
GET      /[base.HomePage]                                                                       60     61     61     62     62     62     63     65    200    500    500   5935
GET      /[breads.BreadPage]                                                                    28     28     28     28     28     29     29     29     39    100    110  65501
GET      /[breads.BreadsIndexPage]                                                              54     55     55     55     56     56     57     57     67    240    240   6044
GET      /[locations.LocationPage]                                                              28     29     29     29     29     29     30     30     39    110    140  35657
GET      /[locations.LocationsIndexPage]                                                        35     35     35     35     36     36     36     37    110    280    280   5937
GET      /admin/api/main/pages/                                                                110    110    110    110    110    110    110    120    120    120    120     64
GET      /admin/login/                                                                         380    390    390    390    400    400    410    410    410    410    410     64
POST     /admin/login/                                                                          69     92    100    120    130    140    200    260    260    260    260     64
GET      /admin/logout/                                                                         16     22     31     32     91    140    140    190    190    190    190     64
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
         Aggregated                                                                             28     28     29     29     52     58     61     62    110    390    500 119330

This time the server running locust used significantly less CPU - didn't need to use multiple locust workers.