Revisit the overhead for zvols

[jmpesp]

Nexus currently allocates a little over 65M per 1G of zvol size to account for on-disk metadata / ZFS overhead. Now that we're using raw zvols, and no longer setting volblocksize so low, this should be revisited.

[jmpesp]

Note: when revisiting, ensure that the overhead calculation will always be non-zero, as there will always be some ZFS overhead.

[leftwo]

I should be able to figure out what the actual requirement is on real hardware. I have several systems now with local storage.

[leftwo]

Some data gathering.

I made a dataset, oxp_09/nocrypt/8810, with 2T reservation and quota

when volblocksize=128

2MX4G4PJ # zfs list -r -o name,used,avail,quota,reservation,rawvol -p oxp_09 
NAME                          USED          AVAIL          QUOTA         RESERV  RAWVOL
oxp_09               2199046270976  3992149086208              0              0       -
oxp_09/nocrypt       2199023353856  3992149086208              0              0       -
oxp_09/nocrypt/8810          98304  2199023157248  2199023255552  2199023255552       -

Here just after the creation

2MX4G4PJ # zfs list -r -o name,used,avail,quota,reservation,rawvol -p oxp_09 
NAME                              USED          AVAIL          QUOTA         RESERV  RAWVOL
oxp_09                   2199045939200  3992149417984              0              0       -
oxp_09/nocrypt           2199023353856  3992149417984              0              0       -
oxp_09/nocrypt/8810             155648  2199023099904  2199023255552  2199023255552       -
oxp_09/nocrypt/8810/vol          57344  2199023099904              -              0      on
2MX4G4PJ # echo "now rawcntl"

Here is after rawcntl ran and the volume is initialized.

2MX4G4PJ # zfs list -r -o name,used,avail,quota,reservation,rawvol -p oxp_09 
NAME                              USED          AVAIL          QUOTA         RESERV  RAWVOL
oxp_09                   2199059566592  3992135790592              0              0       -
oxp_09/nocrypt           2199023353856  3992135790592              0              0       -
oxp_09/nocrypt/8810      1099646795776  1099376459776  2199023255552  2199023255552       -
oxp_09/nocrypt/8810/vol  1099646697472  1099376459776              -              0      on

Delete and starting over:

2MX4G4PJ # zfs list -r -o name,used,avail,quota,reservation,rawvol -p oxp_09 
NAME                          USED          AVAIL          QUOTA         RESERV  RAWVOL
oxp_09               2199058472960  3992136884224              0              0       -
oxp_09/nocrypt       2199023353856  3992136884224              0              0       -
oxp_09/nocrypt/8810          98304  2199023157248  2199023255552  2199023255552       -

After create now with 4k volblksz:

2MX4G4PJ # zfs list -r -o name,used,avail,quota,reservation,rawvol -p oxp_09 
NAME                              USED          AVAIL          QUOTA         RESERV  RAWVOL
oxp_09                   2199059517440  3992135839744              0              0       -
oxp_09/nocrypt           2199023353856  3992135839744              0              0       -
oxp_09/nocrypt/8810             155648  2199023099904  2199023255552  2199023255552       -
oxp_09/nocrypt/8810/vol          57344  2199023099904              -              0      on

After running rawcntl and the volume has been initialized:

2MX4G4PJ # zfs list -r -o name,used,avail,quota,reservation,rawvol -p oxp_09 
NAME                              USED          AVAIL          QUOTA         RESERV  RAWVOL
oxp_09                   2199060697088  3992134660096              0              0       -
oxp_09/nocrypt           2199023353856  3992134660096              0              0       -
oxp_09/nocrypt/8810      1103831089152  1095192166400  2199023255552  2199023255552       -
oxp_09/nocrypt/8810/vol  1103830990848  1095192166400              -              0      on