Design decision: storing large arrays of data

[avirshup]

Possibilities:

1. As a table of values (easy to rearrange atoms; hard to add new fields)
2. As a set of arrays (easy to add new fields, hard to rearrange atoms)
3. As a list of objects (explicit but makes the file very large)

Examples:

// 1) Storing fields as tables: creates an mmCIF/PDB-like layout
{atoms={type:'table[atom]',
        fields=['name', 'atomic_number', 'mass/Dalton', 'residue_index', 'position/angstrom', 'momentum/angstrom*amu*fs^-1']
        entries=[
          ['CA', 6, 12.0, 0, [0.214,12.124,1.12], [0,0,0]],
          ['N', 7, 14.20, 0, [0.214,12.124,1.12], [0,0,0]],
          ...}

// 2) Storing fields as arrays: much more compact, but harder to read and edit
{atoms={ num_atoms=2341,
        names:['CA','CB','OP' ...], 
        atomic_numbers:[6,6,8, ...],
        masses:{val:[12.0, 12.0, 16.12, ...], units:amu},
        residue_indices:[0,0,0,1,1, ...],
        positions:{val:[[0.214,12.124,1.12], [0.214,12.124,1.12], ...], units:angstrom},
        momenta:{val:[[0,0,0], [1,2,3], ...], units:angstrom*amu*fs^-1}
        }

// 3) Storing the fieldnames for each atom: readable, but makes the file huge
{atoms=[
  {name:'CA', atnum:6, residue_index:0,
   mass:{value:12.00, units:'Daltons'},
   position:{value:[0.214,12.124,1.12], units:'angstroms'},
   momentum:{value:[0.0, 0.0, 0.0], units:'angstrom*dalton*fs^-1'},
  },
  {name:'N', atnum:7, residue_index:0,
   mass:{value:14.20, units:'Daltons'},
   position:{value:[0.214,12.124,1.12], units:'angstroms'},
   momentum:{value:[0.0, 0.0, 0.0], units:'angstrom*dalton*fs^-1'},
  },
  ...
  }]
}

[davidlmobley]

This strikes me as being relevant to @pgrinaway as he's thinking about systems where the number of atoms might be changing over time, if I understand correctly... For example, having a system where the ligand might vary over the course of a simulation...?