Review PhET-iO QA process

[zepumph]

From a meeting with @kathy-phet @KatieWoe @Nancy-Salpepi @zepumph, we are trying to simplify the QA process and decrease time/cost where possible. The PhET-iO release (especially for RC testing), was targeted as a spot where we could improve.

I'll start by going through the PhET-iO RC matrix and seeing what isn't important to test anymore. There are three states for each item:

• keep for every test
• gray out and conditionally test when there is extra time
• remove from the matrix.

[Nancy-Salpepi]

Wondering if batch migration can be grayed out or removed from the matrix?

[zepumph]

My recommendations:

QA Book

1. Remove the second about /doc, all doc is accessible from the wrapper index, so really this is captured by clicking through all links in the wrapper index (api doc/examples/client guide).
2. Remove the login instructions
3. Studio doc says: Column 1 says: Standard PhET-iO Wrapper: Add to this to make sure that the standard wrapper is also getting tested. We don't just want to test that it is created correctly, but that the resulting standard wrapper works correctly.
4. State doc: please add an action to play with the downstream simulation when the set-state-rate is 0, and make sure that the upstream state is always correctly applied to the downstream simulation.
5. State doc: I would test with ?phetioDebug=true first, as issues may be more obvious there.
6. State doc says: "Play with the upstream sim and see if any errors come up.", I would say, "repeat actions above"

PhET-iO Testing Matrix

1. I agree with @Nancy-Salpepi, let's gray out batch migration
2. Remove the login test
3. Remove Events: colorized
4. Remove Events: JSON
5. Events: Recording:
   • Remove "Replace &console with &localFile"
   • Make it once per test (not by-device)
   • We just want to confirm that the wrapper index link works, and that it prints a lot of things to the console.
6. Screenshot should be a single cell (once for the whole test)
7. Active should be a single cell (once for the whole test)

In general, I would say that ~85% of the phet-io test should be in the studio/standard wrappers/state wrappers. The simulation standalone doesn't usually differ from the phet brand too much. Perhaps we could re-organize the columns to centralize around those features.

Let me know if you have any thoughts or questions. I'm happy to meet up again if you'd like. Also taggin @samreid so that he is aware of phet-io testing changes, and can add his thoughts if desired.