Bump test runner to latest starter2024#71
Conversation
|
I'm also considering just removing the existing setup where the test runner iterates over functions containing the tests and conditionally calls them. I've had a few complaints that this system was too complex. On CPO, the check blocks are typically at the top level not inside functions and all would be run at the same time. I can add legacy tests ensuring the test runner won't regress for existing solutions and then remove this scaffolding completely. |
bin/run.sh
Outdated
| sanitized_test_output=$(echo "${test_output}" | sed "s@${text_to_remove}@@g") | ||
| sanitized_test_output=$(echo "${sanitized_test_output}" | sed '/./,$!d; s/^[[:space:]]*//; s/\n/ /g') | ||
| # Punting on a diagnostic line discrepancy between macOS and Docker for syntax errors | ||
| sanitized_test_output=$(echo "${sanitized_test_output}" | sed -E '/^There were [0-9]+ potential parses\./d; /^Parse failed, next token is /d; /./,$!d; s/^[[:space:]]*//') |
There was a problem hiding this comment.
Just for readability.
| sanitized_test_output=$(echo "${sanitized_test_output}" | sed -E '/^There were [0-9]+ potential parses\./d; /^Parse failed, next token is /d; /./,$!d; s/^[[:space:]]*//') | |
| sanitized_test_output=$(echo "${sanitized_test_output}" | sed -E ' | |
| /^There were [0-9]+ potential parses\./d | |
| /^Parse failed, next token is /d | |
| /./,$!d | |
| s/^[[:space:]]*// | |
| ') |
The third command, is that to delete leading blank lines?
There was a problem hiding this comment.
Yeah, that drops a leading empty line across the board.
There was a problem hiding this comment.
The first pattern was what the Docker image was spitting out. The second pattern was what my Mac was spitting out. I think those could be their own sed section for clarity since they're only needed for the error syntax test. The others are the earlier patterns for deleting the leading blank lines as well as trimming whitespace. We still need those, but for all tests.
|
I removed the TestRun setup as mentioned. This means I'll also need to PR changes to the testing docs as well as updating the |
Co-authored-by: Isaac Good <IsaacG@users.noreply.github.com>
Co-authored-by: Isaac Good <IsaacG@users.noreply.github.com>
Co-authored-by: Isaac Good <IsaacG@users.noreply.github.com>
| sanitized_test_output=$(echo "${sanitized_test_output}" | sed '/./,$!d; s/^[[:space:]]*//; s/\n/ /g') | ||
| if [[ "${legacy_harness}" = false ]]; then | ||
| # Punting on a diagnostic line discrepancy between macOS and Docker for syntax errors | ||
| sanitized_test_output=$(echo "${sanitized_test_output}" | sed -E ' |
There was a problem hiding this comment.
Note heredocs <<< are also an option here.
3 participants
misc-legacy-essentials2020is the only new test. The rest are renamed for clarity following how I structured tests for the Arturo test runner.In the Dockerfile, I patched out two large dependencies that wouldn't be needed for the current Exercism exercises. I should add more tests as part of this PR to make sure the test runner appropriately returns a helpful error message if a student tries to use something using those missing dependency and error out properly.
I also updated run.sh because I had a puzzling discrepancy emerged where my macOS host and the Docker image were both erroring out on the syntax error test for the right reason but with different diagnostic error messages. I should take a closer look at that.