Skip to content

Report less test_code #80

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
SaschaMann merged 4 commits into from
Jun 6, 2022
Merged

Report less test_code #80

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
673753b
Report less test_code
cmcaine May 23, 2022
8e2b056
Fix test_code(r)
cmcaine May 23, 2022
89f1d93
Add test_logs to CI
cmcaine May 23, 2022
85e32d7
Introduce a new constant to increase clarity
cmcaine Jun 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:

# Change this to add or remove tests
env:
TESTS: all_passing,all_passing_with_debugging,everything_at_once,one_fail,one_passing,truncated_output,grains,complex-numbers
TESTS: all_passing,all_passing_with_debugging,everything_at_once,one_fail,one_passing,truncated_output,grains,complex-numbers,test_logs
JULIA_NUM_THREADS: 2

steps:
Expand Down
13 changes: 10 additions & 3 deletions src/tojson.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ tests collapsed into a single report.
"""
const TEST_RESULT_COLLAPSE_THRESHOLD = 2
const MAX_REPORTED_FAILURES_PER_TESTSET = 5
const MAX_REPORTED_PASSING_TEST_CODE_PER_COLLAPSE = 5

"""
tojson(output::String, ts::ReportingTestSet)
Expand Down Expand Up @@ -85,11 +86,11 @@ function tojson(output::String, ts::ReportingTestSet)
output = truncate_output(output)

function test_code(result::Test.Result)
if startswith(string(result.test_type), "test_throws")
if hasproperty(result, :test_type) && startswith(string(result.test_type), "test_throws")
"@test_throws $(result.data) $(result.orig_expr)"
elseif result isa Test.LogTestFailure
"@test_logs $(join(result.patterns, ' ')) $(result.orig_expr)"
elseif result.test_type == :test_unbroken
elseif hasproperty(result, :test_type) && result.test_type == :test_unbroken
"@test_broken $(result.orig_expr)"
elseif result isa Test.Broken
macro_name = result.test_type === :skipped ? "@test_skip " : "@test_broken "
Expand Down Expand Up @@ -168,10 +169,16 @@ function tojson(output::String, ts::ReportingTestSet)

if collapse_passing_tests
collapsed_name = num_passing == num_results ? name : "$name » $num_passing tests"
# If many tests pass then the reported test_code will be excessively large, so we truncate it.
if num_passing > MAX_REPORTED_PASSING_TEST_CODE_PER_COLLAPSE
code = join(map(test_code, passing_tests[1:MAX_REPORTED_PASSING_TEST_CODE_PER_COLLAPSE]), '\n') * "\n..."
else
code = join(map(test_code, passing_tests), '\n')
end
push!(tests, Dict(
"name" => collapsed_name,
"status" => "pass",
"test_code" => join(map(test_code, passing_tests), '\n')
"test_code" => code,
))
end

Expand Down
2 changes: 1 addition & 1 deletion test/fixtures/grains/results.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"tests": [
{
"name": "beginning squares",
"test_code": "@test on_square(1) == 1\n@test on_square(2) == 2\n@test on_square(3) == 4\n@test on_square(4) == 8\n@test on_square(16) == 32768\n@test on_square(32) == 2147483648\n@test total_after(1) == 1\n@test total_after(3) == on_square(1) + on_square(2) + on_square(3)",
"test_code": "@test on_square(1) == 1\n@test on_square(2) == 2\n@test on_square(3) == 4\n@test on_square(4) == 8\n@test on_square(16) == 32768\n...",
Copy link
Copy Markdown
Contributor Author

@cmcaine cmcaine Jun 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@exercism/reviewers Can someone approve and merge this, please?

The PR truncates the emitted test_code from test sets that contain many passing tests. The PR is part of an ongoing effort to let the exercism web editor support some of our parametrized tests. You can see the effect of the PR in the diff of the fixture/snapshot test above.

The PR also fixes a function that generates the test_code strings from a given test and ensures that that codepath is tested in CI.

Contrary to normal exercism practice, this PR should not be squashed (though it doesn't matter all that much).

"status": "pass"
},
{
Expand Down