-
-
Notifications
You must be signed in to change notification settings - Fork 10
Upgrade to version 2 spec #19
Copy link
Copy link
Open
Labels
x:action/improveImprove existing functionality/contentImprove existing functionality/contentx:knowledge/advancedComprehensive Exercism knowledge requiredComprehensive Exercism knowledge requiredx:module/test-runnerWork on Test RunnersWork on Test Runnersx:size/largeLarge amount of workLarge amount of workx:type/codingWrite code that is not student-facing content (e.g. test-runners, generators, but not exercises)Write code that is not student-facing content (e.g. test-runners, generators, but not exercises)
Description
Metadata
Metadata
Assignees
Labels
x:action/improveImprove existing functionality/contentImprove existing functionality/contentx:knowledge/advancedComprehensive Exercism knowledge requiredComprehensive Exercism knowledge requiredx:module/test-runnerWork on Test RunnersWork on Test Runnersx:size/largeLarge amount of workLarge amount of workx:type/codingWrite code that is not student-facing content (e.g. test-runners, generators, but not exercises)Write code that is not student-facing content (e.g. test-runners, generators, but not exercises)
We've recently updated the test runner interface specification to allow for two types of test runners, identified via a new
versionproperty in theresults.jsonfile (see the spec for the version property):version: 1test runners are quite basic, and only detect if either all tests passed or not. These test runners capture the console output of the test runner and output that. They do not include details on individual tests passing/failingversion: 2test runners contain detailed information on individual tests passing/failing. We've recently added a required field namedtest_code, which contains the code the test ran to verify the behavior (see the spec for the test_code property).The test runner's output is currently what I refer to as version 1.5: it contains the individual test information, but does not yet contain the
test_codekey. If possible, the test runner should be updated to also include the test code, in which case theversionproperty should also be added with2as its value.You are completely free on how you want to extract the test code, either via the AST or via finding the test code in the source code text via string manipulation.
Let me know if there are any questions.