Use the leetcode assessment of test correctness#149
Open
dpkatz wants to merge 1 commit intokaiwk:masterfrom
Open
Use the leetcode assessment of test correctness#149dpkatz wants to merge 1 commit intokaiwk:masterfrom
dpkatz wants to merge 1 commit intokaiwk:masterfrom
Conversation
The code currently decides whether or not the answer is the expected
answer by comparing the `code_answer` and `expected_code_answer`
output blocks in the response to see if they're the same. This can be
an issue for problems where the output is a list but the problem
specifies that the order doesn't matter. For instance, for problem
49 (Group Anagrams) the problem states that "You can return the answer
in **any order**." However, running my solution to that problem gave:
FAIL:
Code Answer:
[["eat","tea","ate"],["tan","nat"],["bat"]]
[[""]]
[["a"]]
Expected Code Answer:
[["bat"],["nat","tan"],["ate","eat","tea"]]
[[""]]
[["a"]]
in the `*leetcode-result-49*` buffer. The problem here is that the
we're not respecting the order-insensitivity of the answer the problem
specifies.
Looking at the result data structure, we see that there is also a
`correct_answer` field which is the Leetcode assessment of whether the
answer was correct or not. For example, in the case shown above the
"correct_answer" field is set to "true".
This change uses the "correct_answer" field to decide whether the test
passed or failed, rather than comparing the output blocks directly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The code currently decides whether or not the answer is the expected answer by comparing the
code_answerandexpected_code_answeroutput blocks in the response to see if they're the same. This can be an issue for problems where the output is a list but the problem specifies that the order doesn't matter. For instance, for problem 49 (Group Anagrams) the problem states that "You can return the answer in any order." However, running my solution to that problem gave:in the
*leetcode-result-49*buffer. The problem here is that the we're not respecting the order-insensitivity of the answer the problem specifies.Looking at the result data structure, we see that there is also a
correct_answerfield which is the Leetcode assessment of whether the answer was correct or not. For example, in the case shown above the "correct_answer" field is set to "true".This change uses the "correct_answer" field to decide whether the test passed or failed, rather than comparing the output blocks directly.
Resolves #148