Remove the startpos field from tokens

[KristofferC]

This makes "untokenization" slightly more annoying (since you need to look at the previous token to know where to start) but I think it is worth it. Some tests needed to update based on that. It is a bit unclear if this is worth the effort.

[c42f]

looks good to me !

[c42f]

This really looks a lot nicer 👍

I guess the only downside of this is we don't get so much useful line information when a test fails as the @test macro occurs inside check_lexing(). A possible alternative could be to rearrange like this

@test lex(str) ==  [
        K"Whitespace" => " "        ,
        K"Identifier" => "str"      ,
...
]

[c42f]

By the way I have a separate token type in JuliaSyntax.SyntaxToken (with some fields needed by the parser) and Tokenize.Token is ephemeral when used inside ParseStream. So there shouldn't be memory overhead from having Token with a few extra fields. At some point we can look at whether to merge those types together.

[KristofferC]

I'll drop the last commit for now and put that in a separate PR since it does make untokenization a bit more awkward and perhaps that is not worth it.

[c42f]

Sure. I'm also fine with it as-is though.

[c42f]

I quite like what you did with the tests, overall.

[KristofferC]

I quite like what you did with the tests, overall.

The issue as you say is worse line info. Even putting the @test outside like you suggest we don't really know which one of all the checks fails.

[c42f]

we don't really know which one of all the checks fails

Yeah this is kinda annoying. It's why I went with the verbose style originally.

It's really a problem with @test ... it's like we need a @test_table macro or something which knows about tables of test cases (ie, the reference arrays you have here) and can report which entry in the table was the problem.

[c42f]

@test_table macro or something which knows about tables of test cases

Although... in current Julia the table would have to be a begin... end block so the macro can know which line each test is on, because a multi-line array literal doesn't come with line number nodes :-/

[KristofferC]

Could maybe use JuliaLang/julia#46138

[c42f]

maybe use JuliaLang/julia#46138

well yeah if stdlib versioning wasn't tied to Julia it might be useful :-/