Rewrite file_source and file_parser

[Pennycook]

Feature/behavior summary

These two files are large, complicated, and poorly documented. The desired functionality is actually very simple:

• "Clean" source files, by stripping comments and combining whitespace.
• Provide an iterator over some representation of the cleaned lines.

Request attributes

• Would this be a refactor of existing code?
• Does this proposal require new package dependencies?
• Would this change break backwards compatibility?

Related issues

No response

Solution description

• Replace file_source with a simpler approach, probably by re-using existing lexers.
• Rewrite file_parser to use a simpler representation of the line_info and LineGroup objects.

Additional notes

We should explore implementing this functionality with pygments; it should be possible to iterate through tokens and discard the ones we don't want. This would also open up the opportunity to rewrite other parts of Code Base Investigator to use pygments tokens.

If we do use pygments, we'd be introducing a new dependency.

[Pennycook]

After #122 is merged, parse_file accounts for ~20% of execution time in my offline stress test.

[Pennycook]

Recent experience (see #144) suggests that it may in fact be preferable to merge the cleaning step into the preprocessor. Our current two-step approach destroys physical line information, because tokenization happens after line continuation is handled. Then a source line spans two (or more) physical lines, all tokens are assigned the line number of the first physical line.