Evaluate Usage of Atom's Highlights

[killercup]

Atom's syntax highlighting stack has been open sourced. This post gives an overview.

Mainly, highlights is of relevance to grock, as it could replace highlight.js.

Important questions to answer:

• How is the language support?
  Excellent.
• How is the styling support?
  Good.
• Does it support hooks? (Cf. Fork Highlight.js and Add Hooks for Comments #5)
  Probably.
• How fast is it? (Compare to highlight.js.)

[killercup]

How is the language support?

Excellent.

It seems like highlights itself supports language definitions either as TextMate grammar or as CommonJS packages with grammar files (cson/json).

Here's a list of language definitions for Atom.

[killercup]

How is the styling support?

Good.

But most themes for Atom are using LESS. The syntax highlighting definitions look like this index.less.

Here's a list of color schemes packages available for Atom.

[killercup]

Does it support hooks?

Probably.

The main highlighter class is quite short (<100 lines), see highlights.coffee. Similar code would have to be written to create a transform method which can be used by grock to highlight vinyl file objects.

It may be possible to detect and extract comments in code by adding a check in the loop to render tokens in scopes, as a comment would be a specific scope. That way, the beginning of a comment could be used to start a new file segment (each segment contains arrays of comments and code lines).

E.g., in the Lua grammar, the scopes comment.block.lua and comment.line.double-dash.lua are defined. Using this (and the prefixes punctuation.whitespace.comment and punctuation.definition.comment.lua), one should be able to correctly separate comments from Lua code.

[killercup]

I just implemented code splitting and highlighting with highlights. It's still quite rough around the edges but it renders grock's own documentation (js/coffee/md, thin style). Code is in the feature/14-highlights branch.

[killercup]

Benchmark: Rendering grock's Documentation

Test	highlights	highlight.js
Time	2.49s	1.90s
Output size (zip)	1.2MB (108kB)	339kB (84kB)

Considering grock takes approx. 1.1s to load on my machine, highlights seems to need at least 70% more time (times are averages of ~20 runs each).

Also worth noticing: The output size of the complete docs directory (incl. assets) is 339kB (84kb as zip) for highlight.js and 1.2MB (108kB as zip) for highlights. The latter seems to be putting everything it knows into the markup, producing truckloads of nested span tags.

[jonruttan]

While fiddling around with the code on this branch trying to see how much of the highlight.js and lib/languages.coffee code I could remove and still have it run, I stumbled upon the fact that that seemed to speed it up quite a bit.

I ran some tests to get some numbers. Like @killercup, my tests are based on rendering Grock's documenation. I used the following command to generate the docs:

./bin/grock --style=thin

Running the command outputs info about the build like so:

[grock] Beginning to process  523 ms
[grock] Writing to docs/
[grock] Done. Generated in 1.03 s

We're only interested in the amount of time it took to generate, so I used the following awk code to extract the number of milliseconds it took to generate the documentation from Grock's output:

awk '/Done. Generated in/ { print $5 * ($6 == "s" ? 1000 : 1) }'

I wanted to be able to compare my results with those which @killercup derived, so I've also averaged 20 iterations.

Here's the full code I used for the tests:

cat /dev/null > /tmp/grock.out.txt;
for i in $(seq 1 20); do 
  ./bin/grock --style=thin | awk '/Done. Generated in/ { print $5 * ($6 == "s" ? 1000 : 1) }' | tee -a /tmp/grock.out.txt;
done;
awk 'NR == 1 { max=$1; min=$1; sum=0 }
{ if ($1>max) max=$1; if ($1<min) min=$1; sum+=$1;}
END {printf "Min: %d ms.\tMax: %d ms.\tAverage: %.1f ms.\n", min, max, sum/NR}' /tmp/grock.out.txt;
rm -f /tmp/grock.out.txt

NOTE: The Min/Max/Avg is from this answer to Is there a way to get the min, max, median, and average of a list of numbers in a single command? on Unix & Linux Stack Exchange.

Running the script on the master branch, using highlight.js, gives the following output:

1000
976
1050
1020
965
958
962
998
1030
1050
1020
991
959
999
966
961
1040
1030
999
1050
Min: 958 ms.    Max: 1050 ms.   Average: 1001.2 ms.

On @killercup's machine this value was 1.90s, or 1900ms, giving a k value between the two machines of:

1900ms. / 1001.2ms. = 1.898

My next test was on the feature/14-highlights branch, which uses atom/highlights, but also requires highlight.js:

1040
1010
986
1010
1000
1040
994
993
997
991
1280
1340
1030
1050
1110
1200
1090
1070
1020
1050
Min: 986 ms.    Max: 1340 ms.   Average: 1065.0 ms.

If we multiply 1065.0ms by the k value from above,

1065.0ms. * 1.898 = 2021ms.

we get 2021ms, which is only 81% of @killercup's measurement of 2490ms. I would have assumed the values would be closer, so I can only speculate as to why there is such a big discrepancy (my tests are writing to an SSD, and atom/highlights is generating 3-4x more data, so it may be due to file I/O.)

Continuing on, I stripped out highlight.js from the branch using the following steps:

• edit package.json, rm l. 46 # "highlight.js": "~8.0.0",
• edit index.coffee, rm l. 1 # getLanguage: require('./getLanguage')
• rm lib/languages.coffee
• edit lib/generator.coffee, rm l. 55 # .pipe t.getLanguage()

Running the tests again gives:

952
983
943
914
926
915
940
940
941
914
925
960
926
941
953
926
944
948
1010
991
Min: 914 ms.    Max: 1010 ms.   Average: 944.6 ms.

This gives the fastest generation times yet.

For one last test, I upgraded the atom/highlights package in packages.json from 0.7.0 to the latest version, 1.2.0:

1130
1110
1110
1150
1110
1110
1130
1100
1110
1120
1140
1110
1180
1110
1130
1110
1130
1110
1090
1090
Min: 1090 ms.   Max: 1180 ms.   Average: 1119.0 ms.

This yields the slowest generation times yet. I wonder what's happened to atom/highlights to make the latest version 175ms. slower?

I'll run some more tests. It's probably worth trying a few of the more recent versions of atom/highlights to see if they're all as slow as version 1.2.0, and whether there's been a trend over time. I'd also like to see how hard it would be to modify atom/highlights so the renderer didn't output a sea of superfluous span tags.

If atom/highlights ends up being peppy enough, and some of its output tags can be trimmed back, then it might be worth re-opening this issue.

[killercup]

@jonruttan, thank you very much for your wonderful investigation!

If using highlights increases the compile time by 20% but makes grock's code much cleaner and easier to extend (e.g. by having the option to include custom language specifications from Atom), we should use it. Far more concerning to me is the amount of output highlights generates.

---

(What follows is speculation. I may be completely wrong. I haven't looked at the code in about a year.)

I noticed that transforms/splitCodeAndComments as well as transforms/highlight seem to call hightlights to parse a file (or parts of it).

The first to split code from comments (as the name suggests), the second to highlight the code. Ideally, a file should only be tokenized once, then split into segments, and then the sub-trees of tokens should be rendered.

The biggest problem with this will be that breaking code at seemingly arbitrary points can lead to invalid highlighting.

Do you think it might be possible to use highlights in such a way that we can insert split points that preserve the scope the code is currently in? Essentially, I'm talking about rewriting this part of highlights' highlightSync method so that instead of an HTML string it will return the segments that grock needs.

[killercup]

It may also be interesting to look more deeply at the implementation of first-mate (which provides the grammar registry and parsing hightlights uses). Skipping highlights and using first-mate directly might also allow us to skip the dependency on season, fs-plus and underscore-plus.

[jonruttan]

You're welcome @killercup, the tests are not a problem. Thanks for maintaining the project.

With this extra info I dug a little further into the code in the features/14-highlights branch. It turns out that all the work is being done in transforms/splitCodeAndComments, and transforms/highlight isn't being used at all.

I looked into your question about using highlights in such a way that we can insert split points that preserve the scope the code is currently in, and I don't think it looks like it would be too hard.

I also did a little work on reducing the size of the generated documentation. As is, the generated documentation is 1236KiB.

By removing the dispensable span tag being added on Line #114 of transforms/splitCodeAndComments it shrunk to 1052KiB.

With a quick hack to atom/highlights' pushScope method I was able to further reduce it to 1020KiB, and I'm pretty sure with some more work in that area I can shrink it some more. Beyond that it should be possible to further modify the way that atom/highlights outputs tags to take better advantage of nesting.

[killercup]

I looked into your question about using highlights in such a way that we can insert split points that preserve the scope the code is currently in, and I don't think it looks like it would be too hard.

Thats great news, @jonruttan!

Are you going to continue working on this? I would love to see a pull request with your changes! (Even work-in-progress, you can just make it merge to the feature/14-branch on this repo.)

it shrunk to 1052KiB

Are the 339kB for highlight.js I quoted above still valid? That means the newly generated code will be about 3 times larger. We should recommend using a compressed file system in the Readme ;)

Edit: I totally forgot to reopen the issue earlier.