New plugin: Asciidoctor

[nicorikken]

I'm working on an Asciidoc parser using Asciidoctor. The Ruby commands are run in boot-jruby. As it is being used for documentation, the intention is to use the diagram back-end for generating PlantUML and Graphviz images to be included in the final doc. Generating pdf versions on the side would be another option, but in my opinion this goes beyond the goals of Perun.

I have a couple of points worth discussing:

1. One of our current documents has too many characters to be handled in the eval statement when handling pods (concerning the render stage currently). I'm considering a file-based solution for HTML inclusion. So the Asciidoctor plugin would generate a (partial) HTML file and adds the referred filename in say a :include key in the metadata. This allows pages to refer to this file, rather than load the value from the :content key. This slurp could also be handled within the Asciidoctor plugin to end up with a similar behavior as say the Markdown plugin.
2. Asciidoc offers several built-in attributes to handle metadata. I plan on parsing the Asciidoc file to extract these attributes and include these into the metadata to be reference.
3. Asciidoc offers the ability to evaluate the file for different inline inclusion macros. My use-case uses these ifdef statements to include code examples for different languages. We intend to generate the document for these different attributes. This evaluation could be done by parsing all ifdef commands and just evaluating all options. A more conservative way would be to either include the attributes in the plugin call, and/or announcing the desired evaluation attributes in a commented asciidoc file header.

[podviaznikov]

@nicorikken having Asciidoctor pluging would be great.
Also in my opinion generating pdfs or ebooks can totally be Perun plugins at some point.

Regarding your questions/points:

1. is it know issue with boot (too many characters eval in the pod)?
2. sounds good. I think it's reasonable. what kinda of metadata would be added
3. didn't quite understand problem, question. Maybe you can provide more context

[nicorikken]

@podviaznikov I assume pdf and epub generation would be based on the final rendering stage, after the rendering stage has been handled.

1. Not that I know of, but in the Boot documentation it focuses mainly on using the filesets for transferring content. Are you hinting at finding a workaround within Boot? Edit: The fileset metadata is limited in size, and therefore the max number of character is limited.
2. This mainly concerns metadata present in current examples, like author, title, version, date, etc.
3. I guess I'll just work out a generic way of handling different version (with the majority of the content identical). I'm thinking about having some options available for which to evaluate (say three different language versions from the same source-document). But in a way the multiple versions of the final content would have to be associated with the same original source-file.

[podviaznikov]

@nicorikken:

1. interesting I didn't know that fileset metadata is limited. If it's indeed the problem we would need global solution. It works for me now with 20 blog posts. Do you know what is the limit? @Deraen, @martinklepsch do you know about such limitation?
2. nice, your initial plan sounds right here
3. sounds like you have a plan:)

[ekroon]

@podviaznikov

The problem with storing output in the metadata is that there is a limitation to what you can store in a string in a .class format (65535 bytes). If you start a boot repl the following shows the problem:

(boot.pod/with-eval-in (boot.pod/make-pod) (pr-str ~(reduce str (take 65535 (repeat "s"))))) 

versus

(boot.pod/with-eval-in (boot.pod/make-pod) (pr-str ~(reduce str (take 65536 (repeat "s"))))) 

which yields

clojure.lang.Compiler$CompilerException: java.lang.ClassFormatError: Unknown constant tag 115 in class file pod$eval234, compiling:(NO_SOURCE_PATH:0:0)
             java.lang.ClassFormatError: Unknown constant tag 115 in class file pod$eval234

So doing something extra with the output after a markdown / asciidoctor generation step can be problematic. A large amount of files to transform will not trigger this problem, but a large input file will. :)

[martinklepsch]

I had a brief chat with @micha about this and he mentioned that the implementation of eval-in* might be the culprit here. Switching from something like pr-str to a streaming protocol like Fressian for this stuff should allow for bigger things to be passed into pods.

[ekroon]

A bit more investigation indicates that the problem is probably in the combination of eval and backtick\template as with-eval-in has the same problem:

(boot.pod/with-eval-in false (identity ~(reduce str (take 65536 (repeat "s")))))

Maybe some kind of helper function to split long strings in chunks is a solution?

[nicorikken]

I've managed to get a workaround via files, allowing longer strings of generated html content. I've now implemented this using an AbsoluteFilePath of the output files to an :include key. I was considering adding some options to bring backwards compatibility with the :content based approach, although it might make more sense to adopt a file-based solution as default to avoid running into the issues mentioned above.

[podviaznikov]

@nicorikken do you have code I can look into?

[dl1ely]

Guys, i would love to see a asciidoctor-plugin. Anything i can help with? @nicorikken can you do a PR soonish?

[nicorikken]

I'm currently working on making a clean pull request from the current working state. Compatibility with other Perun features is left to be desired (due to the file-based handling), and dependencies can be reduced by wrapping the handling in a pod (which I lacked to do). In the meantime check my personal clone to watch the progress.

[nicorikken]

Now that was relatively easy. I now have a working version. The file-based method lacks compatibility with the other methods used in the Perun base. And of course many features can be added to better suit the Asciidoctor use-case. But still there is a working prototype: https://github.com/nicorikken/perun

[martinklepsch]

@nicorikken have you looked at this again since send! has been introduced? send! is experimental but I think it might solve the issue you were encountering.

boot-clj/boot#339

[nicorikken]

@martinklepsch I haven't looked this problem in a while, but that feature seems to fix the problem. I just published my in-house work on the Perun project in a separate repo. Still a bit rough though. The send! feature seems to be the right way of addressing the issue of passing around large lumps of text, so that would seem a worthwhile improvement on the Perun codebase. Maybe I'll have a go at it at some point.

[nicorikken]

I'm currently prototyping Asciidoctorj in Clojure. If I get the desired features to work, I want to redo my original PR. I'll probably consider the send implementation a secondary step, as this is equally an issue for the Markdown handling. It is just that my use-case involves large documents.