Use a lazy builtins file generated from docs for igor pro

[jneen]

Factors out igorpro/builtins.rb which is lazily loaded on first lexer initialization. Docs are generated from the HTML at https://docs.wavemetrics.com/igorpro/commands

[jneen]

@MichaelHuth Can I ask you to take a look here? I want to make sure the way I'm generating these builtin lists from the docs makes some amount of sense, and I'm not super familiar with Igor Pro. In particular, if there's a more conveniently available machine-readable source for this, it might be better than scraping the docs site.

[jneen]

Also tagging @t-b as an Igor Pro user.

[t-b]

@jneen Thanks for tagging us. Fetching Operations and Functions from their documentation website (which is only available since a couple of weeks) is a good idea. But this removes all non-builtins. These are popular addons (called XOPs). The missing ones start with:

• zeromq
• Axon
• MCC
• HDF
• fDAQmx_
• VDT
• VISA
• TUFXOP
• NI
• NC
• MFR
• JSONXOP
• ITC
• GPIB
• GIS
• FCALL
• DAQmx
• vi
• TDM

As newer Igor Pro versions only contain more operations/functions it is also possible to just add everything back in which is present in the old list but not in the new one.

[jneen]

I see! Are these XOPs documented (ideally in a machine-readable way) somewhere we can fetch?

[MichaelHuth]

There are Operations with the same name as Functions in Igor Pro, e.g. "note". Thus, this line needs to be removed.

[jneen]

Rouge can only highlight them as one kind of token, though - which is best in this case?

[jneen]

To be clear, if there is context that can distinguish them, Rouge can detect that, but the current IgorPro lexer only checks inclusion in these:

          elsif self.class.igorFunction.include? m[0].downcase
            token Name::Builtin
          elsif self.class.igorOperation.include? m[0].downcase
            token Keyword::Reserved
            push :operationFlags

(also we're checking inclusion of the lowercased version, which means anything in either of these sets with a capital letter is currently ignored. this generator downcases the source as well.)

[MichaelHuth]

As anyone can write XOPs there is probably no machine readable documentation for the general case. For the XOPs that we developed there is sphinx generated documentation, e.g. at
JSON-XOP: https://docs.byte-physics.de/json-xop/
or
TUF-XOP: https://docs.byte-physics.de/tuf-xop/

more XOP docu can be found through the links here https://www.byte-physics.de/en/projects.html

So I guess you still need some list for the non-builtins.

[jneen]

To be blunt, I would rather not maintain a giant definitive list of all known functions in this framework that I don't use, unless there is a way to automate it. I think if there are XOPs that we can fetch and process automatically in the generator task, I'm happy to include them, though.

[t-b]

I think I'm missing the motivation for the PR.

All the relevant content changes to the IgorPro lexers where done by some of our guys e.g.

git log --pretty="%ae" lib/rouge/lexers/igorpro.rb
michael.huth@byte-physics.de
larouxn@gmail.com
ashmaroli@users.noreply.github.com
mike@inqk.net
ashmaroli@gmail.com
thomas.braun@byte-physics.de
ashmaroli@gmail.com
matthias.kastner@byte-physics.de
matthias.kastner@byte-physics.de
matthias.kastner@byte-physics.de
matthias.kastner@byte-physics.de
matthias.kastner@byte-physics.de
matthias.kastner@byte-physics.de
matthias.kastner@byte-physics.de
gfuji@cpan.org
matthias.kastner@byte-physics.de
matthias.kastner@byte-physics.de
matthias.kastner@byte-physics.de
matthias.kastner@byte-physics.de
matthias.kastner@byte-physics.de
matthias.kastner@byte-physics.de
matthias.kastner@byte-physics.de

and if there is something wrong we are happy to fix it. But I don't see the point in autogenerating the list of operations/functions as that only changes every few years. And can only be done correctly with the program manually.