Stable cfg

[lapp0]

Fixes dottxt-ai#684

Fixes dottxt-ai#959

Fixes dottxt-ai#885

Problem

CFG would encounter runtime errors in main regularly to the point that it rarely succeeds.

• No terminal matches 'e' in the current parser context
• tokens couldn't span multiple terminals, meaning the set of token sequences which could be generated wasn't equal to the set of token sequences accepted by the grammar
• We enforce early termination in cfg due to inconsistent meaning of is_final_state. In some portions of the code it means must terminate, in others it means may terminate.

Solution

Reimplement outlines.fsm.guide.CFGGuide using outlines.fsm.parsing.PartialLark. Credit to Brandon Willard for creating a robust module to handle our CFG structured generation needs.

CFGGuide

• Fix early termination bug
• Fix failure to

Grammars

• Add outlines.grammars.sql_select. enabling integration testing with a more complex grammar

Testing

• Add integration and ASV benchmark test samples for outlines.grammars (arithmetic, json, sql_select)

TODO

• Enable generate.cfg
• Pull integration tests from Add More Grammars to outlines.grammars, Benchmark and Verify Their Integrity dottxt-ai/outlines#587
• ASV tests
• Fix CFG
• Add to outlines.processors
• Unit / integration tests for outlines.processors.CFGLogitsProcessor
• Stop using PartialLark.open_from_package, it has undesired side effects
• Show detailed profile and performance weaknesses