use llguidance library for constraints (including json schemas)

[mmoskal]

Fixes #754

This imports the llguidance library for dealing with constraints (regexp, json schema, CFGs in lark syntax) and removes the aici submodule which was implementing regexps and CFGs (in yacc syntax) before. The code for both is related (I wrote the original versions of both).

This is WIP - putting it out there for comments about how likely it is to be merged.

TODO

• remove testing scriptgs (scripst/*.sh)
• remove aici folder
• update python APIs to accept "json_schema"
• update examples

[github-actions]

Code Metrics Report

  ===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 C Header                2           35           28            0            7
 Dockerfile              1           41           22           10            9
 Happy                   1          442          369            0           73
 JSON                   12          105          104            0            1
 Python                 57         2435         2094           67          274
 Shell                   1           57           22           18           17
 Plain Text              3         3723            0         2413         1310
 TOML                   18          594          529            2           63
 YAML                    2           21           19            2            0
-------------------------------------------------------------------------------
 Jupyter Notebooks       4            0            0            0            0
 |- Markdown             2           77           32           31           14
 |- Python               2          205          178            1           26
 (Total)                            282          210           32           40
-------------------------------------------------------------------------------
 Markdown               44         3314            0         2510          804
 |- BASH                 6          101           98            0            3
 |- JSON                 1           12           12            0            0
 |- Python               7          121          109            0           12
 |- Rust                12          406          344            0           62
 |- TOML                 2           75           63            0           12
 (Total)                           4029          626         2510          893
-------------------------------------------------------------------------------
 Rust                  289        88547        79444         1844         7259
 |- Markdown           140         1499           25         1366          108
 (Total)                          90046        79469         3210         7367
===============================================================================
 Total                 434        99314        82631         6866         9817
===============================================================================
  

[EricLBuehler]

Hi @mmoskal! Thanks for the PR.

I took a look at the code and it looks great already. As you noted, adding some examples and updating the APIs + docs would be great (maybe some tests?). After that, this is something I would certainly be excited to merge.

[mmoskal]

Thanks for the review! I probably won't have time for it this week, but hopefully the one after that.

[EricLBuehler]

@mmoskal friendly ping :) Let me know if I can help!

[mmoskal]

It seems I won't have much time in the next two-three weeks. If anyone else is willing to take on the TODOs that would be great!

I'll be extending the llguidance APIs to be able to perform multithreaded mask computation in background (one of the things I will be busy with). Enabling this will get the constraint computation out of the hot path of sampling - but that should be a one-line change here.

[EricLBuehler]

@mmoskal that sounds exciting - I totally understand however! I tried to recreate this PR to quickly merge it and get something working but it seems many of the APIs have changed since the code here was written. Looking forward to the enhancements and hopefully getting this merged!

[mmoskal]

@EricLBuehler should be good for review!

[EricLBuehler]

Thanks, this looks really great! I was wondering if you could please add more examples, namely:

• Server: add an example of JSON schema and maybe a llguidance grammar
• Python: similarly, add examples of Lark, JSON schema, and maybe a llguidance grammar
• Rust: similarly, add examples of Lark, JSON schema, and maybe a llguidance grammar

Otherwise, the code looks great. I left a few comments, and it looks like formatting and clippy are failing in CI.

[EricLBuehler]

Wow, this is really cool! I also saw a JSON yacc grammar here which might be nice to include as a seperate example, too?

[mmoskal]

Added, but I'm not sure this is the right place to point people to - getting a grammar for specific JSON is somewhat tricky (for example, you need to care about quoting in strings).

I have also added example in Llguidance format that combine Lark with JSON Schema - that's a more robust way of doing this.

[mmoskal]

Added samples in rust, python and for server; fixed clippy and fmt. Should be good to go!

[EricLBuehler]

Thank you so much! This is really amazing and looks great.

I'll keep my eye on llguidance's development for new features!

[mmoskal]

Thanks for merging!

There is a few areas where the integration can be improved, I filed #963, #964, #965 to track.