Question on RecurrentGPT/TaleStudio

[TomLucidor]

Sorry for dialing @IlyaGusev but there are just a few questions to ask regarding summarization: https://github.com/IlyaGusev/TaleStudio

1. Are there ways to determine the general summarization quality of open LLM models?
2. How are chapters in a book handled (either in reading or in writing)? Are there ways to maintain coherence?
3. Is this compatible with things like SillyTavern or multi-agent tools?
4. Have you seen this? https://github.com/pszemraj/textsum

P.S. Not sure if @aiwaves-cn are bailing from the project or not cus we saw things like this (okay they are a custom ghostwriter llm company so expected) #30

[IlyaGusev]

Hey, I'm not sure why you are asking it here and not in the TaleStudio repository, but I'll answer here anyway:

1. If there are ground truth summaries, classical metrics such as ROUGE and BLUE will work. If there are no references, there are LLM-based metrics, such as https://github.com/lilakk/BooookScore.
2. You can add new input fields, such as an outline, that can help in maintaining coherence between chapters in writing.
3. Not really, because this repo and TaleStudio use only "native" OpenAI calls. However, it is possible to integrate them into SillyTavern.
4. Nope, but I see it now :)

[TomLucidor]

@IlyaGusev there is no issues page to ask the Q, sorry about this, got a whole lot more other stuff to ask tbh... Could you open that up?

[IlyaGusev]

Just in case, I have opened issues at https://github.com/IlyaGusev/TaleStudio