Allow the @assistant to read the "quoted" text

[andreyk1991]

How to use GitHub

• Please use the 👍 reaction to show that you are interested into the same feature.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

---

Feature request

Which Nextcloud Version are you currently using: (see administration page)
31.0.8

Is your feature request related to a problem? Please describe.
The feature relates to an inconvenience. Within a talk app chat, in order to pass a user's post to @Assistant, I have to either copy/paste it within chat and include @Assistant, or copy/paste it into nextcloud assistant and then paste the result back into chat.

In the picture you can see the bot not being able to read what post I'm actually trying to translate, even though it's in the quoted text.

Describe the solution you'd like
For solution, I'd like talk app have the ability to pass the text and/or image to the assistant when the post was initiated using the "reply" button.

Describe alternatives you've considered
Usually just having to copy/paste, or process images separately within nextcloud assistant.

[edward-ly]

True, the bot is currently not able to see any previous messages, not just the one being replied to. I'm not completely sure if we currently have the capabilities to do so, though, but it does make sense for the bot to be able to see the previous messages whenever it is relevant.

@oleksandr-nc We could either retrieve the quoted post and prepend it to the prompt string, or retrieve the last N messages in the conversation and submit a core:text2text:chat task instead of just core:text2text. Any ideas on how to do either one with the nc_py_api library?

[oleksandr-nc]

I have not looked there in a while, so I do not remember.
If there is no such API, you need to look If Talk provides this API, you feel free to create a PR and add it.

[andreyk1991]

Here is something that works for me as of now.

main.py
Created a separate bot on the basis of the official talk-bot, registered to its own daemon to avoid breaking official appstore stuff.

It's basic text2text, and only passes the quoted text - does not follow/see past the "quoted" text, but still much more useful than having to copy/paste stuff into at assistant in chat as a separate thread/post.

It miserably fails at passing images.
No errors or anything, but the description is nowhere near to what the image actually is.
If I figure out how to make it properly switch between text2text and image2text, I'll share the update.
For the purposes though, it does the intended job.

[edward-ly]

The workaround looks pretty hacky, but still very impressive! The text2text task does not support image input, indeed, and neither does text2text:chat (we currently don't support chat history and image analysis at the same time), but if you manage to make text2text:chat work too, we would happily review any pull requests for it!