AI Integration RFC

[nelsonkopliku]

Add AI integration RFC.

Visual here

[arbulu89]

Hey @nelsonkopliku ,
Nice stuff!
I have not been involved in the topic, and most probably my input is not the most relevant one, but after reading the docs I learned some interesting things and exposes the feature nicely. Well done!

I have just a question:
If we had the chance the release some sort of "tech-preview" release, without any required support now or in the near-future for this feature (like I need to do some private demo tomorrow without any commitment), would you still choose the same? (elixir implementation in web and custom protocol).

At the end, as you repeat quite often, this environment is evolving really quickly and it is actually really overwhelming.
The thing that I most fear is that we choose something that we regret sooner than later and could have long-lasting effects in our delivery.
In that sense, I at least agree that a elixir based house-made code looks the simpler to deliver, as we don't need to package more things (which is the most frightening thing).

It is not a question for you, most probably more for @abravosuse , but I wonder if we could have any option to make this feature some sort of non-supported tech-preview in the 1st release. Like if we decided to use a 3rd party container, we could just ship it in PackageHub or so without real long-term support commitment.

[arbulu89]

comment:
In relation to API key as input in the UI. It is more a "product" thing more than a technical comment.
I understand that this comes as a desire to continue moving Trento to a "all in UI" kind of tool.
But we need to be aware of the impact this has, specially since we would be storing a really sensitive API key value in our database.
I guess that we can protect this encrypting the code, etc, but I reckon we should be careful of asking the users such a thing.

This anyway opens more questions, like:

• This keys might be really personal, so, how are we going to treat them? User based, installation wide?, etc

[nelsonkopliku]

The api key to interact with LLMs is meant to be user based. Each user should provide his personal api key.

And yes, it is an option that those api keys will be encrypted and because of that fall in trento's secrets rotation.

[nelsonkopliku]

Thanks @arbulu89, happy to provide room for a broader audience to be involved in the topic.

I have just a question: If we had the chance the release some sort of "tech-preview" release, without any required support now or in the near-future for this feature (like I need to do some private demo tomorrow without any commitment), would you still choose the same? (elixir implementation in web and custom protocol).

It depends.
If we're talking about a really quick and dirty thing where neither security is a concern, where velocity is more important than other things, would go the python/js langchain backend implementation, as done in this POC, for instance.

As soon as we start talking about "coherent experience in the product", authentication, authorization, activity logging, trento's installation experience as a whole... well then I think we have to make some more thorough thinking at least.

this environment is evolving really quickly and it is actually really overwhelming.

Yes, it can be overwhelming. The most sensible thinking has been made to consider the team capacity and expertise, expectations, strategy and goals.

The thing that I most fear is that we choose something that we regret sooner than later and could have long-lasting effects in our delivery. In that sense, I at least agree that a elixir based house-made code looks the simpler to deliver, as we don't need to package more things (which is the most frightening thing).

Same concerns here as well. There is a risk in any decision we'd make, and on the other hand we cannot get paralyzed into analysis.
I trust that we will be able to adjust as the need raises, as we go forward.

It is not a question for you, most probably more for @abravosuse , but I wonder if we could have any option to make this feature some sort of non-supported tech-preview in the 1st release. Like if we decided to use a 3rd party container, we could just ship it in PackageHub or so without real long-term support commitment.

I'll leave it to Alberto to provide his thoughts about this, however here's my two cents:

• unless very clear and strong expectation management, shipping something usually translates to that something becoming the commitment
• this option would fall in the initial part of this comment (authnz, activity logging, deployment complexity, etc...)

I am not against alternatives, it is just a matter of agreeing on what we want, and this content comes from what has been discussed and shared so far.

[gagandeepb]

Thanks for this, Nelson! I left a few comments.

[gagandeepb]

naming: I believe that 'Trento Native AI Assistant' is more representative of the overall feature.

[nelsonkopliku]

I changed here 8e2b20d and highlighted the native nature in the "proposed path forward"

[gagandeepb]

suggestion: s/widget/widget supported by user-configurable server-side Agentic/AI capabilities.

[nelsonkopliku]

what does that refer to, more specifically?

[balanza]

suggestion: ‏I would make the difference clear between the context coming from the domain knowledge, and the context coming from the current state of the application. They might have different requirements and lead to different architectures

[nelsonkopliku]

I am partially understanding.

The main context is the state of the application (accessible via APIs and/or UI state), and that state is derived from domain knowledge.

So, in a sense, by having access to the state you already got access to the domain knowledge that led to that state.

What am I not considering? Or what would you suggest adding/rephrasing?

[balanza]

The main context is the state of the application (accessible via APIs and/or UI state),

I mean, there will be a moment in which the model queries the Trento API besides what is actually rendered on screen. Anyway, it's just phrasing, not a big deal. As long as the concept is covered, I'm fine 👍🏼

[balanza]

I see no evil here.

All the taken decisions are reasonable and, more importantly, they look easy to reverse in case we find better options along the way. Not much more to discuss here from my side, we can discuss more details on the PoC, I think.

I'm more concerned about the deferred decisions (RAG yes/no, UI interaction, MCP). I wonder if they can disrupt the overall architecture once we address them.

[nelsonkopliku]

Can you help me better understand the concerns?

RAG yes/no
it is in the plans, just not for the very first iteration.
There has been some exploration around this specific library in the elixir ecosystem https://github.com/georgeguimaraes/arcana.

The main open point around RAG is whether we demand the user to do the ingestion on their servers, or if we ship pre-crunched data that we might do in our building pipelines. Or a combination of both (SUSE provided vector database + the ability for the customer to add up their own content)

Another RFC will very likely follow for this.

UI interaction/MCP: can you elaborate the concerns?

[balanza]

I mean, many decisions are deferred, and it's not clear to me whether they have been framed already or whether they might have an impact on the decisions actually taken in this RFC.

Given that I wasn't part of the discussion that led to this RFC, there might be some understanding in the team that I'm failing to grasp. Looking forward to the next iterations to contribute, though,

[balanza]

My 2 cents here.

Precisely because the topic is relatively new in the industry, and best practices are not yet consolidated, I would follow one protocol for the sake of keeping the pace with the community. I would just assess how impactful/heavy it is.

(I have no particular sympathy for this one; it can be any other).

[nelsonkopliku]

Yes, I am not against following a standard, I actually aim to that, and I have no particular preference over any.

In the context of AG-UI it is worth considering the following

• There are means of translating existing protocols to AG-UI
• There is work in progress in the AG-UI ecosystem to support Elixir ag-ui/pull/1046 and ag-ui/pull/1293
• We can implement it ourselves also partially using the parts of the protocol that are relevant to us

I don't exclude we might actually implement it, however I am wondering about the following:

• if we implement it right away, when the elixir support comes we might need to rework in favor of the community provided library
• if we don't implement it right away, when the elixir support comes we might need to rework in favor of the community provided library
• implementing it right away, necessarily means adding overhead

Additionally

• it is true that if we go AG-UI protocol from the beginning (even though with our own implementation) when a community supported elixir library/sdk comes, we would ideally (because things never go as planned) need to adapt only the backend, because the UI side would already be AG-UI compliant
• with a totally custom protocol, we'd need to adapt backend and UI

That said, I'd find it reasonable keeping the amount of rework in the future limited, and thus implement the standard (hoping it remains so 🙈) protocol ourselves (even partially) right away.

However I don't have visibility, yet, on the impact/delay it might have on releasing the first version in next trento release.

We need to make the decision about our short and mid-term goals, the debt we want to take and where to re-scope.

Bottom line, I don't exclude any option. We need to look closer into that and make the call.

[balanza]

I think in the topic of AI agents, we cannot prevent/limit future rework as the scenarios are going to change unpredictably. So either way, we might need to revisit some parts in the near future. And what is claimed to be a standard today can be legacy in 6 months 🤷🏼

My 2 cents were more on the R&D investment: on the UI, we might study someone else's work rather than tailor our own solutions, so as to focus on things more related to our domain, like how to have the model understand the cluster topology.

Anyway, no big deal either way

[antgamdia]

It is available in this branch of the mcp-server repo:
https://github.com/trento-project/mcp-server/tree/TRNT-4140-liz-alt