Skip to content

Conversation

@mBerasategui-ehu
Copy link
Contributor

Added 2 scripts in scripts/langsmith:
-dbizi_dataset.py (creates a dataset in LangSmith with 15 questions about dbizi)
-evaluate_assistant.py (an llm-as-a-judge evaluates a given lamb assistant with the given dataset in LangSmith)

To try them, first configure these LangSmith env variables in backend/.env:
-LANGCHAIN_TRACING_V2=true
-LANGCHAIN_API_KEY=YOUR_LANGCHAIN_API_KEY
-LANGCHAIN_PROJECT=lamb-assistants
-LANGCHAIN_ENDPOINT=https://api.smith.langchain.com

And these 2 variables in evaluate_assistant.py:
-JWT_TOKEN = "your_jwt_token_here"
-ASSISTANT_ID = 1 (this should be the id of the dbizi assistant in lamb)

Then, run the scripts:
-first, dbizi_dataset.py
-then, evaluate_assistant.py

@mBerasategui-ehu
Copy link
Contributor Author

Fixed issues in evaluate_assistant.py and added dbizi_dataset_eus.py (same as dizi_dataset.py but in basque).

@mBerasategui-ehu
Copy link
Contributor Author

Now these variables are loaded from lamb-kb-server-stable/backend/.env instead of having them hardcoded:
API_BASE_URL = os.getenv("API_BASE_URL")
JWT_TOKEN = os.getenv("JWT_TOKEN")
ASSISTANT_ID = int(os.getenv("ASSISTANT_ID"))
DATASET_NAME = os.getenv("DATASET_NAME")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant