try again

aprilk-ms · aprilk-ms · commit 75a7613db44b · 2025-02-03T09:55:02.000Z
diff --git a/.github/workflows/ai-eval.yml b/.github/workflows/ai-eval.yml
@@ -19,7 +19,7 @@ jobs:
       AZURE_TENANT_ID: ${{ vars.AZURE_TENANT_ID }}
       AZURE_SUBSCRIPTION_ID: ${{ vars.AZURE_SUBSCRIPTION_ID }}
       AZURE_CREDENTIALS: ${{ secrets.AZURE_CREDENTIALS }}
-      EVAL_INPUT_FILE_PATH: ${{ github.workspace }}/evaluation-input-offline.jsonl
+      EVAL_INPUT_FILE_PATH: ${{ github.workspace }}/evaluation-input.jsonl
       EVAL_CONFIG_FILE_PATH: ${{ github.workspace }}/evaluate-config.json
     steps:
       - uses: actions/checkout@v4
diff --git a/evaluation-input-offline.jsonl b/evaluation-input-offline.jsonl
diff --git a/evaluation-input.jsonl b/evaluation-input.jsonl
@@ -0,0 +1,3 @@
+{"id": "6d8b0033-52ba-4dcf-87b0-dd4eebfe01d7", "description": "{\"context\": {\"system-prompt\": \"dummy\"}}", "query": "Tell me about Tokyo?", "response": "Tokyo, the capital of Japan, is one of the most populous and vibrant cities in the world. It combines traditional culture with cutting-edge technology and modernity, making it a fascinating destination for both residents and visitors.\n\n### Geography\nTokyo is located on the eastern coast of Honshu, Japan's main island. The city is part of the Kanto region and is bordered by the Tokyo Bay to the south. It is known for its unique blend of urban and natural landscapes, including parks, rivers, and even mountains in its outskirts.\n\n### Population\nAs of my last knowledge update in October 2021, the Greater Tokyo Area is the most populous metropolitan area globally, with a population exceeding 37 million people. The city's population is characterized by a mix of local residents and a diverse expatriate community.\n\n### Culture and Society\nTokyo boasts a rich cultural heritage, featuring ancient temples, shrines, and traditional arts alongside modern architecture and bustling shopping districts. The city is home to:\n\n- **Historical Sites:** Senso-ji Temple in Asakusa, Meiji Shrine in Shibuya, and the Imperial Palace are popular spots representing Japan's history and spirituality.\n- **Museums:** There are numerous museums, including the Tokyo National Museum, the Tokyo Metropolitan Art Museum, and the teamLab Borderless digital art museum.\n- **Shopping and Entertainment:** Districts like Shibuya, Shinjuku, and Harajuku are famous for their fashion, youth culture, and nightlife. Akihabara is known as the center for otaku culture and electronics.\n\n### Transportation\nTokyo has an extensive and efficient public transportation system, including trains, subways, and buses. The Tokyo Metro and Japan Railways (JR) serve millions daily, making it easy to navigate the city and explore surrounding regions.\n\n### Economy\nTokyo is a major economic hub, hosting numerous multinational corporations, financial institutions, and technology companies. The Tokyo Stock Exchange is one of the largest stock exchanges in the world. Key industries include finance, technology, and tourism.\n\n### Cuisine\nTokyo\u2019s culinary scene is diverse, offering everything from Michelin-starred fine dining to casual street food. Popular dishes include sushi, ramen, tempura, and izakaya fare. Tsukiji Outer Market and various street vendors showcase the city's food culture.\n\n### Events and Festivals\nTokyo hosts various events and festivals throughout the year, including the famous cherry blossom season in spring and the vibrant Sumida River Fireworks Festival in summer. Other notable events include the Tokyo Marathon and various cultural celebrations.\n\n### Environmental Initiatives\nIn recent years, Tokyo has focused on sustainability and environmental initiatives, such as green spaces, public transportation improvements, and waste reduction programs. The city aims to promote eco-friendly practices and respond to climate change challenges.\n\n### Challenges\nLike many major cities, Tokyo faces challenges including overpopulation, high living costs, and natural disasters such as earthquakes and typhoons, which necessitate ongoing urban planning and disaster preparedness.\n\nOverall, Tokyo is a dynamic metropolis that offers a unique blend of the traditional and the modern, enriching the lives of its residents and providing unforgettable experiences for visitors.", "ground_truth": "Tokyo is the capital of Japan and the largest city in the country. It is located on the eastern coast of Honshu, the largest of Japan's four main islands. Tokyo is the political, economic, and cultural center of Japan and is one of the world's most populous cities. It is also one of the world's most important financial centers and is home to the Tokyo Stock Exchange.", "context": "dummy"}
+{"id": "3957da85-0242-45a6-92f8-c761cc800447", "description": "{\"context\": {\"system-prompt\": \"dummy\"}}", "query": "Where is Italy?", "response": "Italy is located in Southern Europe and looks a bit like a high-heeled boot kicking a ball, which is actually the island of Sicily! It's bordered by France to the northwest, Switzerland and Austria to the north, and Slovenia to the northeast, with the Mediterranean Sea to the south. It\u2019s famous for its rich history, delicious food, and fashion\u2014pretty much everything the boot kicks is fabulous! \n\nSpeaking of kicking, why did the Italian soccer player bring string to the game? Because he wanted to tie the score! ", "ground_truth": "Italy is a country in southern Europe, located on the Italian Peninsula and the two largest islands in the Mediterranean Sea, Sicily and Sardinia. It is a unitary parliamentary republic with its capital in Rome, the largest city in Italy. Other major cities include Milan, Naples, Turin, and Palermo.", "context": "dummy"}
+{"id": "b37e2c5d-a215-4767-ad3d-4e96c6de9ce6", "description": "{\"context\": {\"system-prompt\": \"dummy\"}}", "query": "Where is Papua New Guinea?", "response": "Papua New Guinea is located in the southwestern Pacific Ocean. It occupies the eastern half of the island of New Guinea, which it shares with Indonesia (the western half), and it includes numerous offshore islands. It's situated northeast of Australia and is part of the region known as Melanesia.\n\nAnd speaking of Papua New Guinea, did you hear about the octopus who won the lottery there? He said he was going to use the money to \"buy a shell-fie!\" \ud83d\udc19\ud83d\udcf8", "ground_truth": "Papua New Guinea is an island country that lies in the south-western Pacific. It includes the eastern half of New Guinea and many small offshore islands. Its neighbours include Indonesia to the west, Australia to the south and Solomon Islands to the south-east.", "context": "dummy"}
diff --git a/src/api/test_ai_eval.py b/src/api/test_ai_eval.py
@@ -31,9 +31,10 @@ def test_generate_ai_eval_input():
             for line in response.text.splitlines():
                 answer += json.loads(line).get("delta", {}).get("content", "") or ""
         
+            description = {"context": {"system-prompt": "dummy"}}
             input = {
                 "id": str(uuid.uuid4()),
-                "description": "dummy",
+                "description": json.dumps(description),
                 "query": data["query"],
                 "response": answer,
                 "ground_truth": data["ground-truth"],

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	+{"id": "6d8b0033-52ba-4dcf-87b0-dd4eebfe01d7", "description": "{\"context\": {\"system-prompt\": \"dummy\"}}", "query": "Tell me about Tokyo?", "response": "Tokyo, the capital of Japan, is one of the most populous and vibrant cities in the world. It combines traditional culture with cutting-edge technology and modernity, making it a fascinating destination for both residents and visitors.\n\n### Geography\nTokyo is located on the eastern coast of Honshu, Japan's main island. The city is part of the Kanto region and is bordered by the Tokyo Bay to the south. It is known for its unique blend of urban and natural landscapes, including parks, rivers, and even mountains in its outskirts.\n\n### Population\nAs of my last knowledge update in October 2021, the Greater Tokyo Area is the most populous metropolitan area globally, with a population exceeding 37 million people. The city's population is characterized by a mix of local residents and a diverse expatriate community.\n\n### Culture and Society\nTokyo boasts a rich cultural heritage, featuring ancient temples, shrines, and traditional arts alongside modern architecture and bustling shopping districts. The city is home to:\n\n- Historical Sites: Senso-ji Temple in Asakusa, Meiji Shrine in Shibuya, and the Imperial Palace are popular spots representing Japan's history and spirituality.\n- Museums: There are numerous museums, including the Tokyo National Museum, the Tokyo Metropolitan Art Museum, and the teamLab Borderless digital art museum.\n- Shopping and Entertainment: Districts like Shibuya, Shinjuku, and Harajuku are famous for their fashion, youth culture, and nightlife. Akihabara is known as the center for otaku culture and electronics.\n\n### Transportation\nTokyo has an extensive and efficient public transportation system, including trains, subways, and buses. The Tokyo Metro and Japan Railways (JR) serve millions daily, making it easy to navigate the city and explore surrounding regions.\n\n### Economy\nTokyo is a major economic hub, hosting numerous multinational corporations, financial institutions, and technology companies. The Tokyo Stock Exchange is one of the largest stock exchanges in the world. Key industries include finance, technology, and tourism.\n\n### Cuisine\nTokyo\u2019s culinary scene is diverse, offering everything from Michelin-starred fine dining to casual street food. Popular dishes include sushi, ramen, tempura, and izakaya fare. Tsukiji Outer Market and various street vendors showcase the city's food culture.\n\n### Events and Festivals\nTokyo hosts various events and festivals throughout the year, including the famous cherry blossom season in spring and the vibrant Sumida River Fireworks Festival in summer. Other notable events include the Tokyo Marathon and various cultural celebrations.\n\n### Environmental Initiatives\nIn recent years, Tokyo has focused on sustainability and environmental initiatives, such as green spaces, public transportation improvements, and waste reduction programs. The city aims to promote eco-friendly practices and respond to climate change challenges.\n\n### Challenges\nLike many major cities, Tokyo faces challenges including overpopulation, high living costs, and natural disasters such as earthquakes and typhoons, which necessitate ongoing urban planning and disaster preparedness.\n\nOverall, Tokyo is a dynamic metropolis that offers a unique blend of the traditional and the modern, enriching the lives of its residents and providing unforgettable experiences for visitors.", "ground_truth": "Tokyo is the capital of Japan and the largest city in the country. It is located on the eastern coast of Honshu, the largest of Japan's four main islands. Tokyo is the political, economic, and cultural center of Japan and is one of the world's most populous cities. It is also one of the world's most important financial centers and is home to the Tokyo Stock Exchange.", "context": "dummy"}
	`2`	+{"id": "3957da85-0242-45a6-92f8-c761cc800447", "description": "{\"context\": {\"system-prompt\": \"dummy\"}}", "query": "Where is Italy?", "response": "Italy is located in Southern Europe and looks a bit like a high-heeled boot kicking a ball, which is actually the island of Sicily! It's bordered by France to the northwest, Switzerland and Austria to the north, and Slovenia to the northeast, with the Mediterranean Sea to the south. It\u2019s famous for its rich history, delicious food, and fashion\u2014pretty much everything the boot kicks is fabulous! \n\nSpeaking of kicking, why did the Italian soccer player bring string to the game? Because he wanted to tie the score! ", "ground_truth": "Italy is a country in southern Europe, located on the Italian Peninsula and the two largest islands in the Mediterranean Sea, Sicily and Sardinia. It is a unitary parliamentary republic with its capital in Rome, the largest city in Italy. Other major cities include Milan, Naples, Turin, and Palermo.", "context": "dummy"}
	`3`	+{"id": "b37e2c5d-a215-4767-ad3d-4e96c6de9ce6", "description": "{\"context\": {\"system-prompt\": \"dummy\"}}", "query": "Where is Papua New Guinea?", "response": "Papua New Guinea is located in the southwestern Pacific Ocean. It occupies the eastern half of the island of New Guinea, which it shares with Indonesia (the western half), and it includes numerous offshore islands. It's situated northeast of Australia and is part of the region known as Melanesia.\n\nAnd speaking of Papua New Guinea, did you hear about the octopus who won the lottery there? He said he was going to use the money to \"buy a shell-fie!\" \ud83d\udc19\ud83d\udcf8", "ground_truth": "Papua New Guinea is an island country that lies in the south-western Pacific. It includes the eastern half of New Guinea and many small offshore islands. Its neighbours include Indonesia to the west, Australia to the south and Solomon Islands to the south-east.", "context": "dummy"}