Sa692 soc prompts st2 st3

[peter-spencer-ons]

✨ Summary

Adds stage 2 and 3 in the two prompt SOC pipeline.
Changes in this PR are required for (ONSdigital/survey-assist-eval#9) to be tested.

Mirrors the same steps as in SIC pipeline (sic-classification-utils).

📜 Changes Introduced

Introduced changes to

• added methods using LLM (evaluation of the codability, generating open follow up question).
• added validation models for responses.
• added prompts (prompt for codability evaluation, and open follow up question )
• added tests to meet the required 80% coverage.

✅ Checklist

Please confirm you've completed these checks before requesting a review.

• Code is formatted using ruff format
• Imports are sorted using ruff
• Code passes linting with ruff and mypy
• Security checks pass using Bandit
• API and Unit tests are written and pass using pytest
• Terraform files (if applicable) follow best practices and have been validated (terraform fmt & terraform validate)
• DocStrings follow Google-style and are added as per ruff recommendations
• Documentation has been updated if needed

🔍 How to Test

Test in (ONSdigital/survey-assist-eval#9)

[ivyONS]

does it really have to have at least 1 alt candidate?

[peter-spencer-ons]

I think it is worth keeping at least one candidate in case we want to verify LLM's selection.

Also we have been using minimum number of candidates 1 for alt_candidates to ensure there is at least one.

[ivyONS]

but if it is clearly codable and it assigned one code, there are unlikely any suitable alternatives, in such a case the list of alternative candidates should be empty

[ivyONS]

where is the level of edu??

[ivyONS]

you have added the level of edu to the sa_rag_soc_code method but not to its prompt