[DMP 2026]: Logs-to-training pipeline for agentic setups

[Gautam-Rajeev]

Ticket Contents

Description

Build a pipeline that ingests production logs (question answering and agentic setup sessions), removes personally identifiable information and other sensitive content, and produces training datasets suitable for improving a language model used in that setup. Agentic portions must be treated as trajectory-oriented behavior cloning (state, tool actions, observations, multi-step recovery), not only single-turn instruction pairs. Exports must support supervised fine-tuning with LoRA and Direct Preference Optimization, and the data design should support eventually training a smaller model to replace a larger teacher model while preserving behavior. The work includes defining schemas, tagging compositional trajectory complexity, diversity and scheduling hooks for training, and validation so trajectories are consistent with tool results where possible.

Goals & Mid-Point Milestone

Goals

• log event schema and parsers for Q&A and agentic traces (user, assistant, tool call, tool result, errors).
• PII detection, redaction or placeholder replacement, and audit sampling workflow with documented residual risk.
• Export paths for SFT (LoRA-ready) JSONL and DPO JSONL aligned to one chat template and inference format.
• Trajectory tagging for compositional complexity (steps, tools, ambiguity, error recovery) and stratified sampling setup for diversity
• Goals Achieved By Mid-point Milestone: working end-to-end prototype on a sampled log subset—PII-stripped JSONL for SFT and a small validated DPO pair set; documented schemas; gold-set alignment for one representative agent workflow; automated checks for schema, tool name validity, and basic trajectory consistency.

Setup/Installation

NA

Expected Outcome

A documented, repeatable pipeline runnable against designated log sources. It normalizes logs, segments Q&A versus agent trajectories, applies configurable PII rules and review hooks, and emits versioned datasets. SFT exports are valid multi-turn chat or completion records matching production templates including tool syntax. DPO exports provide shared prompts with chosen and rejected completions from governed sources (feedback, failure pairs, or approved synthetics). Each row or shard carries metadata for trajectory complexity and domain so trainers can apply staged mixtures or model-aware sampling without ad-hoc rewrites. Evaluation hooks or scripts exist to compare teacher and student checkpoints on held-out behavioral and tool-use checks. Student-training filters respect smaller context and tool sets.

Acceptance Criteria

• No training artifact ships without passing the configured PII pipeline and a documented audit sample.
• SFT JSONL validates against the chosen trainer dry run (LoRA) on toy and production-shaped samples without template mismatch.
• DPO JSONL validates against a small DPO dry run with required prompt, chosen, and rejected fields.
• Agent trajectories excluded or flagged when tool calls contradict observations or fail schema validation.
• Documentation lists field definitions, split strategy (no near-duplicate leakage across train and preference sets), and how complexity tags map to recommended training schedules.
• Clear criteria defined for when a smaller replacement model is acceptable relative to the teacher on the agreed eval set.

Implementation Details

Python-first CLI or service modules for ingest, transform, and export. Rule-based and model-assisted PII detection as appropriate; consistent synthetic placeholders. JSONL as primary interchange. Optional integration with Hugging Face datasets, PEFT, and TRL-style trainers for validation only unless scope expands. Tagging pipeline computes step counts, tool sets, recovery flags, and optional loss-based hardness when a reference model is available. No storage of raw secrets in config or outputs. Tests for redaction, schema validation, and split integrity.

Mockups/Wireframes

Not applicable.

Product Name

OpenAgriNet

Organisation Name

COSS

Domain

⁠Agriculture

Tech Skills Needed

Python

Mentor(s)

@Gautam-Rajeev

Category

Data Science

[vandit98]

Hi @Gautam-Rajeev, I’d love to work on this for DMP 2026.

I’ve been reading through the problem statement and this is the project I want to prioritize. My background is in Python/Go backend systems, ML/LLM pipelines, data workflows, and open-source work through DMP 2024, so the log-to-training-data + agent trajectory part feels like a strong fit.

I started with a small implementation-first prototype so I can come into the proposal with something concrete, not just a plan: #2

The first slice covers:

• canonical event schema for Q&A and agentic traces
• deterministic PII/secrets redaction with placeholders
• session segmentation into Q&A vs. tool-use trajectories
• trajectory complexity tags for staged training schedules
• LoRA-ready SFT JSONL export
• human-review DPO candidate JSONL export
• basic tool-call/tool-result validation
• sample logs, schema docs, implementation plan, and tests

A few assumptions I’d love to confirm:

1. What raw log formats should we optimize for first: JSONL traces, app logs, OpenTelemetry-style spans, or something else?
2. Which PII classes are highest priority beyond emails, phone numbers, IDs, secrets, and URLs?
3. Do you expect DPO pairs to come from explicit feedback, edited answers, failed/successful trajectories, or mentor-approved synthetic negatives?
4. Is there a preferred chat template or trainer stack for the first LoRA/DPO dry run?

If this direction matches your expectations, I’d be happy to iterate on the prototype and shape it into the DMP proposal.

[dikshant182004]

hi @Gautam-Rajeev ,i would like to be part of this project as this seems my forte .juat a quick question , is there any contribution criteria for the selection or a good proposal with proper understanding of the project ask is enough .

[ManyaJain111]

Hi, I am Manya Jain

I am interested in working in this project for DMP 2026. I had a few clarification questions that would help scope an implementation correctly:

• Will contributors be provided with any sample or anonymized production logs for experimentation, or is the expectation to prototype the pipeline using synthetic/public proxy datasets first?
• Among privacy preservation, downstream model quality, and pipeline scalability, which metric should be treated as the primary success criterion for early implementations?
• Do you already have a preferred canonical event schema for conversations/tool traces, or would contributors be expected to propose one from scratch as part of the solution?
• For the “smaller replacement model” goal, are there target constraints in mind (parameter count, latency, context length, hardware budget), or should contributors define those assumptions themselves?

looking forward to synergize .....

[abd-RAHEEM]

Hi, I'm interested in this project, I have research work in LoRA and will submit my proposal in DMP application.

[Gautam-Rajeev]

Hi @Gautam-Rajeev, I’d love to work on this for DMP 2026.

I’ve been reading through the problem statement and this is the project I want to prioritize. My background is in Python/Go backend systems, ML/LLM pipelines, data workflows, and open-source work through DMP 2024, so the log-to-training-data + agent trajectory part feels like a strong fit.

I started with a small implementation-first prototype so I can come into the proposal with something concrete, not just a plan: #2

The first slice covers:

• canonical event schema for Q&A and agentic traces
• deterministic PII/secrets redaction with placeholders
• session segmentation into Q&A vs. tool-use trajectories
• trajectory complexity tags for staged training schedules
• LoRA-ready SFT JSONL export
• human-review DPO candidate JSONL export
• basic tool-call/tool-result validation
• sample logs, schema docs, implementation plan, and tests

A few assumptions I’d love to confirm:

1. What raw log formats should we optimize for first: JSONL traces, app logs, OpenTelemetry-style spans, or something else?
2. Which PII classes are highest priority beyond emails, phone numbers, IDs, secrets, and URLs?
3. Do you expect DPO pairs to come from explicit feedback, edited answers, failed/successful trajectories, or mentor-approved synthetic negatives?
4. Is there a preferred chat template or trainer stack for the first LoRA/DPO dry run?

If this direction matches your expectations, I’d be happy to iterate on the prototype and shape it into the DMP proposal.

Thanks for your response :) In response to your questions:

• We will have json traces of logs, we are using Pydantic + Langfuse, so we should have json with messages format of question, response, tool calls, tool response.
• The users will mostly mention phone number and some govt id, I am okay with using some already available model/method for this- ideally proposal should not expend too much effort on this.
• There is very little explicit feedback, unless the user themselves mentions something in the next question/response. The bot will have some 'persona' which you can consider as a prompt explaining the bot behaviour- any response to the user that doesn't follow this persona is not good. Any sub-optimal pathway of tool calls is not good, the faster we arrive at all the data required to answer a question the better- usually our trajectories do not need more than 3-4 tool calls and not too heavy reasoning. Also, DPO is just a method; I am happy to let you take any approach you want. We have GPU constraints - we can run on 8 *H100 cards and we want to tune <32B models. I think data creation will be more critical than training approach, focus on LoRA first and DPO/RL later.
• I am okay with anything, it should support latest models like Gemma4 effectively- it should allow for multi-card and possibly multi node training. I think TRL is a decent starting point.

Generally, I want proposal to focus on ideas on how to extract high quality data , using the logs data as samples/seed. We need not directly use the logs. There should be effort on how to get diverse data, varying difficulty etc.
We should ideally have setup by the time you start some environment where we mock all the tools, so you can look at generating more data using these. However, extensive prompting of closed source models will be too expensive for us. Anything that is easy to host on one H100/H200*8 node works

Hi, I am Manya Jain

I am interested in working in this project for DMP 2026. I had a few clarification questions that would help scope an implementation correctly:

• Will contributors be provided with any sample or anonymized production logs for experimentation, or is the expectation to prototype the pipeline using synthetic/public proxy datasets first?
• Among privacy preservation, downstream model quality, and pipeline scalability, which metric should be treated as the primary success criterion for early implementations?
• Do you already have a preferred canonical event schema for conversations/tool traces, or would contributors be expected to propose one from scratch as part of the solution?
• For the “smaller replacement model” goal, are there target constraints in mind (parameter count, latency, context length, hardware budget), or should contributors define those assumptions themselves?

looking forward to synergize .....

Thanks for response!

• I don't think we'll be able to provide sample responses before the proposal deadlines. You can use any agentic tool calls logs data
• Downstream model quality. I don't think we'll need a dataset beyond 50k data points.. so scalability uptil here is good enough.
  Schema will be the usual pydantic/ langfuse. It will user questions, responses and tool calls and their reponses. You can transform data to whatever schema you prefer. Sample data could look like this

[
    {
        "user_question": "",
        "bot_response": "",
        "agent_turns": [
            {
                "parts": [
                    {
                        "tool_name": "fetch_agristack_data",
                        "args": {},
                        "tool_call_id": "",
                        "id": null,
                        "provider_details": null,
                        "part_kind": "tool-call"
                    }
                ],
                "usage": {
                    "input_tokens": 11345,
                    "cache_write_tokens": 0,
                    "cache_read_tokens": 0,
                    "output_tokens": 40,
                    "input_audio_tokens": 0,
                    "cache_audio_read_tokens": 0,
                    "output_audio_tokens": 0,
                    "details": {
                        "cache_creation_input_tokens": 0,
                        "cache_read_input_tokens": 0,
                        "input_tokens": 11345,
                        "output_tokens": 40
                    }
                },
                "model_name": "",
                "timestamp": "2026-01-20T05:54:15.673313Z",
                "kind": "response",
                "provider_name": "",
                "provider_url": "",
                "provider_details": {
                    "finish_reason": "tool_use"
                },
                "provider_response_id": "",
                "finish_reason": "tool_call",
                "run_id": "",
                "metadata": null
            },
            {
                "parts": [
                    {
                        "tool_name": "fetch_agristack_data",
                        "content": "> Farmer Information (Agristack)\nResponses:\n      Provider: Agristack - Farmer Details Providers\n      Farmer Details:\n        Name:n        Mobile: 87***2\n        Date of Birth: 19***1\n        Gender: Male\n        Caste Category: General\n        Village Name: \n        Is PoCRA village?: No\n        Taluka Name: \n        District Name: \n        Sub-District LGD Code: \n        District LGD Code: \n        Village LGD Code: \n        Total Plot Area: hectares\n        Identifier Name:\n      Locations:\n        Loha, Maharashtra, India (19.066, 77.174)",
                        "tool_call_id": "",
                        "metadata": null,
                        "timestamp": "2026-01-20T05:54:16.270192Z",
                        "part_kind": "tool-return"
                    }
                ],}
            {
                "parts": [
                    {
                        "tool_name": "weather_forecast",
                        "args": {
                            "latitude": 19.066,
                            "longitude": 77.174,
                            "days": 5
                        },
                        "tool_call_id": "toolu_018HGFp98z1XxwTFN8VwjnX3",
                        "id": null,
                        "provider_details": null,
                        "part_kind": "tool-call"
                    }
                ],

• We want smallest possible model that does the task with high context length. Max model size is ~ 32B (dense). We want to cover 4-5 tool calls and final response in around 4 seconds.

[Gautam-Rajeev]

hi @Gautam-Rajeev ,i would like to be part of this project as this seems my forte .juat a quick question , is there any contribution criteria for the selection or a good proposal with proper understanding of the project ask is enough .

There is no particular contribution criteria as such.
Detailed plan of what you would do to contribute is enough; (useful) contributions are of course welcome, but not necessary

[Eshaan-byte]

Hi @Gautam-Rajeev, I'm Eshaan, I'd like to work on this for DMP 2026.

I've read through the thread and your answers to the earlier questions, so I won't repeat those. A few things I'm still curious about:

Languages in the logs- are conversations mostly in English, or is there Hindi/regional language mixed in? Matters for PII detection and picking the right tokenizer down the line.

How many tools does the agent have? I see fetch_agristack_data and weather_forecast in the sample. Roughly, how many others are there? Helps me scope the mock environment and validation.

Is the persona prompt something we'd have access to in the pipeline? Since you mentioned responses that don't follow the persona are bad, it'd be useful to feed it into an automated scorer (like LLM-as-judge) for generating preference pairs instead of doing it all manually.

Roughly how many agentic sessions exist in the logs right now? Even a ballpark (hundreds vs thousands vs tens of thousands) would help figure out how much synthesis/augmentation we'd need to get to ~50k.

Working on a proposal and prototype, happy to iterate based on your feedback.

[hadakoi]

Hi @Gautam-Rajeev, interested in working on this for DMP 2026.

The core problem to my understanding is building a pipeline that ingests production logs from an agentic Q&A system, strips PII, and produces training datasets for fine-tuning a smaller model to replace a larger teacher while preserving behavior. The agentic portions need to be treated as full trajectories (state, tool calls, observations, recovery) rather than flat instruction pairs, with exports supporting both SFT (LoRA) and DPO.

After reading through your responses, my understanding of the direction:

• Log source is Pydantic/Langfuse JSON with a user_question → agent_turns (tool-call/tool-return) → bot_response structure
• PII handling should use Microsoft Presidio with a placeholder system and audit log, not a custom pipeline
• Logs are structural templates and constraint priors for data generation, not direct training data rather the focus is diversity and quality via controlled synthetic expansion, especially for edge cases like tool failure and ambiguous queries
• LoRA first, DPO/RL later; target ≤32B dense model, ~4s for 4–5 tool calls on H100x8

Data Generation & Training Pipeline

Rather than treating logs as direct training data, I'd use them as seeds for controlled expansion through a grounded synthetic generation process. For generation, I'd use an open model with strong tool-use (e.g., Llama 3.1 70B for generation, Qwen or Phi-3 for smaller variants). The key is grounding it through a mock tool executor to avoid hallucination:

1. Generator proposes tool call — LLM generates {tool_name, args, tool_call_id} for a given query and prior context
2. Mock executor validates & returns — Schemas are pre-specified (extracted from Langfuse traces); executor validates args against schema, then simulates realistic returns (success with mock data, or controlled failures: "tool timeout", "invalid input", "no results")
3. LLM continues trajectory — Sees the mock result, decides next tool call or final response
4. Repeat until conclusion — Trajectories naturally end when LLM decides to respond or hits max steps

This keeps synthetic data consistent with real tool behavior and avoids unrealistic failure modes that could harm downstream model quality. For scaling without expensive closed APIs: run the open model locally on a single H100 (or one node during proposal phase), batch generation by query type, and re-use mock results across variations.

The resulting trajectories are tagged by complexity (step count, tool set, recovery flags) to enable stratified sampling. For SFT, we export LoRA-ready JSONL: multi-turn conversations with tool calls formatted in the production template. For DPO, preference pairs are scored within persona clusters (fixed base prompt + variable domain/toolset overlays), rather than across all personas. The scoring is rule + model hybrid:

• Rule layer: tool call validity (schema match, arg correctness, logical sequence)
• Model layer: coherence and instruction adherence (does response match persona?)
• Combined score: score(response | trajectory, persona_context) not just score(response | prompt)

Preference pairs come from failure/success trajectory pairs (suboptimal vs. optimal tool sequences for the same query), synthetic variants generated via the mock environment, and human-flagged persona violations. The training stack is OpenRLHF with built-in PPO generation pipelines and PEFT (LoRA), supporting multi-GPU and multi-node setups.

Couple Doubts I would like to have clarified

1. Mock tool schemas: Pre-specified from Langfuse schema definitions, or should we infer from log samples?
2. Persona overlays: Should domain/toolset overlays be extracted directly from metadata, or inferred/tagged during log processing?
3. DPO pair budget: Rough target for preference pairs (~5k, ~10k)? Or does quality matter more than scale given 50k limit on total data?

Happy to iterate on this before the proposal deadline.

[Adarsh765-jpg]

Hi @Gautam-Rajeev,

I’ve gone through the issue and your responses, and I’m shaping my proposal around a log-driven data pipeline. I had two quick clarifications:

For signals like persona adherence and tool usage efficiency, should these be used as hard filters (dropping low-quality trajectories), or stored as metadata/labels for downstream sampling or training?
Will the persona be available in a structured form (e.g., system prompt or defined rules) so it can be used for automated validation, or should it be inferred from the logs?

[sharanyaa23]

Hi @Gautam-Rajeev, I’ve submitted a small prototype PR for the log-to-training pipeline based on my current understanding.

I wanted to clarify a couple of things as I refine the approach further:

• Roughly how many tools/APIs does the agent interact with in production? This would help in shaping validation and inefficiency detection.
• For things like response quality and inefficient behaviour, should I remove such cases completely or keep them with labels for training?

Also, I wasn’t able to access the Discord link from the website, is there any other active channel for discussion?

Happy to contribute further based on your feedback.

[theUtkarshRaj]

Hi @Gautam-Rajeev — I’ve submitted PR #4 with a working end-to-end prototype based on the clarified Langfuse/Pydantic + JSON trace direction.

In addition to core ingest → redact → SFT/DPO export, I focused specifically on downstream data-governance and training-readiness layers:

• Split integrity safeguards (disjoint SFT / DPO / eval)
• Validation CLI for schema + trajectory checks
• Hard-negative DPO templates (persona/tool inefficiency variants)
• Gold example fixtures for representative agent workflows
• CI + tests + packaging

My goal was to complement baseline pipeline prototypes by emphasizing production-readiness, quality controls, and reviewer usability. Happy to modularize or refine scope based on preferred project direction.