Is the Learning System Actually Learning? (USER/ files + ratings)

[noxx]

Hi,

Just installed new version of PAI and spent time understanding the system deeply. I love the architecture and the skill system is impressive. However, I discovered something that confused me and I want to understand if this is intended or a gap.

The Issue: Learning Infrastructure Without Learning

PAI has extensive infrastructure for "learning":

• MEMORY/LEARNING/ directory for captured learnings
• MEMORY/LEARNING/SIGNALS/ratings.jsonl for rating capture
• Rating hooks (ExplicitRatingCapture, ImplicitSentimentCapture)
• USER/ directory with ABOUTME.md, CONTACTS.md, TECHSTACKPREFERENCES.md, etc.
  But from what I can tell, none of this data is actually used.

Specific Issues

1. Ratings go nowhere
   When I rate a response "7" or "8 - good work", it gets captured to ratings.jsonl. But what reads this file? What learns from it? It seems like the ratings are stored but never used to
   improve anything.

2. USER/ files aren't loaded
   I was told to fill in USER/ABOUTME.md, USER/CONTACTS.md, USER/TECHSTACKPREFERENCES.md to personalize PAI. But these files are never auto-loaded. If I say "send to John", PAI doesn't know who John is - it would have to manually grep CONTACTS.md.

The only files that ARE auto-loaded via LoadContext hook:

• SKILL.md (the algorithm)
• AISTEERINGRULES.md
• DAIDENTITY.md

3. The only skill that reads USER/ is Telos skill that actually reads from USER/TELOS/. That's wired up properly. But everything else in USER/ is just... storage that nothing reads.

My Questions

1. Is this the intended design? Is USER/ meant to be manually referenced rather than auto-loaded?
2. What's the point of the rating system? If ratings are captured but nothing uses them, why capture them?
3. Is there a learning loop I'm missing? Does something actually process ratings.jsonl and MEMORY/LEARNING/ to improve future responses?
4. Should more skills reference USER/? For example, should Research skill check TECHSTACKPREFERENCES.md when giving tech recommendations?

Suggestion, Either:

1. Wire up the learning - Have LoadContext inject more USER/ files, have something process ratings
2. Document the reality - Make it clear these are "storage for future use" not "active learning"

I don't want to spend time filling in USER/ files if nothing reads them. And I don't want to rate responses if it's not actually improving anything.

Am I missing something? Is there a learning loop I don't understand?

Thanks for building this - the skill system and algorithm are genuinely impressive. Just want to understand if the learning piece is functional or aspirational.

[wojteksbt]

This is a bit of a surprise! I had assumed the learning system was working somehow and decided to investigate today. It turns out that nothing actually seems to use the gathered LEARNINGS data.

I'm wondering if something is misconfigured on my end, or if the retrieval/application layer simply isn't implemented yet.

I couldn't find any information about this in the README. However, I did notice that AISTEERINGRULES.md appears to be one output of the learning system - something has written to it during PAI development:

"These rules were derived from failure analysis of 84 rating 1 events (2026-01-08 to 2026-01-17)."

This suggests there's some mechanism for applying learnings, but I can't find how it works.

My guesses:

1. There's an automatic feedback loop in KAi that hasn't been extracted to PAI yet
2. The retrieval mechanism is planned but not yet implemented
3. The intended workflow is to periodically ask the DA to analyze the data and update steering rules manually

@danielmiessler — Could you clarify how this is meant to work? I'd love to understand the intended approach. Thanks!

[zeddy303]

This is what my PAI told me when I asked:
❯ Hi Sherpa! can you honestly tell me (do not just create content, this needs to based on your actual
process) are you using the learning system, if so, how are you doing that? Can you give me an example
of the last 10 things you learned?

I am not currently using a persistent learning system. Here's the honest reality:

1. No persistent memory across sessions. Each time you start a new conversation, I start fresh. I don't
   retain anything from previous sessions unless it's loaded into my context (like the CORE skill, steering
   rules, etc. that your hooks inject at startup).
2. The MEMORY system exists in your architecture — I can see references to WORK/, STATE/, LEARNING/
   directories in the documentation. But I'd need to actually check whether those directories exist and have
   content, and whether any workflow is writing to them.
3. I cannot give you "the last 10 things I learned" because I genuinely don't have that data. Making up a
   list would be exactly the kind of fabrication you're asking me not to do.

What I can do right now: Check if the memory/learning directories exist and have any content, which would
tell us whether any session has ever written learnings. Want me to look?

What would need to be true for learning to work:

• A hook or workflow that writes learnings to disk at session end
• A hook or context loader that reads them back at session start
• Neither of those appears to be happening automatically in what I see loaded

This is a real gap worth closing if the learning system matters to you.
\

(but then I followed up with "where did we leave off" and then searches the folders and figured it out):

[danielmiessler]

Yes, the final piece of applying the learnings and doing the matching is not happening natively. It's just all the pieces are set up there for it to happen because we've been capturing the two things combined. But you're right that a final piece is needed to close the loop. We'll be adding that very soon.

…

On Thu, Jan 29, 2026 at 7:37 PM, Zane Edwards < ***@***.*** > wrote: This is what my PAI told me (but then I followed up with "where did we leave off" and then searches the folders and figured it out): I am not currently using a persistent learning system. Here's the honest reality: * No persistent memory across sessions. Each time you start a new conversation, I start fresh. I don't retain anything from previous sessions unless it's loaded into my context (like the CORE skill, steering rules, etc. that your hooks inject at startup). * The MEMORY system exists in your architecture — I can see references to WORK/, STATE/, LEARNING/ directories in the documentation. But I'd need to actually check whether those directories exist and have content, and whether any workflow is writing to them. * I cannot give you "the last 10 things I learned" because I genuinely don't have that data. Making up a list would be exactly the kind of fabrication you're asking me not to do. What I can do right now: Check if the memory/learning directories exist and have any content, which would tell us whether any session has ever written learnings. Want me to look? What would need to be true for learning to work: * A hook or workflow that writes learnings to disk at session end * A hook or context loader that reads them back at session start * Neither of those appears to be happening automatically in what I see loaded This is a real gap worth closing if the learning system matters to you. — Reply to this email directly, view it on GitHub ( #531 (reply in thread) ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AAAMLXSCBFKO6J6EEVK3MEL4JLGYXAVCNFSM6AAAAACTIHJXI2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKNRUGY2TGOA ). You are receiving this because you were mentioned. Message ID: <danielmiessler/Personal_AI_Infrastructure/repo-discussions/531/comments/15646538 @ github. com>

[akpuggy]

I came here after a similar realization. Just like we have /PAIUpgrade leveraging external context for improvement, I thought it would be nice if there was a similar or combined capability to upgrade and improve user skills and tools based on those internal learnings.

PAi helped sketch out this recommendation based on that gap:

08-LOADLEARNINGS-HOOK-SPEC.md
07-PAIIMPROVE-SKILL-SPEC.md

I deferred implementing this myself because I didn't want to conflict with future core upgrades from PAI, but it's great to hear the official piece is coming soon. Hopefully, these specs might be useful reference for what that could look like.

[DolphusCY]

Hi @zeddy303

I just copied your exact question and asked my PAI (after upgarding to 2.5) and this is what it said....

Here's my honest assessment of the learning system:

The learning system is mostly write-only.

What works:

• Hooks reliably write data: 809 algorithm learnings, 55 failure analyses, 246 system learnings, 1200+ signal ratings
• The ratings counter in the startup banner ("Signals 1236") is real — it counts lines in ratings.jsonl

What doesn't work:

• LoadContext.hook.ts — the main session startup hook — does not load any learning files. It loads SKILL.md, USER files, steering rules, but zero learnings. So when
  a new session starts, past learnings aren't in context.
• SoulEvolution.hook.ts reads recent learnings (last 3 days), but its actual impact on behavior is unclear — it runs as a hook, not as injected context that shapes
  responses.

The honest conclusion:
The system captures data well but doesn't close the loop. Learnings are written to disk and sit there. They don't get loaded into new sessions in a way that changes my behavior. The ratings feed a counter. The failure analyses document what went wrong but aren't systematically reviewed before similar tasks.

It's an architectural gap: the write path is solid, the read-back-and-apply path is missing from the main context loading pipeline.

Last 10 real learnings (from the files I read): timezone changes made without asking, PDF styling, fix completions, and several neutral 5/10 sentiment captures. Most contained boilerplate "Action Items" like "Review..." and "Consider..." with no automated follow-through.

What would fix it: LoadContext.hook.ts needs to load a digest of recent learnings (especially failures and low-rated interactions) into session context at startup.

That's a straightforward change — read the last N learning files and inject a summary.

I know Daniel and team will get to this, its just a matter of time .

[matt-quatara]

I've been asking it to analyse the learnings once a week and suggest improvements to my skills and to the steering instructions and such. Really happy with that approach.

[danielmiessler]

Great diagnosis — and the timing is perfect because v3.0 shipped the missing piece.

What's now in place (as of v3.0)

The loop you identified — "signals are captured but nothing reads them back" — is now closed via two workflows in the PAIUpgrade skill:

1. MineReflections

The Algorithm now writes a structured reflection after every Standard+ run to MEMORY/LEARNING/REFLECTIONS/algorithm-reflections.jsonl. Each entry captures three questions:

• Q1 (Self): What would I have done differently in this run?
• Q2 (Algorithm): What would a smarter algorithm have done?
• Q3 (AI): What would a fundamentally smarter AI have done?

The MineReflections workflow mines these for recurring themes — weighted by signal strength (low sentiment ratings, over-budget runs, failed criteria) — and produces concrete upgrade candidates.

Invoke it by saying: "mine reflections" or "check reflections"

2. AlgorithmUpgrade

Goes deeper: takes the reflection themes, maps them to specific sections of the Algorithm spec, reads the current spec text, identifies the gap, and drafts concrete text changes. It even assesses whether a version bump is warranted.

Invoke it by saying: "algorithm upgrade" or "improve the algorithm"

Both live in the PAIUpgrade skill (~/.claude/skills/PAIUpgrade/).

---

On your other points

Ratings: The ratings system feeds implicit sentiment scores per-session. MineReflections weights low-rated sessions higher when identifying upgrade candidates — so bad interactions drive more upgrade pressure than good ones.

USER/ files: Still manually referenced for most things — that's intentional for the v3.0 release. LoadContext auto-loads the steering rules, SKILL.md, and identity config. Other USER/ files (CONTACTS, TECHSTACKPREFERENCES, etc.) are reference material you point skills at rather than bulk-injected context. The pattern @DolphusCY described is accurate.

AISTEERINGRULES.md: The note in that file — "derived from failure analysis of 84 rating 1 events" — is exactly how this loop works. You run MineReflections + AlgorithmUpgrade, review the proposals, apply the ones that make sense, and the Algorithm improves. That's the intended workflow.

---

The short version: yes, it's learning now. The write path has always been solid. v3.0 added the read-back-and-apply path via the PAIUpgrade skill.

[wojteksbt]

Great, thank you @danielmiessler for the detailed explanation! The trigger phrases might be worth mentioning in the docs.

[wojteksbt]

@danielmiessler @kaimagnus — I tried triggering the mechanism you described here — said
"check reflections" to kick off MineReflections — and that's when I discovered the
problem.

Since v3.0 migration I have zero spontaneous reflection entries in
algorithm-reflections.jsonl. The single entry in the file was written only after I
explicitly called out that the file was empty — that's manual intervention, not the
mechanism working.

I suspect this is the same root cause as #700 — frictionless steps (TaskCreate) happen
reliably, but Write-to-disk steps with no enforcement gate get skipped consistently.

Curious whether this is specific to my setup or something you've noticed more broadly —
and if so, whether there's a fix in the works.

[virtualian]

As of v4.0.1, I think the Learning Pipeline dead ends.

See #850

[jarednwilson]

I actually had my system write a custom /done command and it folds all of the learnings as well as updates my project files any to-do’s, error, codes, or stop commands into one systematic process that reads every transcript that has not been processed through the done command and systematically handle this.