Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,16 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

## [Unreleased]

### Added

- `POST /v1/engrams/search` — semantic search (spreading activation) with a JSON body. Accepts `query`, `limit`, `detail`, `level`.
- `POST /v1/episodes/search` — text search over episode content with a JSON body. Same fields as engram search.
- `POST /v1/entities/search` — text search over entity names and aliases with a JSON body.

### Changed

- **Automatic background decay.** Engram now runs activation decay on a configurable interval (default: every hour) without requiring clients to call `POST /v1/activation/decay` on a schedule. Configure via the new `decay` config block (`interval`, `lambda`, `floor`). The endpoint remains for manual/one-off use.
- `GET /v1/engrams`, `GET /v1/episodes`, `GET /v1/entities` no longer accept a `?query=` parameter. Use the new `POST /{resource}/search` endpoints instead. GET list endpoints are now list-only with filter/cursor params.

---

Expand All @@ -37,7 +44,7 @@ Initial public release. Engram is a standalone episodic memory service for AI ag
- `POST /v1/activation/decay` — apply exponential activation decay (forgetting)
- `DELETE /v1/memory/reset` — destructive wipe of all memory
- `GET /health` — public health check endpoint
- `?query=` semantic search on list endpoints (engrams, episodes, entities)
- `?query=` semantic/text search on list endpoints (engrams, episodes, entities) — moved to dedicated `POST /{resource}/search` endpoints in Unreleased
- `?detail=full` verbosity flag — minimal responses by default (id + primary field only)
- `?level=N` pyramid summary compression on engrams and entities (4 / 8 / 16 / 32 / 64 words)

Expand Down
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,8 +108,10 @@ curl -X POST http://localhost:8080/v1/episodes \
-d '{"content": "Alice mentioned she prefers morning standups.", "source": "slack", "author": "alice"}'

# Query memory (spreading activation retrieval)
curl "http://localhost:8080/v1/engrams?query=Alice+meeting+preferences" \
-H "Authorization: Bearer your-secret-key"
curl -X POST http://localhost:8080/v1/engrams/search \
-H "Authorization: Bearer your-secret-key" \
-H "Content-Type: application/json" \
-d '{"query": "Alice meeting preferences", "limit": 10}'

# Trigger consolidation manually
curl -X POST http://localhost:8080/v1/consolidate \
Expand Down
63 changes: 52 additions & 11 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,19 +12,22 @@ A full OpenAPI 3.0 specification is at [`openapi.yaml`](../openapi.yaml).
|--------|------|-------------|
| `GET` | `/health` | Service health check (public) |
| `POST` | `/v1/episodes` | Ingest a raw episode |
| `GET` | `/v1/episodes` | List episodes; `?query=`, `?channel=`, `?unconsolidated=`, `?before={id}`, `?level=N` |
| `GET` | `/v1/episodes` | List episodes; `?channel=`, `?unconsolidated=`, `?before={id}`, `?level=N` |
| `POST` | `/v1/episodes/search` | Text search over episode content |
| `GET` | `/v1/episodes/count` | Episode count; `?channel=`, `?unconsolidated=` filters |
| `GET` | `/v1/episodes/{id}` | Get episode by ID or 5-char prefix |
| `POST` | `/v1/episodes/summaries` | Batch fetch pyramid summaries for episode IDs |
| `POST` | `/v1/episodes/{id}/edges` | Add a typed edge between two episodes |
| `POST` | `/v1/thoughts` | Ingest a free-form thought (shorthand for episodes) |
| `GET` | `/v1/engrams` | List engrams; `?query=` triggers spreading activation |
| `GET` | `/v1/engrams` | List engrams; `?threshold=` filter |
| `POST` | `/v1/engrams/search` | Semantic search via spreading activation |
| `GET` | `/v1/engrams/{id}` | Get engram by ID; `?level=N` for pyramid compression |
| `DELETE` | `/v1/engrams/{id}` | Delete an engram |
| `GET` | `/v1/engrams/{id}/context` | Engram + source episodes + linked entities |
| `POST` | `/v1/engrams/{id}/reinforce` | Boost activation and optionally blend embedding |
| `POST` | `/v1/engrams/boost` | Batch activation boost |
| `GET` | `/v1/entities` | List entities; `?query=`, `?type=` filters |
| `GET` | `/v1/entities` | List entities; `?type=` filter |
| `POST` | `/v1/entities/search` | Text search over entity names and aliases |
| `GET` | `/v1/entities/{id}` | Get entity by canonical ID |
| `GET` | `/v1/entities/{id}/engrams` | All engrams linked to an entity |
| `POST` | `/v1/consolidate` | Trigger consolidation pipeline manually |
Expand Down Expand Up @@ -125,15 +128,28 @@ Edge types: `REPLIES_TO`, `FOLLOWS`, `RELATED_TO`.
List episodes. Returns `[{id, content}]` by default.

Query params:
- `query` — substring search over episode content (does not combine with other filters)
- `channel` — filter by channel value
- `unconsolidated=true` — only return episodes not yet part of any engram
- `before={id}` — return only episodes older than the given episode ID (full or 5-char prefix); used for cursor-based pagination. Returns `400` if the ID is not found.
- `level=N` — apply pyramid compression to the `content` field before returning. Same levels as engrams: `4`, `8`, `16`, `32`, `64`. Episodes without a pre-generated summary at the requested level return raw content.
- `detail=full` — return all fields (applies after `?level=N` compression)
- `limit` — max results (default 50)

All filters except `query` compose: `?channel=X&unconsolidated=true&before={id}&level=8` is a valid combination.
All filters compose freely: `?channel=X&unconsolidated=true&before={id}&level=8` is a valid combination.

#### `POST /v1/episodes/search`

Text search over episode content. Returns `[{id, content}]` by default.

Request:
```json
{"query": "morning standup", "limit": 10, "detail": "full", "level": 8}
```

- `query` — required; substring search over episode content
- `limit` — max results (default 10)
- `detail` — set to `"full"` for all fields
- `level` — pyramid compression level applied to returned content

#### `GET /v1/episodes/{id}`

Expand Down Expand Up @@ -195,7 +211,7 @@ GET /v1/episodes?channel=guild:general&limit=70&before={ep30_id}&unconsolidated=
GET /v1/episodes/count?channel=guild:general&unconsolidated=true
```

The third query returns an empty list once all older episodes have been consolidated — they are then reachable via `GET /v1/engrams?query=...` (spreading activation search). Setting `consolidation.max_buffer` to match the bot's fetch limit (e.g. `100`) ensures episodes are always accessible via one path or the other.
The third query returns an empty list once all older episodes have been consolidated — they are then reachable via `POST /v1/engrams/search` (spreading activation search). Setting `consolidation.max_buffer` to match the bot's fetch limit (e.g. `100`) ensures episodes are always accessible via one path or the other.

---

Expand All @@ -206,11 +222,23 @@ The third query returns an empty list once all older episodes have been consolid
List consolidated engrams. Returns `[{id, summary}]` by default.

Query params:
- `query` — semantic search via spreading activation; returns ranked results
- `detail=full` — return all fields
- `level` — pyramid compression level applied to every returned engram (default `0` = verbatim)
- `limit` — max results when using `?query=` (default 10)
- `threshold` — filter by minimum activation level (list-all mode only)
- `threshold` — filter by minimum activation level

#### `POST /v1/engrams/search`

Semantic search via spreading activation. Returns ranked engrams matching the query.

Request:
```json
{"query": "Alice meeting preferences", "limit": 10, "detail": "full", "level": 0}
```

- `query` — required; natural language search. Seeds spreading activation via semantic KNN, lexical BM25, and entity matching.
- `limit` — max results (default 10)
- `detail` — set to `"full"` for all fields
- `level` — pyramid compression level applied to returned engrams

#### `GET /v1/engrams/{id}`

Expand Down Expand Up @@ -282,11 +310,24 @@ Request:
List extracted named entities. Returns `[{id, name}]` by default.

Query params:
- `query` — text search over entity names and aliases
- `detail=full` — return all fields
- `type` — filter by entity type
- `level` — pyramid compression level (same semantics as engrams)
- `limit` — max results (default 100; default 10 when using `?query=`)
- `limit` — max results (default 100)

#### `POST /v1/entities/search`

Text search over entity names and aliases. Returns `[{id, name}]` by default.

Request:
```json
{"query": "Alice", "limit": 10, "detail": "full", "level": 0}
```

- `query` — required; text search over entity names and aliases
- `limit` — max results (default 10)
- `detail` — set to `"full"` for all fields
- `level` — pyramid compression level applied to returned entities

#### `GET /v1/entities/{id}`

Expand Down
200 changes: 128 additions & 72 deletions internal/api/handlers.go
Original file line number Diff line number Diff line change
Expand Up @@ -325,58 +325,78 @@ func (s *Services) nerEntityEngrams(queryStr string) []string {

// --- Engrams ---

func (s *Services) handleListEngrams(w http.ResponseWriter, r *http.Request) {
full := parseDetail(r)
level := parseLevel(r)
queryStr := r.URL.Query().Get("query")

// Semantic search path
if queryStr != "" {
limit := parseLimit(r, 10)

// Run embedding and NER concurrently — both are network calls.
embCh := make(chan []float64, 1)
seedCh := make(chan []string, 1)

if s.EmbedClient != nil {
go func() {
emb, err := s.EmbedClient.Embed(queryStr)
if err != nil {
s.Logger.Warn("query embedding failed", "err", err)
embCh <- nil
return
}
embCh <- emb
}()
} else {
embCh <- nil
}
type searchEngramsRequest struct {
Query string `json:"query"`
Limit int `json:"limit,omitempty"`
Detail string `json:"detail,omitempty"`
Level int `json:"level,omitempty"`
}

// handleSearchEngrams handles POST /v1/engrams/search.
// Accepts a JSON body to support arbitrarily large query strings.
func (s *Services) handleSearchEngrams(w http.ResponseWriter, r *http.Request) {
var req searchEngramsRequest
if err := decode(r, &req); err != nil {
writeError(w, http.StatusBadRequest, "invalid_request", err.Error())
return
}
if req.Query == "" {
writeError(w, http.StatusBadRequest, "missing_field", "query is required")
return
}

go func() { seedCh <- s.nerEntityEngrams(queryStr) }()
full := req.Detail == "full"
limit := req.Limit
if limit <= 0 {
limit = 10
}

queryEmb := <-embCh
extraSeeds := <-seedCh
// Run embedding and NER concurrently — both are network calls.
embCh := make(chan []float64, 1)
seedCh := make(chan []string, 1)

result, err := s.Graph.Retrieve(queryEmb, queryStr, limit, extraSeeds...)
if err != nil {
writeError(w, http.StatusInternalServerError, "retrieval_error", err.Error())
return
}
applyEngramLevels(s.Graph, result.Engrams, level)
if full {
for _, e := range result.Engrams {
e.Embedding = nil
}
writeJSON(w, http.StatusOK, result.Engrams)
} else {
cards := make([]map[string]any, 0, len(result.Engrams))
for _, e := range result.Engrams {
cards = append(cards, engramCard(e))
if s.EmbedClient != nil {
go func() {
emb, err := s.EmbedClient.Embed(req.Query)
if err != nil {
s.Logger.Warn("query embedding failed", "err", err)
embCh <- nil
return
}
writeJSON(w, http.StatusOK, cards)
}
embCh <- emb
}()
} else {
embCh <- nil
}

go func() { seedCh <- s.nerEntityEngrams(req.Query) }()

queryEmb := <-embCh
extraSeeds := <-seedCh

result, err := s.Graph.Retrieve(queryEmb, req.Query, limit, extraSeeds...)
if err != nil {
writeError(w, http.StatusInternalServerError, "retrieval_error", err.Error())
return
}
applyEngramLevels(s.Graph, result.Engrams, req.Level)
if full {
for _, e := range result.Engrams {
e.Embedding = nil
}
writeJSON(w, http.StatusOK, result.Engrams)
} else {
cards := make([]map[string]any, 0, len(result.Engrams))
for _, e := range result.Engrams {
cards = append(cards, engramCard(e))
}
writeJSON(w, http.StatusOK, cards)
}
}

func (s *Services) handleListEngrams(w http.ResponseWriter, r *http.Request) {
full := parseDetail(r)
level := parseLevel(r)

// Threshold filter path
thresholdStr := r.URL.Query().Get("threshold")
Expand Down Expand Up @@ -535,24 +555,42 @@ func (s *Services) handleGetEngramContext(w http.ResponseWriter, r *http.Request

// --- Episodes ---

type searchEpisodesRequest struct {
Query string `json:"query"`
Limit int `json:"limit,omitempty"`
Detail string `json:"detail,omitempty"`
Level int `json:"level,omitempty"`
}

// handleSearchEpisodes handles POST /v1/episodes/search.
func (s *Services) handleSearchEpisodes(w http.ResponseWriter, r *http.Request) {
var req searchEpisodesRequest
if err := decode(r, &req); err != nil {
writeError(w, http.StatusBadRequest, "invalid_request", err.Error())
return
}
if req.Query == "" {
writeError(w, http.StatusBadRequest, "missing_field", "query is required")
return
}
limit := req.Limit
if limit <= 0 {
limit = 10
}
episodes, err := s.Graph.SearchEpisodesByText(req.Query, limit)
if err != nil {
writeError(w, http.StatusInternalServerError, "db_error", err.Error())
return
}
applyEpisodeLevels(s.Graph, episodes, req.Level)
writeEpisodeList(w, episodes, req.Detail == "full")
}

func (s *Services) handleListEpisodes(w http.ResponseWriter, r *http.Request) {
full := parseDetail(r)
level := parseLevel(r)
queryStr := r.URL.Query().Get("query")
limit := parseLimit(r, 50)

// Text search path (does not support before/level/unconsolidated filters)
if queryStr != "" {
episodes, err := s.Graph.SearchEpisodesByText(queryStr, limit)
if err != nil {
writeError(w, http.StatusInternalServerError, "db_error", err.Error())
return
}
applyEpisodeLevels(s.Graph, episodes, level)
writeEpisodeList(w, episodes, full)
return
}

channel := r.URL.Query().Get("channel")
unconsolidated := r.URL.Query().Get("unconsolidated") == "true"

Expand Down Expand Up @@ -710,24 +748,42 @@ func (s *Services) handleAddEpisodeEdge(w http.ResponseWriter, r *http.Request)

// --- Entities ---

type searchEntitiesRequest struct {
Query string `json:"query"`
Limit int `json:"limit,omitempty"`
Detail string `json:"detail,omitempty"`
Level int `json:"level,omitempty"`
}

// handleSearchEntities handles POST /v1/entities/search.
func (s *Services) handleSearchEntities(w http.ResponseWriter, r *http.Request) {
var req searchEntitiesRequest
if err := decode(r, &req); err != nil {
writeError(w, http.StatusBadRequest, "invalid_request", err.Error())
return
}
if req.Query == "" {
writeError(w, http.StatusBadRequest, "missing_field", "query is required")
return
}
limit := req.Limit
if limit <= 0 {
limit = 10
}
entities, err := s.Graph.FindEntitiesByText(req.Query, limit)
if err != nil {
writeError(w, http.StatusInternalServerError, "db_error", err.Error())
return
}
applyEntityLevels(s.Graph, entities, req.Level)
writeEntityList(w, entities, req.Detail == "full")
}

func (s *Services) handleListEntities(w http.ResponseWriter, r *http.Request) {
full := parseDetail(r)
level := parseLevel(r)
queryStr := r.URL.Query().Get("query")
limit := parseLimit(r, 100)

// Text search path
if queryStr != "" {
entities, err := s.Graph.FindEntitiesByText(queryStr, limit)
if err != nil {
writeError(w, http.StatusInternalServerError, "db_error", err.Error())
return
}
applyEntityLevels(s.Graph, entities, level)
writeEntityList(w, entities, full)
return
}

entityType := r.URL.Query().Get("type")
var entities []*graph.Entity
var err error
Expand Down
Loading