# Shared Agent Memory RFC

## Summary

This document defines a shared memory architecture for agents working across many projects for one human operator, with one active machine at a time.

The design goal is not a generic vector database. The goal is a durable memory system that helps agents understand:

- who Sebas is, what he prefers, and how he tends to decide
- what each project is trying to do
- which decisions and constraints already exist
- which knowledge from one project should inform another

The v1 architecture is intentionally simple:

- `SQLite/libSQL` is the canonical store
- embeddings are auxiliary, not central
- all agents talk to one local `memory-service`
- one local `memory-engine` handles ingestion, extraction, consolidation, embeddings, and index refresh
- memory is captured through explicit actions, not passive continuous listening

This keeps the system local-first, auditable, and easy to evolve without committing to distributed complexity too early.

## Goals

- Share durable memory across agents and projects
- Preserve global personal context about Sebas
- Preserve project-specific facts, conventions, and decisions
- Reuse useful knowledge across repositories
- Support hybrid retrieval: metadata + keyword/full-text + semantic search
- Keep memory explainable and evidence-backed
- Work well with many local agent processes on one machine

## Non-Goals

- Multi-writer conflict resolution across simultaneously active machines
- Passive always-on ingestion of ambient real-world conversation
- Dedicated external vector database in v1
- Rich graph-native storage in v1
- Markdown-vault-first storage as the canonical source of truth

## Core Architecture

The system has four main pieces:

1. `SQLite/libSQL` database
   The source of truth for all memory records, links, sources, ingestion history, and retrieval logs.

2. `memory-service`
   A local shared API surface used by all agents and tools. It is the only supported read/write path into memory.

3. `memory-engine`
   A local background worker responsible for asynchronous processing: extraction, consolidation, embedding generation, and index maintenance.

4. Agent clients
   Any coding agent, assistant, or automation process that queries or writes memory through the service.

### Design Choice

`SQLite/libSQL` is chosen over Postgres and OpenViking for v1 because the actual operating model is:

- one user
- one active machine at a time
- local self-hosting
- moderate write concurrency
- low operational tolerance for heavy infrastructure

This is enough to support a serious memory layer without overbuilding.

## Execution Model: Memory Pipeline Engine

The system must explicitly define when memory enters, who triggers it, and when it becomes durable.

### Components

- `memory-service`
  Accepts writes and reads. Creates ingestion events. Exposes search and context APIs.

- `memory-engine`
  Runs as an event-driven worker. Pulls jobs from a small local queue stored in the database or managed by the service.

### What Triggers Ingestion

Only these ingestion moments are allowed in v1:

1. `manual write`
   A user or agent explicitly writes a memory record, preference, project note, or decision through the API.

2. `session close`
   An agent run, coding session, or task execution finishes and submits its transcript, summary, or artifacts for extraction.

3. `conversation capture`
   A human conversation is intentionally saved into the system as notes, transcript, import, or agent-written summary.

4. `scheduled maintenance`
   Background maintenance runs to generate embeddings, refresh indexes, suggest consolidation, and detect duplicates.

### What Does Not Trigger Ingestion

The system does not do any of the following in v1:

- passive live listening to real-world conversations
- always-on capture from microphones or chat streams
- automatic ingestion of all repo text by default
- promotion of raw conversation snippets directly into durable truth

### Human Conversation Policy

Human conversations are harder than agent sessions because the system should not infer permission or significance by default.

In v1, a conversation enters memory only if one of these happens:

- a transcript or note is pasted/imported explicitly
- an agent writes a summary and it is intentionally saved
- a user captures a conversation artifact through the API

This means the memory system is `explicit-capture`, not `ambient-capture`.

### State Flow

All memory processing follows this lifecycle:

1. `capture`
   Raw material is submitted. This creates a `source` and an `ingestion_event`.

2. `extract`
   The engine derives candidate memories from the raw material. These land as `inbox` records.

3. `consolidate`
   The engine or a later approval pass deduplicates, merges, links, and scopes those candidates.

4. `promote`
   A consolidated record becomes `active` and available as durable memory.

5. `index`
   Embeddings and full-text indexes are updated asynchronously.

### Timing Rules

- Manual writes for profile or project facts are synchronous.
- Session-close extraction is asynchronous.
- Conversation capture is asynchronous after explicit save/import.
- Embedding generation must never block storage.
- Retrieval must still work when embeddings are missing by falling back to metadata and full-text search.

### Ownership Rules

- Agents may auto-create `episode` records and candidate `inbox` facts.
- Durable `profile` facts about Sebas should require explicit confirmation or a promotion rule.
- `decision` records should preserve rationale and source before activation.
- Contradictory or low-confidence memories should remain explicit, not silently overwritten.

### Failure Rules

- Failed jobs remain retryable in `ingestion_events`.
- Raw source material is never deleted because extraction failed.
- Every promoted memory must retain source/evidence references.
- Partial indexing failure must not make the record unreadable.

## Memory Model

The system stores multiple kinds of memory. These are not all equally trustworthy.

### Memory Classes

- `profile`
  Global facts, preferences, worldview hints, working style, communication preferences, recurring intent.

- `project`
  Facts tied to one project, repo, product, or codebase.

- `decision`
  Important choices with rationale, alternatives, and scope.

- `artifact`
  Reusable documents, links, snippets, references, or procedures.

- `episode`
  Session observations, conversation fragments, run summaries, transient context.

- `task_hint`
  Reusable next-step patterns or operational hints that help agents act better.

### Stable Types

#### `MemoryRecord`

Canonical stored unit. Required fields:

- `id`
- `type`
- `scope`
- `status`
- `project_id`
- `repo_id`
- `agent_id`
- `source_kind`
- `title`
- `content`
- `summary`
- `confidence`
- `freshness`
- `created_at`
- `updated_at`
- `observed_at`
- `source_ref`
- `evidence_ref`
- optional embedding vector

#### `MemoryScope`

Allowed values:

- `global`
- `project`
- `repo`
- `agent`
- `session`

#### `MemoryStatus`

Allowed values:

- `inbox`
- `active`
- `superseded`
- `contradicted`
- `archived`

#### `SourceKind`

Allowed values:

- `manual`
- `conversation`
- `run`
- `document`
- `import`

#### `IngestionEvent`

Tracks pipeline execution:

- event id
- trigger type
- source reference
- requested scope
- job state
- timestamps
- processor outcome
- retry count
- error message if any

#### `RetrievalResult`

Contains:

- memory record
- combined score
- score breakdown by retrieval component

[260 more lines in file. Use offset=261 to continue.]


from __future__ import annotations

import json
import threading
from pathlib import Path
from urllib.error import HTTPError
from urllib.request import Request, urlopen

from http.server import ThreadingHTTPServer

from shared_agent_memory import MemoryEngine, MemoryService
from shared_agent_memory.http_api import build_handler
from shared_agent_memory.schema import SCHEMA_VERSION

def make_service(tmp_path: Path) -> MemoryService:
    return MemoryService(str(tmp_path / "memory.sqlite3"))

def _request_json(url: str, *, method: str = "GET", payload: dict | None = None) -> tuple[int, dict]:
    body = None if payload is None else json.dumps(payload).encode("utf-8")
    request = Request(url, data=body, method=method)
    if body is not None:
        request.add_header("Content-Type", "application/json")
    with urlopen(request) as response:  # noqa: S310 - local test server only
        return response.status, json.loads(response.read().decode("utf-8"))

def _start_api(service: MemoryService) -> tuple[ThreadingHTTPServer, str]:
    server = ThreadingHTTPServer(("127.0.0.1", 0), build_handler(service))
    thread = threading.Thread(target=server.serve_forever, daemon=True)
    thread.start()
    host, port = server.server_address
    return server, f"http://{host}:{port}"

def test_manual_profile_fact_is_immediately_searchable(tmp_path: Path) -> None:
    service = make_service(tmp_path)
    service.upsert_profile_fact("Communication", "Sebas prefers concise, direct communication.")

    results = service.search("concise communication", scopes=["global"])

    assert results["results"]
    assert results["results"][0]["memory"]["type"] == "profile"
    assert results["results"][0]["memory"]["schema_version"] == SCHEMA_VERSION

def test_captured_conversation_stays_inbox_until_consolidated(tmp_path: Path) -> None:
    service = make_service(tmp_path)
    engine = MemoryEngine(service)

    service.capture_conversation("chat-1", "project", "We agreed to keep one active machine at a time.")
    processed = engine.process_pending()

    memory_id = processed[0]["memory_id"]
    memory = service.get_memory(memory_id)
    assert memory["status"] == "inbox"

    search_without_inbox = service.search("active machine", scopes=["project"])
    assert search_without_inbox["results"] == []

    promoted = service.consolidate([memory_id])
    assert promoted["promoted"] == [memory_id]

    search_with_active = service.search("active machine", scopes=["project"])
    assert search_with_active["results"][0]["memory"]["id"] == memory_id

def test_context_bundle_combines_global_project_and_task_artifacts(tmp_path: Path) -> None:
    service = make_service(tmp_path)
    service.create_project("proj_alpha", "Alpha")
    service.create_repo("repo_alpha", "Repo Alpha", project_id="proj_alpha")
    service.upsert_profile_fact("Preferences", "Sebas wants agents to preserve clarity.")
    service.create_memory(
        {
            "type": "project",
            "scope": "project",
            "project_id": "proj_alpha",
            "repo_id": "repo_alpha",
            "title": "Project goal",
            "content": "Alpha shares memory across projects.",
        }
    )
    service.create_memory(
        {
            "type": "decision",
            "scope": "project",
            "project_id": "proj_alpha",
            "repo_id": "repo_alpha",
            "title": "Storage engine",
            "content": "Use SQLite as canonical store.",
        }
    )
    service.create_memory(
        {
            "type": "artifact",
            "subtype": "research_artifact",
            "scope": "project",
            "project_id": "proj_alpha",
            "repo_id": "repo_alpha",
            "task_id": "task_alpha",
            "title": "Task note",
            "content": "Relevant artifact.",
        }
    )

    bundle = service.context_for(project="proj_alpha", repo="repo_alpha", task="task_alpha")

    assert bundle["profile_facts"]
    assert bundle["project_memories"]
    assert bundle["active_decisions"]
    assert bundle["task_relevant_artifacts"]

def test_export_import_preserves_operational_records_without_approvals_or_handoffs(tmp_path: Path) -> None:
    service = make_service(tmp_path)
    service.create_project("proj_alpha", "Alpha")
    service.create_repo("repo_alpha", "Repo Alpha", project_id="proj_alpha")
    task = service.create_task(
        task_id="task_alpha",
        title="Resume V2 task",
        intent="Need durable task state.",
        project_id="proj_alpha",
        repo_id="repo_alpha",
        owner_agent="personal-agent",
        run_id="run_alpha",
    )
    service.start_task_run(task["id"], "personal-agent", input_payload={"mode": "runner"})
    service.create_artifact(
        task_id=task["id"],
        artifact_type="brief",
        title="Brief",
        content="Keep resumable state in shared DB.",
        source_ref="personal-agent",
    )

    exported = service.export()

    second = MemoryService(str(tmp_path / "second.sqlite3"))
    second.import_data(exported)
    bundle = second.task_bundle(task["id"])

    assert "approvals" not in bundle
    assert "handoffs" not in bundle
    assert bundle["task"]["run_id"] == "run_alpha"
    assert bundle["runs"]
    assert bundle["artifacts"]

def test_list_memories_can_filter_by_v2_facets(tmp_path: Path) -> None:
    service = make_service(tmp_path)
    service.create_memory(
        {
            "id": "run_123",
            "type": "episode",
            "subtype": "research_run",
            "scope": "global",
            "title": "Research run",
            "content": "Goal: inspect",
            "run_id": "run-123",
            "origin_agent": "personal-agent",
        }
    )
    service.create_memory(
        {
            "type": "artifact",
            "subtype": "research_source",
            "scope": "global",
            "title": "Source",
            "content": "source",
            "run_id": "run-123",
            "url": "https://example.com/spec",
            "domain": "example.com",
            "origin_agent": "personal-agent",
        }
    )

    matches = service.list_memories(subtype="research_source", run_id="run-123", domain="example.com", origin_agent="personal-agent")

    assert len(matches) == 1
    assert matches[0]["url"] == "https://example.com/spec"

def test_search_works_without_embeddings(tmp_path: Path) -> None:
    service = make_service(tmp_path)
    service.create_memory(
        {
            "type": "artifact",
            "subtype": "research_claim",
            "scope": "global",
            "title": "Claim",
            "content": "SQLite remains optional for embeddings.",
            "embedding": None,
        }
    )

    results = service.search("optional embeddings", scopes=["global"])

    assert results["results"]
    assert results["results"][0]["memory"]["subtype"] == "research_claim"

def test_http_api_supports_status_ingest_and_search(tmp_path: Path) -> None:
    service = make_service(tmp_path)
    server, base_url = _start_api(service)
    try:
        status_code, status_payload = _request_json(f"{base_url}/api/status")
        assert status_code == 200
        assert status_payload["ok"] is True

        created_code, created_payload = _request_json(
            f"{base_url}/api/ingest",
            method="POST",
            payload={
                "id": "mem_http_1",
                "type": "artifact",
                "subtype": "research_claim",
                "scope": "global",
                "title": "HTTP memory",
                "content": "Remote API keeps shared memory reachable.",
                "status": "active",
            },
        )
        assert created_code == 201
        assert created_payload["id"] == "mem_http_1"

        search_code, search_payload = _request_json(f"{base_url}/api/search?q=reachable&scope=global")
        assert search_code == 200
        assert search_payload["results"]
        assert search_payload["results"][0]["memory"]["id"] == "mem_http_1"
    finally:
        server.shutdown()
        server.server_close()

def test_http_api_supports_tasks_runs_and_bundles(tmp_path: Path) -> None:
    service = make_service(tmp_path)
    server, base_url = _start_api(service)
    try:
        task_code, task = _request_json(
            f"{base_url}/api/tasks",
            method="POST",
            payload={
                "task_id": "task_http_1",
                "title": "Remote task",
                "intent": "Keep task state remote.",
                "owner_agent": "personal-agent",
            },
        )
        assert task_code == 201
        assert task["id"] == "task_http_1"

        run_code, run = _request_json(
            f"{base_url}/api/task-runs",
            method="POST",
            payload={"task_id": task["id"], "agent_id": "personal-agent", "input_payload": {"mode": "remote"}},
        )
        assert run_code == 201
        assert run["task_id"] == task["id"]

[293 more lines in file. Use offset=261 to continue.]


**Reviewing RFC and schema**

I think I need to look over the consolidation RFC and schema. It seems important to read through the relevant sections on promotion and merging. I want to ensure I understand everything clearly to proceed effectively. Maybe I’ll take some notes as I go, just to keep things organized. Let’s make sure I have all the details I need before moving forward!


- matched scope
- evidence reference
- explanation text

#### `ContextBundle`

The final payload an agent consumes. It should include:

- global profile facts
- relevant project/repo facts
- active decisions
- recent episodes when useful
- citations/explanations

### Relationships

Memory links support these relations:

- `supports`
- `contradicts`
- `supersedes`
- `applies_to`
- `derived_from`
- `related_to`

## Logical Schema

The implementation must support at least these logical tables or equivalent collections:

- `projects`
- `repos`
- `memories`
- `memory_links`
- `sources`
- `ingestion_events`
- `retrieval_logs`
- `profiles` or `saved_views`

### Schema Notes

- `memories` is the canonical table for all memory types.
- `sources` stores original artifacts: transcript refs, note refs, document refs, run refs.
- `memory_links` stores semantic and operational relations.
- `ingestion_events` is both audit log and retry queue anchor.
- `retrieval_logs` make memory behavior inspectable.

### Indexing Requirements

The system should maintain:

- metadata indexes on scope/project/repo/type/status
- FTS index over `title`, `summary`, `content`
- vector index for embeddings if available in chosen runtime

Embeddings remain inside the main store in v1. No separate vector database.

## Retrieval Behavior

Hybrid retrieval is mandatory.

### Retrieval Pipeline

1. Apply metadata filters first
   Filter by scope, project, repo, type, status, and recency bounds.

2. Run full-text retrieval
   Use keyword and lexical matching for explicit facts and exact terms.

3. Run semantic retrieval
   Use embeddings for concept-level similarity.

4. Merge and rerank
   Combine keyword score, semantic score, scope relevance, recency, and confidence.

5. Build response explanation
   Return why each item was selected.

### Default Search Scope

For most agent requests, search should consider:

- global profile memory
- current project and repo memory
- linked cross-project memories only when relevance is above threshold

This prevents unrelated noise while still allowing useful transfer.

### Retrieval Guarantees

- Retrieval must not depend solely on embeddings.
- Records with missing embeddings must still be discoverable.
- Results must include evidence or source references whenever possible.
- Returned context should prefer fewer, stronger memories over many weak ones.

## Ingestion and Consolidation

The system uses two ingestion lanes.

### Lane 1: Manual Curated Ingestion

Used for:

- personal profile facts
- communication preferences
- worldview and working style
- project conventions
- explicit decisions

These writes are synchronous and can become durable immediately if written in structured form.

### Lane 2: Automatic Extracted Ingestion

Used for:

- agent session transcripts
- run summaries
- imported conversations
- documents and notes requiring extraction

These writes first create `inbox` records. They are candidates, not durable truth.

### Consolidation Rules

Consolidation should:

- detect duplicates or near-duplicates
- merge equivalent facts
- preserve provenance
- create contradiction links when claims disagree
- promote only sufficiently grounded memories to `active`

### Truth Policy

- `episode` records may be stored automatically.
- candidate facts extracted from episodes stay `inbox` until consolidation.
- profile/worldview memories should be promoted with explicit confirmation or policy.
- latest write does not automatically replace canonical truth.

## Service API

The exact transport can be local HTTP, local RPC, or MCP-compatible wrapping. The behavioral API should include:

- `ingest(memory_input)`
- `capture_session(source_ref, scope)`
- `capture_conversation(source_ref, scope, capture_mode)`
- `search(query, scopes, filters)`
- `context_for(project, repo, agent, task)`
- `consolidate(inbox_items)`
- `upsert_profile_fact(...)`
- `explain(memory_id | retrieval_id)`
- `export()`
- `import()`

### API Expectations

- all writes go through the service
- all agent context reads go through the service
- service responses should be deterministic enough to debug
- search and context calls should expose explanation fields, not just payloads

## Operational Model

This system should be treated as shared infrastructure for all local agents.

### Concurrency

Multiple agents may read and write at once on the same machine. The design assumes:

- one active machine
- many local processes
- moderate concurrent writes

The service layer should serialize or coordinate writes as needed so agents do not each invent their own storage rules.

### Backup and Portability

V1 backup strategy can be simple:

- regular snapshot/export of the database
- optional export of sources or source references
- manual transfer between machines when switching active device

Because there is no simultaneous multi-device usage requirement, this is acceptable for v1.

## Acceptance Scenarios

The future implementation should satisfy these scenarios:

1. An agent in project A retrieves Sebas global preferences plus project A conventions without project B noise.

2. A decision recorded in project A appears in project B only when linked or strongly relevant.

3. A manually saved profile preference is available immediately even before embeddings exist.

4. A conversation imported from outside the system stays raw until explicitly captured, then becomes `inbox`, then later may be promoted.

5. Session-close extraction does not block the agent workflow.

6. Two contradictory facts can coexist with explicit status and evidence.

7. Retrieval combines lexical and semantic matches and can explain result selection.

8. Export/import preserves ids, timestamps, links, statuses, and ingestion history.

9. Multiple local agents can safely use the same shared memory service.

## Deferred Ideas / Future Extensions

These ideas are useful, but should not be implemented in v1.

### OpenViking-Inspired Ideas

- Hierarchical context loading (`L0/L1/L2`)
  Useful later because agents often need summary-first then drill-down retrieval.
  Not now because the core memory model and retrieval service should exist before multi-level context packaging.

- Filesystem-style browsing and recursive retrieval
  Useful later because it gives humans and agents a navigable knowledge shape.
  Not now because SQL-backed canonical memory is simpler and more reliable for v1.

- Richer separation of user, agent, resource, and skill namespaces
  Useful later because it makes larger agent ecosystems cleaner.
  Not now because current scope is one operator and one shared memory domain.

### Arscontexta-Inspired Ideas

- Optional Markdown vault as a capture surface
  Useful later because Markdown is portable, easy to inspect, and good for human-curated notes.
  Not now because canonical truth should stay in one structured store first.

- Reflection and reweave workflows over notes
  Useful later because long-running knowledge bases benefit from periodic restructuring.
  Not now because the system first needs reliable ingestion, status handling, and consolidation.

### Other Deferred Extensions

- Graph/entity memory
  Useful later when cross-project entity relationships become a retrieval bottleneck.
  Not now because links in SQL are enough for early stages.

- Multi-device sync and conflict handling
  Useful later if more than one machine becomes active in parallel.
  Not now because the operating model explicitly avoids that complexity.

- Live conversation ingestion
  Useful later only if a clear privacy, consent, and review model exists.
  Not now because passive capture is too risky and noisy.

## Recommended v1 Principle

Treat memory as a structured, evidence-backed, consolidating knowledge system.

Do not treat memory as:

- a dump of all conversations
- a pure vector search problem
- an autonomous truth machine

The system should help agents remember better, not hallucinate with more confidence.


# Memory Consolidation + Opportunistic Scheduler RFC

Status: draft
Owner: Sebas
Date: 2026-03-26

## Intent

Add a periodic memory consolidation stage to `agents-database` that is explicit, auditable, and catch-up friendly on a laptop. The scheduler must run at the next available moment if the machine was offline at the planned time.

## Requirements (selected)

- Keep the existing capture→extract→consolidate→promote→index flow.
- Explicit provenance, contradictions, and no silent overwrite.
- Preserve originals; no hard deletes during consolidation.
- Hybrid retrieval remains functional without embeddings.
- LLM steps use **MiniMax M2.5 Free** (only when required).
- Scheduler is **opportunistic** (runs when available, not only at exact times).
- Ship a complete, v1-safe solution; iterate later.

## Non-Goals

- External cron-only scheduling.
- Always-on cloud infra or multi-device coordination.
- Autonomous truth rewriting without explicit promotion rules.

## Proposed Architecture

### 1) Opportunistic Scheduler (Next Available)

Instead of relying on cron/launchd timing alone, schedule lives in the DB and the engine runs jobs when it is awake and detects a missed or due run.

Trigger points:

- Engine startup.
- Periodic engine tick (e.g., every 15 minutes while running).
- Any write operation that touches the DB (optional fast check).

Current implementation status:

- `maintenance-daemon` provides the periodic tick path.
- `MemoryService` now runs an opportunistic due-job check on service startup.
- `MemoryService` now runs a fast due-job check after normal write operations.

Semantics:

- Each job has a `next_due_at` and `window`.
- If `now >= next_due_at`, run immediately.
- If machine was asleep/offline, the job runs on the next available tick.
- After run, compute the next due time based on cadence and configured windows.

This avoids missed runs without requiring exact runtime availability.

### 2) Consolidation Pipeline

Stages per job run:

1. **Select candidates** from `memories` with status `inbox` or `episode` older than a minimum age.
2. **Dedupe** via similarity signals:
   - exact match (content + scope + type)
   - lexical overlap
   - embedding similarity (if present)
   - metadata / temporal proximity
3. **Conflict detection**: if contradictory claims, create `memory_conflicts` + `memory_links` with `contradicts`.
4. **Promotion**: only promote to `active` when above confidence threshold.
5. **Archival**: archive duplicates; never delete originals.

LLM usage:

- Only for synthesis/abstraction when similarity is mid-range (not near-exact).
- Use MiniMax M2.5 Free model for summaries/synthesis.
- Hard budget caps per run; fallback to non-LLM merge if budget is exceeded.

### 3) Cost + Safety Guardrails

- Max LLM calls per batch.
- Budget cap per day/week.
- Preserve originals toggle on by default.
- Manual review queue for low-confidence or conflict-heavy items.

## Data Model Additions

### maintenance_jobs

Stores periodic job definitions and state.

```sql
CREATE TABLE IF NOT EXISTS maintenance_jobs (
  id TEXT PRIMARY KEY,
  job_type TEXT NOT NULL,               -- e.g. consolidation
  cadence TEXT NOT NULL,                -- daily | weekly | interval
  interval_minutes INTEGER,             -- for interval cadence
  window_start TEXT,                    -- optional time window start (local time)
  window_end TEXT,                      -- optional time window end (local time)
  next_due_at TEXT NOT NULL,
  last_run_at TEXT,
  last_status TEXT,
  last_summary TEXT,
  enabled INTEGER NOT NULL DEFAULT 1,
  metadata_json TEXT NOT NULL,
  created_at TEXT NOT NULL,
  updated_at TEXT NOT NULL
);
```

### maintenance_runs

Audit trail per execution.

```sql
CREATE TABLE IF NOT EXISTS maintenance_runs (
  id TEXT PRIMARY KEY,
  job_id TEXT NOT NULL,
  status TEXT NOT NULL,                 -- running | completed | failed
  started_at TEXT NOT NULL,
  completed_at TEXT,
  summary TEXT,
  stats_json TEXT NOT NULL,
  error_message TEXT,
  FOREIGN KEY(job_id) REFERENCES maintenance_jobs(id)
);
```

### memory_conflicts

Explicit conflict registry.

```sql
CREATE TABLE IF NOT EXISTS memory_conflicts (
  id TEXT PRIMARY KEY,
  memory_id TEXT NOT NULL,
  conflicting_memory_id TEXT NOT NULL,
  reason TEXT NOT NULL,
  created_at TEXT NOT NULL,
  metadata_json TEXT NOT NULL,
  FOREIGN KEY(memory_id) REFERENCES memories(id),
  FOREIGN KEY(conflicting_memory_id) REFERENCES memories(id)
);
```

Indexes:

```sql
CREATE INDEX IF NOT EXISTS idx_maintenance_jobs_due ON maintenance_jobs(next_due_at);
CREATE INDEX IF NOT EXISTS idx_maintenance_runs_job ON maintenance_runs(job_id);
CREATE INDEX IF NOT EXISTS idx_memory_conflicts_mem ON memory_conflicts(memory_id);
```

## Consolidation Heuristics (v1)

- Candidate age: >= 7 days for `episode` and `inbox`.
- Dedup threshold:
  - exact match: immediate archive + link
  - high similarity (>= 0.90): merge
  - medium similarity (0.75–0.89): LLM synthesize (MiniMax M2.5 Free)
  - low similarity: no merge
- Promotion threshold: confidence >= 0.70
- Conflict threshold: similarity below merge threshold but overlapping entities/keywords with contradictory polarity.

## LLM Configuration

- Provider/model: **MiniMax M2.5 Free**
- Use only when `strategy == synthesize` or `abstract`.
- Token budget caps per run; hard stop with non-LLM fallback.

Execution (v1):

- Calls the configured CLI runner and passes the prompt via stdin.
- Job metadata can override the command via `llm_command`.

## Scheduler Details (v1)

### Job definition

- Daily light runs at 10:00 and 15:00: dedupe + promotion checks (no LLM by default).
- Weekly deep run: include LLM synthesis and abstraction.

### Next-available algorithm (pseudo)

```python
for job in maintenance_jobs where enabled=1:
  if now >= job.next_due_at:
     run_job(job)
     job.last_run_at = now
     job.next_due_at = compute_next_due(job)
```

`compute_next_due(job)` respects `window_start/window_end`; if outside window, schedule the next window start. If missed, run immediately and schedule the next window.

## Gotchas

- Job windows are interpreted as UTC (based on stored ISO timestamps).
- If the configured runner is unavailable, LLM synthesis is skipped without failing the job.
- `maintenance-daemon` is cooperative; run it via your preferred login/session manager for continuous checks.
- Launchd templates need path edits if your repo lives elsewhere.

CLI usage:

```bash
.venv/bin/python -m shared_agent_memory.cli --db data/shared-agent-memory.sqlite3 maintenance-status
.venv/bin/python -m shared_agent_memory.cli --db data/shared-agent-memory.sqlite3 maintenance-tick
.venv/bin/python -m shared_agent_memory.cli --db data/shared-agent-memory.sqlite3 maintenance-config job_consolidation_daily_10 --window-start 10:00 --recompute-next
.venv/bin/python -m shared_agent_memory.cli --db data/shared-agent-memory.sqlite3 maintenance-daemon --interval-seconds 900
```

Launchd template:

- `docs/launchd/agents-database-maintenance.plist`

Linux systemd user service:

- `systemd/user/agents-database-maintenance.service`
- install with `ln -sf /mnt/rpi/agents-database/systemd/user/agents-database-maintenance.service /home/sebas/.config/systemd/user/agents-database-maintenance.service`
- enable with `systemctl --user daemon-reload && systemctl --user enable --now agents-database-maintenance.service`
- requires `loginctl enable-linger sebas` if the machine should restart the user service after reboot without waiting for login

## Progress Log (living)

- 2026-03-26: RFC drafted with opportunistic scheduler + consolidation pipeline.
- 2026-03-26: Implemented maintenance tables, scheduler hooks, and consolidation runner (v1).
- 2026-03-27: Added maintenance config + daemon CLI commands.
- 2026-03-30: Fixed the maintenance daemon wrapper to run from a repo checkout by exporting `PYTHONPATH` and preferring `./.venv/bin/python`.
- 2026-03-30: Added and enabled a Linux `systemd --user` service with `Restart=always`; verified auto-restart after kill and confirmed `Linger=yes` for reboot persistence.
- 2026-03-31: Added opportunistic due-job checks inside `MemoryService` startup and normal write flows, with tests covering startup catch-up and write-triggered catch-up.

## Open Questions

- Whether to run the engine as a long-lived background process or on-demand via CLI.
- Which storage to use for LLM usage counters (likely `maintenance_runs.stats_json`).

## Proposed Next Steps

1. Add tests for daemon/bootstrap operation and job catch-up across missed windows.
2. Add LLM budget counters per run/day to `maintenance_runs.stats_json` or a dedicated table.
3. Decide whether to keep Linux `systemd --user` as the primary runtime and treat launchd as secondary documentation.
4. Decide whether opportunistic write-path checks should stay on every write or be narrowed to specific write classes if load grows.


**Considering system implementation**

I need to act on the existing system and think about implementing a compatible slice in the agents-database. It seems likely there's already some consolidation in place. So, I should inspect the implementation functions to see if they can be merged or superseded. I’ll start by searching through the code to find what I need. This will help me understand what options I have for making improvements.

bash

cd /mnt/rpi/agents-database && grep -R "def consolidate\|supersede\|contradict\|memory_links\|maintenance_jobs\|MemoryEngine\|upsert_profile_fact\|create_memory" -n shared_agent_memory tests | sed -n '1,260p'

cd /mnt/rpi/agents-database && find shared_agent_memory -maxdepth 2 -type f | sed -n '1,240p'


tests/test_memory_service.py:11:from shared_agent_memory import MemoryEngine, MemoryService
tests/test_memory_service.py:39:    service.upsert_profile_fact("Communication", "Sebas prefers concise, direct communication.")
tests/test_memory_service.py:50:    engine = MemoryEngine(service)
tests/test_memory_service.py:73:    service.upsert_profile_fact("Preferences", "Sebas wants agents to preserve clarity.")
tests/test_memory_service.py:74:    service.create_memory(
tests/test_memory_service.py:84:    service.create_memory(
tests/test_memory_service.py:94:    service.create_memory(
tests/test_memory_service.py:152:    service.create_memory(
tests/test_memory_service.py:164:    service.create_memory(
tests/test_memory_service.py:186:    service.create_memory(
tests/test_memory_service.py:281:    service.create_memory(
tests/test_memory_service.py:316:def test_maintenance_jobs_exist_by_default(tmp_path: Path) -> None:
tests/test_memory_service.py:319:    jobs = {job["id"] for job in service.list_maintenance_jobs()}
tests/test_memory_service.py:328:    engine = MemoryEngine(service)
tests/test_memory_service.py:372:    service.create_memory(
tests/test_memory_service.py:382:    service.create_memory(
grep: shared_agent_memory: No such file or directory
grep: tests/__pycache__/test_memory_service.cpython-313-pytest-9.0.2.pyc: binary file matches


find: ‘shared_agent_memory’: No such file or directory


**Finding package path**

I need to determine where the package path is, and it looks like "src" could be the right place to check. It's essential to locate this correctly to ensure everything functions as expected. I wonder if there are specific guidelines or commands I can use to verify this path. I'll think through my options carefully before proceeding. Maybe I’ll also consider common practices to streamline this process!

cd /mnt/rpi/agents-database && find . -maxdepth 3 -type f | grep -E 'memory_service|http_api|schema|maintenance|cli|engine' | sed -n '1,260p'


./docs/launchd/agents-database-maintenance.plist
./scripts/run-maintenance-daemon.sh
./systemd/user/agents-database-maintenance.service
./tests/test_memory_service.py
./tests/__pycache__/test_memory_service.cpython-313-pytest-9.0.2.pyc
./src/shared_agent_memory/schema.py
./src/shared_agent_memory/engine.py
./src/shared_agent_memory/cli.py
./src/shared_agent_memory/http_api.py
./maintenance-daemon.log


SCHEMA_VERSION = "shared-agent-memory-v2"

SCHEMA_SQL = f"""
PRAGMA foreign_keys = ON;

CREATE TABLE IF NOT EXISTS projects (
    id TEXT PRIMARY KEY,
    name TEXT NOT NULL,
    description TEXT,
    created_at TEXT NOT NULL,
    updated_at TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS repos (
    id TEXT PRIMARY KEY,
    project_id TEXT,
    name TEXT NOT NULL,
    path TEXT,
    created_at TEXT NOT NULL,
    updated_at TEXT NOT NULL,
    FOREIGN KEY(project_id) REFERENCES projects(id)
);

CREATE TABLE IF NOT EXISTS sources (
    id TEXT PRIMARY KEY,
    source_kind TEXT NOT NULL,
    title TEXT,
    content TEXT NOT NULL,
    source_ref TEXT,
    metadata_json TEXT NOT NULL,
    created_at TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS ingestion_events (
    id TEXT PRIMARY KEY,
    trigger_type TEXT NOT NULL,
    source_id TEXT NOT NULL,
    requested_scope TEXT NOT NULL,
    job_state TEXT NOT NULL,
    processor_outcome TEXT,
    retry_count INTEGER NOT NULL DEFAULT 0,
    error_message TEXT,
    created_at TEXT NOT NULL,
    updated_at TEXT NOT NULL,
    processed_at TEXT,
    FOREIGN KEY(source_id) REFERENCES sources(id)
);

CREATE TABLE IF NOT EXISTS memories (
    id TEXT PRIMARY KEY,
    schema_version TEXT NOT NULL DEFAULT '{SCHEMA_VERSION}',
    type TEXT NOT NULL,
    subtype TEXT,
    scope TEXT NOT NULL,
    status TEXT NOT NULL,
    project_id TEXT,
    repo_id TEXT,
    agent_id TEXT,
    origin_agent TEXT,
    run_id TEXT,
    task_id TEXT,
    url TEXT,
    domain TEXT,
    source_kind TEXT NOT NULL,
    title TEXT NOT NULL,
    content TEXT NOT NULL,
    summary TEXT NOT NULL,
    confidence REAL NOT NULL,
    freshness REAL NOT NULL,
    created_at TEXT NOT NULL,
    updated_at TEXT NOT NULL,
    observed_at TEXT NOT NULL,
    source_ref TEXT,
    evidence_ref TEXT,
    embedding_json TEXT,
    metadata_json TEXT NOT NULL,
    FOREIGN KEY(project_id) REFERENCES projects(id),
    FOREIGN KEY(repo_id) REFERENCES repos(id)
);

CREATE TABLE IF NOT EXISTS memory_links (
    id TEXT PRIMARY KEY,
    from_memory_id TEXT NOT NULL,
    to_memory_id TEXT NOT NULL,
    relation TEXT NOT NULL,
    created_at TEXT NOT NULL,
    metadata_json TEXT NOT NULL,
    FOREIGN KEY(from_memory_id) REFERENCES memories(id),
    FOREIGN KEY(to_memory_id) REFERENCES memories(id)
);

CREATE TABLE IF NOT EXISTS retrieval_logs (
    id TEXT PRIMARY KEY,
    query_text TEXT NOT NULL,
    filters_json TEXT NOT NULL,
    results_json TEXT NOT NULL,
    created_at TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS maintenance_jobs (
    id TEXT PRIMARY KEY,
    job_type TEXT NOT NULL,
    cadence TEXT NOT NULL,
    interval_minutes INTEGER,
    window_start TEXT,
    window_end TEXT,
    next_due_at TEXT NOT NULL,
    last_run_at TEXT,
    last_status TEXT,
    last_summary TEXT,
    enabled INTEGER NOT NULL DEFAULT 1,
    metadata_json TEXT NOT NULL,
    created_at TEXT NOT NULL,
    updated_at TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS maintenance_runs (
    id TEXT PRIMARY KEY,
    job_id TEXT NOT NULL,
    status TEXT NOT NULL,
    started_at TEXT NOT NULL,
    completed_at TEXT,
    summary TEXT,
    stats_json TEXT NOT NULL,
    error_message TEXT,
    FOREIGN KEY(job_id) REFERENCES maintenance_jobs(id)
);

CREATE TABLE IF NOT EXISTS memory_conflicts (
    id TEXT PRIMARY KEY,
    memory_id TEXT NOT NULL,
    conflicting_memory_id TEXT NOT NULL,
    reason TEXT NOT NULL,
    created_at TEXT NOT NULL,
    metadata_json TEXT NOT NULL,
    FOREIGN KEY(memory_id) REFERENCES memories(id),
    FOREIGN KEY(conflicting_memory_id) REFERENCES memories(id)
);

CREATE TABLE IF NOT EXISTS profiles (
    id TEXT PRIMARY KEY,
    name TEXT NOT NULL,
    profile_json TEXT NOT NULL,
    created_at TEXT NOT NULL,
    updated_at TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS tasks (
    id TEXT PRIMARY KEY,
    schema_version TEXT NOT NULL DEFAULT '{SCHEMA_VERSION}',
    run_id TEXT,
    title TEXT NOT NULL,
    intent TEXT NOT NULL,
    kind TEXT NOT NULL,
    status TEXT NOT NULL,
    priority INTEGER NOT NULL DEFAULT 3,
    project_id TEXT,
    repo_id TEXT,
    parent_task_id TEXT,
    origin TEXT,
    owner_agent TEXT,
    blocked_reason TEXT,
    requires_human_input INTEGER NOT NULL DEFAULT 0,
    due_at TEXT,
    created_at TEXT NOT NULL,
    updated_at TEXT NOT NULL,
    metadata_json TEXT NOT NULL,
    FOREIGN KEY(project_id) REFERENCES projects(id),
    FOREIGN KEY(repo_id) REFERENCES repos(id),
    FOREIGN KEY(parent_task_id) REFERENCES tasks(id)
);

CREATE TABLE IF NOT EXISTS task_runs (
    id TEXT PRIMARY KEY,
    task_id TEXT NOT NULL,
    agent_id TEXT NOT NULL,
    status TEXT NOT NULL,
    input_payload_json TEXT NOT NULL,
    result_summary TEXT,
    error_message TEXT,
    started_at TEXT NOT NULL,
    completed_at TEXT,
    created_at TEXT NOT NULL,
    updated_at TEXT NOT NULL,
    metadata_json TEXT NOT NULL,
    FOREIGN KEY(task_id) REFERENCES tasks(id) ON DELETE CASCADE
);

CREATE TABLE IF NOT EXISTS artifacts (
    id TEXT PRIMARY KEY,
    task_id TEXT NOT NULL,
    artifact_type TEXT NOT NULL,
    title TEXT NOT NULL,
    content TEXT NOT NULL,
    format TEXT NOT NULL,
    status TEXT NOT NULL,
    source_ref TEXT,
    created_at TEXT NOT NULL,
    updated_at TEXT NOT NULL,
    metadata_json TEXT NOT NULL,
    FOREIGN KEY(task_id) REFERENCES tasks(id) ON DELETE CASCADE
);

CREATE INDEX IF NOT EXISTS idx_memories_scope ON memories(scope);
CREATE INDEX IF NOT EXISTS idx_memories_status ON memories(status);
CREATE INDEX IF NOT EXISTS idx_memories_project ON memories(project_id);
CREATE INDEX IF NOT EXISTS idx_memories_repo ON memories(repo_id);
CREATE INDEX IF NOT EXISTS idx_memories_type ON memories(type);
CREATE INDEX IF NOT EXISTS idx_memories_subtype ON memories(subtype);
CREATE INDEX IF NOT EXISTS idx_memories_run ON memories(run_id);
CREATE INDEX IF NOT EXISTS idx_memories_task ON memories(task_id);
CREATE INDEX IF NOT EXISTS idx_memories_url ON memories(url);
CREATE INDEX IF NOT EXISTS idx_memories_domain ON memories(domain);
CREATE INDEX IF NOT EXISTS idx_memories_origin_agent ON memories(origin_agent);
CREATE INDEX IF NOT EXISTS idx_ingestion_state ON ingestion_events(job_state);
CREATE INDEX IF NOT EXISTS idx_tasks_status ON tasks(status);
CREATE INDEX IF NOT EXISTS idx_tasks_owner ON tasks(owner_agent);
CREATE INDEX IF NOT EXISTS idx_tasks_run ON tasks(run_id);
CREATE INDEX IF NOT EXISTS idx_task_runs_task ON task_runs(task_id);
CREATE INDEX IF NOT EXISTS idx_task_runs_status ON task_runs(status);
CREATE INDEX IF NOT EXISTS idx_artifacts_task ON artifacts(task_id);
CREATE INDEX IF NOT EXISTS idx_maintenance_jobs_due ON maintenance_jobs(next_due_at);
CREATE INDEX IF NOT EXISTS idx_maintenance_runs_job ON maintenance_runs(job_id);
CREATE INDEX IF NOT EXISTS idx_memory_conflicts_mem ON memory_conflicts(memory_id);

CREATE VIRTUAL TABLE IF NOT EXISTS memories_fts USING fts5(
    memory_id UNINDEXED,
    title,
    summary,
    content
);
"""


from __future__ import annotations

from typing import Any

from .service import MemoryService, make_id, utcnow

class MemoryEngine:
    def __init__(self, service: MemoryService) -> None:
        self.service = service

    def process_pending(self, limit: int | None = None) -> list[dict[str, Any]]:
        events = self.service.list_pending_events()
        if limit is not None:
            events = events[:limit]

        processed: list[dict[str, Any]] = []
        for event in events:
            try:
                processed.append(self._process_event(event))
                self.service.mark_event(event["id"], "completed", outcome="extracted")
            except Exception as exc:  # pragma: no cover - defensive
                self.service.mark_event(event["id"], "failed", error_message=str(exc))
        return processed

    def run_due_maintenance(self, *, limit: int | None = None, job_id: str | None = None) -> list[dict[str, Any]]:
        if job_id:
            jobs = [self.service.get_maintenance_job(job_id)]
        else:
            jobs = self.service.list_due_maintenance_jobs(limit=limit)
        results: list[dict[str, Any]] = []
        for job in jobs:
            run = self.service.create_maintenance_run(job["id"])
            try:
                stats = self._run_maintenance_job(job)
                self.service.finish_maintenance_run(
                    run["id"],
                    status="completed",
                    summary=f"{job['job_type']} completed",
                    stats=stats,
                )
                next_due = self.service._compute_next_due(
                    utcnow(),
                    job["cadence"],
                    job.get("interval_minutes"),
                    job.get("window_start"),
                    job.get("window_end"),
                )
                self.service.update_maintenance_job(
                    job["id"],
                    next_due_at=next_due,
                    last_run_at=utcnow(),
                    last_status="completed",
                    last_summary=f"{job['job_type']} completed",
                )
                results.append({"job_id": job["id"], "status": "completed", "stats": stats})
            except Exception as exc:
                self.service.finish_maintenance_run(
                    run["id"],
                    status="failed",
                    summary=f"{job['job_type']} failed",
                    stats={},
                    error_message=str(exc),
                )
                self.service.update_maintenance_job(
                    job["id"],
                    last_run_at=utcnow(),
                    last_status="failed",
                    last_summary=str(exc),
                )
                results.append({"job_id": job["id"], "status": "failed", "error": str(exc)})
        return results

    def _run_maintenance_job(self, job: dict[str, Any]) -> dict[str, Any]:
        job_type = job.get("job_type")
        if job_type == "consolidation":
            return self.service.consolidate_candidates(config=job.get("metadata") or {})
        return {"skipped": True, "reason": f"unknown job_type {job_type}"}

    def _process_event(self, event: dict[str, Any]) -> dict[str, Any]:
        with self.service.store.connection() as conn:
            source = conn.execute("SELECT * FROM sources WHERE id = ?", (event["source_id"],)).fetchone()
            if source is None:
                raise KeyError(event["source_id"])
            source_dict = dict(source)

        content = source_dict["content"]
        summary = self.service._make_summary(content)
        memory = self.service.ingest(
            {
                "id": make_id("mem"),
                "type": "episode",
                "scope": self._normalize_scope(event["requested_scope"]),
                "status": "inbox",
                "source_kind": source_dict["source_kind"],
                "title": source_dict["title"],
                "content": content,
                "summary": summary,
                "confidence": 0.6,
                "freshness": 1.0,
                "observed_at": source_dict["created_at"],
                "source_ref": source_dict["source_ref"],
                "evidence_ref": source_dict["id"],
                "embedding": self.service._text_embedding(content),
                "metadata": {
                    "ingested_by": "memory-engine",
                    "trigger_type": event["trigger_type"],
                    "captured_at": utcnow(),
                },
            }
        )
        return {"event_id": event["id"], "memory_id": memory["id"]}

    def _normalize_scope(self, requested_scope: str) -> str:
        if requested_scope in {"global", "project", "repo", "agent", "session"}:
            return requested_scope
        return "session"


**Inspecting service methods**

I need to take a closer look at the service methods specifically surrounding consolidate_candidates. I’m thinking about the relevant details that might impact their functionality and how well they perform their tasks. There may be various aspects to consider, including efficiency and any potential issues that could arise. It's essential to ensure everything is working as it should. Let's dive deeper into this to understand it better!

cd /mnt/rpi/agents-database && grep -n "def consolidate_candidates\|def ingest\|def _compute_next_due\|def list_due_maintenance_jobs\|def create_maintenance_job\|def update_maintenance_job\|def mark_memory" -n src/shared_agent_memory/*.py | sed -n '1,200p'


src/shared_agent_memory/service.py:128:    def ingest(self, memory_input: dict[str, Any]) -> dict[str, Any]:
src/shared_agent_memory/service.py:318:    def consolidate_candidates(self, *, config: dict[str, Any]) -> dict[str, Any]:
src/shared_agent_memory/service.py:1350:    def list_due_maintenance_jobs(self, *, now: str | None = None, limit: int | None = None) -> list[dict[str, Any]]:
src/shared_agent_memory/service.py:1368:    def update_maintenance_job(
src/shared_agent_memory/service.py:1616:    def _compute_next_due(


from __future__ import annotations

import json
import math
import subprocess
import uuid
from dataclasses import dataclass
from datetime import UTC, datetime, timedelta
from typing import Any
from urllib.parse import urlparse

from .schema import SCHEMA_VERSION
from .store import SQLiteStore

def utcnow() -> str:
    return datetime.now(UTC).isoformat()

def make_id(prefix: str) -> str:
    return f"{prefix}_{uuid.uuid4().hex}"

def normalize_text(value: str) -> str:
    return " ".join(value.lower().split())

def _line_value(content: str, label: str) -> str:
    prefix = f"{label}:"
    for line in content.splitlines():
        if line.startswith(prefix):
            return line[len(prefix) :].strip()
    return ""

@dataclass(slots=True)
class SearchResult:
    memory: dict[str, Any]
    score: float
    explanation: str
    facet_score: float
    lexical_score: float
    semantic_score: float

    def as_dict(self) -> dict[str, Any]:
        return {
            "memory": self.memory,
            "score": self.score,
            "score_breakdown": {
                "facet": self.facet_score,
                "lexical": self.lexical_score,
                "semantic": self.semantic_score,
                "confidence": self.memory["confidence"],
                "freshness": self.memory["freshness"],
            },
            "matched_scope": self.memory["scope"],
            "evidence_ref": self.memory["evidence_ref"],
            "explanation": self.explanation,
        }

class MemoryService:
    def __init__(self, db_path: str) -> None:
        self.store = SQLiteStore(db_path)
        self._maintenance_check_running = False
        self.store.initialize()
        self.migrate_v2()
        self._ensure_default_jobs()
        self._run_opportunistic_maintenance(limit=10)

    def create_memory(self, memory_input: dict[str, Any]) -> dict[str, Any]:
        now = utcnow()
        metadata = dict(memory_input.get("metadata") or {})
        subtype = memory_input.get("subtype")
        run_id = memory_input.get("run_id")
        task_id = memory_input.get("task_id")
        url = memory_input.get("url")
        domain = memory_input.get("domain") or (urlparse(url).netloc if url else None)
        origin_agent = memory_input.get("origin_agent") or metadata.get("origin_agent") or memory_input.get("agent_id")
        record = {
            "id": memory_input.get("id", make_id("mem")),
            "schema_version": memory_input.get("schema_version", SCHEMA_VERSION),
            "type": memory_input["type"],
            "subtype": subtype,
            "scope": memory_input["scope"],
            "status": memory_input.get("status", "active"),
            "project_id": memory_input.get("project_id"),
            "repo_id": memory_input.get("repo_id"),
            "agent_id": memory_input.get("agent_id"),
            "origin_agent": origin_agent,
            "run_id": run_id,
            "task_id": task_id,
            "url": url,
            "domain": domain,
            "source_kind": memory_input.get("source_kind", "manual"),
            "title": memory_input["title"],
            "content": memory_input["content"],
            "summary": memory_input.get("summary") or self._make_summary(memory_input["content"]),
            "confidence": float(memory_input.get("confidence", 1.0)),
            "freshness": float(memory_input.get("freshness", 1.0)),
            "created_at": memory_input.get("created_at", now),
            "updated_at": now,
            "observed_at": memory_input.get("observed_at", now),
            "source_ref": memory_input.get("source_ref"),
            "evidence_ref": memory_input.get("evidence_ref"),
            "embedding_json": self._encode_embedding(memory_input.get("embedding")),
            "metadata_json": self.store.dumps(metadata),
        }
        with self.store.connection() as conn:
            conn.execute(
                """
                INSERT OR REPLACE INTO memories (
                    id, schema_version, type, subtype, scope, status, project_id, repo_id, agent_id, origin_agent,
                    run_id, task_id, url, domain, source_kind, title, content, summary, confidence, freshness,
                    created_at, updated_at, observed_at, source_ref, evidence_ref, embedding_json, metadata_json
                ) VALUES (
                    :id, :schema_version, :type, :subtype, :scope, :status, :project_id, :repo_id, :agent_id, :origin_agent,
                    :run_id, :task_id, :url, :domain, :source_kind, :title, :content, :summary, :confidence, :freshness,
                    :created_at, :updated_at, :observed_at, :source_ref, :evidence_ref, :embedding_json, :metadata_json
                )
                """,
                record,
            )
            self._upsert_fts(conn, record)
        self._run_opportunistic_maintenance(limit=1)
        return self.get_memory(record["id"])

    def ingest(self, memory_input: dict[str, Any]) -> dict[str, Any]:
        return self.create_memory(memory_input)

    def capture_session(self, source_ref: str, scope: str, content: str, **metadata: Any) -> dict[str, Any]:
        return self._capture_source(
            trigger_type="session_close",
            source_kind="run",
            source_ref=source_ref,
            scope=scope,
            content=content,
            metadata=metadata,
        )

    def capture_conversation(
        self, source_ref: str, scope: str, content: str, capture_mode: str = "manual_import", **metadata: Any
    ) -> dict[str, Any]:
        metadata = {"capture_mode": capture_mode, **metadata}
        return self._capture_source(
            trigger_type="conversation_capture",
            source_kind="conversation",
            source_ref=source_ref,
            scope=scope,
            content=content,
            metadata=metadata,
        )

    def search(
        self,
        query: str,
        scopes: list[str] | None = None,
        filters: dict[str, Any] | None = None,
        limit: int = 10,
        include_inbox: bool = False,
    ) -> dict[str, Any]:
        filters = filters or {}
        query_terms = [term for term in normalize_text(query).split() if term]
        sql = ["SELECT * FROM memories WHERE 1=1"]
        params: list[Any] = []

        if scopes:
            sql.append(f"AND scope IN ({','.join('?' for _ in scopes)})")
            params.extend(scopes)
        if not include_inbox:
            sql.append("AND status = ?")
            params.append("active")
        params.extend(self._memory_filter_sql(sql, filters))

        with self.store.connection() as conn:
            rows = [self._row_to_memory(row) for row in conn.execute(" ".join(sql), params)]

        results: list[SearchResult] = []
        query_embedding = self._text_embedding(query) if query_terms else []
        for row in rows:
            haystack = normalize_text(" ".join([row["title"], row["summary"], row["content"]]))
            lexical = self._lexical_score(query_terms, haystack)
            semantic = self._semantic_score(query_embedding, row.get("embedding")) if query_terms else 0.0
            facet = self._facet_score(row, filters)
            if query_terms and lexical == 0 and semantic == 0 and facet == 0:
                continue
            score = facet * 0.45 + lexical * 0.3 + semantic * 0.15 + row["confidence"] * 0.06 + row["freshness"] * 0.04
            explanation = (
                f"facets={facet:.2f} lexical={lexical:.2f} semantic={semantic:.2f} "
                f"confidence={row['confidence']:.2f} freshness={row['freshness']:.2f}"
            )
            results.append(SearchResult(row, score, explanation, facet, lexical, semantic))

        ordered = sorted(
            results,
            key=lambda item: (
                item.facet_score,
                item.lexical_score,
                item.semantic_score,
                item.memory["confidence"],
                item.memory["freshness"],
                item.memory["updated_at"],
            ),
            reverse=True,
        )[:limit]
        payload = [item.as_dict() for item in ordered]

        retrieval_id = make_id("ret")
        with self.store.connection() as conn:
            conn.execute(
                "INSERT INTO retrieval_logs (id, query_text, filters_json, results_json, created_at) VALUES (?, ?, ?, ?, ?)",
                (
                    retrieval_id,
                    query,
                    self.store.dumps({"scopes": scopes, "filters": filters, "include_inbox": include_inbox}),
                    self.store.dumps(payload),
                    utcnow(),
                ),
            )
        return {"retrieval_id": retrieval_id, "results": payload}

    def context_for(
        self, project: str | None = None, repo: str | None = None, agent: str | None = None, task: str | None = None
    ) -> dict[str, Any]:
        profile_facts = self.list_memories(status="active", scope="global", memory_type="profile", limit=8)
        project_filters = {"project_id": project, "repo_id": repo}
        active_decisions = self.list_memories(
            status="active",
            scope="project" if project else None,
            memory_type="decision",
            project_id=project,
            repo_id=repo,
            limit=8,
        )
        project_memories = self.list_memories(
            status="active",
            project_id=project,
            repo_id=repo,
            limit=12,
        )
        project_memories = [item for item in project_memories if item["type"] in {"project", "decision", "artifact", "episode"}][:12]
        recent_episodes = self.list_memories(
            status="active",
            memory_type="episode",
            project_id=project,
            repo_id=repo,
            limit=10,
        )
        task_relevant_artifacts = self.list_memories(
            status="active",
            memory_type="artifact",
            project_id=project,
            repo_id=repo,
            task_id=task,
            limit=10,
        )
        if task and not task_relevant_artifacts:
            task_relevant_artifacts = self.search(task, filters={**project_filters, "type": "artifact"}, limit=5)["results"]
        citations = [
            item["id"]
            for item in [*profile_facts, *project_memories, *active_decisions, *recent_episodes]
            if isinstance(item, dict) and "id" in item
        ]
        return {
            "profile_facts": profile_facts,
            "project_memories": project_memories,
            "active_decisions": active_decisions,
            "recent_episodes": recent_episodes,
            "task_relevant_artifacts": task_relevant_artifacts,
            "citations": citations,
            "agent_id": agent,
        }

    def consolidate(self, inbox_items: list[str]) -> dict[str, Any]:
        promoted: list[str] = []
        linked: list[dict[str, str]] = []
        with self.store.connection() as conn:
            for memory_id in inbox_items:
                row = conn.execute("SELECT * FROM memories WHERE id = ?", (memory_id,)).fetchone()
                if row is None:
                    continue
                memory = self._row_to_memory(row)
                duplicate = conn.execute(
                    """
                    SELECT id FROM memories
                    WHERE id != ?
                      AND status = 'active'
                      AND type = ?
                      AND scope = ?
                      AND content = ?
                    LIMIT 1
                    """,
                    (memory_id, memory["type"], memory["scope"], memory["content"]),
                ).fetchone()
                if duplicate:
                    link_id = make_id("lnk")
                    conn.execute(
                        """
                        INSERT INTO memory_links (id, from_memory_id, to_memory_id, relation, created_at, metadata_json)
                        VALUES (?, ?, ?, ?, ?, ?)
                        """,
                        (
                            link_id,
                            memory_id,
                            duplicate["id"],
                            "related_to",
                            utcnow(),
                            self.store.dumps({"reason": "exact_duplicate"}),
                        ),
                    )
                    conn.execute("UPDATE memories SET status = ?, updated_at = ? WHERE id = ?", ("archived", utcnow(), memory_id))
                    linked.append({"from": memory_id, "to": duplicate["id"], "relation": "related_to"})
                    continue
                conn.execute("UPDATE memories SET status = ?, updated_at = ? WHERE id = ?", ("active", utcnow(), memory_id))
                promoted.append(memory_id)
        return {"promoted": promoted, "linked": linked}

    def consolidate_candidates(self, *, config: dict[str, Any]) -> dict[str, Any]:
        stats = {
            "candidates": 0,
            "promoted": 0,
            "archived": 0,
            "linked": 0,
            "conflicts": 0,
            "llm_summaries": 0,
        }
        age_days = int(config.get("candidate_age_days", 7))
        max_candidates = int(config.get("max_candidates", 200))
        high_threshold = float(config.get("dedupe_high_threshold", 0.9))
        mid_threshold = float(config.get("dedupe_mid_threshold", 0.75))
        promote_threshold = float(config.get("promote_confidence", 0.7))
        conflict_threshold = float(config.get("conflict_threshold", 0.55))
        llm_enabled = bool(config.get("llm_enabled", False))
        llm_provider = config.get("llm_provider") or "opencode"
        llm_command = config.get("llm_command") or ["opencode", "-m", "MiniMax M2.5 Free"]
        cutoff = (datetime.now(UTC) - timedelta(days=age_days)).isoformat()

        with self.store.connection() as conn:
            rows = conn.execute(
                """
                SELECT * FROM memories
                WHERE (
                    status = 'inbox'
                    OR (type = 'episode' AND status = 'active')
                )
                  AND created_at <= ?
                ORDER BY created_at ASC
                LIMIT ?
                """,
                (cutoff, max_candidates),
            ).fetchall()

            candidates = [self._row_to_memory(row) for row in rows]
            stats["candidates"] = len(candidates)

            for candidate in candidates:
                related = conn.execute(
                    """
                    SELECT * FROM memories
                    WHERE status = 'active'
                      AND id != ?
                      AND type = ?
                      AND scope = ?
                    ORDER BY updated_at DESC
                    LIMIT 50
                    """,
                    (candidate["id"], candidate["type"], candidate["scope"]),
                ).fetchall()
                related_memories = [self._row_to_memory(row) for row in related]

                best_match = None
                best_score = 0.0
                for other in related_memories:
                    lexical = self._lexical_score(
                        normalize_text(candidate["content"]).split(),
                        normalize_text(other["content"]),
                    )
                    semantic = self._semantic_score(candidate.get("embedding") or [], other.get("embedding"))
                    score = lexical if semantic == 0.0 else (lexical * 0.6 + semantic * 0.4)
                    if score > best_score:
                        best_score = score
                        best_match = other

                if best_match and candidate["content"] == best_match["content"]:
                    self._link_memories(conn, candidate["id"], best_match["id"], "related_to", "exact_duplicate")
                    conn.execute(
                        "UPDATE memories SET status = ?, updated_at = ? WHERE id = ?",
                        ("archived", utcnow(), candidate["id"]),
                    )
                    stats["archived"] += 1
                    stats["linked"] += 1
                    continue

                if best_match and best_score >= high_threshold:
                    self._link_memories(conn, candidate["id"], best_match["id"], "related_to", "high_similarity")
                    conn.execute(
                        "UPDATE memories SET status = ?, updated_at = ? WHERE id = ?",
                        ("archived", utcnow(), candidate["id"]),
                    )
                    stats["archived"] += 1
                    stats["linked"] += 1
                    continue

                if best_match and best_score >= mid_threshold:
                    if llm_enabled:
                        synthesized = self._llm_summarize_pair(
                            candidate,
                            best_match,
                            provider=llm_provider,
                            command=llm_command,
                        )
                        if synthesized:
                            conn.execute(
                                "UPDATE memories SET summary = ?, updated_at = ?, metadata_json = ? WHERE id = ?",
                                (
                                    synthesized,
                                    utcnow(),
                                    self.store.dumps(
                                        {
                                            **candidate.get("metadata", {}),

[1286 more lines in file. Use offset=421 to continue.]


                                            "synthesis_sources": [candidate["id"], best_match["id"]],
                                            "synthesis_provider": llm_provider,
                                            "synthesis_command": llm_command,
                                        }
                                    ),
                                    candidate["id"],
                                ),
                            )
                            stats["llm_summaries"] += 1
                    self._link_memories(conn, candidate["id"], best_match["id"], "related_to", "mid_similarity")
                    stats["linked"] += 1

                if best_match and best_score >= conflict_threshold and self._conflict_signal(candidate, best_match):
                    self._record_conflict(conn, candidate["id"], best_match["id"], "potential_contradiction")
                    self._link_memories(conn, candidate["id"], best_match["id"], "contradicts", "heuristic")
                    stats["conflicts"] += 1

                if candidate["status"] == "inbox" and candidate.get("confidence", 0.0) >= promote_threshold:
                    conn.execute(
                        "UPDATE memories SET status = ?, updated_at = ? WHERE id = ?",
                        ("active", utcnow(), candidate["id"]),
                    )
                    stats["promoted"] += 1

        return stats

    def upsert_profile_fact(
        self, title: str, content: str, evidence_ref: str | None = None, source_ref: str | None = None
    ) -> dict[str, Any]:
        return self.create_memory(
            {
                "type": "profile",
                "scope": "global",
                "title": title,
                "content": content,
                "summary": self._make_summary(content),
                "source_ref": source_ref,
                "evidence_ref": evidence_ref,
                "metadata": {"kind": "profile_fact"},
            }
        )

    def explain(self, identifier: str) -> dict[str, Any]:
        with self.store.connection() as conn:
            retrieval = conn.execute("SELECT * FROM retrieval_logs WHERE id = ?", (identifier,)).fetchone()
            if retrieval:
                return {
                    "kind": "retrieval",
                    "id": retrieval["id"],
                    "query_text": retrieval["query_text"],
                    "filters": self.store.loads(retrieval["filters_json"]),
                    "results": self.store.loads(retrieval["results_json"]),
                }
            memory = conn.execute("SELECT * FROM memories WHERE id = ?", (identifier,)).fetchone()
            if memory:
                mem = self._row_to_memory(memory)
                sources = conn.execute("SELECT * FROM sources WHERE source_ref = ? OR id = ?", (mem["source_ref"], mem["source_ref"])).fetchall()
                return {"kind": "memory", "memory": mem, "sources": [dict(source) for source in sources]}
        raise KeyError(f"Unknown identifier: {identifier}")

    def export(self) -> dict[str, Any]:
        tables = [
            "projects",
            "repos",
            "sources",
            "ingestion_events",
            "memories",
            "memory_links",
            "memory_conflicts",
            "retrieval_logs",
            "maintenance_jobs",
            "maintenance_runs",
            "profiles",
            "tasks",
            "task_runs",
            "artifacts",
        ]
        exported: dict[str, list[dict[str, Any]]] = {}
        with self.store.connection() as conn:
            for table in tables:
                rows = conn.execute(f"SELECT * FROM {table}").fetchall()
                exported[table] = [dict(row) for row in rows]
        return exported

    def import_data(self, payload: dict[str, Any]) -> None:
        tables = [
            "projects",
            "repos",
            "sources",
            "ingestion_events",
            "memories",
            "memory_links",
            "memory_conflicts",
            "retrieval_logs",
            "maintenance_jobs",
            "maintenance_runs",
            "profiles",
            "tasks",
            "task_runs",
            "artifacts",
        ]
        with self.store.connection() as conn:
            for table in tables:
                rows = payload.get(table, [])
                if not rows:
                    continue
                columns = list(rows[0].keys())
                placeholder = ",".join("?" for _ in columns)
                sql = f"INSERT OR REPLACE INTO {table} ({','.join(columns)}) VALUES ({placeholder})"
                conn.executemany(sql, [tuple(row.get(col) for col in columns) for row in rows])
                if table == "memories":
                    for row in rows:
                        self._upsert_fts(conn, row)
        self.migrate_v2()
        self._run_opportunistic_maintenance(limit=10)

    def get_memory(self, memory_id: str) -> dict[str, Any]:
        with self.store.connection() as conn:
            row = conn.execute("SELECT * FROM memories WHERE id = ?", (memory_id,)).fetchone()
            if row is None:
                raise KeyError(memory_id)
            return self._row_to_memory(row)

    def list_memories(
        self,
        *,
        status: str | None = "active",
        scope: str | None = None,
        memory_type: str | None = None,
        subtype: str | None = None,
        project_id: str | None = None,
        repo_id: str | None = None,
        source_ref: str | None = None,
        evidence_ref: str | None = None,
        run_id: str | None = None,
        task_id: str | None = None,
        origin_agent: str | None = None,
        url: str | None = None,
        domain: str | None = None,
        metadata: dict[str, Any] | None = None,
        limit: int = 50,
    ) -> list[dict[str, Any]]:
        sql = ["SELECT * FROM memories WHERE 1=1"]
        params: list[Any] = []
        if status is not None:
            sql.append("AND status = ?")
            params.append(status)
        if scope is not None:
            sql.append("AND scope = ?")
            params.append(scope)
        filters = {
            "type": memory_type,
            "subtype": subtype,
            "project_id": project_id,
            "repo_id": repo_id,
            "source_ref": source_ref,
            "evidence_ref": evidence_ref,
            "run_id": run_id,
            "task_id": task_id,
            "origin_agent": origin_agent,
            "url": url,
            "domain": domain,
        }
        params.extend(self._memory_filter_sql(sql, {key: value for key, value in filters.items() if value is not None}))
        for key, value in (metadata or {}).items():
            sql.append("AND json_extract(metadata_json, ?) = ?")
            params.extend((f"$.{key}", value))
        sql.append("ORDER BY updated_at DESC LIMIT ?")
        params.append(limit)
        with self.store.connection() as conn:
            rows = conn.execute(" ".join(sql), params).fetchall()
            return [self._row_to_memory(row) for row in rows]

    def list_pending_events(self) -> list[dict[str, Any]]:
        with self.store.connection() as conn:
            rows = conn.execute("SELECT * FROM ingestion_events WHERE job_state = 'pending' ORDER BY created_at ASC").fetchall()
            return [dict(row) for row in rows]

    def mark_event(self, event_id: str, state: str, outcome: str | None = None, error_message: str | None = None) -> None:
        with self.store.connection() as conn:
            conn.execute(
                """
                UPDATE ingestion_events
                SET job_state = ?, processor_outcome = ?, error_message = ?, updated_at = ?, processed_at = ?
                WHERE id = ?
                """,
                (state, outcome, error_message, utcnow(), utcnow(), event_id),
            )

    def create_project(self, project_id: str, name: str, description: str | None = None) -> None:
        now = utcnow()
        with self.store.connection() as conn:
            conn.execute(
                "INSERT OR REPLACE INTO projects (id, name, description, created_at, updated_at) VALUES (?, ?, ?, ?, ?)",
                (project_id, name, description, now, now),
            )
        self._run_opportunistic_maintenance(limit=1)

    def create_repo(self, repo_id: str, name: str, project_id: str | None = None, path: str | None = None) -> None:
        now = utcnow()
        with self.store.connection() as conn:
            conn.execute(
                "INSERT OR REPLACE INTO repos (id, project_id, name, path, created_at, updated_at) VALUES (?, ?, ?, ?, ?, ?)",
                (repo_id, project_id, name, path, now, now),
            )
        self._run_opportunistic_maintenance(limit=1)

    def create_task(
        self,
        *,
        title: str,
        intent: str,
        kind: str = "task",
        status: str = "open",
        priority: int = 3,
        project_id: str | None = None,
        repo_id: str | None = None,
        parent_task_id: str | None = None,
        origin: str | None = None,
        owner_agent: str | None = None,
        blocked_reason: str | None = None,
        requires_human_input: bool = False,
        due_at: str | None = None,
        metadata: dict[str, Any] | None = None,
        task_id: str | None = None,
        run_id: str | None = None,
    ) -> dict[str, Any]:
        now = utcnow()
        record = {
            "id": task_id or make_id("task"),
            "schema_version": SCHEMA_VERSION,
            "run_id": run_id,
            "title": title,
            "intent": intent,
            "kind": kind,
            "status": status,
            "priority": priority,
            "project_id": project_id,
            "repo_id": repo_id,
            "parent_task_id": parent_task_id,
            "origin": origin,
            "owner_agent": owner_agent,
            "blocked_reason": blocked_reason,
            "requires_human_input": 1 if requires_human_input else 0,
            "due_at": due_at,
            "created_at": now,
            "updated_at": now,
            "metadata_json": self.store.dumps(metadata or {}),
        }
        with self.store.connection() as conn:
            conn.execute(
                """
                INSERT INTO tasks (
                    id, schema_version, run_id, title, intent, kind, status, priority, project_id, repo_id,
                    parent_task_id, origin, owner_agent, blocked_reason, requires_human_input, due_at,
                    created_at, updated_at, metadata_json
                ) VALUES (
                    :id, :schema_version, :run_id, :title, :intent, :kind, :status, :priority, :project_id, :repo_id,
                    :parent_task_id, :origin, :owner_agent, :blocked_reason, :requires_human_input, :due_at,
                    :created_at, :updated_at, :metadata_json

[1026 more lines in file. Use offset=681 to continue.]


**Evaluating memory management**

I’m thinking I might not need a merge mechanism beyond simply promoting existing links. I should inspect the process of creating or updating memory and validate status. I'll look for conflicts and consider migrating status values. Proper promotion could leverage existing candidates and various states like inbox, active, and archived. Even though there's no canonical merge, it’s important to align on current consolidation and links. Someone has asked for an inspection, so I need to summarize and possibly propose corrections. Checking the maintenance configuration, API, and tests around consolidation will help.

cd /mnt/rpi/agents-database && grep -n "consolidate_candidates\|memory_conflicts\|maintenance_jobs\|superseded\|contradicted\|inbox" tests/test_memory_service.py | sed -n '1,220p'


48:def test_captured_conversation_stays_inbox_until_consolidated(tmp_path: Path) -> None:
57:    assert memory["status"] == "inbox"
59:    search_without_inbox = service.search("active machine", scopes=["project"])
60:    assert search_without_inbox["results"] == []
316:def test_maintenance_jobs_exist_by_default(tmp_path: Path) -> None:
319:    jobs = {job["id"] for job in service.list_maintenance_jobs()}
384:            "id": "dec_inbox",
387:            "status": "inbox",
392:    service.consolidate_candidates(
404:        conflicts = conn.execute("SELECT COUNT(*) AS count FROM memory_conflicts").fetchone()["count"]


        patch_code, patched = _request_json(
            f"{base_url}/api/tasks/{task['id']}",
            method="PATCH",
            payload={"status": "in_progress", "run_id": run["id"]},
        )
        assert patch_code == 200
        assert patched["status"] == "in_progress"
        assert patched["run_id"] == run["id"]

        bundle_code, bundle = _request_json(f"{base_url}/api/tasks/{task['id']}/bundle")
        assert bundle_code == 200
        assert bundle["task"]["id"] == task["id"]
        assert bundle["runs"][0]["id"] == run["id"]
    finally:
        server.shutdown()
        server.server_close()

def test_http_api_supports_search_by_id(tmp_path: Path) -> None:
    service = make_service(tmp_path)
    service.create_memory(
        {
            "id": "mem_lookup_1",
            "type": "artifact",
            "scope": "global",
            "title": "Lookup memory",
            "content": "Direct id lookup should work.",
        }
    )
    server, base_url = _start_api(service)
    try:
        status_code, payload = _request_json(f"{base_url}/api/search?id=mem_lookup_1")
        assert status_code == 200
        assert payload["retrieval_id"] == "mem_lookup_1"
        assert payload["results"][0]["id"] == "mem_lookup_1"
    finally:
        server.shutdown()
        server.server_close()

def test_http_api_rejects_missing_search_query(tmp_path: Path) -> None:
    service = make_service(tmp_path)
    server, base_url = _start_api(service)
    try:
        try:
            _request_json(f"{base_url}/api/search")
        except HTTPError as exc:
            assert exc.code == 400
        else:
            raise AssertionError("Expected HTTP 400 for missing query")
    finally:
        server.shutdown()
        server.server_close()

def test_maintenance_jobs_exist_by_default(tmp_path: Path) -> None:
    service = make_service(tmp_path)

    jobs = {job["id"] for job in service.list_maintenance_jobs()}

    assert "job_consolidation_daily_10" in jobs
    assert "job_consolidation_daily_15" in jobs
    assert "job_consolidation_weekly" in jobs

def test_maintenance_tick_records_run(tmp_path: Path) -> None:
    service = make_service(tmp_path)
    engine = MemoryEngine(service)

    now = "2026-03-20T10:00:00+00:00"
    service.update_maintenance_job("job_consolidation_daily_10", next_due_at=now)
    results = engine.run_due_maintenance(job_id="job_consolidation_daily_10")

    assert results
    job = service.get_maintenance_job("job_consolidation_daily_10")
    runs = service.list_maintenance_runs(job_id="job_consolidation_daily_10")

    assert job["last_status"] == "completed"
    assert runs

def test_service_startup_runs_due_maintenance(tmp_path: Path) -> None:
    db_path = tmp_path / "memory.sqlite3"
    service = MemoryService(str(db_path))
    now = "2026-03-20T10:00:00+00:00"

    service.update_maintenance_job("job_consolidation_daily_10", next_due_at=now)
    restarted = MemoryService(str(db_path))
    job = restarted.get_maintenance_job("job_consolidation_daily_10")
    runs = restarted.list_maintenance_runs(job_id="job_consolidation_daily_10")

    assert job["last_status"] == "completed"
    assert runs

def test_write_operations_trigger_due_maintenance_check(tmp_path: Path) -> None:
    service = make_service(tmp_path)
    now = "2026-03-20T10:00:00+00:00"

    service.update_maintenance_job("job_consolidation_daily_10", next_due_at=now)
    service.create_project("proj_alpha", "Alpha")

    job = service.get_maintenance_job("job_consolidation_daily_10")
    runs = service.list_maintenance_runs(job_id="job_consolidation_daily_10")

    assert job["last_status"] == "completed"
    assert runs

def test_conflicts_only_for_selected_types(tmp_path: Path) -> None:
    service = make_service(tmp_path)
    service.create_memory(
        {
            "id": "dec_active",
            "type": "decision",
            "scope": "project",
            "status": "active",
            "title": "Decision A",
            "content": "Feature X is enabled for launch.",
        }
    )
    service.create_memory(
        {
            "id": "dec_inbox",
            "type": "decision",
            "scope": "project",
            "status": "inbox",
            "title": "Decision B",
            "content": "Feature X is not enabled for launch.",
        }
    )
    service.consolidate_candidates(
        config={
            "candidate_age_days": 0,
            "max_candidates": 10,
            "conflict_threshold": 0.1,
            "dedupe_high_threshold": 0.99,
            "dedupe_mid_threshold": 0.99,
            "promote_confidence": 1.0,
            "llm_enabled": False,
        }
    )
    with service.store.connection() as conn:
        conflicts = conn.execute("SELECT COUNT(*) AS count FROM memory_conflicts").fetchone()["count"]

    assert conflicts == 1

def test_migrate_v2_rewrites_legacy_rows_and_drops_legacy_tables(tmp_path: Path) -> None:
    service = make_service(tmp_path)
    with service.store.connection() as conn:
        conn.execute(
            """
            INSERT INTO memories (
                id, type, scope, status, source_kind, title, content, summary, confidence, freshness,
                created_at, updated_at, observed_at, source_ref, evidence_ref, embedding_json, metadata_json
            ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
            """,
            (
                "legacy_run",
                "episode",
                "global",
                "active",
                "run",
                "Research run: Legacy cleanup",
                "Goal: Legacy cleanup\nScope: Repo\nAssumptions: None\nSummary: Pending",
                "Pending",
                0.9,
                0.8,
                "2026-03-19T00:00:00+00:00",
                "2026-03-19T00:00:00+00:00",
                "2026-03-19T00:00:00+00:00",
                "personal-agent:run:run-123",
                "personal-agent:run:run-123",
                None,
                service.store.dumps({"legacy_system": "personal-agent", "legacy_kind": "research_run", "legacy_run_id": "run-123"}),
            ),
        )
        conn.execute(
            """
            INSERT INTO tasks (
                id, title, intent, kind, status, priority, created_at, updated_at, metadata_json
            ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
            """,
            (
                "task_legacy",
                "Legacy task",
                "migrate me",
                "task",
                "open",
                3,
                "2026-03-19T00:00:00+00:00",
                "2026-03-19T00:00:00+00:00",
                service.store.dumps({"legacy_kind": "task", "legacy_run_id": "run-123"}),
            ),
        )
        conn.execute(
            """
            CREATE TABLE approvals (
                id TEXT PRIMARY KEY,
                task_id TEXT NOT NULL,
                kind TEXT NOT NULL,
                status TEXT NOT NULL,
                risk_level TEXT NOT NULL,
                payload_json TEXT NOT NULL,
                resolution_note TEXT,
                requested_at TEXT NOT NULL,
                resolved_at TEXT,
                created_at TEXT NOT NULL,
                updated_at TEXT NOT NULL
            )
            """
        )
        conn.execute(
            """
            INSERT INTO approvals (
                id, task_id, kind, status, risk_level, payload_json, resolution_note, requested_at, resolved_at, created_at, updated_at
            ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
            """,

[74 more lines in file. Use offset=480 to continue.]


**Considering consolidation tests**

I realize I might need to gather more information about consolidation tests. It seems important to understand their purpose in evaluating soil or material properties. I wonder what specific aspects I should focus on or if there are common methodologies I should look into. Exploring examples or guidelines could really help in grasping the key concepts surrounding these tests. It's a good idea to deepen my understanding before moving forward!


    service.update_maintenance_job("job_consolidation_daily_10", next_due_at=now)
    service.create_project("proj_alpha", "Alpha")

    job = service.get_maintenance_job("job_consolidation_daily_10")
    runs = service.list_maintenance_runs(job_id="job_consolidation_daily_10")

    assert job["last_status"] == "completed"
    assert runs

def test_conflicts_only_for_selected_types(tmp_path: Path) -> None:
    service = make_service(tmp_path)
    service.create_memory(
        {
            "id": "dec_active",
            "type": "decision",
            "scope": "project",
            "status": "active",
            "title": "Decision A",
            "content": "Feature X is enabled for launch.",
        }
    )
    service.create_memory(
        {
            "id": "dec_inbox",
            "type": "decision",
            "scope": "project",
            "status": "inbox",
            "title": "Decision B",
            "content": "Feature X is not enabled for launch.",
        }
    )
    service.consolidate_candidates(
        config={
            "candidate_age_days": 0,
            "max_candidates": 10,
            "conflict_threshold": 0.1,
            "dedupe_high_threshold": 0.99,
            "dedupe_mid_threshold": 0.99,
            "promote_confidence": 1.0,
            "llm_enabled": False,
        }
    )
    with service.store.connection() as conn:
        conflicts = conn.execute("SELECT COUNT(*) AS count FROM memory_conflicts").fetchone()["count"]

    assert conflicts == 1

def test_migrate_v2_rewrites_legacy_rows_and_drops_legacy_tables(tmp_path: Path) -> None:
    service = make_service(tmp_path)
    with service.store.connection() as conn:
        conn.execute(
            """
            INSERT INTO memories (
                id, type, scope, status, source_kind, title, content, summary, confidence, freshness,
                created_at, updated_at, observed_at, source_ref, evidence_ref, embedding_json, metadata_json
            ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
            """,
            (
                "legacy_run",
                "episode",
                "global",
                "active",
                "run",
                "Research run: Legacy cleanup",
                "Goal: Legacy cleanup\nScope: Repo\nAssumptions: None\nSummary: Pending",
                "Pending",
                0.9,
                0.8,
                "2026-03-19T00:00:00+00:00",
                "2026-03-19T00:00:00+00:00",
                "2026-03-19T00:00:00+00:00",
                "personal-agent:run:run-123",
                "personal-agent:run:run-123",
                None,
                service.store.dumps({"legacy_system": "personal-agent", "legacy_kind": "research_run", "legacy_run_id": "run-123"}),
            ),
        )
        conn.execute(
            """
            INSERT INTO tasks (
                id, title, intent, kind, status, priority, created_at, updated_at, metadata_json
            ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
            """,
            (
                "task_legacy",
                "Legacy task",
                "migrate me",
                "task",
                "open",
                3,
                "2026-03-19T00:00:00+00:00",
                "2026-03-19T00:00:00+00:00",
                service.store.dumps({"legacy_kind": "task", "legacy_run_id": "run-123"}),
            ),
        )
        conn.execute(
            """
            CREATE TABLE approvals (
                id TEXT PRIMARY KEY,
                task_id TEXT NOT NULL,
                kind TEXT NOT NULL,
                status TEXT NOT NULL,
                risk_level TEXT NOT NULL,
                payload_json TEXT NOT NULL,
                resolution_note TEXT,
                requested_at TEXT NOT NULL,
                resolved_at TEXT,
                created_at TEXT NOT NULL,
                updated_at TEXT NOT NULL
            )
            """
        )
        conn.execute(
            """
            INSERT INTO approvals (
                id, task_id, kind, status, risk_level, payload_json, resolution_note, requested_at, resolved_at, created_at, updated_at
            ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
            """,

[74 more lines in file. Use offset=480 to continue.]


**Advising on system alignment**

I believe I have enough information to provide advice. I should mention that the existing system already includes consolidation and conflict links, which could make our scripts redundant. It seems I might need to either revert or align with these existing features. I could ask the user if they want me to adapt to these hooks. They did ask if it was implemented, and I confirmed it. Now, I should share my findings and propose the next steps concisely.

Sí. Y ahora ya vi el sistema real.

Lo que existe en `agents-database`:
- `SQLite` canon
- `memory-service`
- `memory-engine`
- ingest/search/context/export/import
- `consolidate_candidates(config)` ya hace:
  - dedup heurístico
  - links `related_to` / `contradicts`
  - promoción `inbox -> active`
  - conflicto explícito en `memory_conflicts`
- scheduler de mantenimiento ya existe:
  - `maintenance_jobs`
  - `maintenance_runs`
  - ticks oportunistas
  - jobs por defecto `daily_10`, `daily_15`, `weekly`

Conclusión:
- no hacía falta inventar otro consolidator/promoter externo como si el core no existiera.
- mi slice anterior fue sobre la interfaz pública, pero no respetó bien el diseño interno del sistema.
- el contrato correcto para extender es **dentro de `MemoryService` / `MemoryEngine`**, no afuera con scripts paralelos.

Lo que voy a hacer ahora, bien:
1. adaptar el pipeline de `pi-config` a los contratos reales
2. eliminar o neutralizar lo que quedó duplicado
3. llevar la nueva semántica a `agents-database` usando:
   - `consolidate_candidates`
   - `memory_links`
   - `memory_conflicts`
   - `maintenance_jobs`
4. no meter más capas paralelas hasta encajar con el core

Si querés, sigo ya con la corrección y te dejo el sistema alineado, no inventado encima.


ok. considera tambien documentar agents-database en el proceso


**Considering documentation updates**

I need to continue by inspecting the relevant documents that may need updates. It looks like I should modify the agents-database and possibly the pi-config documents. Also, I might need to write to memory based on decisions we make. Let's focus on responding briefly and taking action on these updates! I want to ensure everything is aligned and functioning properly moving forward.


# Agents Database in System V1

`agents-database` is the durable store for System V1.

Canonical topology and machine bootstrap live in `~/personal-agent/docs/system-v1.md`.

## V1 Role

- canonical shared DB for memories, operational tasks, task runs, approvals, artifacts, and handoffs
- resumability anchor for multi-repo work
- migration target for legacy `personal-agent` state

## Resume From Shared DB

Use the local CLI against `data/shared-agent-memory.sqlite3`:

```bash
python3 -m shared_agent_memory.cli --db data/shared-agent-memory.sqlite3 task-bundle task_<id>
python3 -m shared_agent_memory.cli --db data/shared-agent-memory.sqlite3 list-memories \
  --source-ref personal-agent:run:<run-id>
```

## Runtime Dependencies

- the configured CLI runner must be available for LLM consolidation steps (MiniMax M2.5 Free).
- local verification tools live in `./.venv` for this repo

## Maintenance

```bash
.venv/bin/python -m shared_agent_memory.cli --db data/shared-agent-memory.sqlite3 maintenance-status
.venv/bin/python -m shared_agent_memory.cli --db data/shared-agent-memory.sqlite3 maintenance-tick --limit 10
.venv/bin/python -m shared_agent_memory.cli --db data/shared-agent-memory.sqlite3 maintenance-daemon --interval-seconds 900
```

Versioned service definitions:

- `systemd/user/agents-database-maintenance.service`
- `scripts/run-maintenance-daemon.sh`

Installed Linux user service:

- `~/.config/systemd/user/agents-database-maintenance.service`
- enabled with `systemctl --user enable --now agents-database-maintenance.service`
- `Restart=always` with `RestartSec=10`
- `loginctl show-user sebas --property=Linger` is `yes`, so the user service is eligible to come back after reboot without an interactive login

Operational checks:

```bash
systemctl --user status agents-database-maintenance.service --no-pager
systemctl --user show agents-database-maintenance.service --property=NRestarts,ExecMainPID,SubState
journalctl --user -u agents-database-maintenance.service -n 50 --no-pager
```

macOS launchd template:

- `docs/launchd/agents-database-maintenance.plist`

Current Linux runtime notes:

- the daemon now starts through `scripts/run-maintenance-daemon.sh`
- the wrapper exports `PYTHONPATH` for the repo checkout and prefers `./.venv/bin/python` when present
- the service was verified to auto-restart after a manual `systemctl --user kill agents-database-maintenance.service`
- `MemoryService` also performs opportunistic catch-up checks on startup and after normal write operations, so due jobs do not depend exclusively on the long-lived daemon

## Boundary

- new durable work: shared DB only
- legacy `~/personal-agent/data/personal-agent.sqlite3`: transition-only inspection and migration source

## RFCs

- `docs/memory-consolidation-rfc.md`


Offset 261 is beyond end of file (237 lines total)