# Sidecar integration and portability plan

## Core answer
Yes, you are using sidecars.
That is not the problem.
The problem is whether they are:
- canonical
- reinstallable
- documented
- backed up
- separable from this one machine's accidental state

Good system shape is not "everything inside Pi".
Good shape is:
- control plane in repo
- sidecars explicit
- runtime ephemeral
- durable data backed up
- bootstrap path real

## Current sidecar classes

### 1. Canonical control-plane sidecars
These should stay sidecars.
Examples:
- `agents-database`
- session-memory dispatcher/sweeper/analyzer
- `pi-telegram.service`
- helper CLIs in `bin/`
- phase-machine runners
- tool-usage telemetry

Reason:
These are orchestration/infrastructure concerns, not Pi extension concerns.

### 2. Pi-adjacent extension layer
These belong in `pi-config` and are already mostly correct.
Examples:
- `subagent`
- `questionnaire`
- `session-switch`
- `session-memory-dispatcher`

Reason:
These shape agent behavior directly.

### 3. Host integration layer
These are the main portability risk.
Examples:
- systemd user services
- nginx routes
- copyparty exposure
- shell profile helpers like `pitmux`
- machine PATH assumptions
- local install state like Bun

Reason:
These often exist partly outside the repo and partly in memory/docs.

### 4. Durable data/services outside repo
These are acceptable, but need explicit restore strategy.
Examples:
- `/home/sebas/agents-database`
- `/home/sebas/runtime`
- `/home/sebas/work`
- service-owned logs/state

Reason:
Not everything should live in git.
But it must have a declared role and backup policy.

## What should not be forced into Pi
Do not try to collapse these into Pi core:
- memory DB service
- system bootstrap
- nginx/systemd exposure
- file server
- local batch tooling
- evidence/output hosting

That would make the system less truthful and harder to restore.

## Main machine-loss risks today

### Risk 1: bootstrap gap
Repo exists, but full machine restore still depends on remembered/manual setup.

### Risk 2: service definitions split
Some service install scripts exist, but not one canonical bootstrap flow.

### Risk 3: external mutable state
Important data lives outside repo:
- agents database
- runtime artifacts
- task workspaces
- outputs

That is fine only if backup/restore policy is explicit.

### Risk 4: host config drift
Nginx, tmux shell helpers, environment paths, and live services can drift away from repo docs.

### Risk 5: package/runtime assumptions
Bun, Pi packages, system packages, copyparty, nginx, Python envs, and other tooling may exist on the machine without one restorable manifest path.

## Portability target
The realistic target is not "clone repo and everything instantly works".
The right target is:

### Tier A: control-plane restore
From GitHub repo alone, restore:
- docs
- scripts
- extensions
- task system
- phase-machine tooling
- telemetry tooling
- service install scripts
- validation tooling

### Tier B: host bootstrap restore
From repo + one bootstrap flow, restore:
- Bun
- Node deps for validation/tooling
- systemd user units
- cron jobs if still needed
- nginx config symlink/copy steps
- Pi config files/paths
- required directories

### Tier C: durable data restore
From backup, restore:
- `agents-database`
- `work/`
- `outputs/`
- selected runtime state if useful

Tier C should never depend on git history alone.

## Recommended integration shape

### Keep current architecture
- Pi extensions for interactive/tool behavior
- sidecar services for memory/ops/runtime
- host integrations for exposure/persistence

Do not over-merge.

### But make repo the installer/orchestrator
`pi-config` should become the canonical bootstrap and audit repo for the whole system.

Meaning:
- every service has install doc/script
- every external path has declared role
- every important package/runtime has installation path
- every durable dataset has backup classification

## Concrete improvements

### 1. Add one bootstrap doc
Create:
- `docs/bootstrap.md`

It should define restore order:
1. clone repo
2. install Bun/Node/Pi prerequisites
3. install repo deps
4. install user services
5. install nginx routes / host integrations
6. restore durable data from backup
7. run validation checks

This is highest leverage.

### 2. Add one bootstrap script layer
Prefer explicit scripts, not one giant magic script.

Suggested:
- `scripts/bootstrap-prereqs.sh`
- `scripts/bootstrap-pi-config.sh`
- `scripts/bootstrap-services.sh`
- `scripts/bootstrap-validate.sh`

Small, composable, rerunnable.

### 3. Add one service manifest doc
Create:
- `docs/service-manifest.md`

Per service record:
- name
- purpose
- code source
- install command
- runtime path
- data path
- log path
- health check
- backup class

This makes sidecars explicit and auditable.

### 4. Add one backup-class manifest
Create:
- `docs/backup-policy.md`

Classify:
- git-only reproducible
- reproducible with bootstrap
- must-back-up durable data
- cache/runtime disposable

Example:
- `pi-config/` -> git
- `agents-database/` -> must back up
- `runtime/pi-loop-runs/` -> disposable or optional archive
- `work/tasks/` -> must back up
- `outputs/` -> likely must back up

### 5. Standardize external host config capture
At least document/install-script these:
- nginx site config
- tmux shell helpers / dotfiles touchpoints
- systemd user services
- copyparty route assumptions
- environment variables required by Telegram

### 6. Add restore validation checklist
After bootstrap/restore, verify:
- `pi --version`
- `bun --version`
- `bun run typecheck` in `pi-config`
- key services healthy
- evidence links resolve
- task-phase tools run
- agents-database status responds

## What should stay outside repo state
Do not commit:
- databases
- logs
- runtime jobs
- outputs with user data by default
- local secrets

But do define where they live and how they are restored.

## Best next implementation sequence

### Step 1
Add docs:
- `docs/bootstrap.md`
- `docs/service-manifest.md`
- `docs/backup-policy.md`

### Step 2
Add bootstrap scripts for:
- prerequisites
- services
- validation

### Step 3
Audit host integrations:
- nginx
- user systemd
- shell/tmux helpers
- telegram env vars

### Step 4
Optionally add one status command:
- `bin/pi-config-audit`

It should report drift/missing pieces.

## Recommendation
Do not remove sidecars.
Do not force everything into Pi.
Do this instead:
- make `pi-config` the canonical bootstrap repo
- make sidecars first-class documented components
- separate reproducible control plane from backed-up durable data
- add restore docs/scripts until machine loss becomes annoying, not catastrophic