# Extension/helper usage tracking assessment

## Current state
There is no general durable usage/success tracking layer for Pi extensions and local helpers.

### What exists
- `subagent` tracks token/cost/turn usage for a single invocation and shows it in the UI.
- `pi-job-*` and `pi-loop` store per-run artifacts, status, exit code, logs, and timestamps.
- session memory pipeline stores conversation/session summaries, not operational success metrics.
- nginx portal exposes active pi-job count, not helper success telemetry.

### What does not exist
- no shared event log for extension/helper invocations
- no per-tool success/failure counters
- no latency or exit-code rollups across tools
- no dashboard/report for "which helpers get used" or "which ones fail often"

## Practical implication
Right now you can inspect some tools manually:
- `subagent`: visible ad hoc in-session
- `pi-job-*`: inspect per job
- `pi-loop`: inspect per run

But you cannot answer cheaply:
- which extension/helper is used most
- which one fails most
- which one is flaky by task/repo/context
- whether a helper improves completion rate

## Best minimal design
Do not build big telemetry first.
Add one append-only local event log.

### Proposed file
- `/home/sebas/runtime/tool-usage/events.jsonl`

### One event per invocation
Fields:
- `ts`
- `kind`: `extension_tool` | `extension_command` | `script` | `job` | `loop`
- `name`
- `cwd`
- `scope`: `global` | `project`
- `session_id` if available
- `task_id` if known
- `status`: `started` | `succeeded` | `failed` | `cancelled`
- `duration_ms`
- `exit_code` for scripts/jobs
- `error_summary` short text
- optional `usage` object for tokens/cost when available

## First targets
Track these first:
- `questionnaire`
- `subagent`
- `pi-job-start` / runner
- `pi-loop`
- `remember-session`
- `switch-session`

Reason: highest leverage and already semi-structured.

## Reporting layer
Then add one small report command:
- `bin/tool-usage-report`

Outputs:
- usage count by tool
- success rate by tool
- avg duration by tool
- recent failures with short error text

## Implementation shape
### Phase 1
Instrument shell helpers:
- `pi-job-runner`
- `pi-loop`
- maybe wrapper logging in `pi-job-start`

Easy because they already have status/exit timestamps.

### Phase 2
Instrument extension tools/commands:
- add a tiny shared TS logger helper under `extensions/_shared/`
- call it from `subagent`, `questionnaire`, `session-memory-dispatcher`, `session-switch`

### Phase 3
Add reducer/report command.

## Non-goals
- no external telemetry service
- no heavy metrics DB
- no always-on dashboard first
- no user tracking/product analytics style system

## Recommendation
Yes, you should add this.
But keep it file-based and append-only first.
That fits the current repo and keeps portability high.