OpenTokenMini
Get the code →Get our help →
For agents
02-measure/collector/AGENTS.md · raw · GitHub

collector

OTLP/HTTP receiver plus normalizer. collector.py takes POST /v1/logs|metrics|traces (JSON or protobuf, optional gzip), redacts content, hashes identity and writes one raw record per request. normalize.py turns raw logs/traces into events (source otel) at <data dir>/events/otel.jsonl. Rules and worked examples: METHOD.md (N-* IDs).

harness --OTLP--> collector.py --sink--> <data>/otel/<signal>-<UTC day>.jsonl --normalize.py--> <data>/events/otel.jsonl

Run

python3 02-measure/collector/collector.py                         # 127.0.0.1:6979 (OTM_PORT or config collector.port)
OTM_PORT=7861 python3 02-measure/collector/collector.py --out /tmp/otm   # scratch instance, other data root
python3 02-measure/collector/normalize.py --days 7 --harness codex       # --in RAW_DIR, --out EVENTS_DIR
curl 127.0.0.1:6979/health                              # {"ok", "protobuf", "auth", "sink"}

Point a harness at it with OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:6979. Responses: 200 {}, 400 bad body, 401 missing bearer, 413 over MAX_BODY, 500 sink error. Tests: python3 -m unittest discover -s 02-measure/tests -p test_collector.py. Check output: python3 02-measure/schemas/check.py <file> raw|event.

Reconfigure

01-deploy/config.json -> collector: sink (connector name), bind, port (null = OTM_PORT or 6979), token_env (env var holding a bearer token; null = no auth), hash_identity (bool). Registry: SINKS in connectors.py, name -> write(record: dict, cfg: dict) -> None; raise to answer 500. Existing: files (default), stdout, forward_otlp (re-sends redacted OTLP JSON to $OTM_FORWARD_URL).

Worked example, print records instead of writing files: set "collector": {"sink": "stdout"} and run python3 02-measure/collector/collector.py > raw.jsonl. Adding a sink foo: write def foo(record, cfg) in connectors.py and add "foo": foo to SINKS.

Pitfalls: every stage has a connectors.py; collector.py loads its siblings by file path (_load), so do the same in tests. Bodies that cannot be decoded are dropped and summarized as undecoded (N-undecoded); KEEP_RAW_UNDECODED opts into unredacted data_b64. Claude Code's ~/.claude/settings.json env overrides process env, so a per-run OTEL_EXPORTER_OTLP_ENDPOINT is ignored when the settings set one.

Adapt, trade-offs, decisions for the human

  • Adapt: bind address, port, auth (token_env) and sink to where the data should live (01-deploy/); a forward_otlp sink or an existing OTel backend can replace the file store.
  • Trade-offs: stdlib only, a single process, files per UTC day. Simple to read and move, but no retention, rotation or HA. Redaction and identity hashing happen at ingest, so raw content is never stored and cannot be recovered later.
  • Decisions for the human: where it runs and who can reach it; whether to hash identity (hash_identity); whether to keep undecodable bodies (KEEP_RAW_UNDECODED, unredacted).