# PM Live-Sync protocol (beta)

Bidirectional live mirror between the **PM_E‑1 editor** (web) and a **PM_K‑1 device**
(firmware). When armed, either side can edit a groove, change tempo/volume,
start/stop, or select a set‑list item, and the other side reflects it in real
time.

It rides the **existing USB‑MIDI SysEx channel** (manufacturer `0x7D`) that the
device link already uses for RTC / version / programs / firmware — no new
transport, no new browser permission.

- **Editor side:** implemented in `src/livesync.js` + hooks in `editor-beta.html`.
- **Device side:** to be implemented in `pico-cp/app.py` (this document is the contract).
- **Browser support:** Web MIDI = Chrome / Edge / Firefox (no Safari), same as the existing "Device audio" feature.

---

## 1. Frames

Every message is one SysEx frame:

```
F0 7D <op> <payload ASCII bytes, each 0x00–0x7F> F7
```

`<op>` lives in the free `0x40` block (existing ops: `0x01` RTC, `0x02/0x03`
version, `0x10` programs, `0x21/22/23` firmware, `0x7E/0x7F` NAK/ACK):

| op   | name    | direction        | payload                                   |
|------|---------|------------------|-------------------------------------------|
| 0x40 | HELLO   | either → either  | `<origin>`                                |
| 0x41 | FULL    | either → either  | `<origin>;<seq>;<running>;<sl>;<item>;<patch>` |
| 0x42 | DELTA   | either → either  | `<origin>;<seq>;<evt>`                     |
| 0x43 | BYE     | either → either  | `<origin>`                                |
| 0x44 | SLSYNC  | either → either  | `<origin>;<seq>;<json>` — live set-list **content** merge (§8) |
| 0x45 | LOGSYNC | either → either  | `<origin>;<seq>;<json>` — practice-log entry merge (§9) |

- **Payload is 7‑bit ASCII** — never emit a byte > `0x7F` (it corrupts the SysEx
  stream and, per `build.sh`, would also break the firmware‑update path). All
  share‑language patch strings are already ASCII.
- `<origin>` — a short per‑session id (the editor uses e.g. `e1a2b3c`). Used to
  drop your own echoes (see §4).
- `<seq>` — a monotonically increasing integer per sender. Informational /
  duplicate‑drop; ordering is guaranteed by USB‑MIDI so no reordering logic is
  required.
- `<running>` — `0` or `1`.
- `<sl>` / `<item>` — set‑list and item index of the loaded program, or `-1`.
- `<patch>` — a share‑language patch string (see §3). It contains `;` and `/`,
  so it is **always the tail**: parse the first 5 `;`‑fields, then rejoin the
  rest as the patch.

---

## 2. DELTA event grammar (`<evt>`)

One mutation, no `;` inside. Reuses the share‑language tokens (see
`src/engine.js` / README "Share language").

| evt                          | meaning                                            |
|------------------------------|----------------------------------------------------|
| `play`                       | start transport                                    |
| `stop`                       | stop transport                                     |
| `bpm=<n>`                    | set tempo (clamped to the firmware's BPM range)    |
| `vol=<pct>`                  | master volume, 0–100                               |
| `sel=<sl>/<item>`            | cue/load a set‑list item                           |
| `beat=<lane>/<step>/<level>` | per‑step dynamics; level `0/1/2/3` = mute/normal/accent/ghost |
| `lane=<lane>/<field>/<value>`| lane field edit (see below)                        |

`<lane>` and `<step>` are **0‑based** indices into the current program's lane
list / that lane's step list (same order both sides).

`lane=` fields and values:

| field     | value                                          |
|-----------|------------------------------------------------|
| `sound`   | voice name (`kick`, `snare`, `hatClosed`, …)   |
| `groups`  | grouping string, e.g. `2+2+3`                  |
| `sub`     | subdivision int: 1 / 2 / 3 / 4 / 6             |
| `swing`   | `0` or `1`                                     |
| `gain`    | dB int, e.g. `-3`                              |
| `poly`    | `0` or `1`                                     |
| `enabled` | `0` or `1` (0 = silenced lane)                 |

> Structural changes that re‑shape the lane list (add lane, remove lane,
> reorder) are **not** sent as deltas. Send a fresh **`0x41` FULL** instead — it
> is simpler and self‑healing. The editor does exactly this (a coalesced
> full‑state push ~150 ms after the last structural/practice edit).

---

## 3. What each side emits vs. applies

The two halves are **asymmetric in what they emit but symmetric in what they
apply** — each must apply *every* op/evt listed above.

**Editor emits:**
- fine `0x42` deltas for `play`/`stop`, `bpm`, `vol`, `sel`, `beat`
- a coalesced `0x41` FULL for any lane‑field / add / remove / practice (trainer,
  ramp, segment bars, countdown) edit
- `0x41` FULL on connect and in reply to a received `0x40`
- a coalesced `0x44` SLSYNC on any set‑list **content** change (and on connect) — §8
- a `0x45` LOGSYNC after each logged session (and a full batch on connect) — §9

**Device should emit** (from its on‑device input handlers):
- `play`/`stop` when button A toggles transport
- `bpm=<n>` when the joystick / tap changes tempo (throttle to ≤ ~10/s)
- `sel=<sl>/<item>` on set‑list navigation
- `beat=<lane>/<step>/<level>` on a touch beat edit (`app.py` ~573–625)
- a `0x41` FULL after any lane add/remove/reorder or multi‑field lane edit
- a periodic `0x41` FULL **heartbeat** (~every 3–5 s) — the device is the
  convergence authority (see §4)
- `0x41` FULL in reply to a received `0x40`

The `patch` in a `0x41` is produced by the device's existing program serializer
(the inverse of `parse_program()` in `app.py`). It must round‑trip through the
editor's `patchToSetup()` — i.e. the same grammar already used for
`programs.json` `prog` strings, plus a leading `t<bpm>` and optional `vol<pct>`.

---

## 4. Echo / loop suppression and conflict policy

Two rules keep the mirror from oscillating:

1. **Applying a remote change never re‑broadcasts.** Wrap every apply in an
   "applying remote" flag (the editor uses `_applyingRemote`) and have all of
   your broadcast hooks early‑out while it is set. This is the primary guard.
2. **Drop your own origin.** On receive, if `origin == myOrigin`, ignore the
   frame. (Belt‑and‑suspenders; also lets the editor's `?loopback=1` self‑test
   work by relabeling echoes as a peer.)

**Convergence:** the **device is authoritative**. Its periodic `0x41` heartbeat
is treated as ground truth, so if both sides edited the same field in the same
instant, they reconcile within one heartbeat. To avoid flicker, a receiver
should **diff the incoming `patch` against its current state and skip the
rebuild if they're equal** (the editor does this in `_applyFull`), only
reconciling transport.

This is single‑user‑friendly (last‑writer‑wins per field). True simultaneous
multi‑editor use is out of scope for the beta.

---

## 5. Handshake & lifecycle

```
editor "Live sync" ON ─► 0x40 HELLO ─────────────►  device
                       ◄──────────── 0x41 FULL ◄──  (device's current state)
editor 0x41 FULL ──────────────────────────────►   (editor's current state)
        … steady state: 0x42 deltas both ways, device 0x41 heartbeat …
editor "Live sync" OFF ─► 0x43 BYE ────────────►   device
```

On connect the editor sends **both** a `0x40` (asking for the device's state)
and a `0x41` (offering its own), so whichever side the user considers "source of
truth" wins immediately. A device that boots with sync idle should simply answer
`0x40` with a `0x41` and start emitting deltas once it has heard from a peer.

---

## 6. Firmware checklist (`pico-cp/app.py`)

- [ ] **Dispatch** `0x40/0x41/0x42` in the SysEx handler (~`app.py:1361‑1415`,
      alongside `0x01/0x02/0x10/0x21‑23`). Ignore frames whose origin is your own.
- [ ] **HELLO (`0x40`)** → reply `0x41` FULL built from current `App` state
      (running, sl/idx, serialized program).
- [ ] **FULL (`0x41`)** → diff vs. current program; if different, load it
      (reuse `parse_program()` / the `programs.json` load path); then reconcile
      `running` (start/stop). Wrap in your remote‑apply flag.
- [ ] **DELTA (`0x42`)** → apply `play/stop/bpm/vol/sel/beat/lane` to `App`
      state, wrapped in the remote‑apply flag so the on‑device handlers don't
      re‑broadcast.
- [ ] **Broadcast** a `0x42` from each on‑device input handler (button A,
      joystick tempo, touch beat edit, set‑list nav, lane editor), guarded by
      the remote‑apply flag. Structural lane changes → `0x41` FULL.
- [ ] **Heartbeat:** emit `0x41` FULL every ~3–5 s while a peer is connected.
- [ ] **BYE (`0x43`)** → mark the peer gone (stop heartbeating/emitting until
      the next HELLO).
- [ ] **SLSYNC (`0x44`)** → merge the JSON manifest of user lists by normalized
      title (replace‑per‑list, append unknown, never delete), reusing
      `load_user_setlists()`‑shaped parsing; `rebuild_setlists()` + reload.
      Emit one after `_persist_user()` and in reply to `0x40`. (§8)
- [ ] **LOGSYNC (`0x45`)** → merge practice entries by (`at`,`name`); append +
      cap + re‑sort. Emit a one‑entry batch after `_log_play()` and a full batch
      in reply to `0x40`. Record `at` (epoch ms) on each play when the RTC is
      set. (§9)
- [ ] **Throttle** high‑rate sources (joystick tempo) and keep frames small —
      the RP2040 USB‑MIDI RX buffer is tiny (the firmware updater already chunks
      at 64 bytes), and live traffic shares the bus with MIDI clock, note‑out,
      and the editor's Active‑Sensing heartbeat. Don't let a flood stall a
      concurrent firmware push.

### Built‑in vs. user set lists (must match the editor)

The PM_K‑1's built‑in playlists (Styles / Practice / Song) are **baked into
firmware and read‑only**; on‑device edits **copy‑on‑write** into the user
"My edits" list. The editor follows the same rule (`userSetlists()` excludes the
seeded titles). So a **remote edit that targets a built‑in must follow the same
copy‑on‑write semantics** on the receiving side, or the two halves will disagree
about where the edit landed. When in doubt, after such an edit send a `0x41`
FULL with the resulting (copied) program so both sides converge on the same
target.

### Out of scope for the beta
- Mirroring device `settings.json` (LED brightness, MIDI config, etc.).
- Multi‑peer / multi‑editor arbitration beyond last‑writer‑wins.

> **No longer out of scope** (now specced in §8 / §9): live set‑list **content**
> sync (`0x44`) and streaming the device practice log up to the browser and back
> (`0x45`). The old `0x10` programs push (Save/Load to device) still exists as the
> explicit, full‑overwrite path; `0x44` is the *incremental, merge‑by‑title* live
> mirror that runs automatically while sync is armed.

---

## 7. Per‑device emit/apply matrix

Both targets implement the **full apply path** for every verb. They differ in what
they **emit**, because on‑device editing differs:

| Device      | Emits                                              | Applies                                     |
|-------------|----------------------------------------------------|---------------------------------------------|
| **PM_K‑1** Kit (touchscreen + joystick) | `play` / `stop` / `bpm` / `sel` / `beat` / `lane` (FULL on structural lane edits) | all of the above |
| **PM_X‑1** Explorer (6 buttons, read‑only beats) | `play` / `stop` / `bpm` / `sel` only (no on‑device beat/lane editing) | all of the above |
| **PM_G‑1** Grid (17×7 LED matrix, 4 buttons, read‑only beats) | `play` / `stop` / `bpm` / `sel` only (no on‑device beat/lane editing) | all of the above |

Editors don't need to special‑case the source — both DELTA streams look identical on
the wire, and the **device id is only exposed on the version query** (SysEx `0x02`
→ `0x03` reply, `<id>;<version>`; pre‑0.0.23 firmware sends bare version → assume
`K`).

---

## 8. Set‑list content sync (`0x44` SLSYNC)

The `0x41` FULL only carries the *one loaded program* (`<patch>`) plus the
*selection* indices. `0x44` carries **set‑list content** — titles + every item's
name + program string — so the two halves converge on the same library while
sync is armed, without the user pressing "Save to device".

**Frame:** `F0 7D 44 <origin>;<seq>;<json> F7`

- `<origin>` / `<seq>` — same as the other ops (echo‑drop + duplicate info).
- `<json>` — a **7‑bit‑safe JSON** manifest of the sender's **user** set lists,
  in the *exact same shape `0x10` already uses* so the firmware can reuse its
  `programs.json` parser:

  ```json
  {"setlists":[{"title":"My set list","programs":[{"name":"Funk","prog":"t120;..."}]}]}
  ```

  Non‑ASCII in titles/names is escaped `\uXXXX` (the editor's existing
  `programsJSON()` 7‑bit‑safe path); the firmware stores it verbatim. The whole
  manifest rides **one SysEx frame** (same as `0x10` — user libraries are a few
  KB and the RX assembler holds 60 000 bytes). It is **never chunked**; if it
  ever grew past the buffer, fall back to the explicit `0x10` push.

### What's included
- **Only user set lists.** Built‑in / seeded lists (firmware `BUILTIN_SETLISTS`;
  editor `SEED_SETLISTS` titles) are read‑only on both halves and **never
  transmitted** — both sides already have identical copies baked in. (Same filter
  as `userSetlists()` / `load_user_setlists()`.)

### Identity & merge rule
- **Set lists match by title** (normalized: lower‑case, alphanumerics only — the
  firmware's `_slkey()`), independent of index. Indices diverge freely between
  halves (the device prepends built‑ins; the web orders differently), so a
  positional match is wrong — **title is the key.**
- **Items match by name** within a list (case‑sensitive, as both UIs key
  practice history by exact name).
- Merge is a **per‑list replace**: a received user list **replaces** the local
  user list of the same normalized title wholesale (its items become the
  received items, in the received order). Lists present locally but absent from
  the message are **left untouched** (additive — sync never deletes a list the
  peer simply didn't send). A received list with no matching local title is
  **appended** as a new user list.
- This is **last‑writer‑wins per list** (consistent with the rest of the
  protocol). The receiver applies under its remote‑apply guard and does **not**
  re‑broadcast a `0x44` in response (no echo storm); the next heartbeat / FULL
  still reconciles the loaded program.

### Copy‑on‑write for built‑ins
A `0x44` never targets a built‑in: it only carries user lists, and the receiver
only ever writes user lists. If a user **edits a built‑in item** on either half,
that edit must first be **forked into a user list** (the firmware's `_save_edit`
already forks built‑in edits into the "My edits" user list; the editor keeps a
separate user list). The fork then rides `0x44` as an ordinary user list. So
copy‑on‑write happens **before** the sync, and the wire only ever sees user
content — the built‑ins on both sides stay pristine and identical.

### When it's emitted
- **Editor:** coalesced ~150 ms after any set‑list structural edit (add/rename/
  reorder list, add/remove/rename item, capture/update an item), and once on
  connect right after the first FULL. Reuses `syncPatchSoon()`‑style debouncing.
- **Device:** after `_persist_user()` succeeds (a save that wrote
  `programs.json`), guarded by the remote‑apply flag, and once in reply to a
  `0x40` HELLO. The device's per‑item *program* edits still ride `0x41` FULL;
  `0x44` is specifically for **library shape** (which lists/items exist).

> The device is the **convergence authority** for the *loaded program* (§4), but
> set‑list content is last‑writer‑wins per list — there is no periodic `0x44`
> heartbeat (it would clobber concurrent edits on the other half). Send it only
> on an actual content change or on connect.

---

## 9. Practice‑log sync (`0x45` LOGSYNC)

Both halves keep a practice history (web: `localStorage` `metronome.logs`;
device: `/history.json`). `0x45` streams entries between them and **merges by a
stable key**, so a session played on the device shows up in the editor's history
graph and vice‑versa.

**Frame:** `F0 7D 45 <origin>;<seq>;<json> F7`

`<json>` is a 7‑bit‑safe JSON batch of **normalized** entries:

```json
{"log":[{"at":1733059200000,"name":"Funk","dur":92,"bpm":120}]}
```

| field  | type        | meaning                                                    |
|--------|-------------|------------------------------------------------------------|
| `at`   | int (ms)    | session start, **Unix epoch milliseconds** — the dedup key |
| `name` | string      | set‑list item name the session was logged against         |
| `dur`  | int (sec)   | session duration in **whole seconds**                      |
| `bpm`  | int         | tempo at the end of the session                           |

This is the **intersection** of the two native schemas (the web's
`{at,name,durationSec,bpm,lanes}` and the device's `{t,bpm,dur,bars,name}`):
`at` ↔ `at`, `dur` ↔ `round(durationSec)` / `dur`, `bpm` ↔ `bpm`, `name` ↔
`name`. Fields each side keeps privately (web `lanes`; device `t`/`bars`) are
**not** transmitted; the receiver fills them from what it has (`t` from `at` via
the RTC, `bars`/`lanes` left absent).

### Timestamps & the device clock
The dedup key is `at` (epoch ms). The editor already pushes the RTC over `0x01`
on connect / heartbeat, so the device **can** produce a real epoch. The firmware
therefore records an `at` (epoch **seconds** × 1000 → ms) on each logged play
*in addition to* its existing `t:"HH:MM"` field, computed from `time.time()`
when the RTC is set; if the RTC is unset (no editor ever connected) it falls
back to `at = 0`, which the merge treats as "no stable key" (see dedup).

### Direction & dedup/merge
- **Bidirectional.** Each half emits its **own** local entries; each applies the
  peer's.
- **Dedup key = `at` + `name`.** On receive, an entry whose (`at`,`name`) already
  exists locally is dropped. Entries with `at == 0` (device logged before any RTC
  sync) are **always appended** (can't be deduped — better a possible duplicate
  than a dropped session); these are rare and the user can delete them.
- Merge is **additive only** — `0x45` never deletes history. (Deleting a stale
  entry on one half does not propagate; out of scope, matches the
  last‑writer‑wins philosophy and avoids a delete echoing into a re‑add.)
- The receiver **caps** its merged log (web keeps all; device keeps newest 200,
  its existing cap) and re‑sorts newest‑first by `at`.

### When it's emitted
- **Editor:** after `logFinalize()` writes a new session (one‑entry batch), and
  a **full batch** once on connect (after the first FULL) so the device catches
  up on everything it missed. Guarded so an *applied* remote entry doesn't
  re‑broadcast.
- **Device:** after `_log_play()` appends a session (one‑entry batch), guarded by
  the remote‑apply flag, and a full batch once in reply to a `0x40` HELLO.

> **Batch size caution (firmware):** a full‑history batch (up to 200 entries) is
> small JSON but still allocates; the device sends its on connect/HELLO only, and
> the editor's on‑connect batch is bounded by the SysEx buffer (60 KB ≈ a few
> thousand minimal entries). If a log ever exceeded that, the editor truncates to
> the newest entries that fit. Per‑session emits are a single entry — negligible.
