pm-grid: live re-read of programs.json (no reboot needed)

When the host writes the drive (SCSI Write sets a dirty flag) and the drive has
been idle ~1.5s AND playback is stopped, the loop re-reads programs.json and
rebuilds the set lists (reload_user) -> a dropped file applies without a reboot.

Read-only path (split read_programs_json out of read_user_setlists; the format
flash-write only happens at boot), so no FAT-corruption risk from dual access.

Note on the recommended write path: the device deliberately does NOT write the
shared FAT while the host has it mounted (that corrupts the host cache - same
reason CircuitPython is one-direction-at-a-time). The practice log should instead
go to the editor via LOGSYNC (0x45); settings.json *read* (device read-only) is a
safe follow-up. Documented in docs/rust-port.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs/rust-port.md

@@ -229,11 +229,23 @@ set lists appended to the built-ins** — drop your `programs.json` on the drive
 appear (B-hold cycles set lists). Set lists are now a runtime `Vec<SetList>` (built-ins → owned +
 drive).
 
-**Still deferred**: *write* the drive from the device (practice log / `settings.json` — needs a
+**Live re-read — ✅ DONE**: the SCSI Write handler sets a `dirty` flag; when the drive has been idle
-write-back `FlashIo`, the read path is done), re-read `programs.json` without a reboot, show the
+~1.5 s (host finished) **and** playback is stopped, the loop re-reads `programs.json` and rebuilds the
-set-list title, SLSYNC/LOGSYNC (`0x44`/`0x45`), the **on-device 808/909 synth → USB Audio input**
+set lists (`reload_user`) — drop a file, it appears **without a reboot**. Read-only → no FAT
-(the standalone-audio alternative, big), firmware push (intended: UF2 now), optional piezo. A/B
+corruption. (NB the boot black-screen regression was the 24 KB heap being too small for `fatfs` +
-bootloader **dropped** by the user.
+owned set lists → alloc panic; heap is now 96 KB and the drive read runs *after* the splash.)
+
+**Dual-access constraint (why the device doesn't *write* the drive):** while the host has the FAT
+mounted, the device writing it corrupts the host's cached view (same reason CircuitPython makes the
+drive one-direction-at-a-time). So device→drive writes (practice log, `settings.json`) are **not**
+done; the practice log should instead go to the editor via **LOGSYNC** (`0x45`, its designed
+channel), or behind a CircuitPython-style boot-mode toggle. `settings.json` *read* (config from a
+file: brightness, MIDI channel, clock on/off) is safe (device read-only) and is a clean follow-up.
+
+**Still deferred**: practice log via **LOGSYNC** + **SLSYNC** (`0x44`/`0x45`), `settings.json` read,
+show the set-list title, the **on-device 808/909 synth → USB Audio input** (the standalone-audio
+alternative, big), firmware push (intended: UF2 now), optional piezo. A/B bootloader **dropped**.
+Also pending: a **hardening pass** (stress the composite USB + flash-write timing; split `main.rs`).
 
 ### Stage 4 — native A/B + secure boot
 Replace the `.mpy`-level A/B hack (`code.py` loads `app.mpy`, rolls back to `app.bak`) with the

rust/pm-grid/src/main.rs

@@ -156,26 +156,9 @@ fn format_pmg1() {
     }
 }
 
-/// Mount the drive; if it isn't a "PM_G-1"-labelled FAT, format it. Then read programs.json (if any)
+/// Read programs.json (if present) into user set lists. READ-ONLY — never writes the FAT, so it's
-/// into user set lists. Runs once at boot (before USB), so the flash write can't disrupt enumeration.
+/// safe to call at runtime even while the host has the drive mounted.
-fn read_user_setlists() -> Vec<SetList> {
+fn read_programs_json() -> Vec<SetList> {
-    let is_ours = {
-        let io = FlashIo::new();
-        match fatfs::FileSystem::new(io, fatfs::FsOptions::new()) {
-            Ok(fs) => fs
-                .read_volume_label_from_root_dir()
-                .ok()
-                .flatten()
-                .as_deref()
-                == Some("PM_G-1"),
-            Err(_) => false,
-        }
-    };
-    if !is_ours {
-        info!("fat: (re)formatting drive as PM_G-1");
-        format_pmg1();
-    }
-
     let io = FlashIo::new();
     let mut out = Vec::new();
     if let Ok(fs) = fatfs::FileSystem::new(io, fatfs::FsOptions::new()) {
@@ -201,6 +184,41 @@ fn read_user_setlists() -> Vec<SetList> {
     out
 }
 
+/// Boot path: if the drive isn't a "PM_G-1"-labelled FAT, (re)format it, then read programs.json.
+/// The format flash-write only happens here (at boot, before USB) — runtime re-reads never write.
+fn read_user_setlists() -> Vec<SetList> {
+    let is_ours = {
+        let io = FlashIo::new();
+        match fatfs::FileSystem::new(io, fatfs::FsOptions::new()) {
+            Ok(fs) => fs
+                .read_volume_label_from_root_dir()
+                .ok()
+                .flatten()
+                .as_deref()
+                == Some("PM_G-1"),
+            Err(_) => false,
+        }
+    };
+    if !is_ours {
+        info!("fat: (re)formatting drive as PM_G-1");
+        format_pmg1();
+    }
+    read_programs_json()
+}
+
+/// Build the runtime set-list table: built-ins (static → owned) followed by the drive's set lists.
+fn build_setlists(user: Vec<SetList>) -> Vec<SetList> {
+    let mut v: Vec<SetList> = SETLISTS
+        .iter()
+        .map(|(title, items)| SetList {
+            title: String::from(*title),
+            items: items.iter().map(|(n, p)| (String::from(*n), String::from(*p))).collect(),
+        })
+        .collect();
+    v.extend(user);
+    v
+}
+
 /// Read a JSON string starting at the opening quote `b[start]`; returns (value, index-after-close).
 fn json_str(b: &[u8], start: usize) -> Option<(String, usize)> {
     let mut s = String::new();
@@ -562,15 +580,7 @@ const SCROLL_GAP: i32 = 13; // blank columns between repeats (>= name region wid
 
 impl App {
     fn new(now_ns: i64, user: Vec<SetList>) -> Self {
-        // built-ins (static) → owned, then append the drive's set lists
+        let setlists = build_setlists(user);
-        let mut setlists: Vec<SetList> = SETLISTS
-            .iter()
-            .map(|(title, items)| SetList {
-                title: String::from(*title),
-                items: items.iter().map(|(n, p)| (String::from(*n), String::from(*p))).collect(),
-            })
-            .collect();
-        setlists.extend(user);
         let prog = setlists[0].items[0].1.clone();
         let track = track_format::parse(&prog);
         let tempo = track.bpm;
@@ -659,6 +669,18 @@ impl App {
         self.reset_clock(now_ns);
     }
 
+    /// Live re-read: replace the user set lists from a fresh programs.json without disturbing the
+    /// currently-playing track (just makes the new lists reachable via B-hold). Clamps indices.
+    fn reload_user(&mut self, user: Vec<SetList>) {
+        self.setlists = build_setlists(user);
+        if self.sl >= self.setlists.len() {
+            self.sl = 0;
+        }
+        if self.item >= self.setlists[self.sl].items.len() {
+            self.item = 0;
+        }
+    }
+
     fn next_track(&mut self, now_ns: i64) {
         let n = self.setlists[self.sl].items.len();
         self.load(self.sl, (self.item + 1) % n, now_ns);
@@ -1420,11 +1442,18 @@ struct ScsiState {
     offset: usize, // bytes transferred so far for the in-progress Read/Write
     sense_key: u8,
     sense_asc: u8,
+    dirty: bool, // host wrote a block → the main loop schedules a programs.json re-read
     write_buf: [u8; FS_BLOCK_SIZE as usize],
 }
 impl ScsiState {
     fn new() -> Self {
-        ScsiState { offset: 0, sense_key: 0, sense_asc: 0, write_buf: [0u8; FS_BLOCK_SIZE as usize] }
+        ScsiState {
+            offset: 0,
+            sense_key: 0,
+            sense_asc: 0,
+            dirty: false,
+            write_buf: [0u8; FS_BLOCK_SIZE as usize],
+        }
     }
 }
 
@@ -1505,6 +1534,7 @@ fn scsi_command(
                     if count > 0 && (st.offset % (FS_BLOCK_SIZE as usize)) == 0 {
                         // a full 4 KB block accumulated → erase+program that sector
                         info!("msc: write block {}", start / FS_BLOCK_SIZE as usize);
+                        st.dirty = true; // drive changed → re-read programs.json once it's idle
                         let faddr = (FILESYSTEM.as_ptr() as u32 & 0x00ff_ffff) + ((start as u32) & !0xfff);
                         cortex_m::interrupt::free(|_| unsafe {
                             rp2040_flash::flash::flash_range_erase_and_program(faddr, &st.write_buf, false);
@@ -1642,6 +1672,8 @@ fn main() -> ! {
     let mut last_frame_us = 0i64;
     let mut hb_us = 0i64;
     let mut rxbuf = [0u8; 64];
+    let mut fs_write_us = 0i64; // last host drive-write time
+    let mut fs_pending = false; // a re-read of programs.json is owed once the drive goes idle
 
     loop {
         let us = now_us();
@@ -1654,6 +1686,20 @@ fn main() -> ! {
                 let _ = scsi_command(cmd, &mut scsi_state);
             });
         }
+        // host wrote the drive → schedule a re-read once it's been idle for a bit
+        if scsi_state.dirty {
+            scsi_state.dirty = false;
+            fs_write_us = us;
+            fs_pending = true;
+        }
+        // re-read programs.json after the drive settles (host done writing) and we're stopped, so a
+        // dropped file applies without a reboot. Read-only → no FAT-corruption risk.
+        if fs_pending && !app.playing && us - fs_write_us > 1_500_000 {
+            fs_pending = false;
+            let user = read_programs_json();
+            app.reload_user(user);
+            info!("fat: re-read drive -> {} set list(s)", app.setlists.len());
+        }
         // ---- drain the MIDI RX endpoint, feeding SysEx (live-sync) bytes ----
         while let Ok(n) = midi.read(&mut rxbuf) {
             if n == 0 {