Reflections from v0.5 bootstrap on effective goal accomplishment — general principles, domain-specific patterns, and process improvements.
When restored images looked wrong, I didn't accept them at face value. Instead: quantified the problem (pixel-level delta) → traced root cause (missing salt) → verified fix worked (0.15-0.28 bytes).
Why: Visual inspection deceives. Metrics are truth.
Application: For image/video/media projects, always measure pixel-level quality. For data transforms, always compare input ↔ output quantitatively.
Before trusting file-based restoration, tested in-memory: load → scramble → restore all in RAM. Result: 0.23 bytes = mathematically perfect.
This proved the algorithm was correct before blaming JPEG codec or file format.
Why: Separates algorithm bugs from I/O artifacts.
Application: For reversible transforms (encryption, compression, permutation), always have in-memory round-trip test as reference. File-based testing inherits I/O codec artifacts.
Understanding JPEG lossy encoding's effect: 0.15-0.28 bytes Δ is good (codec artifact); 49 bytes Δ would be bad (permutation failed).
This let me distinguish signal from noise in metrics.
Why: False negatives (rejecting good results) waste time; false positives (accepting bad results) hide bugs.
Application: Characterize your I/O medium's noise floor before testing. For JPEG: ~0.5-2 byte uncertainty/pixel. For crypto: zero uncertainty. This changes what metrics you trust.
The bug (unrestored images) was solved by capturing context: capture salt at scramble → use immediately for restore.
Result: perfect restoration. The algorithm was correct; the problem was information loss between steps.
Why: Intermediate state loss is a common pipeline bug.
Application: When a pipeline fails, ask 'What state is being lost between steps?' Often the fix is capturing and threading that state through.
Forcing regions to 8×8 block boundaries eliminates partial-block edge cases and makes permutation math clean.
Users expect pixel-aligned regions anyway.
Application: For block-based transforms (JPEG, video codecs, DFT), always snap geometric requests to block grid. Document the snapping.
HMAC-SHA256 counter-mode gives reproducible permutations (same key → same shuffle), cross-language compatibility, and no random seed management.
True RNG requires seed storage. Deterministic generation requires only key.
Application: For reversible operations (scrambling, diffusion, erasure coding), use deterministic PRNG keyed from user passphrase. Never trust true RNG for reproducibility.
Permutation alone isn't cryptographically secure (99.3% pixel change looks 'secure' but isn't). However: it's a perfect foundation for v1 (add diffusion), demonstrates the core concept, and restoration metrics prove it works.
Permutation is the hardest part technically; diffusion layers on top.
Application: For security phasing: implement hard part (reversible core) first with weak security. Add layers later. Users benefit now; future versions add strength without rewrite.
v0.5 deliberately doesn't embed metadata in APP15 because: core algorithm is decoupled from metadata strategy, multiple strategies can be tried in v1 without rewriting transform code.
Samples work fine with external salt capture.
Why: Premature I/O design locks you into one approach.
Application: For versioned features, defer complex I/O (metadata, headers, serialization) until core algorithm is proven. This lets you iterate on algorithm without breaking file format.
Moved all documentation to notes system early: easy to link/cross-reference, easy to update without committing, git stays clean (code only).
Why: Code reviews bloat when .md docs change. Notes iteration is separate from code review.
Application: Use git for stable deliverables (code, scripts, samples). Use notes for documentation that evolves. Link from CLAUDE.md to notes.
Recorded git baseline in memory with commit hash, tag, metrics achieved, next phase dependencies.
Why: Future sessions can recover exact state. Git history alone doesn't capture why a commit matters.
Application: On every major milestone, record git hash + context in memory. Makes context-switching painless.
Used actual images from WebDAV (3840×2160, HEIC-converted), not toy data. Caught real EXIF orientation issues, tested actual JPEG quality, observed real file sizes.
Why: 10×10 test images hide bugs that 4K images expose.
Application: Use production-scale data early. Don't wait for 'real' data later.
Pattern: scrypt(pass, salt) → (perm_key + check_key). Split output into separate keys for different purposes, prevents cross-contamination, allows future versioning.
Application: For any KDF, split output strategically. Don't reuse one key for multiple roles.
Counter-mode stream generates random values, rejection-sample to unbiased randbelow(). Simpler than rejection-free algorithms, standard math, deterministic stream.
Application: For deterministic shuffling, combine counter-mode PRNG with Knuth-Yates rejection sampling.
Correct: permute_blocks(array, region, p, 8) in-memory, then save_image(array, 'file.jpg', quality=95) with one lossy encode.
Wrong: load → permute → save → load → permute → save (double-encodes).
Why: JPEG encode happens once, pixel permutation happens cleanly in memory first.
Application: For image transforms + JPEG: do all pixel manipulation in-memory, JPEG encode only once at end.
1. Clear definition of success (samples with verified restoration)
2. Metrics-first debugging (quantified rather than assumed)
3. Staged validation (in-memory → file-based → samples)
4. Root cause analysis (salt loss, not algorithm failure)
5. Decoupled concerns (core algorithm independent of metadata strategy)
6. Context capture (commit hashes in memory for recovery)
Baseline at git tag v0.5-baseline (commit 9acab30) — all code checked in, working tree clean.
Samples on WebDAV demonstrate working tool.
Documentation at ideas/jpeg-obscura/* in notes system.
Use metrics, not intuition. Test algorithms in-memory before files. Defer complex I/O decisions until core is proven. Document git context in memory, not just git history.