Date: 2026-08-31 • Context: Implementing pixel-domain diffusion on v0.5 foundation
✓ permutation.py remains unchanged from v0.5
✓ New diffusion.py isolated from permutation logic
✓ transform.py wrappers (permute_and_diffuse_blocks) compose cleanly
Result: Minimal code churn, easy to debug, easy to replace diffusion for v1.5
Learning: Module-level separation enabled focused changes; do this from day 1
✓ In-memory round-trip test proved algorithm perfect (0.0 bytes Δ)
✓ File round-trip test showed JPEG+diffusion interaction (~9.8Δ)
✓ This separation of concerns revealed exactly where artifacts came from
Learning: Always test core algorithm in-memory first; saves days of debugging
✓ v1 and v0.5 coexist without conflicts (schema_version field)
✓ CLI auto-detects and routes correctly
✓ v1.5 can continue using same versioning (no migration needed)
Learning: Versioning saves future pain; add it early even if only 1 version exists
✓ Same code for permutation key (v0.5) and diffusion key (v1)
✓ Domain separation via prefix strings (no code changes)
✓ read_byte() convenience method was 3-line addition
Learning: Invest in abstractions that work across versions; pays off immediately
✓ Used actual 4K photographs, not toy 10×10 test images
✓ Caught JPEG+diffusion interaction immediately (not in unit tests)
✓ Discovered HEIC-conversion subtleties (important for docs)
Learning: Real data exposes bugs that synthetic data misses
Problem: Discovered ~9.8Δ in testing, but design doc said nothing about it
Cause: Didn't think through JPEG lifecycle carefully enough during design
Fix for next time: Design phase must trace data through all transformations
Impact: Added confusion about whether the implementation was wrong (it wasn't)
Problem: Said 'defer to v1.5' but then didn't design what v1.5 looks like
Cause: Made 'defer' decision without understanding v1.5's constraints
Fix for next time: When deferring, describe what you'll do instead (not just 'later')
Impact: v1.5 planning is clear now, but caused some uncertainty mid-v1
Problem: Pixel-by-pixel XOR is single-threaded; could be faster
Cause: Focused on correctness, not optimization
Fix for next time: Design for parallelization from start (easier than retrofitting)
Impact: Minor (still ~1 second per 4K image is acceptable), but noted for v1.5
✓ YES, correct choice. Separation enabled clarity.
Alternative: Could have interleaved (permute one block, diffuse it, repeat)
Why separation is better: Clean separation of concerns, easier to verify, easier to replace
✓ YES, correct choice. Allowed faster delivery, proved concept works.
Alternative: Jump straight to coefficient domain
Why v1 first: Pixel domain is simpler, allowed validation of diffusion logic
Trade-off accepted: 9.8Δ for 2-3 day faster delivery (proved worth it)
✓ YES, correct for v1. Kept scope manageable.
Alternative: Embed in APP15 now
Why deferred: Metadata embedding is complex (JPEG structure); v1.5 can do it properly
No regrets: v1 works fine with external params; v1.5 will be cleaner with embedding
✓ YES, correct choice. Adds security, minimal complexity overhead.
Alternative: Single key for both
Why two keys: Domain separation principle; allows independent evolution
✓ Clear module names (diffusion.py does one thing)
✓ Docstrings explaining purpose and parameters
✓ Deterministic algorithm (no random state, reproducible bugs)
✓ Self-inverse property (un-diffuse is same as diffuse)
• Coefficient-domain version should learn from module separation
• Performance profiling would help prioritize v1.5 optimizations
• Automated end-to-end tests (original → scramble → restore → measure Δ)
Layer 1: Unit tests (self-inverse, determinism, small arrays)
Layer 2: In-memory integration (full algorithm, no file I/O)
Layer 3: File round-trip (real JPEG, realistic sizes)
Layer 4: Visual inspection (human review of samples)
Layer 1 would have caught: wrong XOR logic, non-determinism
Layer 2 caught: core algorithm perfect (0.0Δ)
Layer 3 caught: JPEG+diffusion interaction (9.8Δ)
Layer 4 confirmed: scrambled regions look like noise, restore works visually
Each layer answered different questions. All were necessary.
For v1.5: Add coefficient-domain specific tests (DCT preservation)
✓ Separate design document before implementation (caught assumptions)
✓ Checkpoint documents with git hashes (easy to recover state)
✓ Implementation docs linked from project index (discoverable)
• Should have documented JPEG lifecycle in design (lossy interaction)
• Should have sketched v1.5 approach earlier (would have informed v1 design)
• Could have created architecture diagrams (text is harder to visualize)
• DCT happens after Huffman decoding (can extract coefficients losslessly)
• Quantization tables (DQT) are separate (coefficients are pre-quantized)
• MCU structure varies by subsampling (4:2:0 vs 4:1:1 vs 4:4:4)
• Restart markers (RST) define block boundaries (important for lossless)
• JPEG = transform (DCT) + quantize + Huffman code
• Working between transform and Huffman = lossless region
• No re-quantization needed (use original QT values)
• Permutation of coefficients is simpler than pixels (larger number range)
✓ Module structure (diffusion.py paradigm works, rewrite content)
✓ Schema versioning (continue with schema_version field)
✓ Testing strategy (in-memory → file → visual)
✓ DeterministicStream (no changes needed)
✗ Replace diffusion.py with coefficient version
✗ Replace transform.py wrappers (coefficient blocks, not pixels)
✗ Add APP15 metadata support
✗ Add JPEG parsing/reconstruction (new dependency on jpegtran or libjpeg)
? Parallelize coefficient permutation (multiple threads per block)
? Add batch processing (scramble multiple images at once)
? Add CLI progress reporting (large files take a few seconds)
Based on v1 experience:
• Design phase: 2-3 hours (JPEG tooling decision, coefficient handling)
• Implementation: 1-2 days (simpler than expected if tooling works)
• Testing: 0.5-1 day (more complex due to JPEG edge cases)
• Total: 2-3 days (similar to v1, but with more unknowns around tooling)
Risk: JPEG tooling complexity could extend this to 4-5 days
v1 was a successful stepping stone:
• Proved diffusion concept works (genuine security achieved)
• Revealed JPEG+diffusion trade-off (acceptable for v1, fixed in v1.5)
• Established versioning/compatibility pattern (v1.5 builds on this)
• Produced working code with high confidence (thorough testing)
v1.5 should be smooth:
• Architecture principles clear (module separation, schema versioning)
• Risks identified (JPEG tooling, coefficient handling)
• Testing strategy proven (in-memory → file → visual)
• Timeline realistic (2-3 days, with contingency for tooling)