# Mars Rover — Project Roadmap

## Phase 1: Image Collection & Database (COMPLETE)

- [x] NASA API integration (v1 API for Curiosity, RSS API for Perseverance)
- [x] MySQL database with photos metadata (mast_az, mast_el, sol, camera, etc.)
- [x] Automated weekly collection (cron: Curiosity Fri 3AM, Perseverance Sat 6AM)
- [x] 818K+ Curiosity photos, 365K+ Perseverance photos indexed
- [x] Camera specifications reference document

**Scripts**: `/mnt/astro/mars_rovers/scripts/` on homelab

---

## Phase 2: Panorama Stitching Pipeline (IN PROGRESS)

### Completed
- [x] Docker container with Hugin 2024 CLI + Python + OpenCV
- [x] CLAHE + vignette preprocessing for cpfind feature matching
- [x] NavCam tile combination (01+04 with 16px overlap blending)
- [x] Automatic panorama sequence detection (sequence_id + azimuth gap analysis)
- [x] Incremental assembly for large panoramas (80+ images)
- [x] Camera-adaptive blending (enblend for NavCam, verdandi for Mastcam-Z)
- [x] Geometry-only optimization (no photometric — prevents overexposure)
- [x] Deployed on homelab (Docker) and AstroLab mini PC (Docker Desktop)
- [x] Git repository: mars-panorama-pipeline

### In Progress
- [ ] Validate incremental assembly on multiple sols
- [ ] Fix remaining horizon alignment artifacts
- [ ] Optimize autooptimiser performance for 50K+ control points

### To Do
- [ ] Batch processing mode (multiple sols in one run)
- [ ] Quality scoring (automated detection of bad panoramas)
- [ ] Web gallery for browsing generated panoramas
- [ ] Curiosity panorama support (no mast_az — needs different approach)

**Repository**: [mars-panorama-pipeline](https://git.dupre-garnier.ca/git_admin/mars-panorama-pipeline)

---

## Phase 3: Equirectangular / Spherical Output

### To Do
- [ ] Validate equirectangular projection covers full 360° x 180°
- [ ] Handle partial coverage (fill sky/ground with gradient or procedural texture)
- [ ] HDR output for better VR rendering (exposure bracketing from multiple images)
- [ ] Multiple resolution tiers (preview 2K, standard 4K, high 8K+)
- [ ] Cubemap conversion option (for real-time VR rendering)
- [ ] Metadata embedding (sol, location, date, rover position)

### Research
- [ ] Investigate nadir/zenith filling techniques for partial panoramas
- [ ] Study how existing Mars VR apps handle incomplete coverage
- [ ] Evaluate HDR merging from differently-exposed NavCam images

---

## Phase 4: 3D Assets

### Rover Model
- [x] Perseverance GLB model obtained (65 meshes, 48 materials, 24 textures, 11MB)
- [ ] Inspect and optimize model for real-time VR (LOD, texture compression)
- [ ] Obtain Curiosity 3D model
- [ ] Correct rover scale and orientation
- [ ] Add articulated parts (mast, arm, wheels) for pose matching

### Terrain Photogrammetry
- [ ] Select sols with dense multi-angle coverage for photogrammetry
- [ ] Process with Meshroom (installed on AstroLab mini PC)
- [ ] Generate terrain mesh for immediate surroundings (~10m radius)
- [ ] Texture terrain mesh from rover images
- [ ] Merge terrain mesh with spherical panorama background
- [ ] Research: stereo NavCam pairs for depth estimation

### Research
- [ ] Evaluate NASA Terrain Relative Navigation (TRN) data for terrain meshes
- [ ] Study existing Mars DTM (Digital Terrain Model) from HiRISE orbital data
- [ ] Investigate using AI depth estimation (MiDaS, Depth Anything) on rover images

---

## Phase 5: VR Application

### Platform
- [ ] Choose framework: Unity (C#) vs Unreal (C++) vs native Android (OpenXR/Kotlin)
- [ ] Meta Quest 3 development setup (Meta Developer Hub, ADB)
- [ ] Basic skybox rendering from equirectangular panorama
- [ ] Place rover model at scene center

### Features — MVP
- [ ] Load panorama as environment sphere texture
- [ ] Render rover GLB at correct position/orientation
- [ ] Free look (head tracking)
- [ ] Teleport movement around rover
- [ ] Sol selector UI (browse available panoramas)

### Features — Enhanced
- [ ] Photogrammetric terrain for near-field realism
- [ ] Transition between sols (fade/morph)
- [ ] Information overlays (camera names, geological features)
- [ ] Day/night lighting based on Mars local time
- [ ] Ambient Mars audio (wind, rover sounds from Perseverance microphone)
- [ ] Multi-user mode (explore together)

### Features — Advanced
- [ ] Animated rover (replay drive path between sols)
- [ ] Integration with Mars weather data (MEDA sensor)
- [ ] Orbital view (zoom out to HiRISE map, then zoom in to rover location)
- [ ] Cross-platform (Quest 3 + PC VR + WebXR)

---

## Technical Decisions Log

| Date | Decision | Rationale |
|------|----------|-----------|
| 2026-04-09 | Curiosity API: v1 (mars.nasa.gov) | Old NASA API dead (404), RSS API Perseverance only |
| 2026-04-10 | CLAHE for cpfind, originals for render | CLAHE enables feature matching but degrades visual quality |
| 2026-04-11 | No photometric optimization (no -m flag) | autooptimiser -m causes severe overexposure |
| 2026-04-11 | NavCam tile overlap = 16px | Empirically measured, confirmed by pixel diff analysis |
| 2026-04-11 | FOV: NavCam combined = 82.17°, MCZ wide = 25.6° | Empirical (NavCam), spec sheet (MCZ) |
| 2026-04-12 | Docker for pipeline | Reproducible, portable, eliminates Hugin version issues |
| 2026-04-12 | verdandi for MCZ, enblend for NavCam | verdandi eliminates seam overexposure on MCZ |
| 2026-04-12 | Incremental assembly for large panoramas | Parallel chunks caused orientation flip; incremental preserves coherence |
| 2026-04-12 | GLB format for rover model | Universal standard, native in Unity/Unreal/Quest |

---

## Resources

- [NASA Mars Rover Photos API](https://mars.nasa.gov/rss/api/)
- [Hugin Panorama Tools](https://hugin.sourceforge.io/)
- [Meta Quest Development](https://developer.oculus.com/)
- [Meshroom Photogrammetry](https://alicevision.org/#meshroom)
- [glTF Specification](https://www.khronos.org/gltf/)
- Camera specs: `/mnt/astro/mars_rovers/camera_specifications.md`
- Panorama detection guide: `/mnt/astro/mars_rovers/panorama_sequence_detection.md`