From 8f49bb3e1e206bb512a6115a5b0efcb963b672f6 Mon Sep 17 00:00:00 2001 From: Franck Garnier Date: Sun, 12 Apr 2026 22:03:11 -0400 Subject: [PATCH] Initial commit: project plan and roadmap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mars Rover VR Project — 5-phase plan from panorama stitching to immersive VR on Meta Quest 3. Phases: 1. Image Collection & Database (complete) 2. Panorama Stitching Pipeline (in progress) 3. Equirectangular/Spherical Output 4. 3D Assets (rover model + photogrammetry terrain) 5. VR Application (Meta Quest 3) Co-Authored-By: Claude Opus 4.6 (1M context) --- .gitignore | 11 +++++ README.md | 81 +++++++++++++++++++++++++++++++ ROADMAP.md | 139 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 231 insertions(+) create mode 100644 .gitignore create mode 100644 README.md create mode 100644 ROADMAP.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..abdf95d --- /dev/null +++ b/.gitignore @@ -0,0 +1,11 @@ +__pycache__/ +*.pyc +*.tif +*.tiff +.env +*.log +node_modules/ +.gradle/ +build/ +*.apk +*.aab diff --git a/README.md b/README.md new file mode 100644 index 0000000..8de3e42 --- /dev/null +++ b/README.md @@ -0,0 +1,81 @@ +# Mars Rover — Immersive VR Experience + +Explore Mars through the eyes of NASA's rovers in virtual reality. + +This project creates immersive VR experiences from real Mars rover data: 360° panoramas, 3D rover models, and photogrammetric terrain, viewable on Meta Quest 3. + +## Project Repositories + +| Repository | Purpose | Status | +|-----------|---------|--------| +| [mars-panorama-pipeline](https://git.dupre-garnier.ca/git_admin/mars-panorama-pipeline) | Automated panorama stitching (Docker + Hugin) | Active | +| mars-rover (this repo) | Project documentation, planning, VR app | Active | + +## Architecture + +``` +Mars Rover Data (NASA API) + | + v +[Phase 1] Image Collection & Database + | - NASA Mars Rover Photos API (Perseverance, Curiosity) + | - MySQL database (photos metadata, mast_az/el positioning) + | - Automated weekly collection (cron) + | + v +[Phase 2] Panorama Stitching Pipeline --> mars-panorama-pipeline repo + | - Docker container (Hugin CLI + Python + OpenCV) + | - CLAHE preprocessing for feature matching + | - Incremental assembly for large panoramas (80+ images) + | - NavCam (enblend) + Mastcam-Z (verdandi) blending + | + v +[Phase 3] Equirectangular Projection + | - 360° spherical panorama output + | - HDR processing for VR + | - Multiple resolution tiers + | + v +[Phase 4] 3D Assets + | - Rover 3D model (GLB format, NASA CAD data) + | - Photogrammetric terrain mesh (Meshroom) + | - Texture mapping from rover images + | + v +[Phase 5] VR Application + - Meta Quest 3 native app (Android/OpenXR) + - Spherical environment from panorama + - 3D rover model at scene center + - Photogrammetric terrain for near-field + - Free movement around rover +``` + +## Data Sources + +| Source | Type | Volume | +|--------|------|--------| +| NASA Mars Rover Photos API | Images + metadata | 818K+ Curiosity, 365K+ Perseverance | +| NASA PDS (Planetary Data System) | Raw science data + XML | ~807 GB | +| NASA 3D Models | Rover GLB/USDZ | 12 MB (Perseverance) | + +## Infrastructure + +| Machine | Role | Specs | +|---------|------|-------| +| Homelab (192.168.1.42) | Database, Docker, NAS, Git | i7-6700T, 32GB, Debian 13 | +| AstroLab (192.168.1.250) | GPU processing, VR dev, Hugin UI | Ryzen 7 7735HS, 29GB, Win11 | +| NAS (/mnt/astro) | Image storage | 4 TB USB | + +## Quick Start + +```bash +# Clone +git clone https://git.dupre-garnier.ca/git_admin/mars-rover.git + +# Generate a panorama (Docker) +docker run --rm \ + -v /mnt/astro/mars_rovers:/data \ + -v /mnt/astro/mars_rovers/images/panorama/perseverance:/output \ + mars-panorama-pipeline \ + --sol 1813 --camera NAVCAM_LEFT +``` diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..dadc63e --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,139 @@ +# Mars Rover — Project Roadmap + +## Phase 1: Image Collection & Database (COMPLETE) + +- [x] NASA API integration (v1 API for Curiosity, RSS API for Perseverance) +- [x] MySQL database with photos metadata (mast_az, mast_el, sol, camera, etc.) +- [x] Automated weekly collection (cron: Curiosity Fri 3AM, Perseverance Sat 6AM) +- [x] 818K+ Curiosity photos, 365K+ Perseverance photos indexed +- [x] Camera specifications reference document + +**Scripts**: `/mnt/astro/mars_rovers/scripts/` on homelab + +--- + +## Phase 2: Panorama Stitching Pipeline (IN PROGRESS) + +### Completed +- [x] Docker container with Hugin 2024 CLI + Python + OpenCV +- [x] CLAHE + vignette preprocessing for cpfind feature matching +- [x] NavCam tile combination (01+04 with 16px overlap blending) +- [x] Automatic panorama sequence detection (sequence_id + azimuth gap analysis) +- [x] Incremental assembly for large panoramas (80+ images) +- [x] Camera-adaptive blending (enblend for NavCam, verdandi for Mastcam-Z) +- [x] Geometry-only optimization (no photometric — prevents overexposure) +- [x] Deployed on homelab (Docker) and AstroLab mini PC (Docker Desktop) +- [x] Git repository: mars-panorama-pipeline + +### In Progress +- [ ] Validate incremental assembly on multiple sols +- [ ] Fix remaining horizon alignment artifacts +- [ ] Optimize autooptimiser performance for 50K+ control points + +### To Do +- [ ] Batch processing mode (multiple sols in one run) +- [ ] Quality scoring (automated detection of bad panoramas) +- [ ] Web gallery for browsing generated panoramas +- [ ] Curiosity panorama support (no mast_az — needs different approach) + +**Repository**: [mars-panorama-pipeline](https://git.dupre-garnier.ca/git_admin/mars-panorama-pipeline) + +--- + +## Phase 3: Equirectangular / Spherical Output + +### To Do +- [ ] Validate equirectangular projection covers full 360° x 180° +- [ ] Handle partial coverage (fill sky/ground with gradient or procedural texture) +- [ ] HDR output for better VR rendering (exposure bracketing from multiple images) +- [ ] Multiple resolution tiers (preview 2K, standard 4K, high 8K+) +- [ ] Cubemap conversion option (for real-time VR rendering) +- [ ] Metadata embedding (sol, location, date, rover position) + +### Research +- [ ] Investigate nadir/zenith filling techniques for partial panoramas +- [ ] Study how existing Mars VR apps handle incomplete coverage +- [ ] Evaluate HDR merging from differently-exposed NavCam images + +--- + +## Phase 4: 3D Assets + +### Rover Model +- [x] Perseverance GLB model obtained (65 meshes, 48 materials, 24 textures, 11MB) +- [ ] Inspect and optimize model for real-time VR (LOD, texture compression) +- [ ] Obtain Curiosity 3D model +- [ ] Correct rover scale and orientation +- [ ] Add articulated parts (mast, arm, wheels) for pose matching + +### Terrain Photogrammetry +- [ ] Select sols with dense multi-angle coverage for photogrammetry +- [ ] Process with Meshroom (installed on AstroLab mini PC) +- [ ] Generate terrain mesh for immediate surroundings (~10m radius) +- [ ] Texture terrain mesh from rover images +- [ ] Merge terrain mesh with spherical panorama background +- [ ] Research: stereo NavCam pairs for depth estimation + +### Research +- [ ] Evaluate NASA Terrain Relative Navigation (TRN) data for terrain meshes +- [ ] Study existing Mars DTM (Digital Terrain Model) from HiRISE orbital data +- [ ] Investigate using AI depth estimation (MiDaS, Depth Anything) on rover images + +--- + +## Phase 5: VR Application + +### Platform +- [ ] Choose framework: Unity (C#) vs Unreal (C++) vs native Android (OpenXR/Kotlin) +- [ ] Meta Quest 3 development setup (Meta Developer Hub, ADB) +- [ ] Basic skybox rendering from equirectangular panorama +- [ ] Place rover model at scene center + +### Features — MVP +- [ ] Load panorama as environment sphere texture +- [ ] Render rover GLB at correct position/orientation +- [ ] Free look (head tracking) +- [ ] Teleport movement around rover +- [ ] Sol selector UI (browse available panoramas) + +### Features — Enhanced +- [ ] Photogrammetric terrain for near-field realism +- [ ] Transition between sols (fade/morph) +- [ ] Information overlays (camera names, geological features) +- [ ] Day/night lighting based on Mars local time +- [ ] Ambient Mars audio (wind, rover sounds from Perseverance microphone) +- [ ] Multi-user mode (explore together) + +### Features — Advanced +- [ ] Animated rover (replay drive path between sols) +- [ ] Integration with Mars weather data (MEDA sensor) +- [ ] Orbital view (zoom out to HiRISE map, then zoom in to rover location) +- [ ] Cross-platform (Quest 3 + PC VR + WebXR) + +--- + +## Technical Decisions Log + +| Date | Decision | Rationale | +|------|----------|-----------| +| 2026-04-09 | Curiosity API: v1 (mars.nasa.gov) | Old NASA API dead (404), RSS API Perseverance only | +| 2026-04-10 | CLAHE for cpfind, originals for render | CLAHE enables feature matching but degrades visual quality | +| 2026-04-11 | No photometric optimization (no -m flag) | autooptimiser -m causes severe overexposure | +| 2026-04-11 | NavCam tile overlap = 16px | Empirically measured, confirmed by pixel diff analysis | +| 2026-04-11 | FOV: NavCam combined = 82.17°, MCZ wide = 25.6° | Empirical (NavCam), spec sheet (MCZ) | +| 2026-04-12 | Docker for pipeline | Reproducible, portable, eliminates Hugin version issues | +| 2026-04-12 | verdandi for MCZ, enblend for NavCam | verdandi eliminates seam overexposure on MCZ | +| 2026-04-12 | Incremental assembly for large panoramas | Parallel chunks caused orientation flip; incremental preserves coherence | +| 2026-04-12 | GLB format for rover model | Universal standard, native in Unity/Unreal/Quest | + +--- + +## Resources + +- [NASA Mars Rover Photos API](https://mars.nasa.gov/rss/api/) +- [Hugin Panorama Tools](https://hugin.sourceforge.io/) +- [Meta Quest Development](https://developer.oculus.com/) +- [Meshroom Photogrammetry](https://alicevision.org/#meshroom) +- [glTF Specification](https://www.khronos.org/gltf/) +- Camera specs: `/mnt/astro/mars_rovers/camera_specifications.md` +- Panorama detection guide: `/mnt/astro/mars_rovers/panorama_sequence_detection.md`