8f49bb3e1e
Mars Rover VR Project — 5-phase plan from panorama stitching to immersive VR on Meta Quest 3. Phases: 1. Image Collection & Database (complete) 2. Panorama Stitching Pipeline (in progress) 3. Equirectangular/Spherical Output 4. 3D Assets (rover model + photogrammetry terrain) 5. VR Application (Meta Quest 3) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
5.9 KiB
5.9 KiB
Mars Rover — Project Roadmap
Phase 1: Image Collection & Database (COMPLETE)
- NASA API integration (v1 API for Curiosity, RSS API for Perseverance)
- MySQL database with photos metadata (mast_az, mast_el, sol, camera, etc.)
- Automated weekly collection (cron: Curiosity Fri 3AM, Perseverance Sat 6AM)
- 818K+ Curiosity photos, 365K+ Perseverance photos indexed
- Camera specifications reference document
Scripts: /mnt/astro/mars_rovers/scripts/ on homelab
Phase 2: Panorama Stitching Pipeline (IN PROGRESS)
Completed
- Docker container with Hugin 2024 CLI + Python + OpenCV
- CLAHE + vignette preprocessing for cpfind feature matching
- NavCam tile combination (01+04 with 16px overlap blending)
- Automatic panorama sequence detection (sequence_id + azimuth gap analysis)
- Incremental assembly for large panoramas (80+ images)
- Camera-adaptive blending (enblend for NavCam, verdandi for Mastcam-Z)
- Geometry-only optimization (no photometric — prevents overexposure)
- Deployed on homelab (Docker) and AstroLab mini PC (Docker Desktop)
- Git repository: mars-panorama-pipeline
In Progress
- Validate incremental assembly on multiple sols
- Fix remaining horizon alignment artifacts
- Optimize autooptimiser performance for 50K+ control points
To Do
- Batch processing mode (multiple sols in one run)
- Quality scoring (automated detection of bad panoramas)
- Web gallery for browsing generated panoramas
- Curiosity panorama support (no mast_az — needs different approach)
Repository: mars-panorama-pipeline
Phase 3: Equirectangular / Spherical Output
To Do
- Validate equirectangular projection covers full 360° x 180°
- Handle partial coverage (fill sky/ground with gradient or procedural texture)
- HDR output for better VR rendering (exposure bracketing from multiple images)
- Multiple resolution tiers (preview 2K, standard 4K, high 8K+)
- Cubemap conversion option (for real-time VR rendering)
- Metadata embedding (sol, location, date, rover position)
Research
- Investigate nadir/zenith filling techniques for partial panoramas
- Study how existing Mars VR apps handle incomplete coverage
- Evaluate HDR merging from differently-exposed NavCam images
Phase 4: 3D Assets
Rover Model
- Perseverance GLB model obtained (65 meshes, 48 materials, 24 textures, 11MB)
- Inspect and optimize model for real-time VR (LOD, texture compression)
- Obtain Curiosity 3D model
- Correct rover scale and orientation
- Add articulated parts (mast, arm, wheels) for pose matching
Terrain Photogrammetry
- Select sols with dense multi-angle coverage for photogrammetry
- Process with Meshroom (installed on AstroLab mini PC)
- Generate terrain mesh for immediate surroundings (~10m radius)
- Texture terrain mesh from rover images
- Merge terrain mesh with spherical panorama background
- Research: stereo NavCam pairs for depth estimation
Research
- Evaluate NASA Terrain Relative Navigation (TRN) data for terrain meshes
- Study existing Mars DTM (Digital Terrain Model) from HiRISE orbital data
- Investigate using AI depth estimation (MiDaS, Depth Anything) on rover images
Phase 5: VR Application
Platform
- Choose framework: Unity (C#) vs Unreal (C++) vs native Android (OpenXR/Kotlin)
- Meta Quest 3 development setup (Meta Developer Hub, ADB)
- Basic skybox rendering from equirectangular panorama
- Place rover model at scene center
Features — MVP
- Load panorama as environment sphere texture
- Render rover GLB at correct position/orientation
- Free look (head tracking)
- Teleport movement around rover
- Sol selector UI (browse available panoramas)
Features — Enhanced
- Photogrammetric terrain for near-field realism
- Transition between sols (fade/morph)
- Information overlays (camera names, geological features)
- Day/night lighting based on Mars local time
- Ambient Mars audio (wind, rover sounds from Perseverance microphone)
- Multi-user mode (explore together)
Features — Advanced
- Animated rover (replay drive path between sols)
- Integration with Mars weather data (MEDA sensor)
- Orbital view (zoom out to HiRISE map, then zoom in to rover location)
- Cross-platform (Quest 3 + PC VR + WebXR)
Technical Decisions Log
| Date | Decision | Rationale |
|---|---|---|
| 2026-04-09 | Curiosity API: v1 (mars.nasa.gov) | Old NASA API dead (404), RSS API Perseverance only |
| 2026-04-10 | CLAHE for cpfind, originals for render | CLAHE enables feature matching but degrades visual quality |
| 2026-04-11 | No photometric optimization (no -m flag) | autooptimiser -m causes severe overexposure |
| 2026-04-11 | NavCam tile overlap = 16px | Empirically measured, confirmed by pixel diff analysis |
| 2026-04-11 | FOV: NavCam combined = 82.17°, MCZ wide = 25.6° | Empirical (NavCam), spec sheet (MCZ) |
| 2026-04-12 | Docker for pipeline | Reproducible, portable, eliminates Hugin version issues |
| 2026-04-12 | verdandi for MCZ, enblend for NavCam | verdandi eliminates seam overexposure on MCZ |
| 2026-04-12 | Incremental assembly for large panoramas | Parallel chunks caused orientation flip; incremental preserves coherence |
| 2026-04-12 | GLB format for rover model | Universal standard, native in Unity/Unreal/Quest |
Resources
- NASA Mars Rover Photos API
- Hugin Panorama Tools
- Meta Quest Development
- Meshroom Photogrammetry
- glTF Specification
- Camera specs:
/mnt/astro/mars_rovers/camera_specifications.md - Panorama detection guide:
/mnt/astro/mars_rovers/panorama_sequence_detection.md