Upload via Autoresearch export_data

.gitattributes

@@ -33,3 +33,15 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+experiments/fourier-splatting/models/kitchen/lod/fourier_k5.ply filter=lfs diff=lfs merge=lfs -text
+experiments/fourier-splatting/models/kitchen/lod/fourier_k6.ply filter=lfs diff=lfs merge=lfs -text
+experiments/fourier-splatting/models/kitchen/lod/fourier_k3.ply filter=lfs diff=lfs merge=lfs -text
+experiments/fourier-splatting/models/kitchen/lod/fourier_k4.ply filter=lfs diff=lfs merge=lfs -text
+experiments/fourier-splatting/models/kitchen/lod/fourier_k1.ply filter=lfs diff=lfs merge=lfs -text
+experiments/fourier-splatting/models/kitchen/lod/fourier_k2.ply filter=lfs diff=lfs merge=lfs -text
+experiments/fourier-splatting/models/apartment/lod/fourier_k5.ply filter=lfs diff=lfs merge=lfs -text
+experiments/fourier-splatting/models/apartment/lod/fourier_k6.ply filter=lfs diff=lfs merge=lfs -text
+experiments/fourier-splatting/models/apartment/lod/fourier_k3.ply filter=lfs diff=lfs merge=lfs -text
+experiments/fourier-splatting/models/apartment/lod/fourier_k4.ply filter=lfs diff=lfs merge=lfs -text
+experiments/fourier-splatting/models/apartment/lod/fourier_k1.ply filter=lfs diff=lfs merge=lfs -text
+experiments/fourier-splatting/models/apartment/lod/fourier_k2.ply filter=lfs diff=lfs merge=lfs -text

experiments/fourier-splatting/FOURIER_SPLATTING_REPORT.md

@@ -0,0 +1,92 @@
+# Fourier Splatting × AnySplat — Experiment Report
+
+**Date:** 2026-10-13  
+**Hardware:** existing SF Compute H100 80 GB node; no new node created  
+**Status:** two-scene prototype, 6,000 scene-optimization iterations per scene
+
+## Question
+
+Can the existing AnySplat video-to-3D pipeline be improved by converting its context-only geometry into Fourier-encoded planar primitives and optimizing the scene, instead of retaining a dense Gaussian cloud?
+
+This experiment adapts the upstream Fourier Splatting implementation (https://github.com/mbjurca/FourierSplatting) to AnySplat geometry. It is a downstream, per-scene refinement experiment—not a new feed-forward model.
+
+## Evaluation protocol
+
+- Apartment: 24 frames total; 12 context views for geometry initialization/training and 12 held-out target views.
+- Kitchen: 32 frames total; 16 context views for geometry initialization/training and 16 held-out target views.
+- All images are center-cropped and resized to 448×448.
+- AnySplat context-only inference initializes the Fourier scene. All-frame AnySplat inference is used to obtain target camera poses and align them into the context reconstruction coordinate frame, matching the evaluation protocol. Held-out target pixels are not used for geometry initialization or Fourier training.
+- Initial seed selection uses AnySplat opacity × maximum scale. Fourier Splatting uses K=6 frequencies with staged activation at iterations 300, 600, 900, 1,200, and 1,500.
+- Each scene is optimized for 6,000 iterations. The final LOD sweep changes active Fourier frequency count while keeping primitive count fixed.
+- Metrics: PSNR (higher is better), SSIM (higher is better), and VGG LPIPS (lower is better). Baseline and Fourier metrics use the same VGG LPIPS implementation.
+
+## Main held-out results
+
+| Scene | Method | PSNR ↑ | SSIM ↑ | VGG LPIPS ↓ | Primitives |
+|---|---|---:|---:|---:|---:|
+| Apartment | AnySplat | 19.1235 | 0.6612 | 0.3154 | 1,858,896 |
+| Apartment | Fourier, K=6 | 20.3469 | 0.7041 | 0.3173 | 99,483 |
+| Kitchen | AnySplat | 21.6155 | 0.6660 | 0.2678 | 1,983,229 |
+| Kitchen | Fourier, K=6 | 25.3078 | 0.8212 | 0.2586 | 120,000 |
+
+### Changes vs AnySplat
+
+- **Apartment:** +1.2234 dB PSNR, +0.0429 SSIM, LPIPS worsens by 0.0019; 18.68× fewer primitives.
+- **Kitchen:** +3.6923 dB PSNR, +0.1552 SSIM, LPIPS improves by 0.0092; 16.53× fewer primitives.
+
+The result is encouraging but not uniform across all metrics: apartment LPIPS is marginally worse, while kitchen improves on all three metrics. These are two indoor scenes, not a benchmark sufficient to claim state of the art.
+
+## Fourier level-of-detail sweep
+
+Primitive count is held constant within each scene. Truncating Fourier frequencies reduces PLY size and trades off perceptual quality.
+
+### Apartment (99,483 primitives)
+
+| K | PLY size (MB) | PSNR ↑ | SSIM ↑ | VGG LPIPS ↓ |
+|---:|---:|---:|---:|---:|
+| 1 | 25.07 | 19.9081 | 0.7048 | 0.3831 |
+| 2 | 25.87 | 20.0612 | 0.7056 | 0.3802 |
+| 3 | 26.66 | 20.3420 | 0.7078 | 0.3551 |
+| 4 | 27.46 | 20.3825 | 0.7071 | 0.3374 |
+| 5 | 28.25 | 20.3771 | 0.7056 | 0.3236 |
+| 6 | 29.05 | 20.3469 | 0.7041 | 0.3173 |
+
+### Kitchen (120,000 primitives)
+
+| K | PLY size (MB) | PSNR ↑ | SSIM ↑ | VGG LPIPS ↓ |
+|---:|---:|---:|---:|---:|
+| 1 | 30.24 | 21.9698 | 0.7638 | 0.3696 |
+| 2 | 31.20 | 23.3089 | 0.7772 | 0.3599 |
+| 3 | 32.16 | 24.3752 | 0.8006 | 0.3186 |
+| 4 | 33.12 | 24.9584 | 0.8137 | 0.2889 |
+| 5 | 34.08 | 25.1819 | 0.8190 | 0.2696 |
+| 6 | 35.04 | 25.3078 | 0.8212 | 0.2586 |
+
+**Takeaway from LOD:** K=1–3 is substantially worse on LPIPS. K=5 is a useful compromise; K=6 gives the best perceptual metric on both scenes. The K=1 to K=6 PLY-size increase is about 15–16%, while the scene representation remains 16–19× smaller in primitive count than the AnySplat output. This report does not claim an equivalent 16–19× reduction in serialized bytes; those AnySplat PLY bytes were not measured in this run.
+
+## Interpretation and limitations
+
+1. **Promising direction:** Fourier Splatting looks useful as an offline scene-refinement / compact-representation stage after AnySplat.
+2. **Not feed-forward:** every scene requires 6,000 optimization iterations. This has not yet been integrated into a low-latency video-to-3D inference path.
+3. **No SOTA claim:** only two indoor scenes have been tested. More scenes, motion, occlusion, camera-path diversity, runtime and byte-level comparisons are required.
+4. **Research-code license:** the upstream Fourier Splatting repository states non-commercial research/evaluation terms. Review licensing before using its implementation in a commercial release.
+
+## Reproduction commands
+
+From the vendored Fourier Splatting repository, with the project environment and compiled rasterizer installed:
+
+~~~bash
+python train.py \
+  --source_path /path/to/fourier_scene \
+  --model_path /path/to/output \
+  --eval --iterations 6000 \
+  --test_iterations 3000 4500 6000 \
+  --save_iterations 3000 4500 6000 \
+  --checkpoint_iterations 3000 4500 6000 \
+  --densify_until_iter 0 --densification_interval 100000 \
+  --fourier_K 6 \
+  --fourier_freq_schedule 300,600,900,1200,1500 \
+  --no_dome --resolution 448
+~~~
+
+The bundle contains the scene transforms and images, seed point clouds, final checkpoints, Fourier PLYs, LOD variants, evaluation outputs, and the preparation/evaluation scripts. Training uses context-only geometry; target images are reserved for held-out evaluation.

experiments/fourier-splatting/README.md

@@ -0,0 +1,70 @@
+---
+license: other
+library_name: pytorch
+pipeline_tag: image-to-3d
+tags:
+- 3d-reconstruction
+- gaussian-splatting
+- fourier-splatting
+- anysplat
+- novel-view-synthesis
+- research
+---
+
+# Polymorphic: Fourier Splatting refinement for AnySplat
+
+Experimental video-to-3D reconstruction workflow. AnySplat initializes scene geometry; Fourier Splatting then optimizes a representation for that specific scene. This release contains scene-specific checkpoints, LOD PLY exports, evaluation results, and scripts.
+
+**Important:** This is not a feed-forward model that instantly converts arbitrary video into a 3D asset. New scenes require per-scene optimization. The PLYs are renderable 3D Gaussian/surfel-style scene representations, not watertight meshes or game-ready GLB files.
+
+## Held-out novel-view results
+
+PSNR and SSIM are higher-is-better; VGG LPIPS is lower-is-better. Baseline and Fourier results use the same VGG LPIPS implementation.
+
+| Scene | Method | PSNR | SSIM | VGG LPIPS | Primitives |
+|---|---|---:|---:|---:|---:|
+| Apartment | AnySplat | 19.1235 | 0.6612 | 0.3154 | 1,858,896 |
+| Apartment | Fourier K=6 | 20.3469 | 0.7041 | 0.3173 | 99,483 |
+| Kitchen | AnySplat | 21.6155 | 0.6660 | 0.2678 | 1,983,229 |
+| Kitchen | Fourier K=6 | 25.3078 | 0.8212 | 0.2586 | 120,000 |
+
+- Apartment: +1.22 dB PSNR, +0.0429 SSIM; LPIPS worsened slightly by 0.0019.
+- Kitchen: +3.69 dB PSNR, +0.1552 SSIM; LPIPS improved by 0.0092.
+- Primitive counts fell by about 18.7× (apartment) and 16.5× (kitchen). This does not establish the same reduction in serialized bytes; baseline PLY sizes were not measured.
+
+## LOD exports
+
+Each scene includes K=1–6 PLY files. Higher K generally improves perceptual quality while increasing file size; K=5 is a possible size/quality compromise, while K=6 had the best LPIPS on both scenes.
+
+| Scene | K | PLY MB | PSNR | SSIM | VGG LPIPS |
+|---|---:|---:|---:|---:|---:|
+| Apartment | 1 | 25.071 | 19.9081 | 0.7048 | 0.3831 |
+| Apartment | 2 | 25.867 | 20.0612 | 0.7056 | 0.3802 |
+| Apartment | 3 | 26.663 | 20.3420 | 0.7078 | 0.3551 |
+| Apartment | 4 | 27.459 | 20.3825 | 0.7071 | 0.3374 |
+| Apartment | 5 | 28.255 | 20.3771 | 0.7056 | 0.3236 |
+| Apartment | 6 | 29.051 | 20.3469 | 0.7041 | 0.3173 |
+| Kitchen | 1 | 30.242 | 21.9698 | 0.7638 | 0.3696 |
+| Kitchen | 2 | 31.202 | 23.3089 | 0.7772 | 0.3599 |
+| Kitchen | 3 | 32.162 | 24.3752 | 0.8006 | 0.3186 |
+| Kitchen | 4 | 33.122 | 24.9584 | 0.8137 | 0.2889 |
+| Kitchen | 5 | 34.082 | 25.1819 | 0.8190 | 0.2696 |
+| Kitchen | 6 | 35.042 | 25.3078 | 0.8212 | 0.2586 |
+
+## Method and evaluation protocol
+
+AnySplat inference on context frames initializes geometry. Fourier Splatting is optimized for 6,000 iterations at 448×448 resolution with staged Fourier-frequency activation. Apartment: 24 frames (12 context, 12 held-out targets). Kitchen: 32 frames (16 context, 16 held-out targets). Held-out target pixels were not used to initialize or train the scene representation; all-frame inference was used only for target camera poses and alignment into the context coordinate frame.
+
+## Contents
+
+- `models/apartment/` and `models/kitchen/`: checkpoints, metrics, and K=1–6 PLY exports.
+- `scripts/`: preparation, evaluation, and LOD export scripts.
+- `FOURIER_SPLATTING_REPORT.md`: detailed protocol and findings.
+- `fourier_results.json`: machine-readable metrics.
+- `THIRD_PARTY_NOTICES.md`: upstream project/license notes.
+
+## Limitations and license
+
+Only two indoor scenes were evaluated; this does not establish generalization or state-of-the-art performance. Reconstruction requires per-scene optimization and depends on camera coverage, scene motion, and AnySplat initialization. The apartment perceptual metric regressed slightly.
+
+AnySplat and Fourier Splatting have separate upstream license terms. Fourier Splatting specifies non-commercial research/evaluation restrictions. Review the bundled license files and obtain required permissions before commercial use or redistribution. This release does not grant broader rights to upstream code, weights, or datasets.

experiments/fourier-splatting/THIRD_PARTY_NOTICES.md

@@ -0,0 +1,13 @@
+# Third-party notices
+
+## Fourier Splatting / Gaussian Splatting code
+
+The upstream Fourier Splatting repository distributes its software under the license included at `licenses/FOURIER_SPLATTING_LICENSE.md`. It permits research and evaluation use and restricts commercial use without prior explicit consent from the licensors. The license also requires redistribution under the same license with the full license text and attribution notices retained. Review it before using or redistributing the code, checkpoints, or derivatives.
+
+## AnySplat
+
+AnySplat's MIT license is included at `licenses/ANYSPLAT_LICENSE`. Retain the copyright and permission notice when redistributing relevant code.
+
+## Scope
+
+This repository is an experimental integration and evaluation release. The notices above do not grant rights to third-party datasets or any assets that may have separate terms.

experiments/fourier-splatting/fourier_results.json

@@ -0,0 +1,143 @@
+{
+  "protocol": {
+    "resolution": [
+      448,
+      448
+    ],
+    "apartment": {
+      "context_views": 12,
+      "heldout_views": 12,
+      "iterations": 6000
+    },
+    "kitchen": {
+      "context_views": 16,
+      "heldout_views": 16,
+      "iterations": 6000
+    },
+    "lpips": "VGG"
+  },
+  "main": [
+    {
+      "scene": "apartment",
+      "method": "AnySplat",
+      "psnr": 19.123464584350586,
+      "ssim": 0.6611772179603577,
+      "lpips_vgg": 0.3154071271419525,
+      "primitives": 1858896
+    },
+    {
+      "scene": "apartment",
+      "method": "Fourier K=6",
+      "psnr": 20.346883535385132,
+      "ssim": 0.7040959944327673,
+      "lpips_vgg": 0.3172961249947548,
+      "primitives": 99483
+    },
+    {
+      "scene": "kitchen",
+      "method": "AnySplat",
+      "psnr": 21.61551284790039,
+      "ssim": 0.6659506559371948,
+      "lpips_vgg": 0.26777008175849915,
+      "primitives": 1983229
+    },
+    {
+      "scene": "kitchen",
+      "method": "Fourier K=6",
+      "psnr": 25.307796597480774,
+      "ssim": 0.8211556077003479,
+      "lpips_vgg": 0.25855699460953474,
+      "primitives": 120000
+    }
+  ],
+  "lod": {
+    "apartment": [
+      {
+        "K": 1,
+        "bytes": 25071277,
+        "PSNR": 19.908107837041218,
+        "SSIM": 0.704794724782308,
+        "LPIPS_VGG": 0.38314763208230335
+      },
+      {
+        "K": 2,
+        "bytes": 25867197,
+        "PSNR": 20.061190923055012,
+        "SSIM": 0.705554003516833,
+        "LPIPS_VGG": 0.38020747403303784
+      },
+      {
+        "K": 3,
+        "bytes": 26663117,
+        "PSNR": 20.342023849487305,
+        "SSIM": 0.7078102082014084,
+        "LPIPS_VGG": 0.3551347454388936
+      },
+      {
+        "K": 4,
+        "bytes": 27459037,
+        "PSNR": 20.382510662078857,
+        "SSIM": 0.7071239352226257,
+        "LPIPS_VGG": 0.3374286914865176
+      },
+      {
+        "K": 5,
+        "bytes": 28254957,
+        "PSNR": 20.37706486384074,
+        "SSIM": 0.7056431223948797,
+        "LPIPS_VGG": 0.32359255105257034
+      },
+      {
+        "K": 6,
+        "bytes": 29050877,
+        "PSNR": 20.346883535385132,
+        "SSIM": 0.7040959944327673,
+        "LPIPS_VGG": 0.3172961249947548
+      }
+    ],
+    "kitchen": [
+      {
+        "K": 1,
+        "bytes": 30241562,
+        "PSNR": 21.96980655193329,
+        "SSIM": 0.7638169042766094,
+        "LPIPS_VGG": 0.3695769216865301
+      },
+      {
+        "K": 2,
+        "bytes": 31201618,
+        "PSNR": 23.308930158615112,
+        "SSIM": 0.7771734595298767,
+        "LPIPS_VGG": 0.3599471654742956
+      },
+      {
+        "K": 3,
+        "bytes": 32161674,
+        "PSNR": 24.375155806541443,
+        "SSIM": 0.8006471283733845,
+        "LPIPS_VGG": 0.3186379112303257
+      },
+      {
+        "K": 4,
+        "bytes": 33121730,
+        "PSNR": 24.958438515663147,
+        "SSIM": 0.8136776946485043,
+        "LPIPS_VGG": 0.28893621638417244
+      },
+      {
+        "K": 5,
+        "bytes": 34081786,
+        "PSNR": 25.18192982673645,
+        "SSIM": 0.8189537934958935,
+        "LPIPS_VGG": 0.26962575130164623
+      },
+      {
+        "K": 6,
+        "bytes": 35041842,
+        "PSNR": 25.307796597480774,
+        "SSIM": 0.8211556077003479,
+        "LPIPS_VGG": 0.25855699460953474
+      }
+    ]
+  }
+}

experiments/fourier-splatting/licenses/ANYSPLAT_LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Lihan Jiang and Yucheng Mao
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

experiments/fourier-splatting/licenses/FOURIER_SPLATTING_LICENSE.md

@@ -0,0 +1,83 @@
+Gaussian-Splatting License  
+===========================  
+
+**Inria** and **the Max Planck Institut for Informatik (MPII)** hold all the ownership rights on the *Software* named **gaussian-splatting**.  
+The *Software* is in the process of being registered with the Agence pour la Protection des  
+Programmes (APP).  
+
+The *Software* is still being developed by the *Licensor*.  
+
+*Licensor*'s goal is to allow the research community to use, test and evaluate  
+the *Software*.  
+
+## 1.  Definitions  
+
+*Licensee* means any person or entity that uses the *Software* and distributes  
+its *Work*.  
+
+*Licensor* means the owners of the *Software*, i.e Inria and MPII  
+
+*Software* means the original work of authorship made available under this  
+License ie gaussian-splatting.  
+
+*Work* means the *Software* and any additions to or derivative works of the  
+*Software* that are made available under this License.  
+
+
+## 2.  Purpose  
+This license is intended to define the rights granted to the *Licensee* by  
+Licensors under the *Software*.  
+
+## 3.  Rights granted  
+
+For the above reasons Licensors have decided to distribute the *Software*.  
+Licensors grant non-exclusive rights to use the *Software* for research purposes  
+to research users (both academic and industrial), free of charge, without right  
+to sublicense.. The *Software* may be used "non-commercially", i.e., for research  
+and/or evaluation purposes only.  
+
+Subject to the terms and conditions of this License, you are granted a  
+non-exclusive, royalty-free, license to reproduce, prepare derivative works of,  
+publicly display, publicly perform and distribute its *Work* and any resulting  
+derivative works in any form.  
+
+## 4.  Limitations  
+
+**4.1 Redistribution.** You may reproduce or distribute the *Work* only if (a) you do  
+so under this License, (b) you include a complete copy of this License with  
+your distribution, and (c) you retain without modification any copyright,  
+patent, trademark, or attribution notices that are present in the *Work*.  
+
+**4.2 Derivative Works.** You may specify that additional or different terms apply  
+to the use, reproduction, and distribution of your derivative works of the *Work*  
+("Your Terms") only if (a) Your Terms provide that the use limitation in  
+Section 2 applies to your derivative works, and (b) you identify the specific  
+derivative works that are subject to Your Terms. Notwithstanding Your Terms,  
+this License (including the redistribution requirements in Section 3.1) will  
+continue to apply to the *Work* itself.  
+
+**4.3** Any other use without of prior consent of Licensors is prohibited. Research  
+users explicitly acknowledge having received from Licensors all information  
+allowing to appreciate the adequacy between of the *Software* and their needs and  
+to undertake all necessary precautions for its execution and use.  
+
+**4.4** The *Software* is provided both as a compiled library file and as source  
+code. In case of using the *Software* for a publication or other results obtained  
+through the use of the *Software*, users are strongly encouraged to cite the  
+corresponding publications as explained in the documentation of the *Software*.  
+
+## 5.  Disclaimer  
+
+THE USER CANNOT USE, EXPLOIT OR DISTRIBUTE THE *SOFTWARE* FOR COMMERCIAL PURPOSES  
+WITHOUT PRIOR AND EXPLICIT CONSENT OF LICENSORS. YOU MUST CONTACT INRIA FOR ANY  
+UNAUTHORIZED USE: stip-sophia.transfert@inria.fr . ANY SUCH ACTION WILL  
+CONSTITUTE A FORGERY. THIS *SOFTWARE* IS PROVIDED "AS IS" WITHOUT ANY WARRANTIES  
+OF ANY NATURE AND ANY EXPRESS OR IMPLIED WARRANTIES, WITH REGARDS TO COMMERCIAL  
+USE, PROFESSIONNAL USE, LEGAL OR NOT, OR OTHER, OR COMMERCIALISATION OR  
+ADAPTATION. UNLESS EXPLICITLY PROVIDED BY LAW, IN NO EVENT, SHALL INRIA OR THE  
+AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR  
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE  
+GOODS OR SERVICES, LOSS OF USE, DATA, OR PROFITS OR BUSINESS INTERRUPTION)  
+HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT  
+LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING FROM, OUT OF OR  
+IN CONNECTION WITH THE *SOFTWARE* OR THE USE OR OTHER DEALINGS IN THE *SOFTWARE*.  

experiments/fourier-splatting/models/apartment/all_args.json

@@ -0,0 +1,69 @@
+{
+  "model_params": {
+    "sh_degree": 3,
+    "source_path": "/home/dev/polymorphic/fourier_scene_apt",
+    "model_path": "/home/dev/polymorphic/runs/fourier_apt_6000",
+    "images": "images",
+    "resolution": 448,
+    "white_background": false,
+    "data_device": "cuda",
+    "eval": true,
+    "render_items": [
+      "RGB",
+      "Alpha",
+      "Normal",
+      "Depth",
+      "Edge",
+      "Curvature"
+    ],
+    "fourier_K": 6,
+    "fourier_sigma": 1.16,
+    "fourier_freq_schedule": "300,600,900,1200,1500",
+    "lambda_fourier_reg": 0.0001,
+    "no_dome": true,
+    "beta_color": false,
+    "sb_lobes": 2
+  },
+  "optimization_params": {
+    "iterations": 6000,
+    "position_lr_init": 0.00016,
+    "position_lr_final": 1.6e-06,
+    "position_lr_delay_mult": 0.01,
+    "position_lr_max_steps": 30000,
+    "feature_lr": 0.0025,
+    "opacity_lr": 0.014,
+    "scaling_lr": 0.005,
+    "rotation_lr": 0.001,
+    "sigma_lr": 0.0008,
+    "lambda_dssim": 0.2,
+    "lambda_dist": 0.0,
+    "lambda_normal": 0.05,
+    "lambda_freq": 0.01,
+    "freq_cutoff_ratio": 0.5,
+    "opacity_reg": 0.0,
+    "scale_reg": 0.005,
+    "densification_interval": 100000,
+    "densify_from_iter": 500,
+    "densify_until_iter": 0,
+    "mcmc_opacity_threshold": 0.005,
+    "mcmc_importance_threshold": 0.04,
+    "mcmc_min_views": 2,
+    "mcmc_birth_rate": 0.3,
+    "mcmc_max_primitives": 5000000,
+    "mcmc_sgld_noise_lr": 1e-05,
+    "mcmc_apply_sgld": true,
+    "mcmc_sgld_noise_lr_pos": 500.0,
+    "mcmc_split_from_iter": 3000,
+    "mcmc_split_grad_threshold": 0.0004,
+    "mcmc_split_screen_threshold": 40.0,
+    "mcmc_split_screen_size": 100.0,
+    "mcmc_splitter_checkpoint": "checkpoints/lobe_splitter",
+    "mcmc_max_screen_size": 1400.0
+  },
+  "pipeline_params": {
+    "convert_SHs_python": false,
+    "compute_cov3D_python": false,
+    "depth_ratio": 0.0,
+    "debug": false
+  }
+}

experiments/fourier-splatting/models/apartment/chkpnt6000.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:124f90c4184cd5a3318cbb21c401e4ecdd6ab8797f21fb982117f9636db1eca1
+size 28655349

experiments/fourier-splatting/models/apartment/lod/fourier_k1.ply

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7c1ad989efc80ecec67be220bb45ee5123b3dfe16a3249e8889c51038fe482b5
+size 25071277

experiments/fourier-splatting/models/apartment/lod/fourier_k2.ply

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0330ec57672a038c50123fc990f5facbfc9fc1a9c04879ecaa186530b11c4f4c
+size 25867197

experiments/fourier-splatting/models/apartment/lod/fourier_k3.ply

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c5d3bbe9035fb40529108efdce8619630c49291e752a412947437f878cb5995e
+size 26663117

experiments/fourier-splatting/models/apartment/lod/fourier_k4.ply

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50a8b4f7e9694c6fc1c90c4ddababeb7c2d0aabd2129f3a35ef90c44e71b0dce
+size 27459037

experiments/fourier-splatting/models/apartment/lod/fourier_k5.ply

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f501f029aa57e8c29de90b1303f0e1afdcd9b5fc4ec6da60994e75748796f84
+size 28254957

experiments/fourier-splatting/models/apartment/lod/fourier_k6.ply

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b8083821b5bd6e03f6cc6139893f9087cb0f4ec33bc1e9a7d588b15815684d4
+size 29050877

experiments/fourier-splatting/models/apartment/lod_metrics.json

@@ -0,0 +1,54 @@
+{
+  "checkpoint_iteration": 6000,
+  "scene": "vrnerf_apartment",
+  "results": [
+    {
+      "K": 1,
+      "PSNR": 19.908107837041218,
+      "SSIM": 0.704794724782308,
+      "LPIPS_VGG": 0.38314763208230335,
+      "primitives": 99483,
+      "heldout_views": 12
+    },
+    {
+      "K": 2,
+      "PSNR": 20.061190923055012,
+      "SSIM": 0.705554003516833,
+      "LPIPS_VGG": 0.38020747403303784,
+      "primitives": 99483,
+      "heldout_views": 12
+    },
+    {
+      "K": 3,
+      "PSNR": 20.342023849487305,
+      "SSIM": 0.7078102082014084,
+      "LPIPS_VGG": 0.3551347454388936,
+      "primitives": 99483,
+      "heldout_views": 12
+    },
+    {
+      "K": 4,
+      "PSNR": 20.382510662078857,
+      "SSIM": 0.7071239352226257,
+      "LPIPS_VGG": 0.3374286914865176,
+      "primitives": 99483,
+      "heldout_views": 12
+    },
+    {
+      "K": 5,
+      "PSNR": 20.37706486384074,
+      "SSIM": 0.7056431223948797,
+      "LPIPS_VGG": 0.32359255105257034,
+      "primitives": 99483,
+      "heldout_views": 12
+    },
+    {
+      "K": 6,
+      "PSNR": 20.346883535385132,
+      "SSIM": 0.7040959944327673,
+      "LPIPS_VGG": 0.3172961249947548,
+      "primitives": 99483,
+      "heldout_views": 12
+    }
+  ]
+}

experiments/fourier-splatting/models/kitchen/all_args.json

@@ -0,0 +1,69 @@
+{
+  "model_params": {
+    "sh_degree": 3,
+    "source_path": "/home/dev/polymorphic/fourier_scene_kitchen",
+    "model_path": "/home/dev/polymorphic/runs/fourier_kitchen_6000",
+    "images": "images",
+    "resolution": 448,
+    "white_background": false,
+    "data_device": "cuda",
+    "eval": true,
+    "render_items": [
+      "RGB",
+      "Alpha",
+      "Normal",
+      "Depth",
+      "Edge",
+      "Curvature"
+    ],
+    "fourier_K": 6,
+    "fourier_sigma": 1.16,
+    "fourier_freq_schedule": "300,600,900,1200,1500",
+    "lambda_fourier_reg": 0.0001,
+    "no_dome": true,
+    "beta_color": false,
+    "sb_lobes": 2
+  },
+  "optimization_params": {
+    "iterations": 6000,
+    "position_lr_init": 0.00016,
+    "position_lr_final": 1.6e-06,
+    "position_lr_delay_mult": 0.01,
+    "position_lr_max_steps": 30000,
+    "feature_lr": 0.0025,
+    "opacity_lr": 0.014,
+    "scaling_lr": 0.005,
+    "rotation_lr": 0.001,
+    "sigma_lr": 0.0008,
+    "lambda_dssim": 0.2,
+    "lambda_dist": 0.0,
+    "lambda_normal": 0.05,
+    "lambda_freq": 0.01,
+    "freq_cutoff_ratio": 0.5,
+    "opacity_reg": 0.0,
+    "scale_reg": 0.005,
+    "densification_interval": 100000,
+    "densify_from_iter": 500,
+    "densify_until_iter": 0,
+    "mcmc_opacity_threshold": 0.005,
+    "mcmc_importance_threshold": 0.04,
+    "mcmc_min_views": 2,
+    "mcmc_birth_rate": 0.3,
+    "mcmc_max_primitives": 5000000,
+    "mcmc_sgld_noise_lr": 1e-05,
+    "mcmc_apply_sgld": true,
+    "mcmc_sgld_noise_lr_pos": 500.0,
+    "mcmc_split_from_iter": 3000,
+    "mcmc_split_grad_threshold": 0.0004,
+    "mcmc_split_screen_threshold": 40.0,
+    "mcmc_split_screen_size": 100.0,
+    "mcmc_splitter_checkpoint": "checkpoints/lobe_splitter",
+    "mcmc_max_screen_size": 1400.0
+  },
+  "pipeline_params": {
+    "convert_SHs_python": false,
+    "compute_cov3D_python": false,
+    "depth_ratio": 0.0,
+    "debug": false
+  }
+}

experiments/fourier-splatting/models/kitchen/chkpnt6000.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f1da364c8d64f8017a770c6338482ce5abecb546a04bd7ee4fa3d5dc31a69da8
+size 34564597

experiments/fourier-splatting/models/kitchen/lod/fourier_k1.ply

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:09b401a466e42dbd2e429a1bf6f6ec75c3ebd0eaebc477c73a1bc01e3d5ea373
+size 30241562

experiments/fourier-splatting/models/kitchen/lod/fourier_k2.ply

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:61436222a425b6750fe2eda119b9a6fcbb1917c8fd34b9d89298045430686889
+size 31201618

experiments/fourier-splatting/models/kitchen/lod/fourier_k3.ply

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5b4d52e47ad1faa55c0c220cdab29b136bb8ab9fd8e14e07f4506cdc727cf571
+size 32161674

experiments/fourier-splatting/models/kitchen/lod/fourier_k4.ply

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:140df0357737cd22b8866f07e2930d253d386de74b42e4a3ec1545f430f69880
+size 33121730

experiments/fourier-splatting/models/kitchen/lod/fourier_k5.ply

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:38edaf45c848aea02e5c2f2604d9b446b9e1097953e2cf2b33e83fe28a466cfb
+size 34081786

experiments/fourier-splatting/models/kitchen/lod/fourier_k6.ply

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:df2fd67624e902d01aaae025b82e83a008fae37975ccfe5f72da9c8a3e0aeeec
+size 35041842

experiments/fourier-splatting/models/kitchen/lod_metrics.json

@@ -0,0 +1,54 @@
+{
+  "checkpoint_iteration": 6000,
+  "scene": "vrnerf_apartment",
+  "results": [
+    {
+      "K": 1,
+      "PSNR": 21.96980655193329,
+      "SSIM": 0.7638169042766094,
+      "LPIPS_VGG": 0.3695769216865301,
+      "primitives": 120000,
+      "heldout_views": 16
+    },
+    {
+      "K": 2,
+      "PSNR": 23.308930158615112,
+      "SSIM": 0.7771734595298767,
+      "LPIPS_VGG": 0.3599471654742956,
+      "primitives": 120000,
+      "heldout_views": 16
+    },
+    {
+      "K": 3,
+      "PSNR": 24.375155806541443,
+      "SSIM": 0.8006471283733845,
+      "LPIPS_VGG": 0.3186379112303257,
+      "primitives": 120000,
+      "heldout_views": 16
+    },
+    {
+      "K": 4,
+      "PSNR": 24.958438515663147,
+      "SSIM": 0.8136776946485043,
+      "LPIPS_VGG": 0.28893621638417244,
+      "primitives": 120000,
+      "heldout_views": 16
+    },
+    {
+      "K": 5,
+      "PSNR": 25.18192982673645,
+      "SSIM": 0.8189537934958935,
+      "LPIPS_VGG": 0.26962575130164623,
+      "primitives": 120000,
+      "heldout_views": 16
+    },
+    {
+      "K": 6,
+      "PSNR": 25.307796597480774,
+      "SSIM": 0.8211556077003479,
+      "LPIPS_VGG": 0.25855699460953474,
+      "primitives": 120000,
+      "heldout_views": 16
+    }
+  ]
+}

experiments/fourier-splatting/scripts/eval_lod_sweep.py

@@ -0,0 +1,51 @@
+import argparse, json, os, sys, time
+from pathlib import Path
+import numpy as np
+import torch
+from PIL import Image
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from arguments import ModelParams, PipelineParams, OptimizationParams
+from scene import Scene, FourierModel
+from gaussian_renderer import render
+from utils.image_utils import psnr
+from utils.loss_utils import ssim
+from lpipsPyTorch import lpips as compute_lpips
+
+parser = argparse.ArgumentParser(description='Fourier Splatting LOD sweep')
+lp = ModelParams(parser)
+op = OptimizationParams(parser)
+pp = PipelineParams(parser)
+parser.add_argument('--load_iteration', type=int, default=6000)
+parser.add_argument('--lod_values', type=str, default='1,2,3,4,5,6')
+args = parser.parse_args()
+dataset, opt, pipe = lp.extract(args), op.extract(args), pp.extract(args)
+gaussians = FourierModel(dataset.sh_degree, fourier_K=dataset.fourier_K,
+                          use_beta_color=dataset.beta_color, sb_lobes=dataset.sb_lobes)
+scene = Scene(dataset, gaussians, load_iteration=args.load_iteration, shuffle=False)
+bg = torch.tensor([1,1,1] if dataset.white_background else [0,0,0], dtype=torch.float32, device='cuda')
+lod_values = [int(x) for x in args.lod_values.split(',')]
+results = []
+vis_dir = Path(dataset.model_path) / 'lod_visuals'
+vis_dir.mkdir(parents=True, exist_ok=True)
+with torch.inference_mode():
+    for k in lod_values:
+        gaussians.active_K = min(k, gaussians.fourier_K)
+        pvals, svals, lvals = [], [], []
+        for j, cam in enumerate(scene.getTestCameras()):
+            pred = torch.clamp(render(cam, gaussians, pipe, bg)['render'], 0, 1).unsqueeze(0)
+            gt = torch.clamp(cam.original_image.cuda(), 0, 1).unsqueeze(0)
+            pvals.append(float(psnr(pred, gt).item()))
+            svals.append(float(ssim(pred, gt).item()))
+            lvals.append(float(compute_lpips(pred, gt, net_type='vgg').item()))
+            if j in (0, len(scene.getTestCameras())//2):
+                side = torch.cat([gt[0], pred[0]], dim=2).permute(1,2,0).cpu().numpy()
+                Image.fromarray((side*255).clip(0,255).astype(np.uint8)).save(vis_dir / f'k{k}_view{j}.png')
+        row = {'K': k, 'PSNR': float(np.mean(pvals)), 'SSIM': float(np.mean(svals)),
+               'LPIPS_VGG': float(np.mean(lvals)), 'primitives': int(gaussians.get_xyz.shape[0]),
+               'heldout_views': len(scene.getTestCameras())}
+        print(json.dumps(row), flush=True)
+        results.append(row)
+out = {'checkpoint_iteration': args.load_iteration, 'scene': 'vrnerf_apartment', 'results': results}
+path = Path(dataset.model_path) / 'lod_metrics.json'
+path.write_text(json.dumps(out, indent=2))
+print('WROTE', path, flush=True)

experiments/fourier-splatting/scripts/eval_nvs_vgg.py

@@ -0,0 +1,65 @@
+"""Held-out novel-view evaluation for pose-free feed-forward splatting.
+context = even frames, targets = odd frames. Target poses come from a joint run on all frames,
+aligned to the context-only reconstruction with a similarity transform (Umeyama) on camera centres."""
+import sys, os, time, json, glob, subprocess
+sys.path.insert(0, os.path.expanduser('~/polymorphic/vendor/AnySplat'))
+import numpy as np, torch, lpips
+from pathlib import Path
+from PIL import Image
+import torchvision.transforms.functional as TF
+from skimage.metrics import structural_similarity
+from src.model.model.anysplat import AnySplat
+
+def umeyama(src, dst):
+    mu_s, mu_d = src.mean(0), dst.mean(0); xs, xd = src-mu_s, dst-mu_d
+    cov = xd.T @ xs / len(src); U,S,Vt = np.linalg.svd(cov)
+    D = np.eye(3); D[2,2] = np.sign(np.linalg.det(U)*np.linalg.det(Vt))
+    R = U @ D @ Vt; s = (S*np.diag(D)).sum() / xs.var(0).sum(); t = mu_d - s*R@mu_s
+    return s, R, t
+
+def load(f, mode, long_edge):
+    im = Image.open(f).convert('RGB'); w,h = im.size
+    if mode == 'square':
+        s = long_edge/ min(w,h); im = im.resize((round(w*s), round(h*s)), Image.BICUBIC); w,h = im.size
+        l,t = (w-long_edge)//2, (h-long_edge)//2; im = im.crop((l,t,l+long_edge,t+long_edge))
+    else:
+        s = long_edge/max(w,h); im = im.resize((max(14,round(w*s/14)*14), max(14,round(h*s/14)*14)), Image.BICUBIC)
+    return TF.to_tensor(im)
+
+def main(video, total, mode, long_edge, outdir):
+    out = Path(outdir); fr = out/'frames'; fr.mkdir(parents=True, exist_ok=True)
+    for f in fr.glob('*.jpg'): f.unlink()
+    dur = float(subprocess.check_output(['ffprobe','-v','error','-show_entries','format=duration','-of','csv=p=0',video]).decode())
+    subprocess.run(['ffmpeg','-hide_banner','-loglevel','error','-i',video,'-vf',f'fps={total/dur}','-q:v','2','-frames:v',str(total),str(fr/'%04d.jpg'),'-y'],check=True)
+    files = sorted(glob.glob(str(fr/'*.jpg'))); ctx_i = list(range(0,len(files),2)); tgt_i = list(range(1,len(files),2))
+    imgs = torch.stack([load(f, mode, long_edge) for f in files]).cuda()
+    model = AnySplat.from_pretrained('lhjiang/anysplat').cuda().eval()
+    for p in model.parameters(): p.requires_grad=False
+    with torch.no_grad():
+        gA, pA = model.inference(imgs[ctx_i].unsqueeze(0)); gB, pB = model.inference(imgs.unsqueeze(0))
+    eA, eB, iB = pA['extrinsic'][0].double().cpu().numpy(), pB['extrinsic'][0].double().cpu().numpy(), pB['intrinsic'][0].float()
+    s, R, t = umeyama(eB[ctx_i,:3,3], eA[:,:3,3])
+    resid = np.linalg.norm((s*(R@eB[ctx_i,:3,3].T)).T + t - eA[:,:3,3], axis=1)
+    ext_t = []
+    for i in tgt_i:
+        E = np.eye(4); E[:3,:3] = R @ eB[i,:3,:3]; E[:3,3] = s*R@eB[i,:3,3] + t; ext_t.append(E)
+    ext_t = torch.tensor(np.stack(ext_t), dtype=torch.float32, device='cuda').unsqueeze(0); intr_t = iB[tgt_i].unsqueeze(0).cuda()
+    H,W = imgs.shape[-2:]
+    with torch.no_grad():
+        r = model.decoder.forward(gA, ext_t, intr_t, torch.ones(1,len(tgt_i),device='cuda')*.1, torch.ones(1,len(tgt_i),device='cuda')*100,(H,W)).color[0].clamp(0,1)
+    gt = imgs[tgt_i]
+    # compare on central square so square/rect modes are scored on the same pixels
+    m = min(H,W); t0,l0 = (H-m)//2,(W-m)//2
+    rc, gc = r[:,:,t0:t0+m,l0:l0+m], gt[:,:,t0:t0+m,l0:l0+m]
+    psnr = (-10*torch.log10(((rc-gc)**2).mean(dim=(1,2,3)))).cpu().numpy()
+    ssim = np.array([structural_similarity(a.permute(1,2,0).cpu().numpy(), b.permute(1,2,0).cpu().numpy(), channel_axis=2, data_range=1.0) for a,b in zip(rc,gc)])
+    lp = lpips.LPIPS(net='vgg').cuda().eval()
+    with torch.no_grad(): lpv = lp(rc*2-1, gc*2-1).flatten().cpu().numpy()
+    res = dict(video=Path(video).name, mode=mode, long_edge=long_edge, input_hw=[H,W], n_ctx=len(ctx_i), n_tgt=len(tgt_i), gaussians=int(gA.means.shape[1]),
+               align_resid_mean=float(resid.mean()), PSNR=float(psnr.mean()), SSIM=float(ssim.mean()), LPIPS=float(lpv.mean()))
+    print(json.dumps(res)); (out/f'eval_{mode}_{long_edge}.json').write_text(json.dumps(res, indent=1))
+    for k in (0, len(tgt_i)//2):
+        Image.fromarray((torch.cat([gc[k],rc[k]],-1).permute(1,2,0).cpu().numpy()*255).astype(np.uint8)).save(out/f'nvs_{mode}_{long_edge}_{k}.png')
+
+if __name__ == '__main__':
+    main(sys.argv[1], int(sys.argv[2]), sys.argv[3], int(sys.argv[4]), sys.argv[5])

experiments/fourier-splatting/scripts/export_lod_ply.py

@@ -0,0 +1,34 @@
+import argparse, json
+from pathlib import Path
+import torch
+from arguments import ModelParams, PipelineParams, OptimizationParams
+from scene import Scene, FourierModel
+
+parser = argparse.ArgumentParser()
+lp = ModelParams(parser); op = OptimizationParams(parser); pp = PipelineParams(parser)
+parser.add_argument('--load_iteration', type=int, default=6000)
+args = parser.parse_args()
+dataset = lp.extract(args)
+gaussians = FourierModel(dataset.sh_degree, fourier_K=dataset.fourier_K,
+                         use_beta_color=dataset.beta_color, sb_lobes=dataset.sb_lobes)
+scene = Scene(dataset, gaussians, load_iteration=args.load_iteration, shuffle=False)
+out = Path(dataset.model_path) / 'lod_models'
+out.mkdir(parents=True, exist_ok=True)
+full_r = gaussians._radii_logits.detach().clone()
+full_p = gaussians._phases_raw.detach().clone()
+full_k = gaussians.fourier_K
+rows = []
+for k in range(1, full_k+1):
+    gaussians._radii_logits = torch.nn.Parameter(full_r[:, :k].clone().requires_grad_(False))
+    gaussians._phases_raw = torch.nn.Parameter(full_p[:, :k].clone().requires_grad_(False))
+    gaussians.fourier_K = k
+    gaussians.active_K = k
+    path = out / f'fourier_k{k}.ply'
+    gaussians.save_ply(str(path))
+    rows.append({'K':k,'path':str(path),'bytes':path.stat().st_size,'primitives':int(gaussians.get_xyz.shape[0])})
+gaussians._radii_logits = torch.nn.Parameter(full_r)
+gaussians._phases_raw = torch.nn.Parameter(full_p)
+gaussians.fourier_K = full_k
+gaussians.active_K = full_k
+(out/'sizes.json').write_text(json.dumps(rows,indent=2))
+print(json.dumps(rows,indent=2))

experiments/fourier-splatting/scripts/make_fourier_scene.py

@@ -0,0 +1,93 @@
+import json, math, sys, time
+from pathlib import Path
+import numpy as np
+import torch
+from PIL import Image
+from plyfile import PlyData, PlyElement
+sys.path.insert(0, '/home/dev/polymorphic/vendor/AnySplat')
+from src.model.model.anysplat import AnySplat
+from src.utils.image import process_image
+
+ROOT = Path('/home/dev/polymorphic')
+OUT = ROOT / 'fourier_scene_apt'
+FRAMES = sorted((ROOT / 'exp_apt' / 'frames').glob('*.jpg'))
+OUT.mkdir(parents=True, exist_ok=True)
+(OUT / 'images').mkdir(exist_ok=True)
+assert len(FRAMES) == 24, len(FRAMES)
+device = 'cuda'
+imgs = torch.stack([process_image(str(f)) for f in FRAMES]).unsqueeze(0).to(device)
+model = AnySplat.from_pretrained('lhjiang/anysplat').to(device).eval()
+for p in model.parameters(): p.requires_grad_(False)
+ctx_i = list(range(0, len(FRAMES), 2))
+tgt_i = list(range(1, len(FRAMES), 2))
+with torch.inference_mode():
+    g_ctx, p_ctx = model.inference((imgs[:, ctx_i] + 1) * 0.5)
+    g_all, p_all = model.inference((imgs + 1) * 0.5)
+
+# Align the all-frame predicted target cameras into the context-only reconstruction frame,
+# matching the held-out AnySplat evaluation protocol.
+src = p_all['extrinsic'][0, ctx_i, :3, 3].double().cpu().numpy()
+dst = p_ctx['extrinsic'][0, :, :3, 3].double().cpu().numpy()
+mu_s, mu_d = src.mean(0), dst.mean(0)
+xs, xd = src-mu_s, dst-mu_d
+cov = xd.T @ xs / len(src)
+U,S,Vt = np.linalg.svd(cov)
+D = np.eye(3); D[2,2] = np.sign(np.linalg.det(U)*np.linalg.det(Vt))
+R = U @ D @ Vt
+scale = (S*np.diag(D)).sum() / (xs**2).sum(axis=1).mean()
+trans = mu_d - scale * R @ mu_s
+
+H, W = imgs.shape[-2:]
+fovxs = []
+def blender_matrix(c2w):
+    # FourierSplatting's Blender loader flips camera Y/Z. Pre-flip so that
+    # loading recovers the original AnySplat camera-to-world matrix.
+    return (c2w @ np.diag([1., -1., -1., 1.])).tolist()
+
+train_frames, test_frames = [], []
+for i, f in enumerate(FRAMES):
+    im = Image.open(f).convert('RGB')
+    w, h = im.size
+    side = min(w, h)
+    left, top = (w-side)//2, (h-side)//2
+    im = im.crop((left, top, left+side, top+side)).resize((W,H), Image.Resampling.LANCZOS)
+    name = f'{i+1:04d}.png'
+    im.save(OUT / 'images' / name)
+    if i in ctx_i:
+        j = ctx_i.index(i)
+        c2w = p_ctx['extrinsic'][0,j].float().cpu().numpy()
+        intr = p_ctx['intrinsic'][0,j].float().cpu().numpy()
+        rec = {'file_path':f'images/{name[:-4]}','transform_matrix':blender_matrix(c2w),'index':i}
+        train_frames.append(rec)
+    else:
+        c2w0 = p_all['extrinsic'][0,i].float().cpu().numpy()
+        c2w = np.eye(4, dtype=np.float32)
+        c2w[:3,:3] = R @ c2w0[:3,:3]
+        c2w[:3,3] = scale * R @ c2w0[:3,3] + trans
+        intr = p_all['intrinsic'][0,i].float().cpu().numpy()
+        rec = {'file_path':f'images/{name[:-4]}','transform_matrix':blender_matrix(c2w),'index':i}
+        test_frames.append(rec)
+    fovxs.append(2.0 * math.atan(1.0 / (2.0 * float(intr[0,0]))))
+
+fovx = float(np.median(fovxs))
+(OUT/'transforms_train.json').write_text(json.dumps({'camera_angle_x':fovx,'frames':train_frames},indent=2))
+(OUT/'transforms_test.json').write_text(json.dumps({'camera_angle_x':fovx,'frames':test_frames},indent=2))
+
+# Seed from context-only AnySplat primitives; no target-frame geometry enters initialization.
+means = g_ctx.means[0].float()
+scales = g_ctx.scales[0].float()
+opacity = g_ctx.opacities[0].float().reshape(-1)
+harm = g_ctx.harmonics[0].float()
+score = opacity * scales.max(dim=1).values.clamp_min(1e-7)
+n = min(120_000, len(score))
+idx = torch.topk(score, n, sorted=False).indices
+xyz = means[idx].cpu().numpy().astype(np.float32)
+rgb = (0.5 + 0.28209479177387814 * harm[idx, :, 0]).clamp(0,1).cpu().numpy()
+verts = np.empty(n,dtype=[('x','f4'),('y','f4'),('z','f4'),('nx','f4'),('ny','f4'),('nz','f4'),('red','u1'),('green','u1'),('blue','u1')])
+verts['x'],verts['y'],verts['z'] = xyz.T
+verts['nx'],verts['ny'],verts['nz'] = 0,0,0
+verts['red'],verts['green'],verts['blue'] = (rgb*255).astype(np.uint8).T
+PlyData([PlyElement.describe(verts,'vertex')]).write(str(OUT/'points3d.ply'))
+meta={'source':'AnySplat context-only geometry initialization','total_video_frames':len(FRAMES),'context_views':len(train_frames),'heldout_views':len(test_frames),'seed_primitives':n,'image_hw':[H,W],'camera_fov_x':fovx,'pose_alignment_scale':float(scale),'pose_alignment_translation_norm':float(np.linalg.norm(trans)),'heldout_geometry_used_for_initialization':False}
+(OUT/'seed_metadata.json').write_text(json.dumps(meta,indent=2))
+print(json.dumps(meta,indent=2),flush=True)

experiments/fourier-splatting/scripts/make_fourier_scene_generic.py

@@ -0,0 +1,64 @@
+import json, math, sys
+from pathlib import Path
+import numpy as np
+import torch
+from PIL import Image
+from plyfile import PlyData, PlyElement
+sys.path.insert(0, '/home/dev/polymorphic/vendor/AnySplat')
+from src.model.model.anysplat import AnySplat
+from src.utils.image import process_image
+
+if len(sys.argv) != 3:
+    raise SystemExit('usage: make_fourier_scene_generic.py FRAMES_DIR OUTPUT_DIR')
+ROOT = Path('/home/dev/polymorphic')
+FRAMES = sorted(Path(sys.argv[1]).glob('*.jpg'))
+OUT = Path(sys.argv[2])
+OUT.mkdir(parents=True, exist_ok=True)
+(OUT/'images').mkdir(exist_ok=True)
+assert len(FRAMES) >= 8 and len(FRAMES) % 2 == 0, f'Need even frame count >=8, got {len(FRAMES)}'
+imgs = torch.stack([process_image(str(f)) for f in FRAMES]).unsqueeze(0).cuda()
+model = AnySplat.from_pretrained('lhjiang/anysplat').cuda().eval()
+for p in model.parameters(): p.requires_grad_(False)
+ctx_i = list(range(0,len(FRAMES),2)); tgt_i = list(range(1,len(FRAMES),2))
+with torch.inference_mode():
+    g_ctx, p_ctx = model.inference((imgs[:,ctx_i]+1)*0.5)
+    g_all, p_all = model.inference((imgs+1)*0.5)
+src = p_all['extrinsic'][0,ctx_i,:3,3].double().cpu().numpy()
+dst = p_ctx['extrinsic'][0,:,:3,3].double().cpu().numpy()
+mu_s,mu_d = src.mean(0),dst.mean(0)
+xs,xd = src-mu_s,dst-mu_d
+cov = xd.T@xs/len(src)
+U,S,Vt = np.linalg.svd(cov)
+D=np.eye(3); D[2,2]=np.sign(np.linalg.det(U)*np.linalg.det(Vt))
+R=U@D@Vt
+scale=(S*np.diag(D)).sum()/(xs**2).sum(axis=1).mean()
+trans=mu_d-scale*R@mu_s
+H,W=imgs.shape[-2:]
+flip=np.diag([1.,-1.,-1.,1.])
+train_frames=[]; test_frames=[]; fovxs=[]
+for i,f in enumerate(FRAMES):
+    im=Image.open(f).convert('RGB'); w,h=im.size; side=min(w,h)
+    im=im.crop(((w-side)//2,(h-side)//2,(w+side)//2,(h+side)//2)).resize((W,H),Image.Resampling.LANCZOS)
+    name=f'{i+1:04d}.png'; im.save(OUT/'images'/name)
+    if i in ctx_i:
+        j=ctx_i.index(i); c2w=p_ctx['extrinsic'][0,j].float().cpu().numpy()
+        intr=p_ctx['intrinsic'][0,j].float().cpu().numpy()
+        train_frames.append({'file_path':f'images/{name[:-4]}','transform_matrix':(c2w@flip).tolist(),'index':i})
+    else:
+        c0=p_all['extrinsic'][0,i].float().cpu().numpy()
+        c2w=np.eye(4,dtype=np.float32); c2w[:3,:3]=R@c0[:3,:3]; c2w[:3,3]=scale*R@c0[:3,3]+trans
+        intr=p_all['intrinsic'][0,i].float().cpu().numpy()
+        test_frames.append({'file_path':f'images/{name[:-4]}','transform_matrix':(c2w@flip).tolist(),'index':i})
+    fovxs.append(2*math.atan(1/(2*float(intr[0,0]))))
+fovx=float(np.median(fovxs))
+(OUT/'transforms_train.json').write_text(json.dumps({'camera_angle_x':fovx,'frames':train_frames},indent=2))
+(OUT/'transforms_test.json').write_text(json.dumps({'camera_angle_x':fovx,'frames':test_frames},indent=2))
+means=g_ctx.means[0].float(); scales=g_ctx.scales[0].float(); opacity=g_ctx.opacities[0].float().reshape(-1); harm=g_ctx.harmonics[0].float()
+score=opacity*scales.max(dim=1).values.clamp_min(1e-7); n=min(120_000,len(score)); idx=torch.topk(score,n,sorted=False).indices
+xyz=means[idx].cpu().numpy().astype(np.float32); rgb=(0.5+0.28209479177387814*harm[idx,:,0]).clamp(0,1).cpu().numpy()
+verts=np.empty(n,dtype=[('x','f4'),('y','f4'),('z','f4'),('nx','f4'),('ny','f4'),('nz','f4'),('red','u1'),('green','u1'),('blue','u1')])
+verts['x'],verts['y'],verts['z']=xyz.T; verts['nx'],verts['ny'],verts['nz']=0,0,0
+verts['red'],verts['green'],verts['blue']=(rgb*255).astype(np.uint8).T
+PlyData([PlyElement.describe(verts,'vertex')]).write(str(OUT/'points3d.ply'))
+meta={'source':'AnySplat context-only geometry initialization','total_video_frames':len(FRAMES),'context_views':len(train_frames),'heldout_views':len(test_frames),'seed_primitives':n,'image_hw':[H,W],'camera_fov_x':fovx,'pose_alignment_scale':float(scale),'heldout_geometry_used_for_initialization':False}
+(OUT/'seed_metadata.json').write_text(json.dumps(meta,indent=2)); print(json.dumps(meta,indent=2),flush=True)