Fitting Neural Textures and PBR Material Maps with ES (No Backprop)
Pangram verdict · v3.3
We believe that this text is a mix of AI and human-written content.
AI likelihood · overall
MixedArticle text · 1,247 words · 7 segments analyzed
The repo, released on September 4, 2026 is here:https://github.com/richgel999/neural_texture_es2ES=Evolution Strategies.Here is a mirror of the README.md file on GitHub, with the Prior Art disclosure.Key phrases: neural texture compression, neural material compression, PBR material compression, derivative-free neural texture optimization, latent texture optimization, block-compressed neural texture, quantization-aware neural texture training, compressed-domain texture optimization.toy neural texture compression trained with Evolution StrategiesA small, self-contained C++ experiment: an RGB image (or up to four same-size RGB textures of one material) is encoded as a shared low-resolution latent texture plus a tiny MLP decoder, and both are trained entirely with Evolution Strategies — no backprop, no autodiff, no training framework. (An optional late-training polish, --mlp-fd, switches the decoder to numerical finite differences; still no backprop.) Dependencies are stb_image, stb_image_write, and OpenMP.Write-up: Fitting a neural texture decoder with ESI(u,v) ≈ MLP( bilinear(Z, u, v), phi(u,v) ) Z is the latent texture, phi a small positional encoding. At decode time each pixel bilinearly samples Z at its UV, appends phi, and runs the MLP. An optional second, coarser latent level (--latent2) is sampled at the same UV and its channels are concatenated onto the first level's.Results512×512 crop of kodim23, 3000 iterations, latent quantized to 8 bits after training, MLP weights counted as fp16:LatentPSNRbpp (raw)bpp (entropy coded)64×64×426.9 dB0.560.4764×64×828.2 dB1.070.87128×128×430.3 dB2.061.65128×128×832.2 dB4.073.27These runs used the original positional encoding uv,fourier:1 (--nfreq 1), which is why the evaluation command below passes it.
The current default is uv only, which scored 0.26–0.32 dB higher where both were run (see METHOD.md); the results quoted later in this file use that default unless stated otherwise.The 128×128×8 run uses a 14 → 24 → 24 → 3 MLP (1035 weights, leaky ReLU, sigmoid output) and trains in about 150 s on a 32-thread CPU.
Quantizing the latent to 8 bits costs 0.04 dB.Output of that run (out_128c8/): target crop, reconstruction after 3000 iterations, and the eight latent channels side by side.TargetReconstruction (32.2 dB)out_128c8/model.bin is the trained model; evaluate it with ntc kodim23.png --load out_128c8/model.bin --latent 128 128 8 --nfreq 1 --iters 0.A 4-layer materialThe PavingStones070 material (normal, roughness, albedo, AO; see Test images) trained jointly from one shared 128×128×4 + 64×64×4 latent and one 10 → 36 → 36 → 12 MLP (2172 weights), 3000 iterations, learning rate annealed over the second half, per-weight finite differences for the decoder over the last quarter. 8-bit latent: 2.64 bpp total, 0.66 bpp per texture. Left is the target, right the reconstruction from the quantized latent (out_m1234/). The training command wasntc m1.png m2.png m3.png m4.png --latent 128 128 4 --latent2 64 64 4 --mlp 36,36 --mlp-pairs 64 --iters 3000 --lr-anneal 0.5 0.05 --mlp-fd 0.75 --out out_m1234 Normal map, 23.2 dBRoughness, 31.5 dBAlbedo, 23.2 dBAmbient occlusion, 29.6 dBout_m1234/model.bin is the trained material; evaluate it with ntc m1.png m2.png m3.png m4.png --load out_m1234/model.bin --latent 128 128 4 --latent2 64 64 4 --mlp 36,36 --iters 0.How the ES training worksMLP: antithetic ES, 32 perturbation pairs per step, each pair evaluated on the same random 4096-pixel minibatch. The estimated gradient is fed to Adam. Optional late phases: full-image minibatch (--mlp-full), per-weight finite differences (--mlp-fd), or frozen decoder (--mlp-freeze).Latent: all latent values are perturbed at once and the full image is decoded twice per pair, 4 pairs per step.
Each pixel's loss change is credited only to the (up to) 4 texels its bilinear tap reads, on each latent level.Ordinary ES already updates every parameter from one antithetic pair, but its variance grows with parameter count: the single scalar loss difference is the sum of thousands of separate local effects, and each texel's share is buried under everyone else's. Here each texel's loss difference is measured only over the pixels in its own bilinear footprint, so noise from the other ~16k texels is discarded instead of averaged. That credit assignment, not the per-evaluation cost, is what makes ES practical on a latent with 131k values using only 4 pairs per step.The latent stays fp32 during training; quantization is applied afterwards and reported at a configurable bit depth.BuildingCMake generating a Visual Studio solution (MSVC), or any C++17 compiler with OpenMP on Linux/WSL. Tested with MSVC 2022 and MSVC 2026 on Windows, and gcc 13 under WSL2. Note that std::normal_distribution differs between standard libraries, so the same seed gives slightly different results on each platform.cmake -S .
-B build -G "Visual Studio 17 2022" -A x64 cmake --build build --config Release For Visual Studio 2026 use -G "Visual Studio 18 2026", which needs CMake 4.2 or newer (the CMake bundled with VS 2026 works).Runningntc image.png --out out --latent 128 128 8 --iters 3000 ntc albedo.png normal.png rough.png --weights 1,1,0.5 --out out_mat --latent 128 128 8 --latent2 64 64 4 Several positional images form a material: they must have the same size after cropping, share the latent and the MLP (3 outputs per texture), and can be weighted in the loss with --weights (relative; a weight of 0 drops that texture from training).
Per-texture PSNRs are printed alongside the overall one, output files gain a _tK suffix, and the bitrate is reported both per material pixel and per texture.Run with no arguments, ntc trains on the checked-in kodim23.png using the default 64×64×4 latent. Images are located relative to the executable, so this works from the build directory as well as the repo root. A named image that cannot be found is an error; the synthetic fallback only applies when no image is named.Progress is printed to stdout (MSE, PSNR, quantized PSNR and bitrate, latent stats, throughput). Reconstructions, a latent visualization per level, and model.bin are written to the output directory periodically; side_by_side.png (target | 8-bit-latent decode) is written at the end.Useful options (ntc --help lists them all):FlagMeaning--latent W H Clatent texture size (default 64 64 4)--latent2 W H Coptional second (typically coarser) latent level, e.g. 32 32 4 (off)--weights w0,w1,...per-texture loss weights for a material, one per positional image (relative; 0 drops a texture)--lat2-sigma FES sigma for the second level (default: same as --lat-sigma)--lr-anneal START FINALdecay both learning rates linearly from 1× at START·iters to FINAL× at the end--mlp W1,W2,...hidden layer widths (default 24,24)--act leaky|relu|tanh|sinehidden activation--pos SPECpositional features: uv, fourier:N, dct:N, local, lfourier:N, lquad, ldct:N, ldct2:N, ldct4:N, none--nfreq Nshorthand for --pos uv,fourier:N (the encoding the headline table used)--qbits Nlatent bit depth for the reported quantized PSNR / bitrate--load model.bin --iters 0evaluate a saved model--mlp-pairs, --mlp-batch, --mlp-sigma, --mlp-lr, --mlp-every, --lat-pairs, --lat-sigma, --lat-lrES hyperparameters--mlp-fd START, --mlp-fd-h Hfrom START·iters, train the MLP by central finite differences per weight--mlp-full START, --mlp-full-pairs Nfrom START·iters, evaluate the MLP ES step on the full image--mlp-freeze STARTfrom START·iters, stop updating the MLP (latent-only phase)--lat-alttwo latent levels: perturb one level per pair, rotating, to remove cross-level crosstalkImages larger than 512×512 are center-cropped by default (--crop); all textures of a material are cropped identically and must match afterwards.Test imageskodim23.png, kodim01.png, kodim02.png: the Kodak test set (768×512, center-cropped to 512×512 by default).
kodim23 is the parrots.m1.png … m4.png: a 4-layer cobblestone material at 512×512, in order normal map, roughness, albedo, ambient occlusion. Derived from PavingStones070 on ambientCG, released under Creative Commons CC0 1.0. Train it as a material with ntc m1.png m2.png m3.png m4.png ....Prior art disclosureThe blog post above and the single-texture results (the original repository, richgel999/neural_texture_es) were published on September 3, 2026. This repository, including the two-level latent, materials, and the items marked "added September 4", was published on September 4, 2026. The following are disclosed here as public prior art.Neural texture representations using learned latent grids with small neural decoders, and Evolution Strategies / simultaneous-perturbation methods for derivative-free optimization, are established ideas.