Skip to content
HN On Hacker News ↗

Deduplicate all files in the wheel cache by charliermarsh · Pull Request #21327 · astral-sh/uv

▲ 231 points 121 comments by tosh 6d ago HN discussion ↗

Pangram verdict · v3.3

We believe that this text is a mix of AI and human-written content.

50 %

AI likelihood · overall

Mixed
47% human-written 53% AI-generated
SEGMENTS · HUMAN 0 of 3
SEGMENTS · AI 1 of 3
WORD COUNT 264
PEAK AI % 71% · §3
Analyzed
Aug 31
backend: pangram/v3.3
Segments scanned
3 windows
avg 88 words each
Distribution
47 / 53%
human / AI fraction
Verdict
Mixed
Pangram v3.3

Article text · 264 words · 3 segments analyzed

Human AI-generated
§1 Mixed · 55%

## Summary When content hashing is enabled, we currently allocate and zero a new 64 KiB buffer for every file we copy and hash during streaming extraction. This PR reuses one buffer across the wheel instead. For the PyTorch wheel used in the benchmarks, that reduces buffer allocations for hashing from 11,120 to one, while keeping the buffer size at 64 KiB per active wheel.

§2 Mixed · 36%

The following measurements compare #21327 at `a188b8e833aef3c3b4b60a32ed9fafe6ac74186a` with this optimization applied on top, before moving the change onto `main`. They are not measurements against `main`. The Linux benchmarks alternate base and candidate, using pinned wheels served over local HTTP with content-addressed caching enabled: | Cold install | #21327 | #21327 + buffer reuse | Change | | --- | ---: | ---: | ---: | | AnyIO | 110 ms | 107 ms | -2.6% | | SymPy | 845 ms | 775 ms | -8.3% | | NumPy | 627 ms | 567 ms | -9.5% | | PyTorch CPU | 6.50 s | 5.99 s | -7.8% | | 14-package environment, concurrency 4 | 6.95 s | 6.47 s | -7.0% | The individual results above use 16 paired rounds; the full environment uses 12.

§3 AI · 71%

AnyIO, SymPy, and NumPy were repeated after an initial 20-pair run: the initial AnyIO timings were noisy, while the initial SymPy and NumPy improvements were 7.8% and 6.9%. All original samples were retained. Cached installs and local-wheel controls showed no consistent change. Across the initial runs, repeats, and controls, we measured 672 installs, excluding warmups and cache priming. Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>