How VictoriaLogs Stores Your Logs in a Columnar Layout

VictoriaLogs keeps each stream’s logs together on disk, and that grouping is what makes them compress so well and lets a query touch only the streams it needs instead of scanning everything.Logs with the same stream fields form one stream.The practical rule for you as an operator: keep stream fields stable and low-cardinality, meaning they should have only a handful of distinct values, such as host, app, pod, or container, and keep high-cardinality values, ones with very many unique entries like trace_id or user_id, as normal fields, not stream fields.Now, after receiving and normalizing the incoming records, VictoriaLogs does not handle them one at a time either. It accumulates them in an in-memory buffer and, about once a second (or sooner if the buffer fills up), turns the whole batch into a small searchable chunk that still lives in RAM (an in-memory part).The buffered batch flushes into one in-memory part.That in-memory buffer is not a single shared queue. If every incoming batch had to line up for the same buffer, they would waste time waiting on each other, so VictoriaLogs splits the buffer into shards, one per CPU core, and spreads incoming batches across them in turn.So on a 3-CPU machine there are 3 buffer shards filling in parallel, and each shard flushes on its own, writing its batch out as a new in-memory part about once a second:The buffer is split into per-CPU shards, each flushing its own in-memory part.A part is one of the core data structures across VictoriaLogs (and the other VictoriaMetrics products): a self-contained bundle of data that is searchable, which is to say queryable.Most of the time, the buffered batch is flushed into an in-memory part, but in some rare cases, if a batch is large enough to exceed the in-memory size limit, the part is written straight to disk as a small or big part instead.Metric vl_insert_flush_duration_seconds: how long it takes to turn a buffered batch into an in-memory part.2. Daily partitions#When a batch is flushed, VictoriaLogs files each log into a partition, and a partition holds exactly one calendar day of logs (in UTC). It reads each log’s timestamp, works out which day it belongs to, and routes it there.In other words, your logs are separated by date.

You can see this directly on disk, one directory per day:$ tree victoria-logs-data/

victoria-logs-data/ └── partitions/ ├── 20260109 ├── 20260110 ├── 20260111 └── 20260112 This per-day layout is not just an implementation detail; it is why two everyday operations are cheap:Retention is achieved by deleting whole day directories. When logs age out past -retentionPeriod (7 days by default), or when disk-based retention kicks in, VictoriaLogs drops entire day folders rather than hunting down individual log lines.Queries are almost always time-bounded (_time:1h, _time:5m), so VictoriaLogs only has to open the day partitions that overlap your time range and can ignore the rest.A partition is not purely a folder on disk. It has two faces: an on-disk side that holds the parts already written out, and an in-memory side that holds the buffer shards and the in-memory parts we just saw.A partition has a memory side and a disk side.When you query a day, the partition serves results from both sides simultaneously, pulling the relevant parts from disk only when needed.Metrics vl_storage_parts counts how many parts exist, broken down by where they live: {type="storage/inmemory"} for parts still in memory, {type="storage/small"} and {type="storage/big"} for parts on disk.

And vl_pending_rows{type="storage"} counts the rows still sitting in the buffer, not yet turned into a part.3. Parts: the unit VictoriaLogs actually stores#We met parts back in section 1: a part is the self-contained, searchable bundle of logs that a buffer turns into when it flushes. What we did not say is that parts come in three flavors, which are really the same data at different stages of its life:In-memory parts are created first, so freshly ingested logs are queryable almost immediately without waiting for disk.Small parts are in-memory parts written out to disk for durability.Big parts are what you get after small parts are combined together over time.In-memory parts combine into small, then big parts.Logs in the in-memory buffer first flush into in-memory parts before they ever touch the disk, and that is what keeps ingestion cheap.Parts reach the disk only in big, infrequent chunks, and small parts are merged into bigger ones while still in memory, so VictoriaLogs makes far fewer disk writes and reads. That is how it can absorb around a GiB of logs per second even on slow, low-IOPS HDDs.There is a trade-off hiding in that first flavor. An in-memory part lives in RAM, so logs become queryable within seconds, but they are not yet safe on disk. VictoriaLogs closes that gap by guaranteeing a flush to disk on a short interval. The flag that controls this is -inmemoryDataFlushInterval (default 5s): how often in-memory data is guaranteed to reach the disk.On disk, every part is a directory with a 16-character hexadecimal name (just an id), sitting inside the partition’s datadb folder. A separate indexdb folder holds the stream catalog for that day:$ tree victoria-logs-data/partitions/20260109/

victoria-logs-data/ └── partitions/ └── 20260109/ ├── indexdb/ └── datadb/ ├── 1882C35B4CE64498/ ├── 1882C35B4CE664F8/ ├── 1882C35B4CE66BDB/ └──

parts.json The parts.json file is simply the list of parts that are currently active, so VictoriaLogs knows which directories to read on startup:$ cat victoria-logs-data/partitions/20260109/datadb/parts.json

["1882C35B4CE64498", "1882C35B4CE664F8", "1882C35B4CE66BDB"] A part is immutable: once written, its files never change. That is what makes snapshots and backups safe and cheap.So what is the difference between small parts and big parts?It comes down to the operating system’s page cache. In plain terms, that is the slice of RAM the OS uses to hold recently read file data, so reading the same bytes again comes from memory rather than the disk. The more RAM the machine has, the larger this cache is, and the more of your logs are served straight from memory.Small parts are written through that cache and kept deliberately small, so they usually stay in RAM and read back quickly. Their size scales with the RAM VictoriaLogs leaves free for the OS to cache (at least 10 MB); anything larger is written as a big part instead. Big parts can grow up to about 1 TB, and they bypass the page cache on write, so a large merge does not evict the hot data that recent queries rely on.Beside the datadb folder we saw in the snippet above sits an indexdb folder. While datadb holds the parts (the actual logs), indexdb is the per-day catalog of streams: which streams exist and what fields they carry. When you query, VictoriaLogs checks this catalog first to figure out which streams could possibly match, so it never even looks at logs in unrelated streams. It deserves its own post, so we will leave it there.4. The mental model of VictoriaLogs’ data structure#Before we open the actual files, it helps to have a mental model of how a part is laid out. The guiding idea is simple: VictoriaLogs arranges data so that reading it back later touches as little of it as possible. Two ideas do most of the work.4.1 Logs are grouped into blocks, by stream and by time#Inside a part, logs are packed into blocks.

A block holds rows from a single stream, and a part contains many blocks. The blocks are arranged by stream and then by time, with the rows inside each block sorted by timestamp.A part's blocks, grouped by stream.A block is capped at roughly 2 MiB of uncompressed data (it can stretch to almost 4 MiB when parts are combined). Every block carries a small piece of metadata called a block header, kept separate from the logs themselves (in index.bin, which we open up in section 5.8). It records:which stream the block belongs to,how many rows it holds,and the minimum and maximum timestamps in the block.Since these headers are tiny and stored apart from the actual log data, VictoriaLogs can scan them to decide which blocks are worth reading, without touching a single log line in the blocks it skips.This is what makes a filtered query cheap. When you ask for one stream over the last hour, VictoriaLogs jumps straight to that stream’s blocks and, using each block header’s min and max timestamp, skips any block whose time range cannot overlap your window. It never reads the unrelated streams or out-of-range blocks at all.4.2 Each field is a column, so queries read only what they ask for#The natural way to picture a few logs is one record per row: each log line is a row, with all of its fields sitting side by side:Three logs, one record per row.Within a block, VictoriaLogs flips this around. Instead of keeping each log together as a row, it stores each field as its own column, with the values still lined up by row:The same logs stored column by column.Keeping a field in its own column means VictoriaLogs only has to touch the columns a request actually uses, instead of reading every field of every row. The more fields your logs carry, the more that matters, and it is a big part of why VictoriaLogs handles wide logs with many fields and many distinct values comfortably.This is also what makes the fields pipe in LogsQL such a cheap way to speed up a query. By default, a query returns every field, which means reading every column.