cc-os/plugins/os-vault/eval/fixture/vault/jq-streaming-large-files.md

type	title	summary	tags	scope	last_updated	date
reference	jq --stream for Huge JSON Files	How to use jq's --stream mode to process JSON files too large to load into memory whole.	type/reference tool/jq domain/data-processing	global	2026-06-28	2026-06-28

jq normally parses an entire JSON document into memory before filtering it, which falls over on multi-gigabyte files. --stream mode avoids this by emitting a flat sequence of [path, leaf-value] pairs as it parses, instead of building the full in-memory tree.

Basic usage

jq --stream 'select(.[0][0] == "records")' huge-file.json

This walks the file incrementally and only ever holds the current path/value pair in memory, not the whole document.

Reconstructing structure

Streamed output is flat, so reassembling a filtered subset back into normal JSON needs fromstream on the way out:

jq -n --stream 'fromstream(inputs | select(.[0][0] == "records"))' huge-file.json

When to reach for this

• The file is larger than you're willing to hold in memory (multi-GB exports, log dumps).
• You only need a small slice of a huge document (e.g., one array's worth of records) and don't want to pay the cost of parsing the rest.
• Note that --stream filters are noticeably slower per-byte than plain jq on data that does fit in memory — it's a memory/time tradeoff, not a free win.