43 lines
1.4 KiB
Markdown
43 lines
1.4 KiB
Markdown
---
|
|
type: reference
|
|
title: jq --stream for Huge JSON Files
|
|
summary: How to use jq's --stream mode to process JSON files too large to load into memory whole.
|
|
tags:
|
|
- type/reference
|
|
- tool/jq
|
|
- domain/data-processing
|
|
scope: global
|
|
last_updated: 2026-06-28
|
|
date: 2026-06-28
|
|
---
|
|
|
|
`jq` normally parses an entire JSON document into memory before filtering it, which falls
|
|
over on multi-gigabyte files. `--stream` mode avoids this by emitting a flat sequence of
|
|
`[path, leaf-value]` pairs as it parses, instead of building the full in-memory tree.
|
|
|
|
## Basic usage
|
|
|
|
```
|
|
jq --stream 'select(.[0][0] == "records")' huge-file.json
|
|
```
|
|
|
|
This walks the file incrementally and only ever holds the current path/value pair in memory,
|
|
not the whole document.
|
|
|
|
## Reconstructing structure
|
|
|
|
Streamed output is flat, so reassembling a filtered subset back into normal JSON needs
|
|
`fromstream` on the way out:
|
|
|
|
```
|
|
jq -n --stream 'fromstream(inputs | select(.[0][0] == "records"))' huge-file.json
|
|
```
|
|
|
|
## When to reach for this
|
|
|
|
- The file is larger than you're willing to hold in memory (multi-GB exports, log dumps).
|
|
- You only need a small slice of a huge document (e.g., one array's worth of records) and
|
|
don't want to pay the cost of parsing the rest.
|
|
- Note that `--stream` filters are noticeably slower per-byte than plain `jq` on data that
|
|
*does* fit in memory — it's a memory/time tradeoff, not a free win.
|