BigJSON: A Viewer That Opens
Gigabytes of JSON Without Loading Them

I kept hitting the same wall: I had JSON files several gigabytes in size, and nothing on my Mac could just open them. IDEs choke on them, and when one does limp a large file open, all it gives you is read-only plain text: no structure, no navigation, no search worth the name. The dedicated viewers people point to are commercial; the most-cited one, Dadroit, caps its free tier at around 50 MB, which is nothing next to a multi-gigabyte dump.

So I built BigJSON: a free, open-source, cross-platform JSON viewer that opens multi-GB files, lets you browse the structure, search and query it, and export the results. It’s read-only, fast, and not bound by file size. A second tool fell out of the same core: jsq, a SQL-shaped command-line query language for those same files.

Both are thin frontends over one Rust engine, and the engine has a single trick that makes all of it possible: it never loads the file into memory. Here’s a number that captures the wall everything else hits. Run jq over a 1 GB file to count a few matching rows and it eats roughly 17 GB of RAM, because it first parses the whole document into a heap tree. That same model is why IDEs and viewers fall over on large files. BigJSON’s engine memory-maps the file and builds a compact index instead, so it opens a gigabyte while holding tens of megabytes. This is how.

Why a Gigabyte of JSON Costs You Sixteen

jq is a per-token stream processor, but to evaluate a filter it first materializes the document into an in-memory tree: every object becomes a hashmap, every array a vector, every number and string a heap-allocated, reference-counted value. That representation is convenient (you can walk it, reshape it, index into it freely) but it’s bulky. A JSON number that occupies three bytes on disk becomes a tagged, boxed value many times that size once it’s a node in a tree.

Measured end to end, the parsed form lands at roughly 16× the file size. A 1 GB file becomes ~16 GB of live heap, and that holds whether you use jq (C) or jaq (a tight Rust rewrite). The language and the implementation language don’t matter here. The model does.

For range scans and projections that emit a row per input element, holding everything is sometimes unavoidable. But the queries that dominate real data work (count the paid orders, sum revenue per region, join events to users and total by tier) collapse millions of rows down to a handful. Paying gigabytes to produce three group rows is pure overhead. That’s the gap jsq goes after.

One Query, One Streaming Pass

The design premise of the query language is narrow on purpose: you don’t get a Turing-complete transformation language. You get one query, shaped like SQL, that runs in a single pass and prints NDJSON. It’s what the CLI speaks, and it’s the language behind the viewer’s query bar.

# Count the paid orders
jsq orders.json 'from .orders[] as o where o.status == "paid" aggregate { n: count() }'

# Revenue per region, top 5
jsq orders.json 'from .orders[] as o
  aggregate { revenue: sum(o.total) } by o.region
  order by .revenue desc
  limit 5'

Every query emits one JSON value per line, so the output still composes with the tools you already use: jsq ... | jq ..., | head, | wc -l. jsq doesn’t replace jq; it sits upstream of it, doing the heavy GB-scale reduction that jq would choke on, and hands the small result downstream.

The grammar is a fixed clause pipeline. Clauses are optional except from, but their order is fixed, which is exactly what makes single-pass evaluation tractable:

If you think in SQL, pandas, or list comprehensions, the mapping is direct: for x in arr is from .arr[] as x, filter is where, groupBy + reduce is aggregate ... by, flatMap is unnest, DISTINCT ON is distinct by. The point of constraining the language is that every one of those clauses can be evaluated as the stream flows past, without ever holding the stream.

Two Frontends, One Engine

The primary product is BigJSON, a cross-platform desktop viewer (Tauri + Svelte). It opens a multi-GB file behind a streaming progress bar, then lets you explore it read-only: a collapsible tree navigator with breadcrumbs, text search across the document, an in-app query bar running the same jsq language, and CSV/JSON export of results. Rows are virtualized, so scrolling a gigabyte stays smooth; the UI only ever holds the handful of rows actually on screen.

The jsq CLI is the secondary frontend, a thin Rust binary that pipes NDJSON for shell composition. Both delegate every semantic decision and every byte of output to the same Rust engine crate.

That parity is a deliberate constraint, not an accident. Because the desktop app links the engine as a normal Rust crate rather than reimplementing anything in TypeScript, there is exactly one parser, one evaluator, one renderer. The viewer’s tree navigator is the clearest payoff: it never holds the JSON tree in JavaScript. It asks the engine for one level of children at a time over the memory-mapped index (a batched call into the engine’s child-iteration API), so expanding a node in a 10 GB file is a few page faults, not a load. And a query behaves identically in the CLI and the UI because it is literally the same code path.

From Text to NDJSON

A query string becomes results through four stages. The split between a surface AST and an engine AST is the load-bearing decision here: the surface form mirrors the SQL-shaped syntax a human writes, and a lowering pass rewrites it into a smaller set of primitive operators the evaluator knows how to stream.

Lowering is where the syntactic sugar dissolves. An aggregate item like revenue: sum(o.total) * 1.1 isn’t a single operation: the lowerer detects the sum(...) reducer buried inside the arithmetic, hoists it into a dedicated reduction slot, and rewrites the output expression to multiply that slot by 1.1. Field-set macros expand, let bindings get substituted into the aggregate block, and from .events[] becomes a primitive iterate-over-children operator. By the time the evaluator runs, there’s no SQL left, just a tree of stream operators.

The evaluator itself is push-based. Instead of pulling values through a chain of iterators, walk_eval dispatches on the operator and calls a sink closure for each value it emits:

fn walk_eval(
    // ...
    sink: &mut dyn FnMut(Value) -> bool,
) { /* ... */ }

That early-termination signal matters more than it looks. A limit after a sort doesn’t just trim the output; it bounds how much the sort has to keep (more on that below), and a limit over a plain scan stops the file walk the instant the quota is met.

Memory-Mapped, Not Heap-Allocated

Here is the core trick, and it’s the whole reason the memory numbers look the way they do. jsq never reads the JSON file into a buffer. It memory-maps it, and it builds an offset index (a sidecar file) that it also memory-maps.

The consequence is that the engine’s owned memory tracks the working set, not the file size. Scanning the gigabyte to compute a sum touches the index records sequentially, the kernel keeps a small window of pages resident, and the resident footprint stays flat. It also means the engine can work on files larger than physical RAM, something the parse-everything approach simply cannot do, because you can’t put 64 GB of parsed tree into 32 GB of heap.

This is also what makes the viewer feel instant: opening a file maps it and builds the index rather than reading it into a buffer, and scrolling pulls only the visible rows’ records off the mmap. BigJSON opens files that wouldn’t fit in RAM at all, which is exactly the case where IDEs give up.

This does produce a measurement subtlety worth being honest about:

A Pre-Order Array of Records

The sidecar isn’t a tree of pointers; it’s a flat array of fixed-size records in pre-order (depth-first, parent before children). Each record is 48 bytes:

#[repr(C)]
pub struct NodeRecord {
  offset: u64,        // byte position in the source mmap
  length: u64,        // source byte span of the value
  key_or_index: u64,  // object: offset into the keys arena; array: slot index
  parent: u32,
  subtree_size: u32,  // records in this subtree, including self (always >= 1)
  child_count: u32,   // direct children, primitives included
  key_length: u32,
  kind: u8,           // Null | Bool | Number | String | Array | Object
  flags: u8,
}                     // + 6 bytes padding -> 48, 8-byte aligned

Two invariants fall out of the pre-order layout, and the rest of the engine leans on both.

The second decision is what gets a record at all. Emitting one record per JSON value would make the index enormous and mostly redundant: a flat object of ten small fields would cost ten records plus itself. So jsq uses a hybrid emit-gate:

So child_count and subtree_size measure genuinely different things, and the struct keeps both: a container can have ten direct children but a subtree_size that only counts the nested containers among them, because the scalar children never became records.

Joining Without Materializing

A join is the classic way to blow up memory: the naive version materializes the cross product. jsq doesn’t. When a query joins on u.user_id == e.user_id, the engine builds a foreign-key index: a hashmap from each key value to the record IDs that carry it.

from .events[] as e
join .users[] as u on u.user_id == e.user_id
aggregate { revenue: sum(e.amount) } by u.tier
order by .tier

Internally that index is a HashMap<ScalarKey, Vec<u32>>, built once per join and cached on the document. ScalarKey is a compact, hashable encoding of a JSON scalar (distinguishing integral from fractional numbers, interning the variants) so grouping and lookups never allocate a string per row. At runtime each event hashes its key once and gets the matching user records in $O(1)$, with no nested-loop scan and no joined rows held in memory.

This is why even a two-hop join (event → user → region) stays flat: it’s two index lookups per event, both $O(1)$, neither materializing anything. In the benchmarks the chained join over 7 million events holds the same ~36 MiB as a bare count.

Why Reductions Stay Flat

The reason aggregates cost almost nothing is that they fold the stream away as it flows. A sum ... by region keeps one accumulator per distinct region (a handful of entries) and updates them row by row. Millions of input rows in, three group rows out, and only the three group rows are ever resident.

Sorting is the operation that looks like it must buffer everything, and jsq avoids it whenever a limit follows:

distinct dedupes against a set of value fingerprints rather than retaining full rows. The recurring theme: every stateful operator buffers only what its output demands, never the input.

Which sets up the one honest exception:

jsq vs jq vs jaq

The benchmark suite runs nine queries (filter/project, counts, single- and multi-metric group-bys, single and chained joins, unnest, post-aggregate having, and a combined kitchen-sink) across 10 MB, 100 MB, and 1 GB files, against jq 1.7.1 and jaq 3.0.0. Including jaq is the point: it isolates the variable. jq vs jaq is C vs Rust; jsq vs both is parse-everything vs streaming.

Here is the group-and-sum query (q3) at all three sizes:

At 1 GB jsq is ~10× faster than jq and ~4× faster than jaq on this query, and holds 34 MiB against 15-17 GB, a ~450-500× memory gap. The telling detail is the jaq column: rewriting jq in Rust buys a real speedup (~1.5-2×) but moves the memory needle essentially zero, because jaq still parses everything. The frugality comes from the architecture, not the language.

What jsq Gives Up

The design comes with three real costs.