Appendix to “Depth Perception for Transformers: Telling Models How Deeply Each Token Is Nested”

1. How to read the numbers

The loss. Every number is validation nats per token: the average negative log-likelihood the model assigns to each token of a fixed set of 1,024 held-out records, at the run's final step. Lower is better.

The difference. Results are reported as Δ = configuration − baseline, in nats per token. Negative means the configuration is better. Every table names its baseline. Headline numbers are also given as a perplexity reduction, 100 × (1 − e^Δ), where positive means better: a Δ of −0.01 is about a 1% lower perplexity.

How big is big. The sequence-only baseline sits at 0.3456 nats per token on Python, 0.8408 on spreadsheets, 3.3142 on web pages and 2.5367 on the larger web run. The effects measured here are between about 0.002 and 0.025. For comparison, removing position altogether on Python costs +0.0297.

Why everything is paired. The final loss of one configuration varies from seed to seed with a standard deviation of 0.0032 on Python (20 seeds), 0.0087 on spreadsheets and 0.0124 on web pages — as large as the effects. So every configuration is trained at the same seeds as its baseline, and we analyse the paired differences, whose spread is far smaller (for sequence + tree against sequence only: 0.0017, 0.0067 and 0.0058 respectively).

The statistics. For each comparison we give the mean paired difference, its standard error, a paired t-test on the per-seed differences (n − 1 degrees of freedom, two-sided p), the 95% interval (mean ± t × standard error), and won k/n. Seed counts matter: if a configuration made no difference, winning 5 of 5 seeds would still happen with probability 0.0625 (two-sided), 10 of 10 with probability 0.002 and 20 of 20 with probability 0.000002. We treat 5 of 5 as suggestive, not conclusive, unless the t-test agrees.

The charts. Hover over, or tab to, any mark for its value. Every chart has a table view underneath with every number it plots. Hover over an underlined term for its definition.

2. Moved from the paper body

Appendix

Every number here is recomputed from the run logs by audit_paper_numbers.py, which reads this file alongside the paper. The interactive appendix at docs/index.html presents the same results with per-seed detail and per-seed markers.

Figure 3. Each configuration's gap to sequence only over training

Figure 3. Perplexity reduction against sequence only at every validation checkpoint (above 0 = better): mean over seeds of the paired gap; band, 95% CI for sequence + tree + depth. Each panel has its own scale; spreadsheets from 25% of training. On code the gain narrows as training proceeds; on web pages it holds or grows.

Figure 4. Gain against the number of extra planes

Figure 4. Perplexity reduction against sequence only (above 0 = better) against the number of extra rotary planes each configuration costs, log scale. Marker: mean over seeds of the paired gap; line: 95% CI; the larger web run is one run each, on its test split. Green: depth and depth with a label; orange: a grid (the statement grid on code; row and column on spreadsheets; exchange, speaker and position on chat), with depth where marked; blue: the full 44-plane tree; violet: tree + depth; grey: all 64 planes rotated by sequence. A cheap coordinate reaches the full tree in every setting; which one depends on the data. Every label-on-depth variant is in the interactive appendix.

Other findings (body §5.2)

Where the gain lands. Split by what is predicted, on the test split, the gain concentrates on structural tokens. On spreadsheets, the end of a row improves by 8.6% under sequence + tree and 8.6% under sequence + grid (10/10), against 1.2% and 1.1% on average; a value copied from the cell above across ≥ 256 tokens by 8.5% and 9.0%. On web pages, the first token of a new text block improves by 4.0–5.4% under every structured configuration (10/10), against 1.2–2.0% on average; full RoPE helps no kind of token. On Python, gains are spread evenly; the largest negative values, on tokens right after a subtree closes, are within noise. These are likelihoods, not task accuracy. Figure 5 breaks the gain down by kind of token.

Figure 5. Perplexity reduction against sequence only on selected kinds of token, test split (right = better); marker, mean over seeds; line, 95% CI; shaded row, all tokens. Each panel has its own scale. Every category, with per-seed markers, is in the interactive appendix.

Reference rotation improves on sequence + tree by 0.88% on spreadsheets (9/10, q < 0.05), by 0.39% when every address is also written out as text (7/10, q < 0.05), and not on Python (−0.15%, 2/5), whose renamed identifiers already make each reference a string match. On cell references themselves the gain is 9.1% against sequence only.

Adding versus replacing; one head versus separate layers. Replacing the sequence index with the tree helps on Python (tree only, 0.41%, 20/20, q < 0.05) but hurts on the larger web run (−1.70%, test), where there are 0.12 nodes per token. Splitting sequence and tree into separate layers is worse than sharing each head on Python (−0.22%, 0/5, q < 0.05) and suggestive of better on spreadsheets (0.73%, 5/5, p = 0.097). Learning curves, these splits and per-category tables are in the interactive appendix.

How the structure is used (body §5.2)

The models rely on the structure. Scoring the same test text with each record's coordinates shuffled among its own tokens costs far more than the structure gained, on every seed (5/5): sequence + grid on spreadsheets gains 1.4% but loses 29% when shuffled (12% when removed); sequence + depth on web pages gains 2.0% and loses 18% (5.4%). On Python, shuffling depth multiplies perplexity 7.5×: with no brackets or indentation in the stream, the model hands its nesting bookkeeping to the rotation. Removing the structure is out of distribution there, so shuffling is the cleaner test.

Attention moves where the gain lands (Figure 6). Averaged over all heads, the grid shifts attention onto earlier cells of the same column (14.2% → 18.5%, 5/5, q < 0.05) and onto the referenced cell (+0.55 points, 5/5, q < 0.05); depth on web pages shifts it onto text at the same depth elsewhere on the page (+1.8 points, 5/5, q < 0.05). On Python the shift is only suggestive (onto ancestor nodes, +0.5 points, 4/5, p = 0.062). Individual heads move 15–37 points in either direction, so heads are reassigned rather than nudged, and a sequence-only model also grows a column-reading head; the grid spreads that behaviour across more of the network. Full tables, and what switching the rotation off does to the attention scores, are in the interactive appendix.

Figure 6. Attention of the average head (all 12 heads, 5 seeds) from a spreadsheet cell to cells in the same row and in the rows above, by column offset, test split. The grid moves attention from cells to the left onto the same column in earlier rows.

Table 1. The five settings

Table 1. The five settings.

Depth dropout, in full

Withholding depth on part of the training data. Training web pages with depth dropped on a random 30% of records — the coordinate absent there, contributing no rotation — keeps most of the reading gain and removes the harm when depth is missing. With depth supplied it gains 1.49% (9/10, q < 0.05) against sequence only, where plain depth gains 2.00% (10/10, q < 0.05); without it, −0.42% (4/10, n.s.), where a model trained with depth always present is 3.30% worse than sequence position alone (0/10). At 50% dropout the pair is 1.23% (9/10, q < 0.05) and −0.07% (5/10, n.s.). One model can therefore serve both regimes: read structured documents with the marker, generate free text without it. Use depth where the structure is in the text or available at inference; where it may be missing, withhold it on part of the training data.

Run-to-run noise and the Python splits (body §4, §5.1)

The noise floor. GPU training here is not bit-reproducible: replicate runs of one configuration at one seed are identical at step 0 and diverge thereafter through floating-point reduction order. Three replicate runs of one configuration at one seed on web pages spread 0.0023 nats at the final step against 0.0109 across the ten published seeds — 21% of the spread the pairing removes, and 11% of the depth effect measured against it.

Why Python's two splits disagree. Python's test split shows smaller gains for every configuration (sequence only scores 0.43 nats/token there against 0.35 on validation). The splits are contiguous record ranges over an unshuffled corpus: mean record length is 108.5 tokens in train, 127.9 in validation and 117.9 in test, and the subsets actually scored are 130.0 and 114.5. Validation is also easier than the training data: at the end of training, loss is 0.536 on training batches against 0.348 on the validation set. The body therefore reads the test column as the less idiosyncratic of the two and treats the Python effect as uncertain.

The two-corpus precursor (body §5.4)

An earlier two-corpus version of the four-corpus experiment, superseded by it and reported here for completeness.

A two-corpus precursor (web pages and spreadsheets, five seeds, test split) found the same split of preferences: depth 2.86% (5/5, q < 0.05) on web against the grid's −0.26% (2/5, n.s.) there, and the grid 1.35% (5/5, q < 0.05) on spreadsheets. It also found that two corpora writing different meanings into one plane group cost nothing against giving each its own, 0.02% (3/5, n.s.). Full tables for both cells follow below.

Published orderings for the injection site (body §6)

GRAPE's own experiments (Zhang, Y. et al., 2026) favour the additive site: 51.73 average accuracy for RoPE against 52.87 for ALiBi and 53.25 for its additive variant at 353M parameters. PaTH's (Yang et al., 2025) favour the multiplicative: 19.01 perplexity against 18.03 for its own and 18.33 for an additive data-dependent logit term. Neither holds the injected coordinate fixed while varying the route.

3. Glossary

Every term below is linked wherever it appears; hover for the definition, click to come here.

setting

One corpus trained at one model size, context length and training length. Configurations are compared only within a setting; loss values from different settings are never compared directly, because their token streams differ.

configuration

One way of filling the rotary planes with position information (for example sequence only, or sequence + tree). Within a setting, configurations differ in nothing else.

Python

Setting: 423,111 Python programs written as a compact pre-order serialisation of their syntax tree; 4.7M-parameter model, 1,024-token context, about eight passes; 5 to 20 seeds per configuration.

Spreadsheets

Setting: 41,245 spreadsheet workbooks written out cell by cell; 4.7M-parameter model, 4,096-token context, about eight passes; 10 seeds per configuration.

web pages

Setting: Web pages (visible text only), whole pages of at most 2,048 tokens; 4.7M-parameter model, a third of a pass; 10 seeds per configuration. The small, many-seed web setting.

the larger web run

Setting: Web pages (visible text only) at a 4,096-token context; 31.5M-parameter model, one full pass over 25.4 billion tokens; one seed per configuration. The large web setting.

seed

A random seed fixes the initial weights and the order of training batches. Two configurations trained at the same seed start from identical weights and see identical batches, so they differ only in the position signal.

paired difference

For one seed, the configuration's loss minus the baseline's loss at the same step. Comparing per seed removes the seed-to-seed variation both runs share.

Δ

Configuration loss minus baseline loss, in validation nats per token, at the final step. Negative means the configuration is better. The baseline is always named.

won k/n

The number of seeds, out of n, on which the configuration's loss is lower than the baseline's.

nats per token

Average negative log-likelihood of each held-out token under the model, using the natural logarithm. Lower is better. Perplexity is e raised to this value.

perplexity reduction

100 × (1 − e^Δ): how much lower, in percent, the configuration's perplexity is than the baseline's. A Δ of −0.01 nats per token is about a 1% perplexity reduction. Positive means better.

milestone

A validation score taken every 5% of a run's planned steps, so a full run has 20. Configurations are compared only at milestones both runs have reached.

pass

One epoch: the number of steps it takes to show the model every training record once.

held-out split

Records never trained on. The validation split is scored at every milestone (a fixed set of 1,024 records); the test split is scored once, at the final step, as an independent check.

tree

The structure a document was generated from: the syntax tree of a program, the sheet / row / cell structure of a workbook, or the element tree (DOM) of a web page.

node

One element of the tree: a syntax-tree node, a spreadsheet cell or formula part, an HTML element or a run of text. Each token is assigned to one node.

path

The list of sibling ranks from the root of the tree down to a token's node — one number per level. Two nodes have the same path only if they are the same node.

sibling rank

A node's position among its parent's children, counting from 0 in document order. (On spreadsheets the row and column levels use the absolute row and column index instead, because empty rows are skipped in the data.)

level

Distance from the root: level 1 is a child of the root, level 2 a grandchild, and so on. Up to 22 levels are encoded.

depth

The number of levels a token's node sits below the root — one integer per token.

within-node index

A token's position inside its own node: 0 for the node's first token, 1 for the next. It restarts at every node, so it is a local copy of sequence position.

nodes per token

Tree nodes divided by tokens in a corpus: how much structure each token carries. About 0.86 for Python, 0.56 for spreadsheets and 0.13 for web text.

interior node

A node that has tokens of its own AND a descendant that also has tokens. Depth can recover the tree only if every ancestor is interior in this sense.

pre-order serialisation

Writing a tree out parent first, then each child's subtree in order (depth-first). All three corpora are written this way.

render

How a spreadsheet is written as text. The values render writes only each cell's content; the addressed render writes 'address:content' (e.g. G4:=E4+F4). Tabs separate cells, newlines rows.

reference

A formula operand that names another cell, such as E4 in =E4+F4; in Python, a use of a variable that names its definition.

plane

One two-dimensional rotary subspace: two of a head's dimensions rotated together by one angle. A 128-wide attention head has 64 planes.

unrotated plane

A plane given angle 0 everywhere. It carries no position and acts as an ordinary content channel.

fast and slow plane

Each tree level has two planes. At initialisation the fast plane turns 0.316 rad per sibling (adjacent siblings stay distinguishable) and the slow plane 1/16 of that, so it can count about 318 siblings before its angle wraps round. Both rates are learned.

wrap

When a plane's angle passes 2π it repeats, so two different ranks become indistinguishable on that plane.

co-location

Both position signals inside the same attention head, so a single attention score can depend on sequence distance and tree position at once.

sequence surrogate

A signal that stands in for sequence position. The within-node index is one: it counts tokens in reading order.

plane-matched control

A configuration with exactly the same number of rotating planes and learned parameters as another, but spending them on a different signal. Used to tell 'the signal helps' from 'extra planes help'.

sequence only

Configuration: 16 planes rotate by the token's index in the sequence (standard partial rotary embedding); every other plane is unrotated. The usual baseline.

tree only

Configuration: the 44 tree planes rotate by the token's path; the sequence planes are unrotated, so the model is never told token order directly.

tree + within-node index

Configuration: the 44 tree planes plus 4 planes for the within-node index; no sequence position.

sequence + tree

Configuration: 16 sequence planes plus the 44 tree planes, inside every head. The main structural configuration.

all signals

Configuration: sequence + tree + within-node index (16 + 44 + 4 = all 64 planes).

sequence + depth

Configuration: 16 sequence planes plus 2 planes rotated by the token's depth; the other 46 planes unrotated. No per-level path at all.

top-level-only control

Configuration: 16 sequence planes plus 2 planes rotated by level 1 of the path only (which child of the root the token descends from). The plane-matched control for sequence + depth.

no position

Configuration: no plane rotates. The model gets no positional information except what causal masking implies.

sequence + grid

Spreadsheet configuration: sequence position on 16 planes plus the first three levels of the path — sheet, row and column numbers — on 6 planes; the formula-nesting levels below the cell are not rotated.

sequence + depth + rank

Configuration for nested documents: sequence position on 16 planes plus two numbers per token on 2 planes each — its node's depth and the node's rank among its parent's children, the tree's analogue of a spreadsheet's row and column. Run on Python and web pages.

full RoPE

Control configuration: all 64 planes rotate by the token's index in the sequence, as in standard RoPE (the baseline rotates 16). It checks that the structured configurations do not win merely by rotating more planes.

sequence + tree + depth

Configuration: sequence + tree plus 2 planes rotated by depth (62 of 64 planes). Best or tied overall; the paper's recommendation is the cheaper sequence + depth. On the depth planes a parent and its child always differ by exactly one learned step, so 'is the parent of' is a constant offset.

layer split

Configuration family: each layer carries ONE signal at full width — sequence layers rotate all 64 planes by sequence position, tree layers rotate 44 tree + 4 within-node planes. Written as one letter per layer (s sequence, t tree, n no rotation): e.g. ststst.

head split

Configuration: in every layer, half the heads carry sequence position at full width and the other half the tree (+ within-node index) at full width. No single head sees both.

reference rotation

Configuration: sequence + tree, except that a token belonging to a reference takes the coordinates of the thing it refers to on the relevant tree planes, so a reference and its target line up in angle.

directed-offset variant

Reference rotation where the reference sits a learned constant angle away from its target instead of exactly on it, so 'refers to' differs from 'is next to'. The constant starts at 0.

4. Setup

The four settings

Only configurations within one setting are compared. The two web settings answer the same question at different cost: The larger web run at a realistic context and model size, which buys one seed per configuration; Web pages ten times over, on whole pages, for a fraction of one the larger web run run. Their losses are on different scales and are never compared directly; only the direction and stability of the effects are compared.

How each corpus becomes a tree

Python. Programs are parsed with Python's own syntax-tree parser and written out as a compact pre-order serialisation of the syntax tree: a reversible text form (all 423,111 programs convert back to the original byte for byte). Local variable names are rewritten by order of definition (x0, x1, …), so every use of a variable carries exactly the same name as its definition. A byte-pair tokenizer with 32,768 entries is trained on this serialisation. Each token belongs to the syntax-tree node that emits it, and its path is the list of sibling ranks from the root down to that node. Typical sibling ranks are small (99th percentile 7, maximum 151). 656 nodes (0.002%) sit deeper than the 22 encoded levels; nothing is dropped.

Spreadsheets. Each workbook is a tree: workbook → sheet → row → cell → the cell's content, which is either a literal value or a formula's expression tree (operators, function calls, references). The path's first three levels are the sheet's index, the absolute row index and the absolute column index (0-based) rather than sibling ranks, because empty rows and cells are not stored and counting siblings would misnumber them; deeper levels are sibling ranks inside the formula. A token belongs to the deepest node that produced its first character; the tab and newline separators belong to no node. Two renders of the same workbooks are used; the table after this section shows one row of a timesheet in each.

In the values render a cell's own address is never written: the grid position reaches the model only through the rotation. References inside formulas (the E4 and F4 of =E4+F4) are written in both renders. For the tokens of =E4+F4 in cell G4 of the first sheet, the path begins (sheet 0, row 3, column 6) and continues with the token's place inside the formula. 3,031,723 references occur in the corpus; 2,991,629 name a cell in the same workbook and 40,094 name a defined name rather than a cell.

Web text. Pages come from the English C4 crawl with its original HTML. The HTML is parsed into its element tree, skipping head, script, style, svg, iframe, noscript and template. Nodes are the remaining elements and every text run containing a non-space character; paths start at the children of body, and a rank counts the parent's retained children. Only the visible text is written into the token stream; all markup is removed, so document structure reaches the model only through the rotation. A token belongs to the text node containing its first non-space byte. Because only text emits tokens, every token-carrying node is a leaf of the element tree.

What is excluded

Two different things happen to over-long web pages. The larger web run crops them, keeping the first 4,096 tokens; Web pages drops them, so every web pages page is seen whole. Separately, both web settings drop pages whose tree is deeper than the 22 encoded levels or wider than 318 siblings at some level, so that no path is cut short or wraps round.

Model, training and evaluation

Model. A decoder-only transformer in the GPT-2 style: pre-layer-norm blocks, a GELU MLP four times the width, no bias terms, no dropout, input and output embeddings tied. The only change is the position signal: no learned position embedding, and rotary embedding applied to queries and keys in every head, with the angles described in the mechanism section.

Optimisation. AdamW (β₁ 0.9, β₂ 0.95), weight decay 0.1 on weight matrices — the rotary rate and offset parameters are excluded from weight decay — gradient clipping at 1.0, bfloat16 autocast. The learning rate rises linearly over the first 2% of steps, then follows a cosine to 10% of its peak at the final step.

Batches. Every step trains on a fixed number of whole documents. Documents are grouped by length into micro-batches of at most 8,192 padded tokens and gradients are accumulated across the micro-batches of a step. No document is split across steps.

Learning rates. One peak rate per setting, used for every configuration in it:

• Python: a sweep of five rates (0.0003 to 0.0048, 560 steps each) on four configurations — sequence only, tree only, tree + within-node index and sequence + tree. All four were best at 0.0012.
• [[web pages]]: a sweep of three rates (600 steps) on sequence + tree; 0.0012 was best.
• Spreadsheets: 0.0012, carried over from a sweep on an earlier build of the spreadsheet corpus with a shorter context, not re-swept for this one.
• [[the larger web run]]: 0.0006, from a sweep on an earlier build of the web corpus. The exception is no position, which diverged at 0.0006 and 0.0003 in preliminary runs and was run at 0.00015 — where it diverged again.

Evaluation. Validation loss is scored every 5% of a run's planned steps (a milestone) on the same 1,024 validation documents. Results use the final step: no checkpoint is selected by its score. The single exception is the the larger web run tree-only run, which diverged and is reported at its best milestone, labelled as such. As an independent check, final checkpoints are also scored on the test split: 8,192 test documents for Python (and 2,048 in an earlier pass over six seeds), 2,048 for the larger web run.

Compilation. Runs use PyTorch's compiler except every web pages run (documents of many different lengths would trigger a recompilation per batch shape) and the the larger web run head-split run (it failed under compilation at four heads). Compilation does not change the arithmetic in principle but is not a bit-identical path, so the the larger web run head-split run is not configured exactly like its comparators.

5. Mechanism

Rotary position embedding in one paragraph. In each attention head, the query and key vectors are cut into pairs of dimensions, and each pair — a plane — is rotated by an angle that depends on the token's position. When the attention score between two tokens is computed, the rotations combine so that each plane contributes a term depending only on the difference between the two tokens' angles. Standard rotary embedding makes every angle a multiple of the token's index in the sequence, at a range of frequencies. Nothing requires that: the angle of any plane can be a function of any coordinate, and the score will then depend on the difference in that coordinate. We give some planes to the token's position in the tree.

The angles. A head of width 128 has 64 planes, numbered here 1 to 64 (plane i rotates dimensions i and i + 64). In every configuration that keeps all signals in each head, the same planes always carry the same signal; a configuration only decides which groups rotate.

• Sequence planes 1–16. Plane i (i = 0 … 15) has angle t × 10000^(−i/16), where t is the token's index in its document. This is the standard rotary spectrum over 16 planes, the partial-rotary arrangement used by GPT-NeoX.
• Tree planes 17–60: 22 levels × 2 planes. For tree level k (1 to 22) and plane p ∈ {fast, slow}, the angle is present_k × θ_k,p + r_k × s_k,p. Here r_k is the sibling rank of the token's ancestor at level k; present_k is 1 if the token's node reaches level k and 0 otherwise; θ_k,p (an offset) and s_k,p (a rate) are learned. Levels a node does not reach contribute angle 0.
• Within-node planes 61–64. Angle = present × θ_p + w × s_p, where w is the token's index inside its own node and θ_p, s_p are learned.
• Depth planes (2, in the depth configurations). Angle = d × s_p, where d is the token's depth (the number of levels its node reaches) and s_p is learned. There is no offset: every token has a depth.

Initial values. All offsets θ start at exactly 0. Each tree level's fast plane starts with a rate of √(2 × (1 − 0.95)) ≈ 0.316 radians per sibling — chosen so that adjacent siblings have a cosine similarity of 0.95 on that plane — and its slow plane with 1/16 of that, so the slow plane counts up to about 318 siblings before its angle wraps round and the fast plane about 20. The four within-node planes start at 0.316 / 4^p (p = 0 … 3), wrapping after about 20, 80, 318 and 1,270 tokens; the two depth planes at 0.316 and 0.316 / 16. All rates and offsets are then learned.

Properties that matter for the comparisons.

• Identical networks at a given seed. The rotary parameters are initialised to fixed values, not drawn at random, so two configurations at the same seed start with identical weights everywhere else and see identical batches.
• No look-ahead. A rank counts earlier siblings only, and a token's node and path depend only on the document up to that token.
• Which plane carries which signal is arbitrary. For a model trained from scratch, permuting the planes is the same as permuting rows of the query and key weight matrices, so only the set of (coordinate, rate) pairs matters, not where in the head they sit.

The configurations beyond the plane allocation

Sequence + depth and its control. Two planes carry the token's depth and nothing else of the tree. Its plane-matched control, the top-level-only configuration, has exactly the same 16 + 2 rotating planes and the same two learned rates, but spends them on level 1 of the path (which child of the root the token descends from) instead of on depth.

Depth and the tree. In a pre-order serialisation, a node's parent is the nearest preceding node one level shallower, so the sequence order plus every token's depth determines the tree exactly — provided every ancestor appears in the token stream. On Python this holds and was checked directly: reconstructing every node's parent from order and depth alone recovered 558,271 parents with no error. On web text it cannot hold: only text nodes emit tokens, so no ancestor of a token-carrying node is ever in the stream.

Sequence + tree + depth. Sequence + tree plus the two depth planes. Because depth goes up by exactly one from parent to child, on those planes a parent and its child always differ by exactly one learned step, with a sign: the relation "is the parent of" becomes a constant angular offset, and a grandparent is two steps away.

Reference rotation on spreadsheets. A token that is part of a reference (the E4 in =E4+F4) has its sheet, row and column levels replaced by those of the cell it names; its deeper levels, its place inside the formula, are kept. A reference and the cell it names then have identical angles on those planes, exactly as two tokens of one cell do. It rotates the same planes as sequence + tree, which is its baseline, on the same data and seeds. The coordinate is computed from the reference's own text, so it uses no future information even when the named cell comes later in the workbook. 17.25% of references point to another sheet, which is why the sheet level is replaced too.

Reference rotation on Python. A use of a variable takes the path of the variable's definition for the levels the definition occupies, and keeps its own path below that, so path length is unchanged. 20.6% of tokens are affected; every definition precedes its uses. Baseline: sequence + tree.

Variants of reference rotation, reported with the results:

• Whole-path overwrite (Python): a use takes its definition's entire path. Definitions sit shallower than uses, so 91.7% of the overwritten paths become shorter, losing 3.13 levels on average.
• Directed offset (spreadsheets): the reference sits a learned constant angle away from its target rather than exactly on it, so "refers to" is distinguishable from "is in the same cell", with a direction. The constant starts at 0, so the variant starts identical to plain reference rotation.
• Own plane group (spreadsheets, an earlier build): references rotate a separate group of planes rather than the tree planes.
• Symbol identity (Python): two planes rotated by the order in which each variable is first mentioned, so all mentions of one variable coincide.

Layer splits and the head split. Instead of sharing every head, each signal gets whole layers or whole heads at full width: a sequence layer rotates all 64 planes by sequence position (frequencies 10000^(−i/64)), and a tree layer rotates 44 tree planes plus the 4 within-node planes, leaving 16 unrotated. On Python's six layers four patterns are trained, written one letter per layer (s sequence, t tree, n no rotation): alternating ststst, sequence-first sssttt, tree-first tttsss and thirds ssttnn. On the larger web run (ten layers) and spreadsheets the layer split is alternating. The head split gives half the heads of every layer the sequence and the other half the tree: one head each in Python's two-head model, two each in the larger web run's four-head model.

6. Results