The layer-wise anatomy of the workspace is preserved and refined at 8B scale: onset at 35.7% depth and four CKA seams in Gemma-4-E4B-it

Authors: Jorge Castillo Sepúlveda, Marco Torres Yévenes, Juan Carlos Lanas

Affiliations: AXIS Dynamics Chile · https://axisdynamics.cl · contacto@axisdynamics.cl

Status: first-stage preprint (v2.1) — not peer-reviewed. Target venue: arXiv cs.LG / Axisdynamics preprint series.

Evidence status: full confirmatory run (200/200 valid documents) under a sealed decision rule, whose formal localization verdict is inconclusive; plus a post-hoc structural diagnosis, explicitly exploratory, that explains that non-localization and defines the follow-up pre-registration (§7).

Abstract

The Anthropic J-lens study (Gurnee, Sofroniew, Lindsey et al., 2026) reports that the verbalizable representations of Claude Sonnet 4.5 crystallize into a global workspace at ~37% relative depth, and leaves open (§9.1) the question of whether that fraction generalizes to other scales and training regimes. We replicate the full J-lens estimator (current-and-future, pre-norm residual targets, centered linear CKA over admitted vocabulary) in google/gemma-4-e4b-it (42 layers, 8B scale, mm-RLHF post-training) under a pre-registered protocol with a sealed bootstrap decision rule. Three results. (1) The depth fraction is preserved: the point estimate of the onset falls at L15 = 35.7%, less than two points from the ~37% of Sonnet 4.5, with the workspace→motor transition at L23. (2) The sealed localization verdict is inconclusive: the 95% stability interval is [L9, L16] (width 7 > threshold 4) and the bootstrap distribution is bimodal (59% at L15, 14% at L9). (3) The non-localization has a structural explanation that we report as an exploratory diagnosis: the network does not have two boundaries but four hierarchical CKA seams — L9, L15, L23 and L40 —, and forcing a three-block segmentation obliges the estimator to choose between alternating anatomies, which is exactly what the bootstrap registers. The hierarchy by marginal weight (ΔQ when removing each seam: L23 +0.034 ≫ L15 +0.016 > L9 +0.012 > L40 +0.009) identifies the workspace exit (L23), not its entry, as the dominant transition of the network. The final segment L40–42 does not constitute a coherent block (intra CKA 0.558 < cross CKA 0.670) and is interpreted as an emission seam, not as a functional band. Since the cost Q decreases monotonically with the number of blocks without a defined elbow, no k is established here; the follow-up pre-registration (§7) replaces the question “how many blocks?” with the correct question: “which seams are individually localized under bootstrap?”, with the registered prediction {L9, L15, L23, L40}.

1. Introduction

Gurnee, Sofroniew, Lindsey et al. (Anthropic, Verbalizable Representations Form a Global Workspace in Language Models, Transformer Circuits Thread, July 2026) identify in Claude Sonnet 4.5 a band of layers where the content of hidden states becomes broadly available for verbalization —a “global workspace” in the sense of the cognitive sciences—, with onset at ~37% relative depth. Their §9.1 leaves open the question that this work addresses: is that depth fraction a property that is preserved at another scale and under another post-training regime, or is it an artifact of Sonnet’s architecture and recipe?

We answer with a direct replication at 8B scale and, along the way, we find something the original question did not anticipate: at this scale, the layer-wise anatomy of the model is richer than the sensory/workspace/motor partition that the original methodology presupposes, and that richness is measurable with the same artifacts that the replication produces.

A methodological note that replicators should know: our first run (protocol v1.1) mis-transcribed the estimator, computing only the position-to-position term ∂z[t]/∂h[t] instead of summing the cotangent over all valid future positions t′≥t. A lens that is short-sighted toward the future measures the local next-token prediction machinery, not global availability for verbalization, and systematically underestimates the onset depth (≈L10 instead of L15). The full detail lives in the changelog of the pre-registration (reference 4); in the body of this paper we use only the corrected estimator, and in §4 we show that the v1.1↔v2 discrepancy also has a structural reading.

2. Method

Model. google/gemma-4-e4b-it, 42 transformer layers (λ = 1…42, without an embedding row in the layer index), mm-RLHF post-training.

Estimator. J-lens current-and-future (addendum eq. 1-2): the cotangent is injected at every valid target position of the document, not only at the position under analysis, and is aggregated in two stages —between positions within the document, then between documents with equal weight—. Pre-norm residual target (addendum eq. 3-4): z is the output of the last block and the readout is r_λ(h) = softcap(W_U · N(J̄_λ h) + b_U), with the RMSNorm applied inside the readout.

CKA and segmentation. Centered linear CKA over the admitted vocabulary (255 893 of 262 144 tokens; hash published in §8) via the closed form G = W_c^T W_c. The segmentation into contiguous blocks minimizes c(a,b) = 2/(m−1)·Σ_{i<j} D_ij over CKA distances D = 1 − CKA, with Q = Σ costos de bloque / 42, minimum block sizes 3 (5 for the workspace block in the sealed 3-block cut), solved by exact enumeration with deterministic tie-breaking.

Corpus. WikiText-103, test split, seed 42, documents of 256 tokens: 200 to fit J̄ + 100 for evaluation without overlap. 200/200 valid documents; ~144 s/document.

Uncertainty. Bootstrap by document, B = 500, seed 42. The 95% stability interval is the shortest contiguous integer interval that contains ≥95% of the replicates (not a classic frequentist CI). The onset is declared localized only if that interval has width ≤4 layers; contrast with m-out-of-n bootstrap (m = 69).

Pre-registered decision windows (unchanged with respect to the frozen text): P-LSGOT predicts onset at L19–23 (45–55% depth), derived from layer-wise analysis on internal documents of the LSGOT series, referenced by hash in the pre-registration (references 2–4). Provenance note: the published LSGOT paper (arXiv:2607.09842) analyzes exclusively last-layer trajectories and does not contain the L21 prediction; that hypothesis lives only in the frozen internal material, and its layer-wise analysis remains committed to the companion work of that series; P-EXTRAP predicts onset at L13–17 (31–40%, extrapolating Sonnet’s ~37%). Windows and localization threshold were fixed before computing any Jacobian on this model with the corrected estimator.

3. Confirmatory result

The point estimate is τ̂₁ = L15 (15/42 = 35.7% relative depth), at the center of the P-EXTRAP window and less than two percentage points from the ~37% of Sonnet 4.5. The workspace→motor transition is τ̂₂ = L23. Cut quality: Q = 0.1109, R₃ = 0.480, G_min = 0.1379. The onset is identical under admitted and full vocabulary.

The sealed rule requires, in addition to a point estimate within the window, that the onset be localized. It is not: the 95% stability interval is [L9, L16] (width 7 > threshold 4; m-out-of-n gives [L9, L17], consistent), and the bootstrap distribution is bimodal — 59% of the 500 replicates at L15 (84% in the L14–17 band), with a secondary mode of 14% at L9. The formal verdict of the run, as it was pre-registered and without reformulation, is:

Inconclusive: onset not localized.

Figure 1. Bootstrap distribution of the 3-block onset (B=500). 59% of the replicates at L15, secondary mode of 14% at L9; the interval [9,16] exceeds the localization threshold.

4. Structural diagnosis (exploratory): four seams, not two boundaries

Everything that follows in this section is post-hoc and exploratory: it was not part of the sealed confirmatory analysis and motivates the follow-up pre-registration of §7, not a confirmed result. Every figure is reproducible from the artifacts in data/ with figures/seam_analysis.py.

4.1 Two nearly degenerate minima

The three-block cost landscape has two nearly degenerate minima: τ₁ = 15 (Q = 0.1109) and τ₁ = 9 (Q = 0.1153), separated by only 3.9%. The segmentation forced to three blocks chooses between two real candidate boundaries depending on which documents enter each resample — it does not converge on noise around a single one.

Figure 2. Segmentation cost as a function of the candidate onset. The point estimate (L15) falls within P-EXTRAP; L21, the center of P-LSGOT, lies past both candidates.

4.2 The bootstrap alternates between two complete anatomies

Relaxing the segmentation to four blocks places the boundaries at L9, L15 and L23 (Q = 0.0993). However — and this point corrects a reading that we held in an earlier draft of this same paper — the bootstrap of that segmentation is not more stable than that of three blocks: the 95% intervals of τ₁, τ₂ and τ₃ have widths 8, 12 and 17 respectively (data/bootstrap_4blocks.json). Moving from three to four blocks does not “resolve” the instability.

What the marginal distributions show, instead, is an alternation pattern: τ₁ concentrates mass at L9 (43%) and L15 (29%); τ₂ at L23 (39%) and L15 (36%); τ₃ at L23 (61%) and L40 (39%). That pattern is consistent with replicates oscillating between two complete four-block anatomies:

Anatomy A: cuts at (L9, L15, L23)
Anatomy B: cuts at (L15, L23, L40)

With the published artifacts we have only marginals, so the alternation is inferred and not demonstrated; the follow-up pre-registration stores the joint distribution precisely to close this point (§7).

Figure 3. Bootstrap stability of the three boundaries of 4 blocks. The bimodalities of τ₂ and τ₃ are the signature of the alternating A/B anatomies.

4.3 The five-block segmentation unifies both anatomies

The direct verification: the optimal five-block segmentation (exact enumeration) places its four seams exactly at L9, L15, L23 and L40 (Q = 0.0905) — the union of anatomies A and B. The two previously discrepant single-boundary estimates (≈L10 from the myopic v1.1 estimator; L15 from the corrected estimator) and the two alternating anatomies of the bootstrap are partial projections of this single structure.

The weight of each seam, measured as the increase in Q when removing it from the five-block cut, establishes a clear hierarchy:

Seam	ΔQ when removed	Reading
L23	+0.0337	workspace exit — the dominant transition of the network
L15	+0.0160	workspace entry (35.7% depth)
L9	+0.0116	end of the early sensory band
L40	+0.0087	start of the emission region

Figure 4. Left: Q decreases monotonically with k (0.1109 → 0.0993 → 0.0905 → 0.0833 for k=3…6) without a defined elbow — the number of blocks is not identifiable by cost. Right: seam hierarchy by marginal weight; L23 dominates.

Two caveats that the analysis itself imposes. First: since Q decreases monotonically in k by construction and the relative improvements show no elbow (10.5%, 8.9%, 8.0%), no value of k is established by this analysis — neither four nor five blocks. The identifiable question is not “how many blocks?” but “which seams are individually localized?”. Second: the final segment L40–42 is not a coherent block — its mean intra CKA is 0.558, lower than its cross-similarity with the motor block (0.670). L40 marks the point where the final layers begin to diverge individually toward the unembedding: an emission seam, a readout phenomenon, not a fifth homogeneous functional band.

Figure 5. Layer×layer CKA similarity. The four seams {L9, L15, L23, L40} are genuine similarity drops visible in the matrix, not artifacts of a segmentation choice.

5. Robustness controls

Vocabulary: identical onset (L15) under admitted vocabulary (255 893 tokens) and full vocabulary (262 144).
Precision: onset invariant under fp16.
CKA validity: the closed form matches the explicit computation over a subset of 20K tokens with a difference of 1e-5.
Causality: the gradient of the estimator with respect to non-causal positions (t′ < t) is exactly 0.00e+00.
Corroborating diagnostics (addendum §7, 500 positions): median kurtosis in [0.18, 9.82]; top-{1,5,10,25} readout precision against the last-layer argmax = 1.000 for all k; effective dimensionality d.99/d in [0.007, 0.973]; mean autocorrelation of the log-ratio −1.713.

Figure 6. Corroborating diagnostics by layer, with L15 and L23 marked. Effective dimension and top-k precision rise markedly near L23 — consistent with readout fidelity close to the output—; kurtosis and autocorrelation show local extremes near L21, within the workspace band.

6. Discussion

Read in the strict sense, the confirmatory result answers the open question of §9.1 of the source paper: at 8B scale and under a different post-training regime, the workspace onset falls at 35.7% depth — the fraction is not obviously an artifact of a single model family, although a single replication does not establish it as a constant.

Read with the care that the sealed rule imposes, the most informative contribution of this work is not the numerical coincidence but the structural diagnosis: the three-block operationalization that we inherited from the original methodology is architecturally misspecified for this model. The network has at least four hierarchical CKA seams, and a procedure authorized to report only two oscillates between partial projections of that structure — which is exactly what the inconclusive verdict registered. The non-localization was not a failed measurement; it was the pre-registration detecting a false structural assumption.

The hierarchical finding deserves emphasis: the dominant seam of the network (L23, ΔQ three times that of L40 and twice that of L15) is not the workspace entry but its exit. If this holds cross-architecture, it suggests that the most marked computational event in the layer-wise organization of these models is the transition from global availability toward motor/emission commitment — a hypothesis with a direct echo in the prior LSGOT signal (§4.4) and testable with the ongoing panel.

We deliberately avoid the vocabulary of dynamical systems (“attractor”, “crystallization”) when describing this anatomy: they are properties of a similarity matrix and a segmentation objective, not claims about the internal dynamics of the model.

Outlook. A parallel run on Qwen2.5-7B-Instruct (28 layers, SFT only) shows the opposite profile: a single, sharp onset at L8 with a 95% interval of zero width (492/500 identical replicates). Whether the sharpness of the seams —not only their location— can be diagnostic of the post-training regime is the subject of the companion paper, once DeepSeek-R1-Distill-7B completes the panel.

7. Limitations and reproducibility

A single confirmed architecture (Gemma-4-E4B-it); the preservation of the depth fraction should not be generalized without the panel of §7.
All of §4 is exploratory and post-hoc; none of the four seams is confirmed under a sealed rule until §7.1–7.3 is completed.
The alternation of anatomies A/B (§4.2) is inferred from marginals; its demonstration requires the joint distribution (§7.2).
Full protocol, estimator addendum (eq. 1-26) and deviations changelog (with SHA-256 hashes) as supplementary material; see data/DATA_SOURCES.md for the provenance of each figure. The raw Jacobian tensor (1.1 GB) is referenced by checksum.
Corpus, seeds, vocabulary hash (6bd2b7d9…0289e3) and bootstrap parameters in §2; the analysis path is deterministic given those values. The figures of §4.3 are regenerated with figures/seam_analysis.py from data/cka_matrix_v2.npy.

8. Conclusion

At 8B scale and under a different post-training regime, the workspace onset of Gemma-4-E4B-it falls at 35.7% relative depth, closely preserving the fraction reported by Anthropic for Sonnet 4.5. The sealed localization verdict —inconclusive— is not a null result but the measurable symptom of a false structural assumption: the network does not have two boundaries but four hierarchical seams {L9, L15, L23, L40}, dominated by the workspace exit at L23. The well-posed question for the next stage is not how many blocks the model has, but which seams survive a pre-registered individual localization criterion — and that pre-registration is defined here.

References

Gurnee, W.*, Sofroniew, N.*, Pearce, A., Piotrowski, M., Kauvar, I., Chen, R., Soligo, A., Bogdan, P., Ong, E., Wang, R., Thompson, T. B., Abrahams, D., Kantamneni, S., Ameisen, E., Batson, J., Lindsey, J.*† (2026). Verbalizable Representations Form a Global Workspace in Language Models. Transformer Circuits Thread, Anthropic. https://transformer-circuits.pub/2026/workspace/index.html (July 6, 2026).
Internal pre-registration preregistro_A1_jlens_L21.md (v1.1, frozen; SHA-256 af754c73c8b92b6a2eb4d3bd0b87c091579971adc698364de04278e5dfca813f).
Internal addendum propuesta_adenda_A1_MATH_v1.0.md (SHA-256 038204e637d3e160547700314e0e45a9a99624af8660b953f509ef195f1e9001).
Internal changelog preregistro_A1_CHANGELOG.md — full record of deviations with dates and hashes.

Data and code availability: the derived artifacts (CKA matrix, bootstrap distributions, curves, confirmatory report, seam_analysis.json) accompany this preprint in data/; figures/generate_figures.py and figures/seam_analysis.py deterministically regenerate all figures and numbers of §4.3. See data/DATA_SOURCES.md for provenance and the checksum of the excluded Jacobian tensor.