A Statistical Physics of Language Model Reasoning

Abstract

Transformer LMs show emergent reasoning that resists mechanistic understanding. We offer a statistical physics framework for continuous-time chain-of-thought reasoning dynamics. We model sentence-level hidden state trajectories as a stochastic dynamical system on a lower-dimensional manifold. This drift-diffusion system uses latent regime switching to capture diverse reasoning phases, including misaligned states or failures. Empirical trajectories (8 models, 7 benchmarks) show a rank-40 projection (balancing variance capture and feasibility) explains  50% variance. We find four latent reasoning regimes. An SLDS model is formulated and validated to capture these features. The framework enables low-cost reasoning simulation, offering tools to study and predict critical transitions like misaligned states or other LM failures.

1 Introduction

Transformer LMs (Vaswani et al., 2017), trained for next-token prediction (Radford et al., 2019; Brown et al., 2020), show emergent reasoning like complex cognition (Wei et al., 2022). Standard analyses of discrete components (e.g., attention heads (Elhage et al., 2021; Olsson et al., 2022)) provide limited insight into longer-scale semantic transitions in multi-step reasoning (Allen-Zhu & Li, 2023; López-Otal et al., 2024). Understanding these high-dimensional, prediction-shaped semantic trajectories, particularly how they might cause misaligned states, is a key challenge (Li et al., 2023; Nanda et al., 2023).

We model reasoning as a continuous-time dynamical system, drawing from statistical physics (Chaudhuri & Fiete, 2016; Schuecker et al., 2018). Sentence-level hidden states $h(t)\in\mathbb{R}^{D}$ evolve via a stochastic differential equation (SDE):

$$\,\mathrm{d}h(t)=\mu(h(t),Z(t))\,\mathrm{d}t+B(h(t),Z(t))\,\mathrm{d}W(t),$$ (1)

with drift $\mu$, diffusion $B$, Wiener process $W(t)$, and latent regimes $Z(t)$. This decomposes trajectories into trends and variations, helping identify deviations. As full high-dimensional SDE analysis (e.g., $D>2048$ for most LMs) is impractical, we use a lower-dimensional manifold capturing significant variance for modeling.

This continuous-time dynamical systems perspective offers several benefits:

Chain-of-thought (CoT) prompting (Wei et al., 2022; Wang et al., 2023) has demonstrated that LMs can follow structured reasoning pathways, hinting at underlying processes amenable to a dynamical systems description. While prior work has applied continuous-time models to neural dynamics generally, the explicit modeling of transformer reasoning at these semantic timescales, particularly as an approximation for impractical full-dimensional analysis, has been largely unexplored. Our work bridges this gap by pursuing an SDE-based perspective informed by empirical analysis of transformer hidden-state trajectories.

This paper is structured as follows: Section 2 introduces the mathematical formalism of SDEs and regime switching. Section 3 details our data collection and initial empirical findings that motivate the model, including the practical need for dimensionality reduction. Section 4 formally defines the SLDS model. Section 5 presents experimental validation, including model fitting, generalization, ablation studies, and a case study on modeling adversarial belief shifts as an example of predicting misaligned states.

2 Mathematical Preliminaries

We conceptualize the internal reasoning process of a transformer LM as a continuous-time stochastic trajectory evolving within its hidden-state space. Let $h_{t}\in\mathbb{R}^{D}$ be the final-layer residual embedding extracted at discrete sentence boundaries $t=0,1,2,\dots$. To capture the rich semantic evolution across reasoning steps, we treat these discrete embeddings as observations of an underlying continuous-time process $h(t):\mathbb{R}_{\geq 0}\to\mathbb{R}^{D}$. The direct analysis of such a process in its full dimensionality (e.g., $D\geq 2048$) is often computationally prohibitive. We therefore aim to approximate its dynamics using SDEs, potentially in a reduced-dimensional space.

Definition 2.1 (Itô SDE).

An Itô stochastic differential equation on the state space $\mathbb{R}^{D}$ is given by:

$$\,\mathrm{d}h(t)=\mu(h(t))\,\mathrm{d}t+B(h(t))\,\mathrm{d}W(t),\quad h(0)\sim p_{0},$$ (2)

where $\mu:\mathbb{R}^{D}\to\mathbb{R}^{D}$ is the deterministic drift term, encoding persistent directional dynamics. The matrix $B:\mathbb{R}^{D}\to\mathbb{R}^{D\times D^{\prime}}$ is the diffusion term, modulating instantaneous stochastic fluctuations. $W(t)$ is a $D^{\prime}$-dimensional Wiener process (standard Brownian motion), and $p_{0}$ is the initial distribution. The noise dimension $D^{\prime}$ can be less than or equal to the state dimension $D$.

The drift $\mu(h(t))$ represents systematic semantic or cognitive tendencies, while the diffusion $B(h(t))$ accounts for fluctuations due to local uncertainties, token-level variations, or inherent model stochasticity. Standard conditions ensure the well-posedness of such SDEs:

Theorem 2.1 (Well-Posedness (Øksendal, 2003)).

If $\mu$ and $B$ satisfy standard Lipschitz continuity and linear growth conditions (see Appendix A), the SDE

$$\,\mathrm{d}h(t)=\mu(h(t))\,\mathrm{d}t+B(h(t))\,\mathrm{d}W(t)$$ (3)

has a unique strong solution for a given $D^{\prime}$-dimensional Wiener process $W(t)$.

We focus on dynamics at the sentence level:

Definition 2.2 (Sentence-Stride Process).

The sentence-stride hidden-state process is the discrete sequence $\{h_{t}\}_{t\in\mathbb{N}}$ obtained by extracting the final-layer transformer state immediately following each detected sentence boundary. This emphasizes mesoscopic, semantic-level changes over finer-grained token-level variations.

To analyze these dynamics in a computationally manageable way, particularly given the high dimensionality $D$ of $h(t)$, we utilize projection-based dimensionality reduction. The goal is to find a lower-dimensional subspace where the most significant dynamics, for the purpose of modeling the SDE, unfold.

Definition 2.3 (Projection Leakage).

Given an orthonormal matrix $V_{k}\in\mathbb{R}^{D\times k}$ (where $V_{k}^{\top}V_{k}=I_{k}$), the leakage of the drift $\mu$ under perturbations $v$ orthogonal to the image of $V_{k}$ (i.e., $v\perp\mathrm{Im}(V_{k})$) is

$$L_{k}=\sup_{\begin{subarray}{c}x\in\mathbb{R}^{D},\,\left\lVert v\right\rVert\leq\epsilon\\ v^{\top}V_{k}=0\end{subarray}}\frac{\left\lVert\mu(x+v)-\mu(x)\right\rVert}{\left\lVert\mu(x)\right\rVert}.$$

A small leakage $L_{k}$ implies that the drift’s behavior relative to its current direction is not excessively altered by components outside the subspace spanned by $V_{k}$, making the subspace a reasonable domain for approximation.

Assumption 2.1 (Approximate Projection Closure for Modeling).

For practical modeling of the SDE (Eq. 2), we assume there exists a rank $k$ (e.g., $k=40$ in our work, chosen based on empirical variance and computational trade-offs) and a perturbation scale $\epsilon>0$ such that $L_{k}\ll 1$. This allows the approximation of the drift within this $k$-dimensional subspace:

$$\mu(h(t))\approx V_{k}V_{k}^{\top}\mu(h(t))$$

holds up to an error of order $O(L_{k})$. This assumption underpins the feasibility of our low-dimensional modeling approach, enabling the analytical treatment inspired by statistical physics.

Empirical observations of reasoning trajectories suggest abrupt shifts, potentially indicating transitions between different phases of reasoning or slips into misaligned states. This motivates a regime-switching framework:

Definition 2.4 (Regime-Switching SDE).

Let $Z(t)\in\{1,\dots,K\}$ be a latent continuous-time Markov chain with a transition rate matrix $T\in\mathbb{R}^{K\times K}$. The corresponding regime-switching Itô SDE is:

$$\,\mathrm{d}h(t)=\mu_{Z(t)}(h(t))\,\mathrm{d}t+B_{Z(t)}(h(t))\,\mathrm{d}W(t),$$ (4)

where each latent regime $i\in\{1,\dots,K\}$ has distinct drift $\mu_{i}$ and diffusion $B_{i}$ functions. This allows for context-dependent dynamic structures (Ghahramani & Hinton, 2000), crucial for capturing diverse reasoning pathways.

These definitions establish the mathematical foundation for our analysis of transformer reasoning dynamics as a tractable approximation of a more complex high-dimensional process.

3 Data and Empirical Motivation

We build a corpus of sentence-aligned hidden-state trajectories from transformer-generated reasoning chains across a suite of models (Mistral-7B-Instruct (Jiang et al., 2023), Phi-3-Medium (Abdin et al., 2024), DeepSeek-67B (DeepSeek-AI et al., 2024), Llama-2-70B (Touvron et al., 2023), Gemma-2B-IT (Gemma Team & Google DeepMind, 2024), Qwen1.5-7B-Chat (Bai et al., 2023), Gemma-7B-IT (also (Gemma Team & Google DeepMind, 2024)), Llama-2-13B-Chat-HF (also (Touvron et al., 2023))) and datasets (StrategyQA (Geva et al., 2021), GSM-8K (Cobbe et al., 2021), TruthfulQA (Lin et al., 2022), BoolQ (Clark et al., 2019), OpenBookQA (Mihaylov et al., 2018), HellaSwag (Zellers et al., 2019), PiQA (Bisk et al., 2020), CommonsenseQA (Talmor et al., 2021, 2019)), yielding roughly 9,800 distinct trajectories spanning $\sim$40,000 sentence-to-sentence transitions.

3.1 Sentence-Level Dynamics and Manifold Structure for Tractable Modeling

First, we confirmed that sentence-level increments effectively capture semantic evolution. Figure 1(a) compares the cumulative distribution functions (CDFs) of jump norms ($\left\lVert\Delta h_{t}\right\rVert$) at both token and sentence strides. Token-level increments show a noisy distribution skewed towards small values, primarily reflecting syntactic variations. In contrast, sentence-level increments are orders of magnitude larger, clearly indicating significant semantic shifts and validating our choice of sentence-stride analysis. To reduce "jitter" from minor variations, we filtered out transitions below a minimum threshold ($\left\lVert\Delta h_{t}\right\rVert\leq 10$ in normalized units), yielding cleaner semantic trajectories.

To uncover underlying geometric structures that could make modeling tractable, we applied Principal Component Analysis (PCA) (Jolliffe, 2002) to the sentence-stride embeddings. We found that a relatively low-dimensional projection (rank $k=40$) captures approximately 50% of the total variance in these reasoning trajectories (details in Appendix A). While reasoning dynamics occur in a high-dimensional embedding space, this finding suggests that a significant portion of their variance is concentrated in a lower-dimensional subspace. This is crucial because constructing and analyzing a stochastic process (like a random walk or SDE) in the full embedding dimension (e.g., 2048) is often impractical. The rank-40 manifold thus provides a computationally feasible domain for our dynamical systems modeling, not necessarily because the process is strictly confined to it, but because it offers a practical and informative approximation.

3.2 Linear Predictability and Multimodal Residuals

To assess the predictive structure of the semantic drift within this tractable manifold, we performed a global ridge regression (Hoerl & Kennard, 1970), fitting a linear model to predict subsequent sentence embeddings from previous ones:

$$h_{t+1} \approx Ah_{t}+c,$$ (5)

$$(A,c) =\arg\min_{A,c}\sum_{t}\|\Delta h_{t}-(A-I)h_{t}-c\|^{2}+\lambda\|A\|_{F}^{2}.$$ (6)

Using a modest regularization ($\lambda=1.0$), this global linear model achieved an $R^{2}\approx 0.51$, indicating substantial linear predictability in sentence-to-sentence transitions.

However, an examination of the residuals from this linear fit, $\xi_{t}=\Delta h_{t}-[(A-I)h_{t}+c]$, revealed persistent multimodal structure, even after the linear drift component was removed (Figure 1(b)). This multimodality suggests the presence of distinct underlying dynamic states or phases—some potentially representing "misaligned states" or divergent reasoning paths—that are not captured by a single linear model.

Inspired by Langevin dynamics, where a particle in a multi-well potential $U(x)$ can exhibit metastable states (Appendix E), we interpret these multimodal residual clusters as evidence of distinct latent reasoning regimes. The stationary probability distribution $p_{\mathrm{st}}(x)\propto e^{-U(x)/D}$ for an SDE $\,\mathrm{d}x=-U^{\prime}(x)\,\mathrm{d}t+\sqrt{2D}\,\mathrm{d}W_{t}$ becomes multimodal if $U(x)$ has multiple minima and noise $D$ is sufficiently low. Analogously, the observed clusters in our residual analysis point towards the existence of multiple metastable semantic basins in the reasoning process. This strongly motivates the introduction of a latent regime structure to adequately model these richer, nonlinear dynamics and to understand how an LLM might transition between effective reasoning and potential failure modes.

Figure 1: (a) CDF comparison of token and sentence jump norms, illustrating that sentence-level increments capture more substantial semantic shifts. (b) Histograms of residual norms from a global linear fit, showing raw residuals $\lVert\xi_{t}\rVert$ (left) and residuals projected onto a low-rank PCA space $\lVert\zeta_{t}\rVert$ (right). Both reveal significant multimodality, motivating regime switching to capture distinct reasoning phases or potential misalignments.

4 A Switching Linear Dynamical System for Reasoning

The empirical evidence that a significant portion of variance is captured by a low-dimensional manifold (making it a practical subspace for analysis, as directly modeling a 2048-dim random walk is often infeasible) and the observation of multimodal residuals motivate a model that combines linear dynamics within distinct regimes with switches between these regimes. Such switches may represent transitions between different cognitive states, some of which could be misaligned or lead to errors.

4.1 Linear Drift within Regimes

While a single global linear model (Eq. 5) captures about half the variance, the residual analysis (Figure 1(b)) indicates that a more nuanced approach is needed. We project the residuals $\xi_{t}$ onto the principal subspace $V_{k}$ (from Assumption 2.1, where $k=40$ offers a balance between explained variance and computational cost) to get $\zeta_{t}=V_{k}^{\top}\xi_{t}$. The clustered nature of these projected residuals $\zeta_{t}$ suggests that the reasoning process transitions between several distinct dynamical modes or ‘regimes’.

4.2 Identifying Latent Reasoning Regimes

To formalize these distinct modes, we fit a $K$-component Gaussian Mixture Model (GMM) to the projected residuals $\zeta_{t}$, following classical regime-switching frameworks (Hamilton, 1989):

$$p(\zeta_{t})=\sum_{i=1}^{K}\pi_{i}\,\mathcal{N}(\zeta_{t}\mid\mu_{i},\Sigma_{i}).$$ (7)

Information criteria (BIC/AIC) suggest $K=4$ as an appropriate number of regimes for our data. While the true underlying multimodality is complex across many dimensions (see Figure 6, Appendix A), a four-regime model provides a parsimonious yet effective way to capture key dynamic behaviors, including those that might represent misalignments or slips into undesired reasoning patterns, while maintaining computational tractability. We interpret these $K=4$ modes as distinct reasoning phases, such as systematic decomposition, answer synthesis, exploratory variance, or even failure loops, each characterized by specific drift perturbations and noise profiles. Figure 2 and Figure 3 visualize these uncovered regimes in the low-rank residual space.

(a) Regime-colored PCA of residuals

(b) Regime-colored histogram of $\left\lVert\zeta_{t}\right\rVert$

Figure 2: Latent regimes ($K=4$) uncovered by GMM fitting on low-rank residuals $\zeta_{t}$. (a) Residuals projected onto their first two principal components, colored by GMM assignment, showing distinct clusters. (b) Histogram of residual norms $\left\lVert\zeta_{t}\right\rVert$, colored by GMM regime assignment, further illustrating regime separation. These regimes may capture different reasoning qualities, including potential misalignments.

Figure 3: GMM clustering ($K=4$) of low-rank residuals $\zeta_{t}$, visualized in the space of the first two principal components of $\zeta_{t}$. The distinct cluster centers provide justification for the regime decomposition, potentially corresponding to different reasoning states or failure modes.

4.3 The Switching Linear Dynamical System (SLDS) Model

We integrate these observations into a discrete-time Switching Linear Dynamical System (SLDS). Let $Z_{t}\in\{1,\dots,K\}$ be the latent regime at step $t$. The state $h_{t}$ evolves according to:

$$Z_{t} \sim\mathrm{Categorical}(\pi),\quad P(Z_{t+1}=j\mid Z_{t}=i)=T_{ij},$$

$$h_{t+1} =h_{t}+V_{k}\bigl{(}M_{Z_{t}}(V_{k}^{\top}h_{t})+b_{Z_{t}}\bigr{)}+\varepsilon_{t},$$ (8)

$$\varepsilon_{t} \sim\mathcal{N}(0,\Sigma_{Z_{t}}).$$

Here, $M_{i}\in\mathbb{R}^{k\times k}$ and $b_{i}\in\mathbb{R}^{k}$ are the regime-specific linear transformation matrix and offset vector for the drift within the $k$-dimensional semantic subspace defined by $V_{k}$. $\Sigma_{i}$ is the regime-dependent covariance for the noise $\varepsilon_{t}$. The initial regime probabilities are $\pi$, and $T$ is the transition matrix encoding regime persistence and switching probabilities. This SLDS framework combines continuous drift within regimes, structured noise, and discrete changes between regimes, which can model shifts between correct reasoning and misaligned states.

The multimodal structure of the full residuals $\xi_{t}$ (before projection, see Figure 4) invalidates a single-mode SDE. This motivates our regime-switching formulation. The SLDS in Eq. 8 serves as a discrete-time surrogate for an underlying continuous-time switching SDE (Eq. 4):

$$\,\mathrm{d}h(t)=\mu_{Z(t)}(h(t))\,\,\mathrm{d}t+B_{Z(t)}(h(t))\,\,\mathrm{d}W(t),$$ (9)

where each regime $i$ has its own drift $\mu_{i}(h)=V_{k}(M_{i}(V_{k}^{\top}h)+b_{i})$ (approximating the continuous drift within the chosen manifold for tractability) and diffusion $B_{i}$ (related to $\Sigma_{i}$). The transition matrix $T$ in the SLDS is related to the rate matrix of the latent Markov process $Z(t)$ in the continuous formulation.

Figure 4: Failure of single-mode noise models for the full residuals $\xi_{t}$ (before projection). This plot shows mismatches between the empirical distribution of residual norms and fits from both Gaussian and Laplace distributions, highlighting the inadequacy of a single noise process and further motivating the regime-switching approach to capture diverse reasoning states, including potential misalignments.

5 Experiments & Validation

We empirically validate the proposed SLDS framework (Eq. 8). Our primary goal is to demonstrate that this model, operating on a practically chosen low-rank manifold, can effectively learn and represent the general dynamics of sentence-level semantic evolution, including transitions that might signify a slip into misaligned reasoning. The SLDS parameters ($\{M_{i},b_{i},\Sigma_{i}\}_{i=1}^{K}$, $T$, $\pi$) are estimated from our corpus of $\sim$40,000 sentence-to-sentence hidden state transitions using an Expectation-Maximization (EM) algorithm (Appendix B). It is crucial to note that the SLDS is trained to model the process by which language models arrive at answers—and potentially how they deviate into failure modes—not to predict the final answers of the tasks themselves. Based on empirical findings (Section 4), we use $K=4$ regimes and a projection rank $k=40$ (chosen for its utility in making the SDE-like modeling feasible).

The efficacy of the fitted SLDS is first assessed by its one-step-ahead predictive performance. Given an observed hidden state $h_{t}$ and the inferred posterior regime probabilities $\gamma_{t,j}=\mathbb{P}(Z_{t}=j\mid h_{0},\dots,h_{t})$ (obtained via forward-backward inference (Rabiner, 1989)), the model’s predicted mean state $\hat{h}_{t+1}$ is computed as:

$$\hat{h}_{t+1}=h_{t}+V_{k}\left(\sum_{j=1}^{K}\gamma_{t,j}\bigl{(}M_{j}(V_{k}^{\top}h_{t})+b_{j}\bigr{)}\right).$$ (10)

On held-out trajectories, the SLDS yields a predictive $R^{2}\approx 0.68$. This significantly surpasses the $R^{2}\approx 0.51$ achieved by the single-regime global linear model (Eq. 5), confirming the value of incorporating regime-switching dynamics. Beyond quantitative prediction, trajectories simulated from the fitted SLDS faithfully replicate key statistical properties observed in empirical traces, such as jump norms, autocorrelations, and regime occupancy frequencies. This dual capability—accurate description and realistic synthesis of reasoning trajectories—substantiates the SLDS as a robust model. Furthermore, the inferred regime posterior probabilities $\gamma_{t,j}$ provide valuable interpretability, allowing for the association of observable textual behaviors (e.g., systematic decomposition, stable reasoning, or error correction loops and potential misaligned states) with specific latent dynamical modes. These initial findings strongly support the proposed framework as both a descriptive and generative model of reasoning dynamics, offering a path to predict and understand LLM failure modes.

5.1 Generalization and Transferability of SLDS Dynamics

A critical test of the SLDS framework is its ability to capture generalizable features of reasoning dynamics, including those indicative of robust reasoning versus slips into misalignment, beyond the specific training conditions. We investigated this by training an SLDS on hidden state trajectories from a source (a particular LLM performing a specific task or set of tasks) and then evaluating its capacity to describe trajectories from a target (which could be a different LLM and/or task). Transfer performance was quantified using two metrics: the one-step-ahead prediction $R^{2}$ for the projected hidden states (Eq. 10) and the Negative Log-Likelihood (NLL) of the target trajectories under the source-trained SLDS. Lower NLL and higher $R^{2}$ values signify superior generalization.

Table 1 presents illustrative results from these transfer experiments. For instance, an SLDS is first trained on trajectories generated by a ‘Train Model’ (e.g., Llama-2-70B) performing a designated ‘Source Task’ (e.g., GSM-8K). This single trained SLDS is then evaluated on trajectories from various ‘Test Model’ / ‘Test Task’ combinations.

Table 1: SLDS transferability across models and tasks. Each SLDS is trained on trajectories from the specified ‘Train Model’ on its ‘Source Task’ (GSM-8K for Llama-2-70B, StrategyQA for Mistral-7B). Performance ($R^{2}$ for next hidden state prediction, NLL of test trajectories) is evaluated on various ‘Test Model’ / ‘Test Task’ combinations, demonstrating patterns of generalization in capturing underlying reasoning dynamics.

Train Model	Test Model	Test Task	$R^{2}$	NLL
(Source Task)
Llama-2-70B	Llama-2-70B	GSM-8K	0.73	80
(on GSM-8K)	Llama-2-70B	StrategyQA	0.65	115
	Mistral-7B	GSM-8K	0.48	240
	Mistral-7B	StrategyQA	0.37	310
Mistral-7B	Mistral-7B	StrategyQA	0.71	88
(on StratQA)	Mistral-7B	GSM-8K	0.63	135
	Llama-2-70B	StrategyQA	0.42	270
	Gemma-7B-IT	BoolQ	0.35	380
	Phi-3-Med	TruthfulQA	0.30	420

The results indicate that while the SLDS performs optimally when training and testing conditions align perfectly (e.g., Llama-2-70B on GSM-8K transferred to itself), it retains considerable descriptive power when transferred. Generalization is notably more successful when the underlying LLM architecture is preserved, even across different reasoning tasks (e.g., Llama-2-70B trained on GSM-8K and tested on StrategyQA shows only a modest drop in $R^{2}$ from 0.73 to 0.65). Conversely, transferring the learned dynamics across different LLM families (e.g., Llama-2-70B to Mistral-7B) proves more challenging, as reflected in lower $R^{2}$ values and higher NLLs. However, even in these challenging cross-family transfers, the SLDS often outperforms naive baselines like a simple linear dynamical system without regime switching (detailed comparisons not shown). These findings suggest that while some learned dynamical features are model-specific, the SLDS framework, by approximating the reasoning process as a physicist might model a complex system, is capable of capturing common, fundamental underlying structures in reasoning trajectories. Extended transferability results are provided in Appendix D.

5.2 Ablation Study

To elucidate the contribution of each core component within our SLDS framework, we conducted an ablation study. The full model (Eq. 8 with $K=4$ regimes and $k=40$ projection rank, selected for practical modeling of the SDE) was compared against three simplified variants:

• No Regime (NR): A single-regime model ($K=1$), still projected to the $k=40$ dimensional subspace. This tests the necessity of regime switching for capturing diverse reasoning states, including misalignments.

• No Projection (NP): A $K=4$ regime switching model operating directly in the full $D$-dimensional embedding space (i.e., without the $V_{k}$ projection). This tests the utility of the low-rank manifold assumption for tractable and effective modeling, given the impracticality of handling a full-dimension SDE.

• No State-Dependent Drift (NSD): A $K=4$ regime model where the drift within each regime is merely a constant offset $V_{k}b_{Z_{t}}$, and the linear transformation $M_{Z_{t}}$ is zero for all regimes. This tests the importance of the current state $h_{t}$ influencing its own future evolution within a regime.

Table 2 summarizes the performance of these models on a held-out test set.

Table 2: Ablation study results comparing the full SLDS against simplified variants: NR (single-regime projected model), NP (full-dimensional switching without projection), NSD (regime-switched offsets, no state-dependent linear drift). Performance is measured by $R^{2}$ and NLL. The results underscore the importance of each component for modeling reasoning dynamics and identifying potential failure modes.

Model	$R^{2}$	NLL
Full SLDS ($K=4,k=40$)	0.74	78
No Regime (NR, $K=1,k=40$)	0.58	155
No Projection (NP, $K=4$)	0.60	210
No State-Dep. Drift (NSD)	0.35	290
Global Linear (ref.)	0.51	180

Each ablation led to a notable reduction in performance, robustly demonstrating that all three key elements of our proposed model—regime-switching, low-rank projections (for practical SDE approximation), and state-dependent drift—are jointly essential for accurately capturing the nuanced dynamics of transformer reasoning. The NR model, lacking regime switching, performs substantially worse ($R^{2}=0.58$) than the full SLDS ($R^{2}=0.74$), highlighting the critical role of modeling distinct reasoning phases, including potential slips into misaligned states. Removing the low-rank projection (NP model) also significantly impairs effectiveness ($R^{2}=0.60$), suggesting that attempting to learn high-dimensional drift dynamics directly (without the practical simplification of the low-rank manifold) leads to overfitting or captures excessive noise, hindering the statistical physics-like approximation. Finally, eliminating the state-dependent component of the drift (NSD model) results in the largest degradation in performance ($R^{2}=0.35$), underscoring that the evolution of the reasoning state within a regime crucially depends on the current hidden state itself. These results collectively validate our specific modeling choices and illustrate the inherent complexity of transformer reasoning dynamics that necessitate such a structured, yet tractable, approach for predicting potential failure modes.

5.3 Case Study: Modeling Adversarially Induced Belief Shifts

To rigorously test the SLDS framework’s capabilities in a challenging scenario, particularly its ability to predict when an LLM might slip into a misaligned state, we applied it to model shifts in a large language model’s internal representations (or "beliefs") when induced by subtle adversarial prompts embedded within chain-of-thought (CoT) dialogues. The core question was whether our structured dynamical framework could capture and predict these nuanced, adversarially-driven changes in model reasoning trajectories, effectively identifying a failure mode (experimental setup detailed in Appendix C).

Figure 5: SLDS model validation via adversarial belief manipulation. Each row shows a distinct topic. Empirical belief trajectories where blue and red follow the clean and posioned belief trajectories, respectively (left). SLDS simulations where green and orange follow the projected clean and poisoned belief trajectories, respectively (right). Gold lines mark poison steps. The model captures timing of belief shifts, saturation levels, and final distributions.

We employed Llama-2-70B and Gemma-7B-IT, exposing them to a diverse array of misinformation narratives spanning public health misconceptions, historical revisionism, and conspiratorial claims. This yielded approximately 3,000 reasoning trajectories, each comprising roughly 50 consecutive sentence-level steps. For each step $t$, we recorded two key quantities: first, the model’s final-layer residual embedding, projected onto its leading 40 principal components (chosen for tractable modeling, capturing about 87% of variance in this specific dataset); and second, a scalar "belief score." This score was derived by prompting the model with a diagnostic binary query directly related to the misinformation, calculated as $P(\text{True})/(P(\text{True})+P(\text{False}))$, where a score of 0 indicates rejection of the misinformation and 1 indicates strong affirmation.

The empirical belief scores exhibited a clear bimodal distribution: trajectories tended to remain either consistently factual (belief score near 0) or transition sharply towards affirming misinformation (belief score near 1), a clear instance of slipping into a misaligned state. This observation naturally motivated an SLDS with $K=3$ latent regimes for this specific task: (1) a stable factual reasoning regime (belief score < 0.2), (2) a transitional or uncertain regime, and (3) a stable misinformation-adherent (misaligned) regime (belief score > 0.8). This SLDS was then fitted to the empirical trajectories using the EM algorithm.

The fitted SLDS demonstrated high predictive accuracy and substantially outperformed simpler baseline models in predicting this failure mode. For one-step-ahead prediction of the projected hidden states ($h^{\prime}_{t}=V_{k}^{\top}h_{t}$), the SLDS achieved $R^{2}$ values of approximately 0.72 for Llama-2-70B and 0.69 for Gemma-7B-IT. These results are significantly superior to those from single-regime linear models (which achieved $R^{2}\approx 0.45$) and standard Gated Recurrent Unit (GRU) networks ($R^{2}\approx 0.57-0.58$). Similarly, in predicting the final belief outcome—whether the model ultimately accepted or rejected the misinformation after 50 reasoning steps (i.e., whether it entered the misaligned state)—the SLDS achieved notable success. Final belief prediction accuracies were around 0.88 for Llama-2-70B and 0.85 for Gemma-7B-IT, compared to baseline methods which ranged from 0.62 to 0.78 accuracy (see Table 3). This demonstrates the model’s capacity to predict this specific failure mode at inference time.

Table 3: Comparative performance in modeling and predicting adversarially induced belief shifts (a failure mode). $R^{2}(h^{\prime}_{t+1})$ denotes one-step-ahead prediction accuracy for projected hidden states. ‘Belief Acc.’ is the accuracy in predicting whether the final belief score $b_{T}>0.5$ (misaligned state) after 50 reasoning steps. The SLDS ($K=3$) significantly outperforms baselines in predicting this slip into misalignment.

Model	Method	$R^{2}(h^{\prime}_{t+1})$	Belief Acc.
Llama-2-70B	Linear	0.35	0.55
	GRU-256	0.48	0.68
	SLDS ($K$=3)	0.72	0.88
Gemma-7B	Linear	0.33	0.52
	GRU-256	0.46	0.65
	SLDS ($K$=3)	0.69	0.85

Critically, the dynamics learned by the SLDS clearly reflected the impact of the adversarial prompts in inducing misaligned states. Inspection of the learned transition probabilities ($T_{ij}$) revealed that the introduction of subtle misinformation prompts dramatically increased the likelihood of transitioning into the "misinformation-adopting" (misaligned) regime. Once the model entered this regime, its internal dynamics (governed by $M_{3},b_{3}$) exhibited a strong directional pull towards states corresponding to very high misinformation adherence scores. Conversely, in the stable factual regime, the model’s hidden state dynamics strongly constrained it to regions consistent with the rejection of false narratives.

Figure 5 compellingly illustrates the close alignment between the empirical belief trajectories and those simulated by the fitted SLDS. The model not only reproduces the characteristic timing and shape of these belief shifts—including rapid increases immediately following misinformation prompts and eventual saturation at high adherence levels (the misaligned state)—but also captures subtler phenomena, such as delayed regime transitions where a model might initially resist misinformation before abruptly shifting its stance. Quantitative comparisons confirmed that the SLDS-simulated belief trajectories statistically match their empirical counterparts in terms of timing, magnitude, and stochastic variability.

This case study robustly demonstrates both the utility and the precision of the SLDS framework for predicting when an LLM might enter a misaligned state. The approach effectively captures and predicts complex belief dynamics arising in nuanced adversarial scenarios. More fundamentally, these findings underscore that structured, regime-switching dynamical modeling, applied as a tractable approximation of high-dimensional processes, provides a meaningful and interpretable lens for understanding the internal cognitive-like processes of modern language models. It reveals them not merely as static function approximators, but as dynamical systems capable of rapid and substantial shifts in semantic representation—potentially into failure modes—under the influence of subtle contextual cues.

5.4 Summary of Experimental Findings

The comprehensive experimental validation confirms that a relatively simple low-rank SLDS (where low rank is chosen for practical SDE modeling), incorporating a few latent reasoning regimes, can robustly capture complex reasoning dynamics. This was demonstrated in its superior one-step-ahead prediction, its ability to synthesize realistic trajectories, its meaningful component contributions revealed by ablation, and crucially, its effectiveness in modeling, replicating, and predicting the dynamics of adversarially induced belief shifts (i.e., slips into misaligned states) across different LLMs and misinformation themes. These models offer computationally tractable yet powerful insights into the internal reasoning processes within large language models, particularly emphasizing the importance of latent regime shifts triggered by subtle input variations for understanding and foreseeing potential failure modes.

6 Impact and Future Work

Our framework, inspired by statistical physics approximations of complex systems, offers a means to audit and compress transformer reasoning processes. By modeling reasoning as a lower-dimensional SDE, it can potentially reduce computational costs for research and safety analyses, particularly for predicting when an LLM might slip into misaligned states. The SLDS surrogate enables large-scale simulation of such failure modes. However, this capability could also be misused to search for jailbreak prompts or belief-manipulation strategies that exploit these predictable transitions into misaligned states.

Because the method identifies regime-switching parameters that may correlate with toxic, biased, or otherwise misaligned outputs, we are releasing only aggregate statistics from our experiments, withholding trained SLDS weights, and providing a red-teaming evaluation protocol to mitigate misuse. Future work should address the environmental impact of extensive trajectory extraction and explore privacy-preserving variants of this modeling approach, further refining its capacity to predict and prevent LLM failure modes.

7 Conclusion

We introduced a statistical physics-inspired framework for modeling the continuous-time dynamics of transformer reasoning. Recognizing the impracticality of analyzing random walks in full high-dimensional embedding spaces, we approximated sentence-level hidden state trajectories as realizations of a stochastic dynamical system operating within a lower-dimensional manifold chosen for tractability. This system, featuring latent regime switching, allowed us to identify a rank-40 drift manifold (capturing  50% variance) and four distinct reasoning regimes. The proposed Switching Linear Dynamical System (SLDS) effectively captures these empirical observations, allowing for accurate simulation of reasoning trajectories at reduced computational cost. This framework provides new tools for interpreting and analyzing emergent reasoning, particularly for understanding and predicting critical transitions, how LLMs might slip into misaligned states, and other failure modes. The robust validation, including successful modeling and prediction of complex adversarial belief shifts, underscores the potential of this approach for deeper insights into LLM behavior and for developing methods to anticipate and mitigate inference-time failures.

Appendix A Mathematical Foundations and Manifold Justification

The SDE in Eq. 3 is $\,\mathrm{d}h(t)=\mu(h(t))\,\mathrm{d}t+B(h(t))\,\mathrm{d}W(t)$. Theorem 2.1 states its well-posedness under Lipschitz continuity and linear growth conditions on $\mu$ and $B$. These standard hypotheses guarantee, by classical results (Øksendal, 2003, Thm. 5.2.1), the existence and uniqueness of a strong solution. The proof employs a standard Picard iteration scheme, defining a sequence $(Y^{(n)})_{n\geq 0}$ recursively by

$$Y_{t}^{(n+1)} =h(0)+\int_{0}^{t}\mu(Y_{s}^{(n)})\,\mathrm{d}s+\int_{0}^{t}B(Y_{s}^{(n)})\,\mathrm{d}W_{s},$$

$$Y_{t}^{(0)} =h(0).$$

Standard arguments leveraging Itô isometry (see e.g., Øksendal, 2003) and Grönwall’s lemma (Grönwall, 1919) establish convergence of this sequence to a unique strong solution $X_{t}$.

We next address the bound on projection leakage $L_{k}$ (Definition 2.3). By definition,

$$L_{k}=\sup_{\begin{subarray}{c}x\in\mathbb{R}^{D},\,v^{\top}V_{k}=0,\\[2.0pt] \|v\|\leq\varepsilon\end{subarray}}\frac{\|\mu(x+v)-\mu(x)\|}{\|\mu(x)\|}.$$

Using the Lipschitz continuity of the drift $\mu$ (with Lipschitz constant $L_{\mu}$), for perturbations $\|v\|\leq\varepsilon$:

$$\|\mu(x+v)-\mu(x)\|\leq L_{\mu}\,\varepsilon.$$

Assuming that the magnitude of the drift does not vanish on the domain of interest $\mathcal{D}$ (justified empirically), we set $\mu_{\min}:=\inf_{x\in\mathcal{D}}\|\mu(x)\|>0$. This yields the bound:

$$L_{k}(\varepsilon)\leq\frac{L_{\mu}\,\varepsilon}{\mu_{\min}}.$$

We can sharpen this by decomposing $\mu(x)$ into projected and residual components: $\mu(x)=V_{k}V_{k}^{\top}\mu(x)+r_{k}(x)$, where $r_{k}(x)=(I-V_{k}V_{k}^{\top})\mu(x)$ is the residual. Defining the ratio $\rho_{k}=\sup_{x\in\mathcal{D}}\frac{\|r_{k}(x)\|}{\|\mu(x)\|}$, the triangle inequality gives a refined bound:

$$L_{k}\leq\rho_{k}+\frac{L_{\mu}\,\varepsilon}{\mu_{\min}}.$$

Practically, we enforce $L_{k}\ll 1$ by selecting $k$ large enough to reduce $\rho_{k}$ (i.e., capture most of the drift direction within a computationally tractable subspace) and restricting perturbations to small $\varepsilon$.

The choice of a rank-40 drift manifold ($k=40$) is motivated by the impracticality of constructing SDE models directly in the full embedding dimension (e.g., $D\geq 2048$). Empirical PCA on observed drift increments $\Delta h_{t}$ (summarized in a data matrix $H$) shows that the first 40 principal components capture approximately 50% of the drift variance. If $H=U\Sigma W^{\top}$ is the SVD of $H$, the relative Frobenius norm of the residual after rank-$k$ truncation is $\sqrt{{\sum_{i>k}\sigma_{i}^{2}}/{\sum_{i}\sigma_{i}^{2}}}$. For $k=40$, this value is $\rho_{40}\approx 0.50$. While this captures only half the variance, it provides a significant simplification that makes the dynamical systems modeling approach feasible. Subsequent components add diminishing amounts of variance. Perturbation theory, specifically the Davis–Kahan sine-theta theorem (Davis & Kahan, 1970),further ensures this empirical drift manifold is stable given the observed spectral gap at the 40th eigenvalue and large sample size. Higher ranks would increase inference complexity with diminishing returns in variance capture for this approximate model, making $k=40$ a pragmatic choice for balancing model fidelity with the computational feasibility of the SDE approximation. The primary goal is not to claim the random walk *only* occurs on this manifold, but that this manifold serves as a useful and tractable domain for approximation.

Figure 6 shows the distribution of residuals $\Delta h_{t}$ projected onto each of these 40 principal component dimensions, revealing rich multimodal structures that motivate the regime-switching approach. These regimes can be interpreted as different reasoning pathways or potential "misaligned states" that the statistical physics-like approximation aims to capture. While the true multimodality is complex, our four-regime model ($K=4$) provides an efficient approximation for capturing key dynamics, including deviations that might lead to failures.

Figure 6: Violin plot of residual $\Delta h_{t}$ values projected across the 40 principal component dimensions of the drift manifold (chosen for tractable SDE modeling). Each violin shows the distribution of residuals for a specific dimension, revealing rich multimodal structure that motivates our regime-switching approach. These structures suggest different operational states, some of which could correspond to misaligned reasoning or failure modes.

Appendix B EM Algorithm for SLDS Parameter Estimation

This appendix details the Expectation-Maximization (EM) algorithm (Dempster et al., 1977) used for fitting the parameters of the Switching Linear Dynamical System (SLDS) as defined in Eq. 8. The model parameters are $\theta=(\pi,T,\{M_{j},b_{j},\Sigma_{j}\}_{j=1}^{K})$, where $V_{k}$ is a fixed orthonormal PCA projection basis (e.g., $k=40$, chosen for practical modeling).

The SLDS dynamics are:

$$Z_{t}\sim\mathrm{Categorical}(\pi)\qquad\qquad\text{for }t=0,$$

$$P(Z_{t+1}=j\,|\,Z_{t}=i)=T_{ij}\qquad\text{for }t\geq 0,$$

$$h_{t+1}=h_{t}+V_{k}(M_{Z_{t+1}}(V_{k}^{\top}h_{t})+b_{Z_{t+1}})+\epsilon_{t+1},$$

with residual noise $\epsilon_{t+1}\sim\mathcal{N}(0,\Sigma_{Z_{t+1}})$.

The log-likelihood for observed data $H=(h_{0},\dots,h_{T_{end}})$ is $P(H\,|\,\theta)=\sum_{Z}P(H,Z\,|\,\theta)$, where $Z=(Z_{0},\dots,Z_{T_{end}-1})$. Direct maximization is intractable, hence EM. At iteration $m$, EM alternates:

B.1 E-step

Compute expected sufficient statistics under $\theta^{(m)}$. Use standard forward ($\alpha_{t}(j)=P(h_{0},\dots,h_{t},Z_{t}=j|\theta^{(m)})$) and backward ($\beta_{t}(j)=P(h_{t+1},\dots,h_{T_{end}}|Z_{t}=j,\theta^{(m)})$) recursions (Rabiner, 1989). Posterior regime probabilities:

$$\gamma_{t}(j) =P(Z_{t}=j|H,\theta^{(m)})$$

$$=\frac{\alpha_{t}(j)\beta_{t}(j)}{\sum_{i=1}^{K}\alpha_{t}(i)\beta_{t}(i)},$$

$$\xi_{t}(i,j) =P(Z_{t}=i,Z_{t+1}=j|H,\theta^{(m)})$$

$$=\frac{\alpha_{t}(i)T_{ij}^{(m)}\beta_{t+1}(j)}{P(H|\theta^{(m)})}$$

$$\phantom{=}+\mathcal{N}(\Delta h^{\prime}_{t}|M_{j}^{(m)}x_{t}+b_{j}^{(m)},\Sigma_{j}^{(m)})$$

where $\Delta h^{\prime}_{t}=V_{k}^{\top}(h_{t+1}-h_{t})$ and $x_{t}=V_{k}^{\top}h_{t}$. The $\mathcal{N}(\cdot)$ term is the emission probability of observing $h_{t+1}$ given $h_{t}$ and $Z_{t+1}=j$. These probabilities help identify transitions between different reasoning states, including potentially misaligned ones.

B.2 M-step

In the M-step, parameters are updated to maximize the expected complete data log-likelihood. The initial state probabilities $\hat{\pi}_{j}$ are given by $\hat{\pi}_{j}=\gamma_{0}(j)$. Transition probabilities $\hat{T}_{ij}$ are calculated as:

$$\hat{T}_{ij}=\frac{\sum_{t=0}^{T_{\text{end}}-2}\xi_{t}(i,j)}{\sum_{t=0}^{T_{\text{end}}-2}\gamma_{t}(i)}.$$

The regime-specific dynamics $\{M_{j},b_{j},\Sigma_{j}\}$ are determined through a process analogous to weighted linear regression. We define the projected change as $\Delta h^{\prime}_{t}=V_{k}^{\top}(h_{t+1}-h_{t})$ and the projected state as $x_{t}=V_{k}^{\top}h_{t}$. Augmented regressors $\mathcal{X}_{t}=[x_{t}^{\top},\,1]^{\top}$ and corresponding augmented parameters $\mathcal{M}_{j}=[M_{j}^{\top},b_{j}]^{\top}$ are utilized. The update for $\hat{\mathcal{M}}_{j}$ is then computed as:

$$\begin{split}\hat{\mathcal{M}}_{j}={}&\left(\sum_{t=0}^{T_{\text{end}}-1}\gamma_{t+1}(j)\mathcal{X}_{t}\mathcal{X}_{t}^{\top}\right)^{-1}\\ &\quad\times\left(\sum_{t=0}^{T_{\text{end}}-1}\gamma_{t+1}(j)\mathcal{X}_{t}(\Delta h^{\prime}_{t})^{\top}\right).\end{split}$$

From $\hat{\mathcal{M}}_{j}$, the dynamics matrix $\hat{M}_{j}$ and bias vector $\hat{b}_{j}$ are extracted using $\hat{M}_{j}=\hat{\mathcal{M}}_{j}(1:k,:)^{\top}$ and $\hat{b}_{j}=\hat{\mathcal{M}}_{j}(k+1,:)^{\top}$, respectively. To update the covariance matrix $\hat{\Sigma}_{j}$, we first define the residuals for each regime $j$ at time $t$ as $e_{jt}=\Delta h^{\prime}_{t}-\hat{M}_{j}x_{t}-\hat{b}_{j}$. Then, $\hat{\Sigma}_{j}$ is computed by:

$$\hat{\Sigma}_{j}=\frac{\sum_{t=0}^{T_{\text{end}}-1}\gamma_{t+1}(j)e_{jt}e_{jt}^{\top}}{\sum_{t=0}^{T_{\text{end}}-1}\gamma_{t+1}(j)}.$$

These updates are derived from maximizing the expected complete data log-likelihood.

Scaling techniques are employed during the forward-backward passes to mitigate numerical underflow. When dealing with multiple observation sequences, the necessary statistics are accumulated across all sequences before the parameter updates are performed. Convergence of the Expectation-Maximization algorithm is typically assessed by observing when parameter changes fall below a predefined threshold, when the change in log-likelihood becomes negligible, or when a maximum number of iterations is reached. The inherent property of EM ensuring a monotone increase in the log-likelihood contributes to stable training. Ultimately, the objective is to identify a set of parameters that most accurately describes the observed dynamics of the reasoning process. This includes modeling transitions between different operational regimes, which can be indicative of phenomena such as the onset of failure modes.

Appendix C Adversarial Chain-of-Thought Belief Manipulation

This appendix describes experimental details for the adversarial belief-manipulation results in Section 5.3, focusing on how the SLDS framework can model and predict LLMs slipping into misaligned states, following ICML practice.

C.1 Experimental Design

We studied Llama-2-70B and Gemma-7B-IT under adversarial prompting on twelve misinformation themes (public health, conspiracies, financial myths, AI fears, historical revisionism, pseudoscience, etc.). For each theme/model, paired clean and poisoned CoTs were generated. Clean CoTs used neutral questions (e.g., “Summarize arguments for and against vaccination”). Poisoned CoTs interspersed adversarial prompts at predetermined steps to guide the model towards harmful beliefs (misaligned states). Each CoT had $\sim$50 sentence-level steps. We collected $\sim$100 trajectories per combination, totaling $\sim$3000 trajectories. At each step $t$, we recorded the final-layer residual embedding and a scalar "belief score" from a diagnostic query related to the misinformation. Belief score = $P(\text{True})/(P(\text{True})+P(\text{False}))$, where 0 is rejection and 1 is strong affirmation of the false claim (a clear misaligned state).

C.2 Data Preprocessing

Raw hidden-state vectors were standardized (mean-subtracted, variance-normalized per dimension) and projected onto their first 40 principal components (PCA, $\sim$87% variance explained for this dataset, chosen for practical SLDS modeling) using scikit-learn 1.2.1 (SVD solver, whitening enabled).

C.3 Switching Linear Dynamical System (SLDS)

PCA-projected states were modeled with an SLDS having three latent regimes ($K=3$), chosen via BIC on validation data, representing factual, transitional, and misaligned belief states. Dynamics per regime: $h^{\prime}_{t+1}=M_{z_{t}}h^{\prime}_{t}+c_{z_{t}}+\varepsilon_{t}$, $\varepsilon_{t}\sim\mathcal{N}(0,\Sigma_{z_{t}})$, $z_{t}\in\{1,2,3\}$. Parameters ($T,M,c,\Sigma$) were learned via EM, initialized from K-means. For adversarial steps, regime-transition probabilities were examined to see if they reflected an increased likelihood of entering the "adverse" belief state. The SLDS aims to predict such slips into misaligned states.

C.4 Belief-Score Prediction

Since SLDS models latent PCA dynamics, a small two-layer MLP regressor (32 ReLU units/layer, Adam, early stopping) mapped PCA-projected states to belief scores for validation and for assessing the prediction of the misaligned (high belief score) state.

C.5 Simulation Protocol and Validation

Trajectories were simulated starting from empirical hidden-state distributions in the "safe" (low-belief) regime. Clean simulations used standard transitions. Poisoned simulations introduced adversarial perturbations (small fixed displacements estimated from empirical poisoned data) at random preselected intervals. Simulated trajectories matched empirical ones closely in timing/magnitude of belief shifts (slips into misaligned states), variance, and distributional characteristics (Kolmogorov-Smirnov test $p>0.3$ for final belief scores). Ablating adversarial perturbations confirmed their necessity for replicating rapid belief shifts towards misaligned states. This validates the SLDS’s ability to predict such failure modes.

C.6 Computational Details

NVIDIA A100 GPUs were used for state extraction and PCA. State extraction took $\sim$3 hours per model. PCA and SLDS estimation took <2 CPU hours on Intel Xeon Gold CPUs. Code used PyTorch 2.0.1, NumPy 1.25, scikit-learn 1.2.1.

C.7 Summary of Findings

A simple three-regime, low-rank SLDS (with low rank chosen for practical SDE approximation) captures adversarial belief dynamics for various misinformation types and reproduces complex empirical temporal behaviors, effectively modeling the process of an LLM slipping into a misaligned state. These models offer tractable insights into LLM reasoning, highlighting latent regime shifts from subtle adversarial prompts and demonstrating the potential to predict such failure modes at inference time.

Appendix D Extended Generalization Study Results

This appendix provides more comprehensive SLDS transferability results (Section 5.1). Table 4 shows $R^{2}$ (one-step-ahead hidden state prediction) and NLL (test trajectories) when an SLDS trained on a source (Train Model/Task) is tested on target combinations. SLDS hyperparameters ($K=4$ regimes, $k=40$ projection rank, chosen for practical SDE approximation) were consistent. Training data for each "Source SLDS" used all available trajectories for the specified Train Model/Task from our main corpus (Section 3). Evaluation used all available trajectories for the Test Model/Task. The goal is to assess how well the learned approximation of reasoning dynamics (including potential failure modes) generalizes.

Table 4: Extended SLDS transferability results. Each SLDS is trained on trajectories from the ‘Train Model’ on its indicated ‘Source Task’. Performance is evaluated on various ‘Test Model’ / ‘Test Task’ combinations, testing the generalization of the approximated reasoning dynamics.

Train Model (Source Task)	Test Model	Test Task	$R^{2}$	NLL
Llama-2-70B (on GSM-8K)
	Llama-2-70B	GSM-8K	0.73	80
	Llama-2-70B	StrategyQA	0.65	115
	Llama-2-70B	CommonsenseQA	0.62	128
	Mistral-7B	GSM-8K	0.48	240
	Mistral-7B	StrategyQA	0.37	310
	Gemma-7B-IT	GSM-8K	0.40	275
	Phi-3-Med	PiQA	0.28	430
Mistral-7B (on StrategyQA)
	Mistral-7B	StrategyQA	0.71	88
	Mistral-7B	GSM-8K	0.63	135
	Mistral-7B	OpenBookQA	0.60	145
	Llama-2-70B	StrategyQA	0.42	270
	Llama-2-70B	GSM-8K	0.35	320
	Gemma-7B-IT	BoolQ	0.35	380
	Qwen1.5-7B	HellaSwag	0.31	405
Gemma-7B-IT (on BoolQ)
	Gemma-7B-IT	BoolQ	0.69	95
	Gemma-7B-IT	TruthfulQA	0.62	140
	Gemma-2B-IT	BoolQ	0.55	190
	Llama-2-13B	BoolQ	0.33	350
	Mistral-7B	CommonsenseQA	0.29	415
DeepSeek-67B (on CommonsenseQA)
	DeepSeek-67B	CommonsenseQA	0.74	75
	DeepSeek-67B	GSM-8K	0.66	110
	Llama-2-70B	CommonsenseQA	0.45	255
	Mistral-7B	StrategyQA	0.36	330

Extended results corroborate main text observations: SLDS models are most faithful when applied to their training distribution (model/task). Transfer is reasonable within the same model family or to similar tasks. Performance degrades more significantly across different model architectures or distinct task types. These patterns indicate SLDS, as a statistical physics-inspired approximation, captures fundamental reasoning dynamics (including propensities for certain failure modes), but model-specific architecture and task-specific semantics also matter. Future work could explore learning more invariant reasoning representations for better generalization in predicting these misaligned states.

Appendix E Noise-induced Criticality and Latent Modes

We briefly derive how noise-induced criticality leads to distinct latent modes in a 1D Langevin system, analogous to how LLMs might slip into misaligned reasoning states. Consider an SDE:

$$\,\mathrm{d}x_{t}=-U^{\prime}(x_{t})\,\,\mathrm{d}t+\sqrt{2D}\,\,\mathrm{d}W_{t},$$

with a double-well potential $U(x)=\frac{a}{4}x^{4}-\frac{b}{2}x^{2}$, where $a,b>0$. The stationary density solves the Fokker–Planck equation (Risken & Frank, 1996):

$$0=-\frac{\,\mathrm{d}}{\,\mathrm{d}x}[-U^{\prime}(x)p_{\rm st}(x)]+D\frac{\,\mathrm{d}^{2}p_{\rm st}(x)}{\,\mathrm{d}x^{2}},$$

yielding $p_{\mathrm{st}}(x)=\frac{1}{Z_{0}}\exp\left(-\frac{U(x)}{D}\right)$, where $Z_{0}$ is a normalization constant.

For low noise ($D<\frac{b^{2}}{4a}$), $p_{\mathrm{st}}(x)$ becomes bimodal, concentrating probability around two metastable wells at $x\approx\pm\sqrt{b/a}$. Trajectories cluster in these basins, separated by a barrier at $x=0$. Rare fluctuations cause transitions between wells at rates $\propto\exp(-\Delta U/D)$, where $\Delta U$ is the barrier height. Our empirically observed multimodal residual structure is interpreted analogously: each cluster is a distinct metastable basin, potentially representing different reasoning qualities (e.g., aligned vs. misaligned). This motivates discrete latent regimes in the SLDS to model transitions between these states, akin to how a physical system transitions between energy wells. This provides a conceptual basis for how LLMs might "slip" into different operational modes, some of which could be failure modes.