12–13 Nov 2026
HUN-REN Headquarters
Europe/Budapest timezone

Auditable LLM Pipelines for Text Measurement: Conceptual Frame Profiling

12 Nov 2026, 11:55
25m
HUN-REN Headquarters

HUN-REN Headquarters

1054 Budapest Alkotmány utca 29.

Speaker

Orsolya Putz

Description

Policy documents do not just report the economy, they describe it through metaphor. The minutes of the U.S. Federal Reserve’s rate-setting committee (FOMC) say that inflation is “climbing” or that commercial paper issuance “almost dried up”. In the first case, an abstract phenomenon, inflation (the target domain), is understood in terms of a concrete experience, vertical movement (the source domain). Together they form the conceptual metaphor INFLATION IS VERTICAL MOVEMENT. In the second, the financial market (target) is understood as a liquid (source): FINANCIAL MARKET IS A LIQUID. The set of metaphors a text relies on is a partial window on its conceptual frame, the system of concepts through which its authors understand the economy. Standard sentiment tools measure how negative a text is, but not which frames it uses. We present Conceptual Frame Profiling (CFP), an LLM-based pipeline that identifies metaphorical expressions and aggregates them into quantitative, document-level frame profiles, and we evaluate it as a measurement instrument.

CFP has three steps. (i) Gemini 2.5 Pro finds metaphorical expressions and classifies them under a fixed, schema-constrained codebook: 9 source domains (e.g., liquid, machine, vertical movement), 5 target domains (e.g., inflation, financial market), and 3 graded states (1 = under control, 2 = tension, 3 = loss of control). (ii) Because LLM output is stochastic, we repeat the extraction three times and keep only expressions confirmed across runs. (iii) The retained expressions are aggregated per document into a frame profile (the shares of the source domains) and a strain index, the mean state of the document’s expressions (range 1–3). It measures how strained the description is, not how strained the economy is.

We applied CFP to 257 FOMC minutes (1992–2026, ~1.76 million words). 76.3% of the identified expressions are reproduced by an independent run. Three annotators judged 84.5–92.8% of the expressions in fifty sampled passages correct, and agreed with each other on source domain at κ = 0.82–0.91 (chance-corrected agreement). A second, open-weights model (Gemma 4) reproduces the source domain classification (κ = 0.82) and the document-level profiles (r = 0.89), so the method does not depend on one proprietary model.

Compared with two standard sentiment measures (Loughran–McDonald dictionary and FinBERT), the strain index partly overlaps with sentiment (r = −0.66 and −0.52). The frame composition, however, is nearly independent of it: sentiment explains at most 6.4% of the variance of any source domain share, and crisis phases still differ from normal periods after controlling for sentiment. In an exploratory analysis, the profiles are concentrated in four source domains, stable over three decades, and shift at the boundaries of historically labelled financial-cycle phases (boom, panic, etc.) together with four financial stress indices.

The codebook, prompts, run outputs, and code are released. The approach is domain-agnostic and could be applied to other large collections of expert text, such as regulatory communication or financial disclosures.

Author

Presentation materials

There are no materials yet.