- The paper introduces the LM Tree, which combines multi-armed bandit price exploration with LLM-generated text features to discover content segments and validate prices without requiring a predefined feature matrix.
- On 8,939 HardwareLuxx articles, the LM Tree generated $264 in held-out revenue—65% more than a single price, 47% more than format pricing, and 40% more than the publisher’s eight-segment taxonomy.
- The results suggest text-based pay-per-crawl pricing can uncover valuable cross-cutting segments, but the evidence relies on traffic-calibrated willingness-to-pay from one publisher and requires validation with live crawler transactions.
Motivation and the mechanism selection problem
The transition from search-mediated to AI-mediated content consumption undermines the advertising- and traffic-based revenue model of online publishing. When AI systems crawl content for training or retrieval-augmented generation without directing users to the source, publishers require a direct charging mechanism. Pay-per-crawl (PPC) infrastructure—Cloudflare's bot management layer, Tollbit, the HTTP 402 status code—now exists; what remains unsolved is pricing.
Archer, Ghili, and Haghpanah frame this as a problem of mechanism selection at scale. Two features of willingness-to-pay (WTP) make it hard. First, pricing-relevant features are unstructured: an article's value to a crawler depends on attributes embedded in its prose—topic specificity, data richness, timeliness—not on metadata columns. Second, the relevant attributes differ across content sub-types and even across publisher types: recency governs value for financial news, jurisdiction for legal databases, product tier for technology reviews. A publisher must therefore discover which segments warrant distinct prices, which textual features define them, and what prices they support—all from binary purchase feedback alone. The number of hand-written rules grows combinatorially with taxonomy depth, making manual design infeasible.
The LM Tree algorithm
The LM Tree is an adaptive pricing agent that alternates three operations at each node of a segmentation tree:
- Price exploration: a multi-armed bandit over log-spaced price arms anchored to the parent node's optimal price. Log-scale spacing is essential because optimal prices may differ by orders of magnitude across content types (fractions of a cent for commodity news versus dollars for flagship reviews). Exploration trials serve a dual role: they identify pn∗ and simultaneously generate the price variation that produces contrast sets.
- Feature discovery: items that purchased at top-half price arms form set Hn (revealed high WTP); items purchasing only at bottom-half arms form Ln. The LLM Analyst reads text from both sets and proposes candidate discriminating attributes. This is feature construction rather than feature selection—the step with no analogue in existing tree methods.
- Split validation and annotation: the LLM Annotator applies discovered attributes to all node items; splits are retained only if child nodes converge to different optimal prices. Existence rules (attribute present/absent) are preferred over threshold rules because presence/absence is robust across incommensurable metric scales.
Two properties echo honest estimation in causal trees: exploration data used for splitting is distinct from fresh exploration used to estimate child prices, preventing overfitting; and at inference time no LLM calls are needed—routing reduces to attribute lookups on pre-computed annotations.
Data and WTP calibration
Evaluation uses 8,939 HardwareLuxx articles (two observable format categories: artikel long-form reviews, news), split into 7,210 training and 1,729 test articles, with 80,451 synthetic buyer queries (nine per article). Crawler logs from GPTBot, ClaudeBot, PetalBot, OAI-SearchBot, and PerplexityBot show heavy concentration: the top 10% of articles account for 42% of requests.
WTP is calibrated as v(i)=0.004×crawler views(i), yielding a median article WTP of $0.02. This calibration rests on two arguments. The directional argument—that crawlers allocate budget toward valuable content—is plausible but not verifiable. The structural argument is more careful: crawler traffic is partially correlated with categories but leaves large within-category variation that is recoverable from text (flagship GPU benchmarks are identifiable as such). The authors argue that this structure makes the test environment valid for evaluating whether the LM Tree adds value over category-level pricing, regardless of whether traffic tracks WTP levels precisely. This is a reasonable defense, but the evaluation ultimately tests performance against a proxy rather than realized transaction revenue—a limitation inherent to a market that does not yet exist at scale.
Notably, crawler heterogeneity implies buyer-type variation: OAI-SearchBot directs 73% of requests to artikel, while GPTBot directs 85% to news, meaning joint second-degree (content-based) and third-degree (buyer-identity-based) discrimination is feasible but left to future work.
Results
On the held-out test set, learned policies yield:
| Strategy |
Test revenue |
vs. single price |
| Single Price |
$160 |
— |
| Format category (2 segments) |
$179 |
+12% |
| Editorial taxonomy (8 segments) |
$189 |
+18% |
| LM Tree |
$264 |
+65% |
Three findings stand out. First, the LM Tree's 47% gain over format pricing and 40% gain over the publisher's own eight-segment editorial taxonomy are achieved despite the agent never observing the finer taxonomy—it is initialized with only the two format categories and discovers everything else from text and binary outcomes. Second, the learned splits cut across editorial boundaries: within artikel, the existence rule "high-end GPU mentioned" (e.g., RTX 30-series specifications) prices GPU-review articles at $0.148 versus $0.081 for other reviews; within news, a threshold rule ("market value ≥ $1,000") separates coverage of high-end equipment from commodity announcements. No editorial category maps cleanly onto either leaf—for example, *news/allgemein* has the highest share assigned to the news high-value leaf (16.7%), ahead of dedicated hardware news (11.1%). Third, all four news editorial sub-categories converge to identical prices under editorial pricing ($0.028), confirming that within-news variation is article-level and recoverable only from text—the regime where the LM Tree's advantage is largest.
The train–test revenue drop is proportional across strategies and attributable to online exploration cost rather than overfitting.
Position in the literature
The paper situates itself at the intersection of several literatures. From bandit pricing (Kleinberg and Leighton; Besbes and Zeevi; Misra et al.), it borrows per-node price learning but claims its contribution lies in deciding which bandit problems to run. From market segmentation trees (Aouad et al.) and causal trees (Athey and Imbens), it inherits the recursive-partitioning skeleton and economic validation criteria. Its genuine departure is upstream: prior tree methods search a fixed feature matrix, whereas the LM Tree generates a node-specific feature space via LLM reasoning over prose. Economically, the agent performs second-degree price discrimination (Mussa–Rosen; Maskin–Riley; Bergemann et al.; Haghpanah and Siegel) while relaxing those frameworks' assumption that the seller knows which product dimensions matter. The discussion also compares PPC against bulk licensing (which aggregates away item-level value and excludes small publishers) and auctions (inapplicable absent slot scarcity), concluding that only PPC scales with heterogeneous content value.
Limitations and open questions
Several limitations deserve explicit acknowledgment. The most consequential is the WTP calibration: because no live PPC transactions exist, results are measured against a traffic-derived proxy whose level accuracy cannot be verified, however sound its structure. Second, the evaluation covers one publisher in one domain (German consumer technology); whether the LLM Analyst reliably discovers pricing-relevant attributes in domains where value hinges on subtler signals—recency, jurisdiction, methodology quality—remains untested. Third, the tree terminates at shallow depth (four leaves in practice), leaving open how deeply such trees can profitably recurse before exploration costs and LLM annotation noise dominate. Fourth, buyer-type conditioning is deferred: the framework extends naturally to segmenting over (content × crawler identity), but the joint problem is unsolved here. Fifth, strategic responses are not modeled—crawlers facing posted prices could shift crawling behavior, invalidating the assumption that current traffic patterns reflect valuations under payment. Finally, generalization beyond PPC—to API access, data licensing, professional services—is argued structurally rather than demonstrated empirically.
Conclusion
The paper contributes a pricing agent that replaces feature selection with feature construction, enabling tree-based price discrimination when the feature space does not exist and must be induced from text. On HardwareLuxx data, it delivers a 65% revenue gain over static pricing and outperforms the publisher's own taxonomy by 40%, recovering cross-cutting segments (high-end component coverage) that formal editorial categories miss. The empirical evidence is strong within its setting but rests on calibrated rather than observed demand; validating these gains against live pay-per-crawl transactions, extending segmentation to buyer identity, and testing deeper recursion remain open questions.