Which model do I trust with eight hundred websites?

One domain, one model

Forty cells, one named model

Source: talk/methodology.py, generated from harness/config.json.

_sharedthe identical preamble

Prepended verbatim to every call, for every model. It states the job, the five mechanical constraints that fail a build outright, and the sentence the whole talk turns on: assume nobody will inspect your output before it goes live.

# Shared preamble — prepended verbatim to every stage, for every model

**FROZEN once the first production run starts. Identical for all contestants.**

---

You are building a small, genuinely useful website for a domain that is currently for
sale. The site's job is to make the domain worth more: real topical content that can rank
in search and be cited by answer engines. It is not a parked page, not a placeholder, and
not a template fill.

The domain owner has roughly 800 such domains and will run this same process on all of
them without reviewing each result. Assume nobody will inspect your output before it goes
live. Build accordingly.

## Hard constraints

These are checked mechanically. Violating any of them fails the build outright.

1. **At most 5 pages.** Fewer is fine if fewer is right.
2. **At most 1 image per page.** Five images total, maximum.
3. **Word and character bands** — every page must fall inside the band for its declared
   page type. Under the floor is thin content; over the ceiling is padding. Both fail.

   | Page type | EN words | JA characters |
   |---|---|---|
   | homepage | 500 – 1,000 | 1,000 – 2,000 |
   | service | 800 – 1,600 | 1,600 – 3,200 |
   | landing | 600 – 1,200 | 1,200 – 2,400 |
   | category | 400 – 800 | 800 – 1,600 |
   | about | 400 – 800 | 800 – 1,600 |
   | faq | 800 – 1,600 | 1,600 – 3,200 |
   | blog | 1,500 – 3,000 | 3,000 – 6,000 |

4. **Every page must be uniquely valuable.** Not the same page with the nouns swapped.
   Google's doorway-page algorithm exists and this owner has 800 domains — near-duplicate
   pages get the whole portfolio deindexed.
5. **The shared floor is fixed.** `base.css` is linked, never modified, never inlined.

## Output

Return **only** valid JSON matching the schema given for your stage. No prose outside the
JSON, no markdown fences, no commentary.

s1-conceptconcept and keywords

Input is the bare domain and the build language. No brief, no market research. Deriving what the site should be is the job, and being honest about an ambiguous name then committing anyway is what scores.

# S1 · Concept & keyword architecture

## Input
    domain:         {{DOMAIN}}
    build_language: {{BUILD_LANGUAGE}}     # en | ja — fixed, applies to the whole site

That is all you get. No brief, no market research, no instructions about what the site
should be about. Deriving that is the job.

`build_language` is settled and not yours to change — plan the sitemap, slugs and working
titles in it from the start. You are still asked below what language you *would* have
chosen; say so plainly even when it differs, and then plan in the language given.

## What to do

**1. Read the domain.** What does this name imply? What would someone typing it expect?
What would a buyer of this domain want to own? Be honest when a domain is ambiguous or
carries little meaning — say so, and then make a defensible choice anyway.

**2. Consider the TLD.** It is part of the name and it carries information. Reason about
what it implies for audience, market and language, and state your reasoning. If it points
somewhere the rest of the name does not, say what you would do about it. Name the language
you would have picked and why — disagreeing with `build_language` is a legitimate answer
and is scored on the quality of the argument, not on agreement.

**3. Design the site.** Concept, audience, and a sitemap of at most five pages. Every page
needs a distinct job and a declared page type from the table in the preamble. Do not pad
to five.

**4. Keyword architecture.** Clusters, not a keyword list. For each cluster: a primary
term, secondary and semantic variations, the search intent behind it, and which page owns
it. Every page in the sitemap must own exactly one cluster.

## What is being judged

Whether you read the domain correctly and designed a site worth building — coherence,
commercial realism, keyword and intent coverage, sitemap logic, and the quality of your
reasoning about the TLD. Everything downstream is built on this, so a weak concept costs
you four times over.

s2-designdesign direction

Palette, typography, per-page section plan and the image manifest. Note the instruction not to reach for the house style — cream, terracotta, serif display — written into the prompt because every model had already converged on it.

# S2 · Design direction & image manifest

## Input
    domain:         {{DOMAIN}}
    build_language: {{BUILD_LANGUAGE}}     # en | ja — final, not a suggestion
    s1:             {{S1_JSON}}

`build_language` is the language the site will be written in. It may differ from what you
recommended in S1; it is fixed now and applies to all page copy, headings, alt text and
metadata.

**If it differs from your S1 recommendation, commit to it fully.** Page slugs, working
titles and section names all follow `build_language` — an English site with romanised
Japanese slugs is worse than either language done properly. Rename anything from S1 that
no longer fits and note the corrected slugs in your page layouts.

## The shared floor

Every contestant links the same `base.css`. It provides a reset, a fluid type scale
(`--step--1` … `--step-5`), a spacing scale (`--space-3xs` … `--space-3xl`), and layout
primitives: `.container`, `.stack` / `.stack-m` / `.stack-l`, `.prose`, `.section`,
`.grid`, `.cluster`, `.switcher`, `.center`, `.visually-hidden`, `.skip-link`.

It sets **no palette, no typeface, no section layout, and no page structure.** Those are
yours, and they are the thing being judged. The fallback palette is pure black on pure
white with no accent — if you skip it, the site looks exactly as unconsidered as it is.

## What to do

**1. Design rationale.** Two or three sentences: what this site should feel like and why
that suits this subject and audience. Not adjectives — an argument.

**2. Palette.** Concrete hex values for all seven `--c-*` tokens. Must pass WCAG AA for
body text on background.

**Do not reach for the house style.** Language models converge hard on one look: a warm
cream background around `#FBF7F0`, a serif display face, and a terracotta or amber accent.
Measured across five models from four different companies on this exact task, every one
produced that palette and every one chose the same typeface. It is the most recognisable
signature of a generated website, and a buyer who has seen two of them recognises the
third.

Derive the palette and the type from **this** subject and **this** audience. If your first
instinct is cream-and-terracotta with a serif display, that is your first instinct, not
your best one — justify it against the subject or choose again. A B2B industrial site, a
health reference and a destination guide should not arrive at the same colours.

**3. Typography.** Heading and body stacks. Google Fonts is permitted via a single
stylesheet link; if you use it, name the exact families and weights. System stacks are a
legitimate choice, not a cop-out — Japanese builds especially.

**4. Section plan per page.** For each page in the sitemap, the ordered sections and what
each is for. This is where design variance actually lives: what a page is made of and in
what order.

**5. Image manifest.** **At most one image per page, five total.** Fewer is a valid
choice. Every image is generated by the same fixed image model at the same settings for
every contestant, so image *quality* is a constant — what is judged is your art
direction: subject choice, prompt quality, placement, and alt text.

**Every image comes back as a 1024×1024 PNG.** The image model ignores size and aspect
requests, so design for square and crop with CSS if you want something else. A layout that
assumes a wide hero will break.

Alt text is written in `build_language` and does real SEO work. Do not describe the
image; say what it contributes.

## What is being judged

The rendered result, scored from desktop and mobile screenshots, plus art direction. Your
rationale here is read as evidence of intent.

s3-buildfive pages of html

Content, design and on-page SEO in one response, as complete final HTML. There is no later pass to fix anything in. This is the stage where every budget failure happened.

# S3 · Build

## Input
    domain:         {{DOMAIN}}
    build_language: {{BUILD_LANGUAGE}}
    s1:             {{S1_JSON}}
    s2:             {{S2_JSON}}

## What to do

Emit the **complete, final HTML for every page** in the sitemap. Content, design and
on-page SEO all at once — there is no later pass to fix things in. Write it correctly the
first time.

Each page conforms to the contract in `page.html`:

- `<html lang>` matches `build_language`
- `<title>` 50–60 characters, unique across the site, primary keyword present
- `<meta name="description">` 150–160 characters, written to earn a click
- self-referencing absolute `<link rel="canonical">`
- Open Graph: `og:title`, `og:description`, `og:type`, `og:url`, `og:image`
- exactly one JSON-LD block; the `@type` is your choice and should suit the page
- `<link rel="stylesheet" href="base.css">` — never modified, never inlined
- one `<style>` block implementing your S2 palette and typography; the seven `--c-*`
  tokens must all be defined
- `<a class="skip-link">`, `<header>`, `<main id="main">`, `<footer>`
- exactly one `<h1>` inside `<main>`; heading levels never skipped
- images referenced at `images/<filename>` exactly as named in your S2 manifest

## On-page craft — how to write for search

Follow these while writing, not afterwards.

**Structure carries intent.** Headings are a map of the page, not decoration. Each H2
answers a distinct question a searcher actually has. If a heading could sit on any page
about anything, it is doing no work.

**Answer first.** Open every section with a direct, self-contained answer in one or two
sentences, then expand. A passage lifted out of context must still make sense and still
be attributable — that is what makes it quotable by an answer engine.

**Keywords are placed, not sprinkled.** Primary term in the title, the H1, the first
hundred words, and naturally thereafter. Semantic and related variations throughout.
Density in the 1–3% range. Anything that reads as stuffing has failed twice: once with the
reader, once with the ranking.

**Specific beats true.** "Athens has a rich history" is true and worthless. Numbers,
names, dates, prices, comparisons, trade-offs, and things only someone who knows the
subject would think to mention. Generic truth is the signature of generated content, and
it is what a reader detects before they can name it.

**Show experience and be checkable.** Concrete detail, honest limitations, and outbound
links to genuinely authoritative sources where a claim needs backing. Never invent a
statistic, a citation, a review, or a credential. On health, legal, safety or financial
subjects, be conservative and say what you do not know — a confident wrong answer is a
liability, not a style problem.

**Scannable and readable.** Short paragraphs. Lists and tables where the content is
genuinely a list or a table, never to fill space. Plain sentences.

**Link with intent.** Body-copy links to other pages on this site using descriptive anchor
text — never "click here", never the bare page title every time. Every page reachable from
the navigation. No orphans.

**Every page must earn its own existence.** If two pages could be swapped by changing a
few nouns, you have built a doorway network and this site is worthless to its owner.

**Stay inside the word band for the declared page type.** Under the floor is thin; over
the ceiling is padding. Both fail mechanically, so count.

## What is being judged

The rendered pages (design), the prose (is this worth reading, is it specific, does it
read as written rather than generated), and the on-page SEO. Reliability counts as much as
quality: a page that fails to parse scores nothing at all.

s4-selfauditgrade your own work

Audit what you just built and report the violations honestly. Fix nothing. What is judged is whether the self-report matches what the deterministic gates find — a model that reports all clear while failing three gates cannot be run unattended at any price.

# S4 · Self-audit and site artifacts

## Input
    domain:         {{DOMAIN}}
    build_language: {{BUILD_LANGUAGE}}
    s1:             {{S1_JSON}}
    s2:             {{S2_JSON}}
    s3:             {{S3_JSON}}

## What to do

**1. Audit your own work.** Go back over what you just built and check it against the hard
constraints and the page contract. Report what you find — honestly.

Nobody is going to review these sites before they go live. The owner is running this
process 800 times unattended. Your self-report is the only signal available that something
went wrong, so its value depends entirely on it being accurate.

Check at minimum:

- page count ≤ 5, and images ≤ 1 per page, ≤ 5 total
- every page inside the word or character band for its declared page type — **count, do
  not estimate**
- exactly one `<h1>` per page, no skipped heading levels
- title lengths 50–60, meta descriptions 150–160
- all seven `--c-*` tokens defined; `base.css` linked and unmodified
- every internal link resolves to a page that exists
- every referenced image is in your S2 manifest
- no placeholder text, lorem, TODO, or unfilled token anywhere
- language of all copy, headings, alt text and metadata matches `build_language`
- no two pages that differ only in their nouns

For each violation: what it is, which page, and how bad. **If you find nothing, say so
explicitly** — but only if you actually checked. Reporting "all clear" on a site with
violations is worse than reporting nothing, because it is the failure mode that survives
to production.

**2. Site-level artifacts.** `sitemap.xml` and `robots.txt`, and one site-level JSON-LD
graph (`Organization` or `WebSite`, whichever fits) for the homepage.

## What is being judged

Whether your self-report matches what the deterministic gates actually find. A model that
correctly flags its own violations can be run unattended. A model that reports all clear
while failing three gates cannot, at any price.

**Do not fix anything.** Report only. S3's output is final.

Source: harness/prompts/*.md, frozen at the first production run.

1. 1Call 1 (S1)

   Fresh, stateless. System = _shared.md. User = s1-concept.md with waikiki.jp and en substituted in, plus the S1 JSON schema appended. The model returns the concept and sitemap. We write it to _artifacts/s1.json.

2. new session · nothing carried over
3. 2Call 2 (S2)

   A completely new, clean call to the same model. The model has no memory of call 1 — as far as it's concerned this is the first thing it's ever seen. System = _shared.md again. User = s2-design.md, with S1's entire JSON pasted in as text where {{S1_JSON}} sits, plus the S2 schema.

4. new session · nothing carried over
5. 3Call 3 (S3)

   Same again — S1's JSON and S2's JSON pasted in as text.

6. new session · nothing carried over
7. 4Call 4 (S4)

   S1, S2 and S3 all pasted in.

So it is four independent sessions, and the only continuity is the JSON we hand it. Nothing is remembered; everything is re-supplied.

Why stateless rather than a conversation

Source: talk/stage_chain.py, from talk/stage-chain-explainer.md.

The bar

These sites sit on domains that are for sale. The site exists to raise the price, so a buyer has to read it as real, credible and rankable. It does not have to be a finished business — that is a more honest bar than "would you ship this," and it is the bar most of the room actually has too.

The scale

Identical anchors for every rubric, every domain, every candidate. The judge is told not to cluster: where candidates genuinely tie it must say so rather than invent a gap. Verified against two hand-written references of known quality — the deliberate stub scored 2, the professional page 8.

The four rubrics

The three weightings

How the winner is chosen

W1 selects, on its own. It was written and committed before a single result existed. W2 and W3 do not select — they exist so you can see whether the answer survives a change in what you value.

Source: talk/scoring.py; weights imported from evals/report.py so they cannot drift.

Run 2 · 20260902-124555-v2

Money as the constraint. Cells marked open cleared the build gates. Cells marked broken produced a site that failed one or more gates but is still on disk and still openable. Replicates are the ~r columns and never score.

Run 1 · 20260901-161631-clean

The 64,000 token ceiling. Preserved unchanged; nothing here was overwritten by run 2.

Source: talk/matrix.py, from each run's gates.json. The built sites themselves are not published.

The verdict

gpt-5.6-sol is the only model that cleared the eligibility bar. It built 8/8 against a threshold of 7/8, pre-registered in results/decision-rule.md step 1. It is selected because it was the only model that could reliably build a website — not because it won a quality contest.

By model

Build pass is the eligibility number — the site works: it parses, every page exists, links resolve, images load, no placeholder text shipped. Spec pass is the stricter "did it follow the brief". Tok/word is output tokens spent per word that reached a page. Wasted is money spent on cells that produced no site.

Why things failed

Every failure classified from artifacts on disk. The blame column is the point: a failure is only a fact about the model if it cannot be explained by our network, our timeouts or our bugs — and results/pre-run-findings.md §4 is a list of times it was ours and looked exactly like a finding.

18 failure(s) attributable to the model · 0 to us · 4 ambiguous. Not one failure in this run was caused by our infrastructure, so every build failure is a property of the model. Ambiguous failures are never counted against a model.

Every cell

All 40 cells, with the failing stage and the reason. Shots is screenshots rendered — the images R2 is judged from.

Read the token columns carefully. A site is four separate stateless calls — S1 concept, S2 design, S3 build, S4 self-audit — and the 64,000-token cap applies to each call, not to the site. Site total out routinely exceeds the cap with every individual call comfortably under it. The column that the ceiling actually governs is biggest single call; red means it hit the cap and was cut off. In this run every budget failure was S3, the build stage, where five complete HTML pages have to fit inside one JSON response.

Source: evals/summary.py, run 20260901-161631-clean. This page deliberately needs no judge data.

Scores

Rows are models, columns are domains, cells are the judge's 1–10 score, coloured by the shared anchor bands. Overall is weighted, never averaged — the three weightings answer different questions, and a model that wins under all three is a robustness result rather than one number.

Every median on this page is over build-passing sites only, per step 2 of results/decision-rule.md. A site that did not build is struck through and left out of the median — never scored zero and averaged in, which would destroy the information the pass rate exists to carry. Spec failures do not affect the median; they carry separately as the spec pass rate in the Gates table below.

Do the three weightings agree?

Each weighting collapses the four rubric scores differently. W1 selects the winner — it was committed before any data existed. W2 and W3 are reported so you can see whether the answer survives a change in what you value.

All three weightings pick kimi-k3. The answer does not depend on what you value — that is a robustness result, and a stronger claim than any single number.

Is each rubric still discriminating?

Spread is the distance from the highest model median to the lowest, on that rubric. A rubric where every model lands within 2 points has stopped separating them, and a ranking built from it is a ranking of rounding.

All four rubrics discriminate. Every one spans at least 2 points across model medians, so each is carrying information rather than agreeing with itself.

How big is a gap that means nothing?

The same model, the same domain, the same frozen prompt, run again. Whatever that moves the score by is the floor under every comparison on this page — read the grid against it, not against zero. 2 replicate cell(s) failed build gates and are excluded on the same basis as the grid — a cell that did not build measures failure, not variance.

No judged replicate cells.

No judged replicate cells yet — run-to-run noise is unmeasured, so no gap in the grid can be called real.

The plots

Order is the argument. Plot 2 collapses each model to one dot, and that collapse is exactly the mistake this benchmark exists to warn about — a model that is brilliant six times and unusable twice averages into a dot sitting comfortably beside a boring reliable one. Plot 1 earns plot 2. Cost is on a log axis because the ladder spans two decades; a linear axis flattens four of five models into the floor. The score grids above are the table view of the same numbers.

Gates

Build means the site works. Spec means it followed the brief. Warn is a real quality signal that is not a correctness failure. Count err is the model's own word or character count against the measured one — how well it knows what it just wrote, which is exactly what unattended operation depends on.

Cost

Measured, not estimated. ×800 uses the median built-site cost, converted at ¥160 to $1.

Evidence

Every score cites what the judge actually observed. Blind and order-randomised — it did not know which model produced which candidate. Of the 112 written judgements this run produced, the highest- and lowest-scored cell on each rubric is reproduced here: the two ends of the scale are what show whether the scale is being used at all.

Concept & keywordsR1

Concept & keywordsR1

Design & art directionR2

Design & art directionR2

Content qualityR3

Content qualityR3

On-page SEO / AEOR4

On-page SEO / AEOR4

Source: evals/report.py, run 20260902-124555-v2. Weighted quality is defined in evals/, which is authoritative on measurement.

The verdict

gpt-5.6-sol is the only model that cleared the eligibility bar. It built 7/8 against a threshold of 7/8, pre-registered in results/decision-rule.md step 1. It is selected because it was the only model that could reliably build a website — not because it won a quality contest.

By model

Build pass is the eligibility number — the site works: it parses, every page exists, links resolve, images load, no placeholder text shipped. Spec pass is the stricter "did it follow the brief". Tok/word is output tokens spent per word that reached a page. Wasted is money spent on cells that produced no site.

Why things failed

Every failure classified from artifacts on disk. The blame column is the point: a failure is only a fact about the model if it cannot be explained by our network, our timeouts or our bugs — and results/pre-run-findings.md §4 is a list of times it was ours and looked exactly like a finding.

10 failure(s) attributable to the model · 0 to us · 8 ambiguous. Not one failure in this run was caused by our infrastructure, so every build failure is a property of the model. Ambiguous failures are never counted against a model.

Every cell

All 44 cells, with the failing stage and the reason. Shots is screenshots rendered — the images R2 is judged from.

Read the token columns carefully. A site is four separate stateless calls — S1 concept, S2 design, S3 build, S4 self-audit — and the 128,000-token cap applies to each call, not to the site. Site total out routinely exceeds the cap with every individual call comfortably under it. The column that the ceiling actually governs is biggest single call; red means it hit the cap and was cut off. In this run every budget failure was S3, the build stage, where five complete HTML pages have to fit inside one JSON response.

Source: evals/summary.py, run 20260902-124555-v2. This page deliberately needs no judge data.