Nobody reads the fine print

ToS Buddy reads a website's terms of service and gives back four short sections, a list of potential dangers and a verdict. before we automated any of it, we saved 86 terms of service by hand. together they come to 691,975 words, and the summary that shipped gets 500 tokens. the model call in the middle is a single request. the work is on either side of it, in finding the right document on a site nobody has looked at and getting the same shape back every time.

We saved 86 by hand first

in january 2024 my senior project was a different idea. it was Scarlett, "an AI personal assistant" you talked to over text. a month later we started over on something everyone does without thinking: clicking "I agree".

the new README didn't have a name yet. it began: "[project_name] is a chrome extension that helps users understand and control their privacy on the internet."

the first dataset was made by hand. we went site by site and saved each terms of service as a text file in tos_docs/. over two days the commit messages read like a progress bar: "manual tos", "more sites", "36 more sites", "86 done".

the chart up top is that folder. the shortest file is IGN's at 775 words, the median is 6,240, and Indeed's is 61,422. the one we summarized first, Amazon's AWS Service Terms, is 39,657 words, about the length of a short novel.

doing it by hand also shows you where companies keep their terms. that turned into the spec for the crawler.

a crawler can't guess paths like these. it has to start where a person would, on the home page, and follow the link that says terms.

the same file sketched the database too: a websites table, a terms_of_service table, and a column called version_identifier. the idea that terms change and we should keep track was there in the first week.

A loose filter, then a strict check

i wrote the crawler. it landed on february 27, in a commit called "crawler can find the link, a little overengineered but it works". it's built on Crawlee and Playwright, which give it a request queue, retries and a real browser. each page gets 60 seconds.

finding the terms is two checks, and they're deliberately unequal.

the first one is loose. on the home page, queue every link whose text or address matches terms|tos|privacy. it's a regex over every anchor, so it catches more than it should, on purpose. missing the real link costs more than visiting a few extra pages.

the second one is strict. open each queued page and read its text. it only counts if it says "terms of service", "terms and conditions" or "user agreement". anything else is dropped.

to see it work, i replayed both checks on two live sites in october 2026, with the same regex, the same three phrases and the same filename rule as the code.

GitHub is the easy case. 143 links on the home page, 2 queued, both say the words. the privacy statement passes too, because it mentions the terms of service, so GitHub gets two files.

if nothing on a site passes, there's a fallback. it asks Google's Custom Search API for the site plus "terms of service", takes the top result, and runs the same strict check on that.

the file name comes from the domain and the page title: anything with "terms" in the title becomes tos, anything with "privacy" becomes privacy_policy. so github_tos.txt and spotify_privacy_policy.txt. the text goes to a storage bucket under that name, next to its summary.

crawling and summarizing a site, with up to a minute per page, takes longer than a serverless function wants to run. so in june the API moved into a Docker container on the Bun image.

Three prompts, one shape

the first summary a model wrote for us was of those 39,657 words of AWS terms. it was five numbered paragraphs and then a section called Potential Dangers.

every summary after it has a Potential Dangers section. that part stayed. the rest changed twice.

two weeks later the prompt got ambitious: nine sections, from privacy and data use to dispute resolution, each with a score from 1 to 100 for transparency, fairness and user-friendliness. the overall score was the average. a 64 doesn't tell you what you're agreeing to. the next prompt went back to sentences.

the prompt that shipped asks for four sections in plain English, a list of potential dangers and an overall assessment. my notes from that stretch say how it was made: Claude Opus to write the prompt, Claude 3 Haiku to run it on every site.

the prompt spells out the JSON it expects, key by key, with an instruction in each value. the site's page reads those keys back in the same order.

the dangers are a list, so the page can print one bullet per risk. the verdict comes last, and the prompt tells the model to base it on the summaries and the dangers it just wrote. so by the time it decides "favorable or unfavorable", the reasons are already on the page above it.

Haiku has 500 tokens for all six fields. that's roughly 370 words, short enough to read before you click agree. if Anthropic rate-limits us, the same prompt goes to GPT-4. big re-runs take a third path: the requests go into a JSONL file for OpenAI's Batch API on gpt-4o, with a 24-hour window, which is slower and stays out of the rate limits.

this is Front's page from may 2024, the first three fields as the model wrote them.

the prompt also asks for the JSON wrapped in triple backticks, and the server keeps whatever sits between the fences. because every summary has the same keys, one stored string can serve every part of the product.

One shape, three readers

two things read that string as soon as it's stored, the site and the Chrome extension. a third, the scripts that summarize sites again, comes in once the index has filled. the site came first. i started it as a fork: my notes from february say "take the frontend UI of analyzemyrepo.com", an open-source project by crowd.dev that grades GitHub repos.

it's the right shape for this. you paste in something and you get a report, and its report ends in a box called "Our overall assessment".

ours ends the same way. overallAssessment is the last key, and the page prints it last, under the dangers.

the name "Tos Buddy" shows up on march 4. Buddy got his face on demo day, april 9, the day of commits called "demo for today" and "right before the demo". he's a smiling teal square, and he's also the extension's icon.

each site's page is pre-rendered from the newest version of its terms and rebuilt in the background every five hours, so a visit never waits on a crawl or a model. anyone can add a site: paste a url, and the add-a-site page posts it to the same endpoint the crawler sits behind.

the extension came next. we rebuilt it at the end of april and finished it on june 7. it's Manifest V3 and asks for two permissions, activeTab and tabs. all it needs is the url of the tab you're on.

pressing Analyze Privacy sends that url through a background service worker to /api/summaryTos. the endpoint keeps only the protocol and the host, so a privacy notice and a pricing page on front.com both become https://front.com and get the same answer. that answer is a database lookup. every summary was written when its site was added, so the button never waits on a model.

the stored string still starts with json, the language tag from the fence the model wrapped it in. the site's page strips it with a regex; the popup drops the first five characters. then the popup loops over whatever keys are inside summary, so a fifth section added to the prompt would show up without a new build.

683 sites through the same door

an empty index isn't useful, so a script fills it. populate_initial_sites.txt has 683 lines, one url each, from google.com down.

for each url the script reads the page title and the meta description. titles are marketing copy, so it splits them on dashes, colons, pipes, .com and a few others, and keeps the shorter side. here's that function on four live titles:

logos come from svgl.app, an open library of brand SVGs. a separate script pulls the whole library into a CSV, and the populate script matches each site to its logo by url. the SVG is stored on the row itself, so a card never waits on a second request.

then the script posts each url to the same endpoint as the add-a-site page. the crawler, the summary and the database row happen the same way for 683 sites as for one. a cleanup script runs after: if a url ended up in the table twice, it keeps the row that has terms. then it drops sites that never got terms, so the index only lists sites with a summary.

Terms change, so summaries keep versions

the scripts are the third reader, and they're why the raw text was kept. terms of service change quietly. a summary from last year can be wrong in ways you can't see. in may i redid the schema, and the first week's version_identifier became real.

check_and_update_tos.py is written for a cron every sunday at midnight. it finds every summary older than 90 days and sends those sites back through the summarizer in one batch. if the new summary differs from the stored one, the row gets the new text and version goes up by one. if it's the same, nothing changes.

worked through for Front: its row is from may 28, so it would come due 90 days later, on august 26, and the next sunday run would pick it up.

update_tos_prompt.py covers the other kind of change, when we change and the terms don't. it downloads every saved terms file from the bucket, runs it through the new prompt and writes the result back. a new prompt costs one model call per site and no crawling.

What the model sees

the model only ever sees one document, and it fills in six fields it was told about in advance. everything else we built, from the regex to the version column, is there to make sure that document is the right one and still current.¹