A verified quote is not a supported claim

This site doesn't write articles directly from somebody else's article. It first extracts a set of facts. Every fact has a claim and a verbatim piece of evidence from the original source, and the system checks that evidence against the page it actually downloaded.

If the quoted evidence isn't really on the page, the fact doesn't get verified.

Sounds pretty solid, right?

I thought so too. I had 201 tests backing me up.

Then I turned another AI model loose on the code and specifically told it to be an asshole.

It invented a claim about a remote code execution vulnerability in Android 14. Complete bullshit.

Then it paired that claim with a perfectly legitimate sentence from a real Google page — except the sentence was about a purchase dialog on tablets and had absolutely nothing to do with a security vulnerability.

And my evidence check passed it.

Worse, the article scored 0.88 out of 1.00 and qualified for automatic publication.

Well, shit.

I was checking the wrong thing

The problem wasn't some obscure typo. The problem was how I'd designed the check.

The AI produces two things:

Claim: what it says the source tells us.

Evidence: the exact text it says supports that claim.

My code then goes back to the source and makes damn sure that evidence actually exists.

And it did! The sentence was 100% real.

It just didn't support the fucking claim.

I'd built a very good answer to:

"Is this quote actually on the page?"

Then treated that as though I'd answered:

"Does this quote support what I'm about to publish?"

Those are absolutely not the same question.

So I added another gate

I wanted the next check to be deterministic. No asking another model, "Hey, does this look right?" and trusting its opinion.

The claim and its evidence now get compared directly.

There are a few rules.

If the claim contains a specific version, date, number or other discrete value, that value needs to exist in the evidence itself. Finding it somewhere else on the page isn't good enough.

The claim is allowed to say less than its evidence. It isn't allowed to magically know more.

Certain words also have to be earned. If we're going to publish words like vulnerability, deprecated, deadline, suspended, or price, the evidence better actually support that language.

Finally, the meaningful words in the claim and evidence need enough overlap to show they're at least talking about the same damn thing.

That overlap threshold ended up at 0.40.

And no, I didn't pull 0.40 out of my ass.

I ran the new check against 385 real facts already in the site's database, looked at the score distribution, and manually inspected the twelve examples closest to the cutoff.

Eleven were legitimate paraphrases.

For example:

Klibs.io now hosts over 4,200 Kotlin Multiplatform projects

against:

Klibs.io Grows to 4,200+ KMP Projects

scores only 0.50. Expanding "KMP" into "Kotlin Multiplatform" actually hurts the lexical score even though the meaning is correct.

That's why simply cranking the threshold higher isn't necessarily safer. At some point you're mostly rejecting perfectly legitimate writing.

And it still isn't magic

This check doesn't prove that a claim is true.

It proves something narrower but useful: the claim and its evidence appear to be talking about the same thing.

There's still a known hole.

A claim like:

AAOS SDV uses Rust as the primary language for new components

can get through when the evidence only says:

for new components developed for AAOS SDV, we prioritized memory safety

Same subject. Enough shared language. But somebody slipped "uses Rust as the primary language" into the claim without the evidence actually saying that.

Lexical comparison can't reliably understand that distinction.

And instead of pretending otherwise, there's now a test specifically documenting that failure case.

That matters. Six months from now I don't want myself — or some AI working on this code — to look at the test suite and assume this check guarantees something it doesn't.

The part that pissed me off

All 201 tests passed while this hole existed.

And there was actually a good reason.

The mock AI provider used by the tests was written alongside the system itself. It had the same assumptions I had when I built the evidence gate.

Every fake "verified" fact in the tests came with evidence that obviously supported its claim.

Of course they passed.

I'd tested whether the system worked when everybody behaved correctly. I hadn't tested what happened when somebody handed it a real quote attached to a bullshit claim.

That's exactly what the adversarial audit did.

Then the second audit round found another variation: my scorer vetoed an article when zero facts were verified.

Sounds reasonable until you realize what that means.

Give it one legitimate fact and a pile of unverified bullshit?

No veto.

That version scored 0.87 and qualified for auto-publication too.

So that's fixed as well. Any unverified fact now stops automatic publication.

Then real data found even more stupid shit

This same week turned up three other bugs that none of the tests found.

My article extractor was basically deleting entire GitHub Blog pages. github.blog uses no-sidebar on its <body> element. My furniture-removal code saw the word sidebar and decided the entire fucking page was a sidebar.

105 KB of HTML went in. 76 characters came out.

Another bug crashed on purely numeric tokens of three or more characters. PHP silently converts numeric array keys into integers, and apparently my test corpus had conveniently managed to use numbers like 35 and 8 without ever stepping on that landmine.

And then there was the AI spending limit.

The hard ceiling wasn't actually hard.

PDO was binding floats as strings, and SQLite has some interesting opinions about comparing numbers and strings. The end result was that a comparison equivalent to:

1.15 <= '1.00'

could evaluate true.

So much for the fucking spending ceiling.

The lesson I'm taking from it

Tests are great. I still have a shitload of them.

But 201 green checkmarks don't mean you tested the assumption that was wrong.

The most useful bugs in this project have come from making the system deal with things I didn't design the happy path around:

real feeds, real HTML, weird data, hostile inputs, another model deliberately trying to make it publish bullshit, and a second implementation looking at the first implementation's assumptions.

That's why I'm doing adversarial audits on this site instead of just letting the model that wrote the code tell me its tests passed.

Because apparently:

"The quote is real."

and

"The claim is supported."

are two entirely different fucking tests.

Now I have both.