Skip to content
osbytes
← back to blog

Slop is a judgment problem, not a provenance problem

2026-07-18by@dillonstreator8 min read
#ai #llm #opinion #engineering

"AI-generated" and "AI slop" get used as synonyms. You see it in reply threads, in code review comments, in editorial policies that ban AI assistance outright: if a model touched it, it's slop. That equivalence felt roughly right in 2023, when most AI output in the wild really was filler. It holds up worse every year I use these tools, and the research is starting to say why.

Over the past year, researchers have started pulling the two terms apart. What's interesting is how consistently they land in the same place: the definitions of slop that survive scrutiny are about effort, judgment, and volume. Provenance doesn't appear in them.

The definition that's emerging

The most direct treatment is Why Slop Matters (Kommers et al., ACM AI Letters, 2026). Rather than a formal definition, the authors identify three prototypical properties of slop:

  1. Superficial competence. A veneer of quality over a deeper lack of substance.
  2. Asymmetry of effort. It took vastly less effort to produce than it would have without AI.
  3. Mass producibility. It exists inside an ecosystem of high-volume generation and consumption.

Notice what's absent. None of the three properties is "a model generated it." A carefully built AI-assisted essay, illustration, or pull request can fail all three tests: substance under the surface, real effort in the loop, produced as a single deliberate artifact rather than one of ten thousand. Under this definition it isn't slop, full stop.

The measurement work points the same direction. Measuring AI "Slop" in Text (Shaib et al., 2025) built a taxonomy from expert interviews and had annotators label AI text at the span level. Two findings stand out. Binary "is this slop?" judgments are somewhat subjective; raters disagree. But the judgments correlate with measurable quality dimensions, coherence and relevance chief among them. Slop, in their framing, is multidimensional and quality-based, not a provenance bit you flip when a model was involved.

Slop as an incentive problem

A response paper, Why AI Slop Matters, but Not Like That (Nishal, Sax, and Kieslich, ACM AI Letters, 2026), pushes the frame further: stop treating slop as an aesthetic category and look at why it gets produced. Their argument is that platforms actively stimulate slop production because volume drives engagement, and that the costs (attention, energy, concentration of distribution power) sit outside the producer's ledger.

Columbia's AI Slop and the Information Ecosystem report (Weedon, François, and Ponak et al., June 2026) fills in the mechanics. Generative AI collapsed the marginal cost of content production; distribution platforms collapsed the cost of reach. Their phrase for the result: "volume itself has become a defining force," overwhelming human attention and institutional filters. Much of the slop economy the report documents isn't even trying to deceive anyone. It's monetization: content optimized to grab attention and rank in search, produced because producing it costs nearly nothing.

That's a spam story, not an AI story. Word processors didn't create spam email, and cameras didn't create stock-photo clickbait, but each collapsed a production cost, and the flood followed the incentive. LLMs are the same shape at a much larger scale: they don't make anyone produce low-value content, they make it nearly free, and "nearly free" plus an engagement-optimized distribution channel is sufficient.

It's not just blog posts

The enterprise version of this is worth naming because it's the version engineers live inside. TechTarget's CIO-facing piece (January 2026) defines enterprise AI slop as low-quality, unverified AI-generated content flowing into corporate systems, and the examples are not marketing copy: AI-assisted code shipped without review, auto-generated docs nobody checked, knowledge-base entries that strip institutional context, synthetic data propagated without validation. One failure mode they describe is a loop: auto-drafted FAQ pages get ingested as retrieval sources, and the unreviewed output of one system becomes the ground truth of the next.

The operative clause in every example is without review. The defect isn't that a model wrote the first draft of the code. It's that no one with judgment stood between the draft and production.

The axis that actually predicts slop

Put the five sources side by side and a consistent contrast falls out:

AI-assisted work AI slop
AI is a tool inside a judgment loop AI replaces the judgment loop
A human reviews and owns the output Little or no meaningful review
Optimized for usefulness Optimized for volume and distribution
One deliberate artifact One of thousands from the same pipeline
Substance survives a close read Style is the whole payload

The predictive axis is effort and retained judgment, not provenance. Human-written SEO filler existed for two decades before ChatGPT and sits comfortably on the right-hand column. An expert's AI-assisted design doc, where the expert chose the problem, checked the claims, and cut what didn't hold, sits on the left.

That reframing also changes what the criticism should be. "An AI wrote this" is a statement about tooling. The complaint people are actually making, when the complaint is legitimate, is "the author didn't do the work": didn't verify the claims, didn't cut the filler, didn't care whether it was true. That's a real failure and it deserves the contempt it gets. It just isn't unique to AI, and treating provenance as a proxy for it misfires in both directions: it condemns careful work made with assistance and gives a pass to careless work made by hand.

What this looks like in practice

If the definitions above are right, then the useful question for any team producing content, code, or documentation with AI in the loop is not "did a model write this?" but "where does judgment enter the pipeline, and can it say no?" A few consequences follow directly.

Gate on review, not on tooling. A policy that bans AI assistance but accepts unreviewed human work filters on the wrong variable. So does its mirror image: a policy that celebrates AI adoption without asking who reads the output before it ships. The enforceable line is that someone with judgment and accountability stood between the draft and production, and would have caught the failure modes the TechTarget examples describe. That's the same line code review has always drawn; AI assistance doesn't move it.

Label provenance honestly, then let the work be judged on its merits. Hiding AI involvement concedes that provenance is the axis. It isn't, and disclosure is what makes that argument in good faith. The inverse holds too: a label is not a substitute for review. "AI-generated, human-reviewed" only means something if the review was real.

Keep the option to ship nothing. Slop economics run on filling the slot because filling it is free. A weekly newsletter that goes out regardless of whether anything happened, a docs pipeline that generates a page for every endpoint whether or not there's anything to say, a bot that comments on every PR — these drift toward slop structurally, whatever model quality does, because volume is the point and judgment has no veto. A pipeline that can conclude "nothing cleared the bar" is resistant to slop in a way no style guide is.

Make effort visible where it matters. Asymmetry of effort is one of the defining properties, and readers calibrate trust partly by inferring how much work went in. Verified claims with linked sources, a repro the author actually ran, a stated position someone can disagree with: these are hard to fake at volume, which is exactly what makes them useful signals. Work that carries them earns a different reading than work that couldn't.

Sources