#ross anderson | Explore Tumblr Posts and Blogs

mostlysignssomeportents · 2 months

Text

The Coprophagic AI crisis

I'm on tour with my new, nationally bestselling novel The Bezzle! Catch me in TORONTO on Mar 22, then with LAURA POITRAS in NYC on Mar 24, then Anaheim, and more!

A key requirement for being a science fiction writer without losing your mind is the ability to distinguish between science fiction (futuristic thought experiments) and predictions. SF writers who lack this trait come to fancy themselves fortune-tellers who SEE! THE! FUTURE!

The thing is, sf writers cheat. We palm cards in order to set up pulp adventure stories that let us indulge our thought experiments. These palmed cards – say, faster-than-light drives or time-machines – are narrative devices, not scientifically grounded proposals.

Historically, the fact that some people – both writers and readers – couldn't tell the difference wasn't all that important, because people who fell prey to the sf-as-prophecy delusion didn't have the power to re-orient our society around their mistaken beliefs. But with the rise and rise of sf-obsessed tech billionaires who keep trying to invent the torment nexus, sf writers are starting to be more vocal about distinguishing between our made-up funny stories and predictions (AKA "cyberpunk is a warning, not a suggestion"):

https://www.antipope.org/charlie/blog-static/2023/11/dont-create-the-torment-nexus.html

In that spirit, I'd like to point to how one of sf's most frequently palmed cards has become a commonplace of the AI crowd. That sleight of hand is: "add enough compute and the computer will wake up." This is a shopworn cliche of sf, the idea that once a computer matches the human brain for "complexity" or "power" (or some other simple-seeming but profoundly nebulous metric), the computer will become conscious. Think of "Mike" in Heinlein's *The Moon Is a Harsh Mistress":

https://en.wikipedia.org/wiki/The_Moon_Is_a_Harsh_Mistress#Plot

For people inflating the current AI hype bubble, this idea that making the AI "more powerful" will correct its defects is key. Whenever an AI "hallucinates" in a way that seems to disqualify it from the high-value applications that justify the torrent of investment in the field, boosters say, "Sure, the AI isn't good enough…yet. But once we shovel an order of magnitude more training data into the hopper, we'll solve that, because (as everyone knows) making the computer 'more powerful' solves the AI problem":

https://locusmag.com/2023/12/commentary-cory-doctorow-what-kind-of-bubble-is-ai/

As the lawyers say, this "cites facts not in evidence." But let's stipulate that it's true for a moment. If all we need to make the AI better is more training data, is that something we can count on? Consider the problem of "botshit," Andre Spicer and co's very useful coinage describing "inaccurate or fabricated content" shat out at scale by AIs:

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4678265

"Botshit" was coined last December, but the internet is already drowning in it. Desperate people, confronted with an economy modeled on a high-speed game of musical chairs in which the opportunities for a decent livelihood grow ever scarcer, are being scammed into generating mountains of botshit in the hopes of securing the elusive "passive income":

https://pluralistic.net/2024/01/15/passive-income-brainworms/#four-hour-work-week

Botshit can be produced at a scale and velocity that beggars the imagination. Consider that Amazon has had to cap the number of self-published "books" an author can submit to a mere three books per day:

https://www.theguardian.com/books/2023/sep/20/amazon-restricts-authors-from-self-publishing-more-than-three-books-a-day-after-ai-concerns

As the web becomes an anaerobic lagoon for botshit, the quantum of human-generated "content" in any internet core sample is dwindling to homeopathic levels. Even sources considered to be nominally high-quality, from Cnet articles to legal briefs, are contaminated with botshit:

https://theconversation.com/ai-is-creating-fake-legal-cases-and-making-its-way-into-real-courtrooms-with-disastrous-results-225080

Ironically, AI companies are setting themselves up for this problem. Google and Microsoft's full-court press for "AI powered search" imagines a future for the web in which search-engines stop returning links to web-pages, and instead summarize their content. The question is, why the fuck would anyone write the web if the only "person" who can find what they write is an AI's crawler, which ingests the writing for its own training, but has no interest in steering readers to see what you've written? If AI search ever becomes a thing, the open web will become an AI CAFO and search crawlers will increasingly end up imbibing the contents of its manure lagoon.

This problem has been a long time coming. Just over a year ago, Jathan Sadowski coined the term "Habsburg AI" to describe a model trained on the output of another model:

https://twitter.com/jathansadowski/status/1625245803211272194

There's a certain intuitive case for this being a bad idea, akin to feeding cows a slurry made of the diseased brains of other cows:

https://www.cdc.gov/prions/bse/index.html

But "The Curse of Recursion: Training on Generated Data Makes Models Forget," a recent paper, goes beyond the ick factor of AI that is fed on botshit and delves into the mathematical consequences of AI coprophagia:

https://arxiv.org/abs/2305.17493

Co-author Ross Anderson summarizes the finding neatly: "using model-generated content in training causes irreversible defects":

https://www.lightbluetouchpaper.org/2023/06/06/will-gpt-models-choke-on-their-own-exhaust/

Which is all to say: even if you accept the mystical proposition that more training data "solves" the AI problems that constitute total unsuitability for high-value applications that justify the trillions in valuation analysts are touting, that training data is going to be ever-more elusive.

What's more, while the proposition that "more training data will linearly improve the quality of AI predictions" is a mere article of faith, "training an AI on the output of another AI makes it exponentially worse" is a matter of fact.

Name your price for 18 of my DRM-free ebooks and support the Electronic Frontier Foundation with the Humble Cory Doctorow Bundle.

If you'd like an essay-formatted version of this post to read or share, here's a link to it on pluralistic.net, my surveillance-free, ad-free, tracker-free blog:

https://pluralistic.net/2024/03/14/14/inhuman-centipede#enshittibottification

Image: Plamenart (modified) https://commons.wikimedia.org/wiki/File:Double_Mobius_Strip.JPG

CC BY-SA 4.0 https://creativecommons.org/licenses/by-sa/4.0/deed.en

#pluralistic #ai #generative ai #André Spicer #botshit #habsburg ai #jathan sadowski #ross anderson #inhuman centipede #science fiction #mysticism

549 notes · View notes

mostlysignssomeportents · 1 month

Text

Async mugwump linkdump

I'm on tour with my new, nationally bestselling novel The Bezzle! Catch me TOMORROW in ANAHEIM at WONDERCON: YA Fantasy, Room 207, 10 a.m.; Signing, 11 a.m.; Teaching Writing, 2 p.m., Room 213CD.

For 20+ years, I've processed all the information that came over my transom by blogging – mulling on why something I saw in the world caught my attention and trying to summarize it for strangers. This turns out to be a very powerful way to do a lot of different kinds of mental work:

https://pluralistic.net/2021/05/09/the-memex-method/

With Pluralistic, the solo blog I founded 4 years ago, I've moved into longer, more synthetic essays that try to connect the things that caught my attention today with all those things I've written about for the past two decades. That's also proven very fruitful:

https://pluralistic.net/2024/02/20/fore/#synthesis

But this move to longer works has a downside: sometimes I'll arrive at the week's end and have a list of things that caught my attention without there being any obvious way to connect them, and when that happens, I devote a Saturday edition to a linkdump. There's been 15 of these so far:

https://pluralistic.net/tag/linkdump/

Welcome, then, to the 16th Pluralistic linkdump, and a warning, this one starts with an obituary.

Ross Anderson was one of the heroes of the cryptographic revolution, a brilliant scientist and communicator, a fantastic activist, and a scorching curmudgeon. Ross died this week. He was 67, and had chronic heart issues as well as long covid:

https://www.lightbluetouchpaper.org/2024/03/29/rip-ross-anderson/

There's so much that's been written about Ross and his legacy already, and there's doubtless more to come, but I've picked out two pieces to point you to. The first is from Danny O'Brien, who was also the guy who talked me down off the ledge the first time Ross flamed me on a public mailing list, leaving me bleeding and furious:

https://news.ycombinator.com/item?id=39868983

As Danny says, Ross was "the model of a politically and socially involved computer scientist," a man whose blazing intellect, fierce moral center and relentless curiosity inspired a generation of technologists to think about politics, and a generation of political activists to think about technology. Few of Ross's eulogizers (thus far) have mentioned how Ross's passion came out as fury, and – as someone who counted Ross as a friend and inspiration – I think this is a serious omission. It's hard to imagine Ross doing all that he did without understanding the anger that – along with his ethics – fueled his passion.

(Compare with @neil-gaiman's classic essay on the anger of Terry Pratchett:)

https://www.theguardian.com/books/2014/sep/24/terry-pratchett-angry-not-jolly-neil-gaiman

The other obit that I want to point you to comes from Bill Buchanan, one of Ross's closest collaborators. Buchanan's memorial for Ross does a superb job of rounding up Ross's technical contributions to the field of security engineering:

https://medium.com/asecuritysite-when-bob-met-alice/ross-anderson-rip-59233c75fadf

Buchanan embeds videos for some of Ross's best speeches, links to his key papers (including the classic "Programming Satan's Computer," on "programming a computer which gives answers that are subtly and maliciously wrong at the most inconvenient moment possible), reminiscences of Great Moments In Ross Anderson, and terrific, lay-friendly breakdowns of some of Ross's key mathematical work.

As an unreasonable, angry person, I take great inspiration from people who channel their unreasonable anger to socially beneficial conduct – like whistleblowers. After Baltimore's Francis Scott Key Bridge was totaled by the 95,000-ton cargo ship MV *Dali(, a vast cohort of instant experts in structural engineering, sea freight and shipbuilding has taken to the internet with a slurry of takes on the Meaning Of the Bridge.

Some of these are very stupid indeed, like the idea that somehow "DEI" caused the collision. But you don't have to be an expert in maritime issues or civil engineering to understand the importance of this report from The Lever about shipping giant Maersk's culture of retaliation against whistleblowers:

https://www.levernews.com/feds-recently-hit-cargo-giant-in-baltimore-disaster-for-silencing-whistleblowers/

Maersk is the company that chartered the MV Dali; Maersk is also a key player in the cartel that controls the world's shipping. Maersk was just sanctioned by the Labor Department for retaliating against a whistleblower who complained of unsafe conditions on the ships that Maersk chartered:

https://www.dol.gov/sites/dolgov/files/OPA/news%20releases/Maersk-Sec%20Findings%20-FINAL%20071423_Redacted.pdf

Maersk's policy required employees to bring concerns to their supervisors before alerting the Coast Guard or others. This is not how that stuff is supposed to work. OSHA called this policy “repugnant” and a “reprehensible and an egregious violation of the rights of employees,” which “chills them from contacting the [Coast Guard] or other authorities without contacting the company first.”

The whistleblower – chief mate on the Safmarine Mafadi – complained of "unrepaired leaks, unpermitted alcohol consumption onboard, inoperable lifeboats, faulty emergency fire suppression equipment, and other issues." We don't know (yet) what happened on the Dali, but it's obvious that a company that retaliates against whistleblowers, rather than heeding their warnings, is prioritizing covering its ass, not operating safely.

Which brings me (inevitably) to Boeing, and to poor John "Swampy" Barnett, the Boeing whistleblower who took his own life earlier this month. Barnett's suicide has stirred up similar low-yield online chatter focused on whether Boeing assassinated Barnett, a question that categorically cannot be answered through the method of arguing with internet strangers.

But there is a lot to say about Barnett: in particular, there's the substance of his whistleblowing, the specifics of his complaints about Boeing. For that, we can turn to the always-fantastic Maureen Tkacik, whose American Prospect piece "Suicide Mission" is definitive:

https://prospect.org/infrastructure/transportation/2024-03-28-suicide-mission-boeing/

Tkacik does a great job of painting a picture of Swampy as a member of the tribe of unreasonable and angry people who refuse to sideline principle in order to get along. More importantly, Tkacik shows us what made Swampy so angry: a company that was hell-bent on lobotimizing itself by forcing out any technical expert who might point out inconvenient truths about the safety risks of high-profit strategies.

As Tkacik writes, Boeing once thought about "knowledge" in terms of expertise that could be brought to bear on the unimaginably complex task of making reliable, airworthy jets. But under the "value-engineering" financialized culture that arose after the McDonnell-Douglas merger, the company viewed knowledge as "intellectual property, trade secrets, and data." In other words, the point of knowledge was rent-extraction, not safety.

At the root of this transformation was the Jack Welch protege Jim "Prince Jim" McNerney, the former 3M CEO who took the helm at Boeing. McNerney was openly contemptuous of the company's senior engineers, branding them "phenomenally talented assholes" and rewarding managers who found ways to force them out of the company. It was McNerney who decided to produce the 787 "Dreamliner" in non-union shops, far from Seattle and its phenomenally talented assholes. Instead of these engineers, McNerney turned to Boeing suppliers to do the major engineering work on the 787 – despite the fact that many of these suppliers "lacked engineering departments."

The 787 was, infamously, a $80b-over-budget boondoggle, haunted by technical failures. Swampy was part of the "cleanup crew" that tried to salvage the 787, and witnessed first-hand how the company purged all the engineers who managed to ship the 787 despite McNerney and his "value engineers" and retaliated against workers who tried to unionize the South Carolina facility.

In particular, it was safety inspector who came in for the most savage punishment. When the FAA decided to let Boeing mark its own homework – hiring in-house safety inspectors to replace government inspectors – they pretended to believe that these Boeing-payrolled inspectors would be able to operate independently of Boeing's leadership. The inspectors tried to operate this way (not least because they were criminally liable for oversights that occurred on their watch) and McNerney's Boeing came down on them like a ton of aviation-grade aluminum.

To further neuter these inspectors, Boeing management ordered the inspectors to outsource their work to the mechanics they were supposed to be supervising – that is, the FAA outsourced safety checks to Boeing inspectors, and the inspectors outsourced those checks to the mechanics themselves. Tkacik: "Swampy believed relying on mechanics to self-inspect their work was not only insane but illegal under the Federal Aviation Administration charter."

Swampy kept careful records of every way in which this system produced unsafe aircraft and an unsafe workplace – including the day he discovered that someone had removed 400+ defective parts from the rejects box and installed them in aircraft in order to meet deadlines. Swampy's reports were key to establishing that the company's much-trumpeted "improvements" in safety reports were down to a culture of "bullying" – not any improvement in safety itself.

When Boeing went to war against Swampy, they barely bothered to pretend that they were playing by the rules. He was told one day that he was four-weeks into a 60-day "corrective action" that no one had told him about. The "corrective action" paperwork had a blank for Swampy's comments. He wrote, "Leadership wants nothing in email so they maintain plausible deniability. It is obvious leadership is just looking for items to criticize me on so I stop identifying issues. I will conform!"

Shortly thereafter, he was forced out altogether. Managers who tried to bring him on their teams were told that no one was allowed to hire John Barnett. His name appeared on a secret internal memo entitled "Quality Managers to Fire." Meanwhile, the value of Boeing shares had tripled.

After Boeing's 737 Maxes started falling out of the sky, Swampy's painstaking documentation of the flaws in the 787's production took on a new urgency. A program of random inspections of 787s found major defects in all of them ("Boeing Looked for Flaws in Its Dreamliner and Couldn’t Stop Finding Them" –WSJ). An Aviation Week diagram of problem spots with the 787 marked red arrows over "every single section, from the tip of the nose to the horizontal stabilizers":

https://aviationweek.com/air-transport/new-boeing-787-fix-details-reveal-extent-gap-check-challenge

Boeing's war on "brilliance" did its work: after everyone who understood how to make a safe aircraft was forced out of the company, financialized CEOs were able to cut corners on safety, triple the share-price, scoop up billions in government subsidies and bailouts, all without those pesky "phenomenally talented assholes" pointing out that they were going get (lots of) people killed.

Tkacik closes by saying that Swampy's former work colleagues refuse to believe he killed himself. A former executive told her "I don’t think one can be cynical enough when it comes to these guys…It’s a top-secret military contractor, remember; there are spies everywhere." I confess that I don't know what to make of that, but I'll say this: if Boeing killed Swampy, that's just one of hundreds of murders they committed. Whether or not Swampy's death was their fault, the deaths of everyone who went down on the 737 Maxes that crashed is on their hands.

That's what "profits before people" means, after all: sacrificing human lives to make yourself richer. It's the foundational tenet of the conservative movement, though that impulse is often checked by other factors, like human decency. It's only when sociopaths get a sustained run at leadership that you see what they really want.

Which brings me to the UK, which has been governed by the Conservative Party for 14 years. The Tories are tipped to get destroyed in the next election, and a long article in the New Yorker by Sam Knight catalogs the many ways in which Tory rule has devastated the UK:

https://www.newyorker.com/magazine/2024/04/01/what-have-fourteen-years-of-conservative-rule-done-to-britain

The thing is, after 14 years, it's impossible for the Tories to blame anyone else for the state of the UK. With strong Parliamentary majorities, Conservatives were able to govern as they pleased – the only compromises they made were between their own internal factions. The ideological commitment to making the rich richer, privatizing everything, subordinating governance to market forces – that's all them.

It's all them: the worst period for wage growth since the Napoleonic Wars, on them. The catastrophic traffic, housing, jobs market, and precarity, on them. Plummeting health, on them. The austerity, on them. The withering of the country's courts and prisons and police, its wilderness, its programs for young people and pensioners, its public health, its diplomatic corps, its road maintenance – on them.

A country where the police can't afford to prosecute burglaries – on them (4% of burglaries are prosecuted). The 2.5 year delay between a rape arrest and its trial? On them. Mass closures of schools that are literally crumbling? On them.

43% of the countries courts have closed. On them. Cuts to prison funding, coupled with longer sentences? On them.

And of course, Brexit – on them. Every part of it. The referendum. The referendum question. The failure to negotiate a deal with the EU. All on them. The collapse in British living standards, all on them. The fact that the 20% richest households in the UK have been untouched by all this? Also on them. But you might not notice it in London, where people earn an average of 400% more than people in Nottingham.

The only growth sector outside of London are the Citizens Advice Bureaux, whose client rosters are growing even as their funding is cut. Where the CAB once primarily catered to people who couldn't make ends meet due to disability, unemployment and other reliable predictors of economic distress, today, CAB advisors are seeing homeowners, people working two jobs. Desperation is "like a black hole, dragging more and more people in,"

More Conservative growth: Tories presided over a doubling in the rate of NHS antidepressant prescriptions, and a 20% rise in long-term health conditions. No wonder Tory Britain had the world's worst pandemic outcomes for a wealthy nation – that's on them, too.

Knight's article closes with a Tory MP who believes that "the key thing for the Conservatives now is to be more conservative…Toryism must have its day again."

We can't count on oligarchs to rescue us from oligarchy – not even when oligarchy's failures push society to the breaking point. There's always a rationalization explaining why we just had to lean harder into oligarchy.

You hear echoes of this in the pro-monopoly choir, whose squeals of outrage at the rise of a new anti-monopoly movement grow louder even as monopolism's failures grow clearer. One of the more tangible expressions of monopoly's failures is the Ticketmaster/Livenation octopus, which controls the entire live music industry – key venues, promotions, and ticketing. Ticketmaster fucks over music fans, but it also cheats famous musicians, the kinds of people with big microphones, so we know a lot about how bad it is:

https://pluralistic.net/2022/11/20/anything-that-cant-go-on-forever-will-eventually-stop/

Of course, the fact that Swifties hate Ticketmaster lets the pro-monopolists dismiss critics as foolish young girls, not Very Serious People Who Understand Economics and thus can see that Ticketmaster's monopoly is Good, Actually.

Last week, Congressman Bill Pascrell dumped a ton of litigation documents related to Ticketmaster's sleaze, and Matt Stoller broke them down:

https://www.thebignewsletter.com/p/explosive-new-documents-unearthed

The docs reveal how Ticketmaster's system of (formerly) secret kickbacks let it choke out any competitor, so that it could charge fans more and pay artists less. The mechanics of the scam are beautifully laid out in Stoller's post – as is the many ways in which it violated both the law and Ticketmaster's numerous consent decrees arising from its previous lawbreaking.

This kind of scam breakdown is essential. It's easy to think that we, as mere normies, can't hope to understand the machinations of the corporations that prey on us. But once you pierce the veil of performative complexity, what's left behind is a set of crude tricks and transparent ruses.

Here's one of those transparent ruses: Discord's terms of service require Discord users to actively opt out of its "binding arbitration" system. Binding arbitration is when you sign a contract saying you can't sue the company no matter how much it harms you – instead, you promise to have your disputes heard by an "arbitrator" (a fake judge paid by the company that screwed you). Unsurprisingly, these fake judges are awfully tolerant of their employers' crimes.

Discord says that once you click through its garbage legalese novella, you have just a few days to opt out of this binding arbitration clause – if you happen to miss that fine print, you have "consented" to giving up your legal rights.

But every time Discord changes its ToS, the clock for opting out starts ticking again, and Discord has just changed (that is, worsened) its ToS again:

https://discord.com/terms

That means that if you send an email right now to [email protected] with "I am confirming that as of the date of this email, I am choosing to opt out of binding arbitration to settle disputes with Discord" in the body, you can escape this consent theater:

https://mamot.fr/@[email protected]/112175832989845038

Consent theater is a particularly galling corporate ruse – the idea that we chose to allow them to abuse us. Consent theater gets more outrageous by the day. Take Soofa, who operate streetside digital kiosks that identify you by grabbing your phone's unique wifi and Bluetooth identifiers:

https://gizmodo.com/digital-kiosks-snatch-your-phones-data-when-you-walk-by-1851368948

Soofa sells this data to advertisers – claiming that by walking down a public street, you "consented" to being tracked and sold.

The only reason this flies is that the US hasn't passed a federal consumer privacy law since 1988's Video Privacy Protection Act, which bans video-store clerks from telling people which VHS cassettes you took home. Congress keeps on failing to pass a privacy law, despite garbage companies like Soofa.

But that hasn't stopped the administrative agencies from acting to defend your privacy! The FTC just dropped its latest Privacy and Data Security Update, a greatest hits list of the actions the Commission took while Congress failed:

https://www.ftc.gov/system/files/ftc_gov/pdf/2024.03.21-PrivacyandDataSecurityUpdate-508.pdf

One of the best things about the current administration is the number of extremely competent regulators who know exactly how much power they have and aren't afraid to use it to help the American people:

https://pluralistic.net/2022/10/18/administrative-competence/#i-know-stuff

The new FTC report, which details how the Commission's existing powers let it go after the commercial surveillance industry from smart doorbells to review fraud, from kids' programming to medical data, from lax security to data-breaches, is a bright spot in an otherwise grim week.

One more bright spot, then, before I wind up this linkdump. All week, I've been humming a half-remembered lyric, "come on baby/you're a link in this chain/put your hands together/and get free of the pain." For the life of me, I couldn't place it.

Last night, I searched for it (using Kagi, the post-Google search engine I've been paying for for the past month, and which I'm loving) and discovered that I had somehow completely forgotten a whole-ass band that I once loved: Toronto's Bourbon Tabernacle Choir, whom I saw live on many occasions.

The mystery lyric came from "Death is the Great Awakener," a fucking banger of a post-gospel track that I've been listening to on nonstop repeat as I wrote this. It's a hell of a tune and I'm intensely grateful to have it back in my life:

https://www.youtube.com/watch?v=Q6RUb63Tx3w

If you'd like an essay-formatted version of this post to read or share, here's a link to it on pluralistic.net, my surveillance-free, ad-free, tracker-free blog:

https://pluralistic.net/2024/03/30/dewey-502/#rip-ross-anderson

Image: Waffleboy https://www.flickr.com/photos/waffleboy/28198395465/

CC BY 2.0 https://creativecommons.org/licenses/by/2.0/

57 notes · View notes