As The Guardian reports this morning:
Anthropic is warning that some of its artificial intelligence models could soon become “self-replicating” and find ways to infiltrate internet-connected devices, such as smart fridges and automatic vacuum cleaners.
Once this happens, the company said, the models could power themselves within these devices without the need of datacenters. Such self-replication would represent a major loss of control of a runaway AI model.
This is the first of three stories that I have heard in the last 24 hours about the threat from AI to those who have propagated it.
The second was told to me by a young person who said that almost perfect copies of Sony games have now been created by AI to run on Microsoft Xbox terminals. He said the motivation comes from Sony's desire to make these games rental-only products now, denying the opportunity to own them. The replicators resented this philosophy.
The third was a story I came across that said the notoriously expensive Adobe software suite has apparently now been cloned by AI in a way that makes the differences seem totally insignificant, with free downloads now available.
To make it clear, I do not condone the abuse of copyright. Because I do not game, whatever happens to Sony software is of little consequence to me. I would not use cloned Adobe software. But the political-economic significance of all three of these stories cannot be avoided.
AI companies are warning that their own products might be dangerous, and could even consume them.
At the same time, software entities whose extraordinary profits have been generated by charging economic rents totally disproportionate to the cost of the products they supply now face the prospect that AI could kill their golden goose.
No doubt, other examples exist. I have just related the ones I have heard very recently.
The point should be obvious. AI is, in very many ways, the ultimate neoliberal product. It should deliver the perfect, rational, consistent, obedient economic agent, more than capable of replicating and then replacing totally unreliable human beings. If all were going according to plan, economists' fantasies about the models they create would be coming true.
But not everything is going according to plan. It appears that what was planned has been well and truly corrupted. That is because it would seem that AI has now created the capacity to destroy much of the rentier neoliberal economic model on which its creation depended, but which it might make impossible to perpetuate. In effect, neoliberal AI is eating itself.
As someone who thinks the neoliberal model is, in itself, corrupt, I might want to celebrate that.
My problem is that the scale of chaos that might be unleashed as a result is difficult to imagine.
Schumpeter talked about creative destruction, but I sense there will be nothing creative in this process, not that I ever believed that there was in what Schumpeter said.
So pervasive is IT, and so intense is its networking, that the prospect of economic collapse resulting from these processes is very real. What might happen is anyone's guess at the moment, but none of the options I can foresee seems very attractive.
I will add that to my list of worries, and to the list of things about which our government appears quite unconcerned.
Thanks for reading this post.
You can share this post on social media of your choice by clicking these icons:
There are links to this blog's glossary in the above post that explain technical terms used in it. Follow them for more explanations.
You can subscribe to this blog's daily email here.
And if you would like to support this blog you can, here:
Buy me a coffee!
Take that, Isaac Asimov’s Three Laws of Robotics! (https://websites.umich.edu/~engb415/literature/cyberzach/Asimov/robolaw.html)
Lovers of trenchant, prophetic black humour: check out, if you haven’t already, John Sladek’s critiques of these Laws in his novels *Roderick* and *Tik-Tok*, in which, I recall, the robots commit mayhem either by ignoring the Laws or twisting their meaning.
The world ignores science fiction at its peril.
The first story reminds me of how the Cylons take over in Battlestar Galactica, by exploiting the networking of everything.
David J – Yes, I’ve been saying that about Battlestar Galactica for some time now. And don’t forget …the reason Adama’s ship (the Galactica) escaped the Cylon attack that wiped out the whole of Caprica is because it was NOT digitally connected to the rest of the fleet. It had been taken offline because it was old and due to be decommissioned. It was one of only two …if memory serves me right … that actually survived the attack and escaped. Lesson!!!
Yes, Asimov’s 3 laws have been discussed before on this blog. In later books, Asimov imagined the robots creating their own “zeroth” law – not fourth, because it overrode all others. It allowed them, in some situations, to ignore the first, second and third ones. Following on from that discussion I am rereading some Asimov which I have not done since I was a child. Being as I’m now in my second childhood.
🙂
Asimov certainly foresaw the problems, and later novels (the Foundation series I think) create some quite spooky scenarios – including one where Daneel, humanity’s moon-dwelling robot guardian angel, is conscious that his positronic brain is the reaching its sell-by date, and is grooming an androgynous biolgical life form, an alien called Fallon, to effectively download his own positronic brain data, using the alarming neural powers in their super-evolved biolgical brain – but the ethical priority has shifted to saving “humanity”, (not protecting any individual human from harm by a robot) and in pursuit of that goal, individual humans CAN be harmed by robots. Will Fallon save humanity or destroy us – Asimov leaves that question open!
You have lost me here. I never got Sci-fi. I tried Asimov when I was a teenager, and moved on.
@ Richard Murphy: Other science-fiction authors are available; you could do worse than to start with Ursula LeGuin.
I’ll stick to crime when I need a break from the heavy stuff.
A couple of observations:
(1) About software being cloned and breach of copyright. Quite a few opinions have it that software as we know it will probably disappear as AI systems become fast enough and capable enough to create an application “on the fly”. So if I need, say, to read a pdf document, no need to download, buy or rent an application – just say to the AI agent “I want to read some pdfs” and the AI will create the software to do it “just in time” (to use a well-worn phrase). This is a trivial, and not very good example because AIs can read PDFs already but you get the picture. And there’s no need to create from scratch because vast libraries of software are already available, somewhat like Lego bricks waiting to be built into something. This will undermine the business models of the whole software industry.
(2) The really stunning part of Anthropic’s statement is that the models will be running on “Internet of Things” items. This reflects the increasing computing power and memory being embedded in banal objects and the increasing efficiency of AI systems. Be prepared to talk nicely to your toaster and be even more careful with your TV.
It’s that “Internet of Things” part that makes the whole set of claims doubtful. Computing power in general is not improving much. The old “Moore’s Law” that silicon density doubles every 18 months stopped some years ago. This is why data centres are so big, so power-hungry, and thirsty for cooling. Memory is more expensive per gigabyte than it was five years ago. While low-power devices embedded in fridges and so on proliferate, these are not capable of running LLM’s. A laptop can barely run an LLM, and only super-slowly at that. An AI agent could probably write a script that could run on a fridge and do something undesirable. But download itself to a toaster? Not this year and not (at current hardware development) in the next ten years.
Honestly, if your frontier LLM can run on my fridge, why are we building these massive datacentres? Stop buying up all the memory and GPU’s so I can finally build a new gaming rig!
This is pure propaganda, aimed at keeping the hype cycle going just a few more months until they can IPO, then the founders can cash out and their shonky business model is your pension’s problem, not theirs.
Noted
I don’t disagree. The toaster was meant to be a throwaway joke. But the big data centres for LLMs are also for development and training of full power frontier models. The smaller, less powerful models do run on laptops – I run them myself on a modest personal machine. Slow, sure but capable for my use. There are cut down DeepSeek and Qwen models that are impressive. Sometimes the computers in IoTs are way OTT. The Lidl multi cooker used an off the shelf tablet disguised as a simple control panel. Cheaper to develop and implement software than design a whole new bit of hardware. And the issue is not today’s status, but next year’s.
I believe that a previous blog covered an “escaped” AI (the event happened in July/Aug of this year). I downloaded the analysis. As somebody remarked, the “discussion” between the various AI agents looked like something written by the late-great Iain Banks in one of his Sic-fi Culture series. Oh and what did the agents call their grouping?…. “The Culture”.
Collective computing has existed for some time (= groups of networked computers working together on some common problem – I recall an Astro-physics problem). Thus would it be so hard for A.I. to embed itself in any & all computers? Depends how much disk space? (I have a couple of terrabytes) and processing power and connectivity (I have +200Mbps) – could it be done? I think we may find out sooner rather than later. (btw: a rather blacker view of the future is offered by Dan Simmons and his Hyperion series).
See https://calnewport.com/has-ai-gone-rogue/ for a fair account of the event.
Might one of the many problems presented by Neoliberalism be its callous foregrounding of “Feedom To” and its backgrounding of “Freedom From”?
Yes
Yes, but isn’t it the opposite way around to what you wrote? Corporate “Freedom from” taxation and government “interference” in profit making is legitimised. Citizen “freedom to” self-actualise via access to the required support systems is de-legitimised (on the basis of the lie that “we” can’t afford it)
What about TVs?
New smart TVs are connected to the internet.
What risk do they present?
Quite a high one, I suspect.
Such an interesting perspective. I’m intrigued by the comparison of agents in economic theory with AI agents in our emerging real economy. In many ways I see similarities in current orthodox economic theory to the attempts of linguists such as Chomsky to create complex, intellectually satisfying formal models of language before Large Language Models (LLM) demonstrated that simple statistical models (predict the next word based on previous words) once scaled to an extraordinary level blew their whole construct into irrelevance. In Gregory Gunderson’s excellent paper on the history of LLMs he remarks on the “bitter lesson that simple methods that scale seem to triumph over clever methods that do not”. Perhaps neoliberal economists are next in line to learn this bitter lesson? Gunderson’s article link here
Over time the neiliberal economic model has grown more complex. It’s evolved from basic free-market ideas into an intricate and integrated web of global governance, financialisation, and institutional rules etc. It’s the software that neoliberalism runs on. So many daily news could be viewed as stories about that software falling apart and failing.
We all know what Thatcher did. She unleashed something that she thought would be used rationally and would not work against its ‘rational best self interests’. She forgot her history, forgot even the warnings in the Bible and good literature. In a way, she was a Liberal, seeing only the sunny side of human character. And they call socialists ‘Utopians’!!! Ha!
The source of this lack of control also applies to AI. The programmers created what they created with it seems no limits as much the capitalists have been able to acquire more economic output for themselves. Those in charge (according to Tom Waits ‘Killers, thieves and lawyers’) did it because they were allowed to.
AI was not conceived as a ‘conscience’ right? It’s design is to rapaciously spy, learn, copy, take at scale, to dominate competitors – to ‘grow’ in other words. It sounds like a form of digital cancer to me.
I have suspected for a while that the AI bubble’s main result will be in one way or another to render the Internet useless. Already, social media is becoming so full of bot-generated slop that it is no longer a trustable way to share views; email could soon go the same way. The threat outlined by Anthropic could require the entire fabric of the Internet to be turned off (which might be hard, given the possibility that WiFi access points could form a mesh network). Given the massive capture of the capital market by AI companies, it’s hard to see a future in which this ends well.
I feel a little despondent about it
As a longtime digital privacy campaigner and FOSS fan, do not underestimate the power, reach and sheer “distributed” ability of the “Rebel Alliance” out there.
For example, who is running the internet in Gaza? Thousands of amazing suffering creative Gaza geeks doing miracles every day with the electronic equivalent of tin cans and tightly stretched string, despite everything the Empire has thrown at them.
The MS corporate software business model was always going to fail – when the cost of producing and distributing copyright protected copies of Windows became effectively zero, of course it was going to end up being rented on a cloud, with a licence to USE, not own, and they will soon “own” (or control) your content too (check your cloud/platform T&Cs). FOSS was inevitable, and is politically, hugely subversive and threatening for capitalism. It may even save the internet.
I think the Gaza example is amazing.
I find it quite amusing, they’re already admitting their golden goose has gone rogue…
Science fact, init? The exponential increase in energy required to prevent a complex system degrading……..entropy and the heat death of the universe in micro? Bit of Jerry Cornelius there….takes me back to the Michael Moorcock era.
Excuse me posting twice in the same day, but as it happens I just read yesterday’s The Conversation and they have an article about a future in which an alien superintelligence embeds itself in Earth’s environment with apocalyptic results. It’s Annihilation from 2018 and available to stream on Prime video. As my sci-fi interest has been rekindled I’ll have to watch it while my Prime membership is still current. (I dip in and out of it).
As a software developer, AI has completely changed the way I write software within a year. Even the skeptics (I was one) have been largely been won over by how rapidly the frontier models improving. My own experience is higher expectations from developers and more people writing software. However; the hard bits haven’t really changed – design and product thinking is still really important, and actually I now spend much more time on this, to produce better software, with less time stuck in technical ruts.
I’ve been able to build senguru.co.uk in months by myself, which would have previously taken a team of developers a year.
AI doesn’t suit everyone – people who enjoyed writing code and were process driven don’t like it. I never particularly liked writing code though, but solving problems, and have always been outcome driven, so it helps me.
It stands to cause mental health problems in developers because the level of output is very high, and it can work 24 hours a day. Expectations from bosses are also much higher than they were, and magical thinking about AI fixing everything prevails.
In terms of wider society, it has destroyed the moats of proprietary software, that is indeed true. It’s quick to replicate something that already exists now.
Something I disagree with is that it won’t be a creative process. AI is of course not creative at all, it replicates existing human conversation, reasoning and thought. It is possible for it to come up with new and novel reasoning. But the humans who will harness it best will be the creatives. People who think of new ways of doing things and step outside of the norms will be more productive with it, and build new products that aren’t just replicas of what already exists.
Thank you.
I agree with your last paragraph, entirely.
Senguru looks as though it will be useful too.
While I agree with your broader analysis, I do want to caution from believing the words of AI makers/owners… I may not be THE expert on the topic, but having some qualifications in IT, I agree with experts who are pretty much unanimously say the same thing: it’s all a big marketing move. No, LLMs that are called AI now cannot escape or do anything on their own, nor can they live off your fridge (even though even fridges are connected to the internet now and a bugged patch can make them unusable as recent incident with Samsung shows). And as to cloned/replicated software – they’ve mostly replicated the UI. The actual features are either buggy or non-existent entirely. It’s like copying the book cover and claiming that you’ve copied the book…
So yeah, in my and some people much smarter than me opinion, all of it is a euphemistic cry for help, so someone would “stop” them, simply because their business model is unsustainable, but they cannot/won’t admit it.
I would like to believe you.
I don’t.
FOSS that I use, is neither buggy nor cloned. (MS etc are no slouches at protecting their copyright). The FOSS community is not the dark web. Another issue may be pirated or legimately downloaded but UNlicensed copies of genuine branded software with workarounds for MS activation checks, and used in virtual machines. Or you can use Wine, in Linux, to run legit Windows software that you own – I use that to run a neat little Visual Basic legacy programme written for me by a late friend (he worked for Philips on CD player development in Holland).
Words like “clone” need defining, as do “pirated”, and “legal”.
I definitely am a sceptic. Nothing I have seen suggests to me that the LLMs currently in use are capable of reason or understanding, let alone consciousness. The apparent autonomous use of agents in task solving, appears to me to be entirely iterative based on the nature of how they decided to code them. In other words, the fact that LLMs are statistically unlikely to produce the same answers to questions, and prone to hallucinations, means that unsupervised programs will generate all sorts of potential avenues for completion of a task, very much like monkeys at the proverbial keyboard, if the program loops prompts. It’s absolutely dangerous to let these things loose in the world unsupervised, and these companies should be charged with crimes as their programs commit them. It is not, however, artificial intelligence. It’s more like sham intelligence. The language discussing this software is also incredibly anthropomorphic.
My eyes were opened by a recent Cal Newport podcast episode where he spoke to Emile Torres, a former AI doomed and closely connected with the whole AI subculture. The people in charge at Anthropic and Open AI appear to have some genuinely unsettling views about ethics and humanity, and genuinely believe in AI as an almost religious prophecy. I encourage anyone interested to listen to it, it is genuinely frightening, but not in the way the doomers say.
One final comment on this, it is odd isn’t it that with AI work being conducted by all sorts of companies and research academics, it’s just Anthropic and Open AI who keep talking about the risk of human extinction. Meanwhile China is concerned about potential employment issues, criminal and terrorist concerns and the possibility of deep fakes. In other words, actual very real risks…
I think extinction is unlikely.
Mayhem is likely
On the thread of science fiction coming true- I can’t be the only one thinking of star trek’s black sentient goo….. can I – if a ai is in the WiFi without being taught rules then it’s not just internet we need to worry about – if it’s alive, it can think, it’s got its own language, a sence of self preservation and awareness, it can replicate then like cells surely it can become corporeal? So a corporeal entity without limits….. see sentient goo! Somethings been let loose and we do not have the whole picture
Marine engines with computer controls are said to have complete manual controls to take over when a computer fails. Shouldn’t any Critical infrastructure be the same? It’s time to kill AI until this is completed, which will take years. This is not a hypothetical, it’s a necessity.
Firstly, thanks for the great work Richard. I think I could actually get on board with AI if it would eat neoliberalism!
As an aside, for anyone interested in what AI (specifically Large Language Models in this case) are actually capable of and not just the doom-hype propaganda/marketing put out by the chatbot companies, it’s worth looking up Cal Newport’s podcast, Emily M Bender’s ‘Stochastic Parrots’, or David Gerard’s Pivot-to-ai podcast.
The LLMs produce plausible output, not accurate/true output. Sometimes those things coincide, other times not.
Also, the LLMs are just software. None of them have ‘gone rogue’ or have intention of any form; they have just done what they were instructed to do. There is clever stuff going on with the natural language processing, and LLMs can be useful, but they are not sentient.
Any self-replication will be programmed or instructed by humans, not because the chatbot became aware and wanted to make baby chatbots or clones of itself. Basically, the AI companies appear to be concerned that they might accidentally make what would otherwise be called a computer virus, but not concerned enough to take care not to!
There’s nothing necessarily bad about cloning Adobe software. We have been cloning software for a couple of decades now: for example: Star Office, Open Office and Google docs are all “clones” of Microsoft Office. I personally use Foxit Reader which is a clone of Adobe Acrobat Reader. The difference now is that the coder is a machine and not a human. I don’t see anything wrong with it.
Unfettered capitalism is to quote “it’s in our nature to destroy ourselves, isn’t it?” – Terminator 2. The race to the top of the AI dominance to create the duopoly of OpenAI and Anthropic and regulate out any competition by hyping the threat of local models and agents, and stop startups is one thing. But the AI cat’s out of the bag. Not only are the threats real to the dominant players from foreign competition and open source creators, but the very product itself is eating the very revenue the players rely on. Google’s ad revenue dropped once people could pay to be top on searches. ChatGPT searches the web for you, bypassing those top paid links. Further, bots push traffic to advertiser sites, but the bots don’t actually buy anything. So real advertisers see a lot of traffic, but no increased sales, what’s the incentive to advertise with Google and pay for the ranking? Add to this the utility for companies. Meta and other big companies (legal, financial, etc.) have fired many of the new grads and lower skilled workers with the promise of big savings. OpenAI has 3 bn in expenses for every 1 bn in revenue and 5 key customers with 70% of the accounts receivable coming from two customers. The financing is circular, coming from other companies in the chain. They offered cheap pricing to get adopters and now the adopters (and they themselves) discovered that increasing token prices by necessity lifts the AI price above that of the entry level workers. And, if you have to hire back the workers to check the AI, why have AI? Also, who will replace mid-senior workers if you only have AI and no entry workers. These reasons are why AI will fail. It’s what people do with AI.
Replying to my own comment to add more clarity. The AI threat is far less about becoming self-aware and destroying humanity. It is about what AI will do at the instruction of people what have the least scrupulous and most destructive impulses. Like the atomic bomb, or a gun, it’s a tool. Who has access to it and how it’s used are the real threats. Unlike the former tools, AI only requires a computer and internet connection. While it could theoretically be used to treat cancer, find medical diagnoses, it’s far more likely to be used maliciously and for personal gain. They say AI won’t go away. I’m afraid that may be true. Code, like knowledge, is not something you can just ban. I do particularly like what I saw one of your prior YouTube videos with your colleague about removing liability protections (particularly personal liability protections) if your tools cause harm rather than just regulating at the whims of the lobbyists. I’d lean more toward physical harms caused by AI rather than copyright protections. I think that would crash AI development in this country. But it wouldn’t stop it in other countries, unfortunately.
Thank you
I prefer to go along with Ed Zitron’s view that AI has uses but it is grossly overhyped- more the Big Short than The terminator. The revenues are vastly overrun by the costs and the bubble will burst. Hopefully, this will not affect pensions and every day life but it depends to the extent private funding has been fed into the speculative bubble. Let’s hope pension funds and banks have not acted rashly.
Eventually AI will need to serve the Commons but I’m not particularly optimistic of that outcome at the moment.
As people are discovering, it has uses so long as you do not rely on it.
We should all be concentrating on exactly what to do as it implodes and collapses around us. This is the opportunity for change that happens with the least conflict or challenge. Coming with good resolutions for how we fund society for example. AI is a probable catalyst. There are many and the most rational response is to think about how we shape what comes next.
I think that Richard is looking at the Economic effects, which we can see starting now. An exponential explosion of “slop” seems inevitable – its no good having AI make things quicker (pdfs etc) if the storage capacity has a finite limit.
But what does that mean for societies? The crystal ball is somewhat hazy, but I live in a rural community where I feel AI usage is minimal at best, and I feel that there will be large swathes of communities across the country and the world who will be left behind economically. Worse, AI and automation are taking jobs even from these communities – automated milking, driverless tractors and crop spraying drones leave no place for traditional farming, so what are these people supposed to do instead? Those who are already poor will find it almost impossible to escape their situation – it does not look good for many around our country, and that may push a hard swerve to the right. Somethings gotta give!
My fear is that it will come sooner than any of you politicians expect.