58 minutes ago
2
Come with them if you want to live.
Anthropic’s CEO Dario Amodei says he has found an outside team that can regulate AI and make sure it doesn’t destroy the world — and it just so happens to have deep ties to his old housemate and the woke Effective Altruism movement that’s already been involved in one major tech meltdown.
In a bombshell essay titled “We Must Pace the Frontier,” Amodei argued that leading AI firms should commit to allowing “embedded third-party evaluators” from a nonprofit called Model Evaluation and Threat Research, or METR, that would maintain “employee-like access” to oversee safety efforts and ensure advanced AI models don’t turn Terminator and wipe out humanity.
The would-be John Connors at METR purport to be fully independent, but the group traces its roots directly back to key figures in Effective Altruism, whose Silicon Valley-based devotees once included disgraced crypto boss Sam Bankman-Fried, one of Anthropic’s biggest early investors.
Effective Altruism, a philosophy that’s become popular with Silicon Valley masters of the universe, advocates for using reason — and fortunes amassed by tech’s best and brightest — to do the most good for humanity over the long term.
In the past, it has been used to justify making as much money as possible with the idea that you can donate more later to do more good – though the movement suffered a big black eye when one of its most famous proponents, Bankman-Fried, saw his crypto empire collapse amid fraud charges that sent him to prison.
Deep EA ties
The overlap between METR and Anthropic is rife with potential conflicts of interest, according to critics.
METR spun off from an Effective Altruism-aligned tech incubator called Alignment Research Center (ARC), which is headed by AI researcher Paul Christiano. He was Amodei’s housemate, coworker and research collaborator in the 2010s when they were both at OpenAI — Anthropic’s biggest rival.
Christiano also served as one of the first five trustees of Anthropic’s Long-Term Benefit Trust.
One staffer on METR is Ajeya Cotra, who is married to Christiano and has been a major organizer in the EA world.
The founder and CEO of the group is Beth Barnes, who was hired by Christiano and was involved in the EA movement at college.
Chief scientist Hjalmar Wijk and staffers Megan Kinniment and Pip Arnott all came from the same research cluster at the University of Oxford — a major feeder for EA.
When reached by phone, a METR official acknowledged that the nonprofit has a lot of overlap with Effective Altruism, but insisted its employees have a diverse array of ideological viewpoints. They’re required to disclose conflicts of interest, such as employees holding equity in firms they are tapped to investigate.
“We don’t think any small group should have authority over what happens with AI, and our goal is to get information from inside these companies into the public domain – external investigators can help bring more information to light,” a METR spokesperson said in a statement.
“Our funders have no say in the projects we work on and we do not accept any funding from AI companies or their employees.”
Amodei, who has pledged to give away 80% of his personal wealth, has deep ties to EA. His sister Daniela married Holden Karnofsky, the founder of two major EA nonprofits.
METR’s involvement is likely to be a nonstarter for the Trump administration, which has had a testy relationship with Anthropic over the past year and has also expressed heavy skepticism of so-called AI “doomers” who argue the technology is on the precipice of disaster.
“METR is essentially Anthropic in another form,” a source familiar with the White House’s thinking told The Post on Tuesday.
Multimillion-dollar fundraising
Effective Altruism’s obsessed adherents believe in funneling their wealth toward the goals they view as most important to humanity, including climate change and strict AI regulation.
Since its founding, METR and its affiliates have received funding from EA kingpins like Democratic megadonor Dustin Moskovitz and Skype cofounder Jaan Tallinn — both of whom are Anthropic investors.
In August, METR announced it had raised $71 million in new funding from outside investors over the previous six months.
Tallinn’s Survival and Flourishing Fund, a philanthropy group, has earmarked up to $752,000 in funding to support METR since its founding, including $324,000 in general cash and a $428,000 matching pledge, according to public records.
Elsewhere, Moskovitz’s Coefficient Giving – formerly known as Open Philanthropy before a hasty rebranding last year – gave $1,515,000 to METR’s incubator, the ARC, in 2022. METR was originally known as ARC Evals before spinning off as an independent nonprofit and changing to its current name in 2023.
METR does not take any compensation from AI labs for its work, nor does it take donations from executives or employees of AI companies, the METR official said. In the case of Moskovitz’s donation to ARC in 2022, the official said the funding was firewalled and not used for METR’s operations.
That’s hardly reassuruing to critics.
“Saying you want to be audited by the ‘independent’ people at METR is a complete joke,” said Perry Metzger, chairman of Alliance for the Future, a Washington, DC-based AI policy group. “It’s people who are funded by your buddies. It’s completely ridiculous.
“If you went out tomorrow and you’re a big bank and you had your audits done by an organization consisting entirely of your own former employees, which was funded by the same investors that are investing in you, do you know what the feds would do with you?” he added. “It would be positively medieval.”
Amodei’s surprise proposal published last Saturday morning came just days after Jacob Coxon, an Anthropic researcher who previously worked at OpenAI, abruptly quit the firm while warning that both AI giants were “gambling with our lives” by pursuing progress without proper oversight.
Without safeguards in place, Amodei warned the internet could be overtaken by AI bots within six to 12 months, “potentially causing hundreds of billions of dollars in damage.”
‘Stop pretending’
The doomsday warnings received sharp pushback in some corners of the tech and political world, with President Trump asserting that they were a “hoax” and smaller AI startups arguing that leaders like Anthropic and OpenAI were just trying to rewrite the law to benefit themselves.
Among the skeptics was former White House AI czar David Sacks, who called on Amodei to “stop pretending METR is independent when it is interviewed with Anthropic’s investors and staff.”
“Most of all, stop pretending the motivation to slow down is purely altruistic,” he added. “You face massive product-liability exposure if your products enable a truly damaging cyberattack.”
It’s also unclear if the AI industry’s push to police itself will pass muster with US regulators.
Federal Trade Commission Chairman Andrew Ferguson warned that “everyone should be deeply suspicious” of any request to grant AI companies an antitrust exemption to coordinate safety efforts.
“I will say generally, though, that if companies are simultaneously coming to Washington and asking for a host of regulations and an antitrust exemption, all of my alarm bells go off,” Ferguson said during a Tuesday event at Georgetown University.
Under Amodei’s proposals, the third-party evaluators – which some have likened to nuclear inspectors – would “verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes,” according to his essay.
“Anthropic is unilaterally committing to this step now,” Amodei wrote. “We intend this to be part of a broader push to redouble efforts on our safety and alignment work.”
OpenAI’s Sam Altman said he agreed with Amodei about the “need to pace the frontier” – adding that “committing to having independent evaluators with employee-like access is a great idea, and we will do the same.” He made no mention of METR.
Anthropic declined to comment.
Additional reporting by Jared Downing and Georgia Worrell
English (US)