A TEAM of technology experts issued a global call on Monday seeking the toughest questions to pose to artificial intelligence (AI) systems, which increasingly have handled popular benchmark tests like child’s play.

Dubbed “Humanity’s Last Exam,” the project seeks to determine when expert-level AI has arrived. It aims to stay relevant even as capabilities advance in future years, according to the organizers, a nonprofit called the Center for AI Safety (CAIS) and the startup Scale AI.

The call comes days after the maker of ChatGPT previewed a new model, known as OpenAI o1, which “destroyed the most popular reasoning benchmarks,” said Dan Hendrycks, executive director of CAIS and an advisor to Elon Musk’s xAI startup.

Mr. Hendrycks co-authored two 2021 papers that proposed tests of AI systems that are now widely used, one quizzing them on undergraduate-level knowledge of topics like US history, the other probing models’ ability to reason through competition-level math. The undergraduate-style test has more downloads from the online AI hub Hugging Face than any such dataset.

At the time of those papers, AI was giving almost random answers to questions on the exams. “They’re now crushed,” Mr. Hendrycks told Reuters.

As one example, the Claude models from the AI lab Anthropic have gone from scoring about 77% on the undergraduate-level test in 2023, to nearly 89% a year later, according to a prominent capabilities leaderboard.

These common benchmarks have less meaning as a result.

AI has appeared to score poorly on lesser-used tests involving plan formulation and visual pattern-recognition puzzles, according to Stanford University’s AI Index Report from April. OpenAI o1 scored around 21% on one version of the pattern-recognition ARC-AGI test, for instance, the ARC organizers said on Friday.

Some AI researchers argue that results like this show planning and abstract reasoning to be better measures of intelligence, though Mr. Hendrycks said the visual aspect of ARC makes it less suited to assessing language models. “Humanity’s Last Exam” will require abstract reasoning, he said.

Answers from common benchmarks may also have ended up in data used to train AI systems, industry observers have said. Mr. Hendrycks said some questions on “Humanity’s Last Exam” will remain private to make sure AI systems’ answers are not from memorization.

The exam will include at least 1,000 crowd-sourced questions due Nov. 1 that are hard for non-experts to answer. These will undergo peer review, with winning submissions offered co-authorship and up to $5,000 prizes sponsored by Scale AI.

“We desperately need harder tests for expert-level models to measure the rapid progress of AI,” said Alexandr Wang, Scale’s chief executive officer.

One restriction: the organizers want no questions about weapons, which some say would be too dangerous for AI to study. — Reuters

STOCKHOLM — Lower voter turnout and increasingly contested results globally are threatening the credibility of elections, an intergovernmental watchdog warned on Tuesday, as its sub-index for free and fair elections suffered its biggest decline on record in 2023.

In its report, the Stockholm-based International Institute for Democracy and Electoral Assistance (IDEA) said 2023 was the eighth consecutive year with a net decline in overall democratic performance, the longest consecutive fall since records began in 1975.

The watchdog bases its Global State of Democracy indexes on more than 100 variables and uses four main categories — representation, rights, rule of law and participation — to categorize performance.

The category of democracy related to free and fair elections and parliamentary oversight, a sub-category of representation, suffered its worst year on record in 2023.

“This report is a call for action to protect democratic elections,” IDEA’s Secretary-General Kevin Casas-Zamora said in the report. “The success of democracy depends on many things, but it becomes utterly impossible if elections fail.”

The think tank said government intimidation and electoral process irregularities, such as fraudulent voter registration and vote-counting, were increasing. It also said that threats of foreign interference, disinformation and the use of artificial intelligence in campaigns added to challenges.

It also said that global voter participation had fallen to 55.5% of eligible voters in 2023 from 65.2% in 2008. Globally, in almost 20% of elections between 2020 and 2024, one of the losing candidates or parties rejected the results.

IDEA said that the democratic performance in the US, which holds a presidential election this year, had recovered somewhat in the past two years, but the assassination attempt on former President Donald Trump in July highlighted continued risks.

“Less than half (47%) of the Americans said the 2020 election was ‘free and fair’ and the country remains deeply polarized,” IDEA said. — Reuters

BACTERIAL ILLNESSES that are resistant to available antibiotic medicines will cause more than 39 million deaths worldwide over the next 25 years and indirectly contribute to an additional 169 million deaths, according to a forecast published on Monday.

By 2050, annual death tolls attributed directly to antibiotic resistance, or associated with it, will reach 1.91 million and 8.22 million, respectively, if remediation measures are not in place, an international team of researchers reported in The Lancet.

Those annual numbers represent increases of nearly 68% and 75% per year, respectively, over death tolls directly and indirectly attributed to antibiotic resistance in 2022, the researchers with the Global Research on Antimicrobial Resistance Project wrote.

The increases will strain health systems and national economies and contribute to annual gross domestic product losses of $1 trillion to $3.4 trillion by 2030, they predict.

The forecast of how the antibiotic resistance burden is likely to evolve was released ahead of a Sept. 26 United Nations (UN) General Assembly High Level Meeting on the subject.

“This landmark study confirms that the world is facing an antibiotic emergency, with devastating human costs for families and communities across the world,” Dame Sally Davies, the Special Envoy on Antimicrobial Resistance for the UK and a member of the UN Interagency Coordination Group on Antimicrobial Resistance, said in a statement. She was not involved in the research.

Resistance to antimicrobials appears to pose the biggest threat to the elderly, with deaths in adults over age 70 increasing by more than 80% between 1990 and 2021, according to the report.

Low- and middle-income countries face a disproportionate burden, with the highest rates of antibiotic-resistance-related deaths occurring in sub-Saharan Africa and South Asia, in particular from multi-drug resistant tuberculosis, researchers said.

Antibiotic resistance occurs when bacteria, viruses, fungi and parasites no longer respond to drugs, either because of genetic changes in these organisms or, more often, because of the misuse and overuse of the drugs to treat, prevent or control infections in humans, animals and plants, according to the World Health Organization.

Access to better care for serious infections, new vaccines to prevent infections, and more judicious medical protocols that limit antibiotic use to appropriate cases could save a total of 92 million lives between 2025 and 2050, they also predict.

Estimates for the new study were made for 22 types of disease-causing organisms, 84 combinations of drugs vs bacteria, and 11 infectious syndromes such as meningitis and sepsis.

The estimates were based on records from 520 million people of all ages in 204 countries from a wide range of sources, including hospital data, death records, and antibiotic use data. — Reuters