A software contest in Amman has handed its top prize to a school-air forecasting model that missed 14 of 41 recorded high-pollution episodes, prompting experts to question whether the competition rewarded fewer alerts at the expense of timely warnings. The model, called Sahm-3, was judged on how well it could reduce unnecessary school closures while forecasting dust levels.
The Civic Signal Challenge announced its winner on 14 October, after 12 teams tested their systems against two years of air readings from schools and municipal monitors. Sahm-3’s creators will receive 240,000 Jordanian dinars (about €290,000) and a six-month pilot contract. The model is not yet used to make closure decisions.
A score built around fewer closures
The judges’ published scoring sheet gave teams points for accurate predictions, but also deducted points for recommending closures that did not happen. Sahm-3 cut projected closure days by 38% against the contest’s reference model. It also failed to flag 14 episodes when PM10 levels passed 180 micrograms per cubic metre, according to a review of the challenge’s test results by researchers at the fictional Levant Institute for Urban Systems.
That gap matters because Amman’s school guidance uses a lower alert threshold than the model’s misses might suggest: officials are asked to consider closures when readings remain above 150 micrograms per cubic metre for several hours. The contest did not include children’s health records or measure exposure inside classrooms, so its scoreboard could not show whether fewer closure recommendations meant less disruption or more time in polluted air.
The prize has gone to the system that was best at making the warning disappear, not the one that proved it could protect a school day.
Raoul Sala, atmospheric-computing researcher
Kunti Ganesan, the challenge’s technical director, said the brief was to “find a practical balance between reliable alerts and avoidable closures”, not to maximise warnings. She said the 14 missed episodes were flagged for review and that the pilot would not begin until the model had passed a separate assessment. “A contest result is not an operating rule,” she said.
The alert gap reaches beyond Amman
Sala said the scoring design could become a template for other cities buying predictive software, especially where dust storms already disrupt classes and public services. If a prize or contract depends heavily on reducing closures, he said, suppliers have an incentive to treat a warning as a cost to eliminate. “The danger is not a bad algorithm alone,” he said. “It is a bad target dressed up as technical progress.”
Pia Pavlin, a data-audit specialist who examined the public results, said the contest’s test monitors were concentrated in central Amman, while several school sites sat farther out on the city’s dry outskirts. That left the model with less evidence about the places most exposed to blowing dust, she said. The organizers said teams received the same dataset and that the pilot would add more school-level sensors.
The city’s Amman Air Desk will decide whether to run the pilot after a review due by 30 November, said Elinor Harris, a programme adviser. The assessment is expected to test missed alerts separately from false alarms and compare sensor readings at school entrances with nearby public monitors. Harris said officials could reject the model even if it keeps its contest ranking.
For now, the result has turned a software prize into a test of what public agencies ask technology to optimise. Schools need forecasts early enough to act, Sala said, but the contest measured success partly by how often the system avoided a closure recommendation. Whether Sahm-3 can do both will be decided outside the competition room.
