Delivery and verification

Bellingcat has been timing language models on geolocation since June 2025. The October round is the one to read, because the test finally looks like the work.

Each model received 52 photographs that had not been published anywhere online. The frames came from Bellingcat staff and the volunteer community. Metadata was stripped. There were no significant landmarks, and the set included snow, fog, and night. The prompt asked for one coordinate pair, the photographer’s position. No list of guesses, and no confidence score to hide behind. Models could use the provider’s own tools: search, code execution, and page fetch. Output was capped at 64,000 tokens, on the default effort setting, with 15 minutes per photo. One rescue prompt was allowed if the first answer came back without coordinates.

That is a harder test than a Street View game. It is also the right comparison for a live case.

The current models are better at the first move. Bellingcat reports that the strongest of them placed more than half of the 52 photos within 25 kilometres, and that several models put at least 10 of the 52 inside one kilometre. Claude’s Opus and Fable sat at the top of the table on either score. Gemini 4 Argon led the Mean Log Score, the metric built so the last few kilometres count. A GeoGuessr-style score treats 500 metres and 5 kilometres as nearly the same result. On a real job those two answers are different buildings.

The last stretch is where a geolocation is actually made: matching features and checking perspective, often on a bad frame, often against a second source. Bellingcat’s write-up says none of the models was foolproof, and that a model can state a wrong position with confidence. In a military or intelligence setting, that miss carries a different cost than a hobby map. Bellingcat also notes that Anthropic’s public comparison used GeoGuessr and images that were already online, including Street View: daytime, navigable, and absent from large parts of the world. Anthropic’s own caveat, as Bellingcat reports it, is that this does not clearly match the job of pinning a location in near real time in a less populated battlefield. Bellingcat’s set was unpublished on purpose, so a strong score is harder to explain as a photo the model had already seen.

Bellingcat also tried a supercharged agent on Claude Opus: extra instructions, satellite tiles, and a lookup against their OpenStreetMap search. On a photograph at the Global Campus of Human Rights in Venice, 11 of 17 off-the-shelf models landed within 5 kilometres. The customised agent, and GeoSleuth, were the ones that named the camera position. The same write-up says those runs are not a straight comparison with the off-the-shelf table, because the timeout and the toolset were different, and that the boosted models still got locations wrong.

Let the model throw away the wrong continent and the wrong city. Then do the last check yourself, on imagery you can defend, with a method you can explain. If the pin is unverified, it is a suggestion, and a suggestion is where the work starts.

The score table and the method notes are in Bellingcat’s piece, Can LLMs supercharge geolocations?.