Smoother API upgrades for OpenStreetMap
Recently, my coding attention has drifted towards the OpenStreetMap API. The venerable version 0.6 API celebrated its 10th birthday earlier this year, so it's had a good run - clocking up double the age of all previous versions combined!
Years ago, whenever we wanted to change the API, we gathered all the OSM developers in one room for a hack weekend, and discussed and coded the changes required there and then. For the most recent change, from 0.5 to 0.6, we had to do some database processing in order to create changesets, and so we simply switched off the API for a long weekend. Nobody could do any mapping. And everyone had to upgrade all their software that same weekend, since anything that used the 0.5 API simply stopped working.
In short, I don't think we can use this approach next time! Moreover, I think this 'big bang' approach to making API changes is actually the main reason that we've stuck with API version 0.6 for so long. Sure, it works, and it's clearly 'good enough'. But it can be better. Yet with such a high barrier to making any changes, it's not surprising that we've got ourselves a little bit stuck on this version.
So I've started work on refactoring the codebase so that we can support multiple API versions at the same time. There's a bunch of backwards-incompatible changes that we want to make to the API, but since those are not fundamentally changing the concepts of OSM, it makes sense to run version 0.6 and 0.7 at the same time. This gives us a smooth transition period, and allows all the applications and tools that use the current API version a chance to upgrade in their own time.
SotM 2019 was in Heidelberg last month, and among other things gave me a chance to talk face-to-face with lead developers from four different OpenStreetMap editors. The editors are the key pieces of software that use the API, so this was a rare chance to check in and find out what changes they would like to see. I spoke with developers of Vespucci, iD, JOSM - and of course Level0! My task now is to figure out which of their requests and suggestions are blocked by having to break API compatibility, or which ones can be implemented in the current API version.
One example of a breaking change is moving to structured error responses, a topic Simon Poole raised with me straight away. Often the API returns a human-readable error message when there's a problem. For example, during a changeset upload, the API could return "Precondition failed: Way 1234 is still used by relations 45,3557,357537". This isn't particularly handy, since editors then need a bunch of regular expressions to try to match each error message, and to carefully parse the element ids out of those human readable sentences. Changing these into structured responses would be more useful for developers, but that requires a version increment.
Another change that I am particularly keen to see, but perhaps not so keen to code, is to make diff uploads more robust. They currently suffer from two main problems. The first is that the diff can be large and therefore complex to parse and apply to the database, and so it can take a long time to receive a response from the API. For mobile editors, in particular, it can be hard to maintain an open connection to the server for long enough to hear back whether it was successfully applied to the database. So I'd like to move to a send-and-poll model, where the API immediately acknowledges receipt of the diff upload, and gives the client a URL that it can poll to check on progress and to receive the results. That way if there's a hiccup on a cellular connection, the editor can just poll again on a fresh connection.
Relatedly, the second problem is what the editor should do if it never hears back from the server after sending a diff upload. If the mapper has added a hundred new buildings, presses save, and the editor gets no response from the API, should the software try again? If it doesn't, then the mapper looses all their work. If it tries the upload again (and again?), it could be creating duplicate map data each time. It's impossible to know what happened - did the request get lost on the way to the server, or did the server receive it, save it, and it was the response that got lost? My proposal is to include an idempotency key in the diff upload. The server will store these, and if it sees the same upload key for a second (or third) time, it will realise the response has been lost somewhere, and no harm is done. A similar approach can then be taken with other uploads, like creating notes or adding comments, to allow safe retries for those too.
There are many other upgrades that I'd like to make to the API, focussed on a broader strategy of making life easier for editor developers. But without the ability to run multiple versions in parallel, none of these changes are likely to happen. So that's my first priority.
This post was posted on 17 October 2019 and tagged development, OpenStreetMap