Things I've learned maintaining OpenStreetMap (LRUG presentation)
On the 10th June 2024 I gave a presentation to LRUG, the London Ruby Users Group, about being a software maintainer for the OpenStreetMap website codebase. The description of the talk was:
Maintaining one of the world’s largest non-commercial websites, OpenStreetMap, is a unique challenge. We’re a small, volunteer-based development team, not professional software developers. I will illustrate some of these challenges with a mixture of technical and organisational tips, tricks and recommendations, that you might find useful for your own teams and projects too.
Here’s a video that the team at LRUG kindly made (32 mins) which has the audio and the slides, and below that you’ll find the transcript.
Transcript
Hi everyone, my name is Andy. Every Wednesday is my volunteering day, it’s when I step away from normal work and I do some volunteer development work. So in the mornings I head to my desk, I open up the OpenStreetMap website github repo page and I have a quick check to see if there’s any pull requests that need my review.
Yeah there’s a few, there’s a few more, in fact there’s loads. At the moment we have 137 open pull requests, more than 500 open issues and you might be thinking the same as me which is where on earth do I even start!
So I’m Andy Allan, I’m one of the two volunteer maintainers for the OpenStreetMap website project.
Actually the other maintainer, Tom Hughes, has come along this evening so I need to be very careful about what I say as he’s an expert on things. When I’m not doing volunteer development work I’m the founder of a company called Thunderforest which builds commercial services based on OpenStreetMap data. This evening my talk is split into three parts, I’m going to give you background about OpenStreetMap, and also some background about the OpenStreetMap website project. I’m going to look at some of the challenges that we face, the organisational approaches that we use to deal with that, and then finally some of the technical implementation details which is great to be in front of a technical audience for a change so I actually get to put some code on the screen today.
So let’s start with some background.
OpenStreetMap, if you haven’t heard about it already, is a global volunteer open data mapping project, where we go out and we map all the details about the whole world from scratch.
The data gets used by hundreds of different websites and applications, businesses small like mine or large like Amazon, Apple, Facebook, they all incorporate OpenStreetMap data into their various mapping products.
Lots of people think of it as either a technical project or a data project but in my mind it’s actually a community project. It’s building the thousands of volunteer mappers who are willing to spend their time going out and mapping the world, and on a typical month we have more than 45,000 volunteer mappers who will be walking the streets adding information about their local areas into the system.
The part of this project that I’m talking about tonight is the Ruby on Rails app that powers the core website. So this is the initial commit by Steve Coast, the founder of OpenStreetMap back in 2006 so those of you with very long memories will realise that this is a Rails 1.0 era project that’s been continually maintained for almost two decades. Like most Rails projects, the split covers a few bases.
The first obviously is the website. We also have an XML and JSON API which allows third-party editors to edit OpenStreetMap and the project also contains the definitions for the core database of the project. It’s kind of hard for me to give an overview of such a large project in a small amount of time so here’s just a list of the models in the Rails app with some of the more interesting ones highlighted.
So as well as the OpenStreetMap data we have lots of community features, like we have user diaries, we can comment on changesets, we can block users, report, spam, all these kinds of things. If you’re familiar with the front page of OpenStreetMap, it’s worth me at this point mentioning that a lot of the stuff you see on the front page is not maintained by me and Tom, they are separate projects. So the rendered map images, the search, the routing engines, even the built-in editor and some of the other built-in tools are all separate projects, but the OpenStreetMap website is the one that pulls everything together.
So we have these user-facing separate projects that have some of the functionality of the site. We also have some behind-the-scenes external projects which are to do with speeding up the API, or distributing the data including publishing the data every minute. But like I say the OpenStreetMap website project ties everything together.
To give you an idea of the activity we’ve had 14,000 commits from over 200 individual people in the last 18 years. We’ve got a few thousand tests. Because it’s a global volunteer project it’s important that it’s translated so we have over 100 languages that the website is translated into.
I was shocked when I made that stat but it’s almost 2,000 different people have been involved in translating the website into these languages and we have just under 3,000 source strings in our en.yml file which leads to almost 200,000 translations and these are done by a separate project and automatically copied over into our repo for publication.
So the main challenge that I want to talk about tonight is that this is not developed in the same way as your typical commercial project. We don’t have a big team of people. We have two volunteer maintainers and a few volunteer developers. And for projects this size, this important, this heavily used, that’s quite different from your typical setup with teams of professional software developers, product managers, designers and so on in the background.
And our volunteer developers come along and make one of these pull requests.
It’s worth bearing in mind that there’s no candidate screening, there’s no technical interviews, no onboarding, no pair programming, no senior developers to point them in the right direction. It’s quite different. And because also they’re volunteers, regardless of technical ability, they also have their own priorities. They have limited availability of how many hours a week they’re likely to dedicate and this has implications for the kind of features that we can develop, and how we want to maintain the project. So I want to dig into this a little bit and see some of the techniques that we use in order to handle the fact that all of our development is done both by volunteers and by people of varying skill levels.
One of the key things I like to focus on is making the developer experience as robust as possible and that involves a lot of linters.
So most of you are familiar with RuboCop and its sibling gems. We also use strong migrations which can catch a migration that might block one of these super huge tables if it needs an exclusive block. We use ERBLint for linting the view files. We use Brakeman to do security testing. We use something that you can barely see on the screen called Annotate which is a nice little gem. It puts comments describing your database tables into the model files and it makes it super easy for people to get started to know what the model does, much more so than digging into the database.
And we put all these into CI. And for those of you who think of RuboCop or any of these linters as a kind of like strict teacher making sure your double quotes are not single quotes or that the spaces in your hashes are consistent, that’s not really how I like to think of it. I like to think of it more as a form of automated code review.
Those kind of syntax layout things are not as interesting as the cops which will teach you something. And the great thing about putting these in CI is that our developers will get instant context specific feedback on the code that they’ve just written. And to me that’s much more important than expecting somebody to have read a Rails book or been on a tutorial or read something general because this is about specifically what they’re focused on.
So for example a Capybara RuboCop thing that might tell you how to make it work with asynchronous methods or going back to the example about database migrations, it turns out loads of the database migrations are safe if you’re doing specific things on specific versions of Postgres. And not only do I not expect the developers to know this, I don’t know it either. So this is super useful for me when I’m doing code reviews to know that I’ve got the backup of all of the collective wisdom of all of these different tools.
The second thing that I like to do is to really concentrate on refactoring. And that’s because our developers tend not to be Ruby experts who are starting work on OpenStreetMap. They’re OpenStreetMap experts who are learning how to do Ruby and Rails by reading what’s here already and copying what we’ve done in the past.
And for an 18-year-old project there’s a lot of things in various corners of the code base which is not how we would do it anymore. So it’s important to refactor this first, otherwise we just get pull requests with - aargh, wouldn’t merge that if I was you.
And the third thing is that we work hard to make sure we follow Rails standards and conventions because we don’t have much internal documentation.
We don’t have much time to explain to people how things work and what they do. So we rely heavily on the fact that if we follow the Rails conventions then there are tutorials, there are the guides, there are YouTube videos which will also explain things. So we shy away from doing our own curious things and try and stick to the main ones.
So for example we got rid of our own file handling to use Active Storage. We got rid of an external job queue in order to use Active Job and things like that. Whenever these new things appear in Rails framework we try and take advantage of them straight away.
When it comes to the technical matters we also need to approach them with the view that again our developers are not Rails experts. And the first topic I want to talk about is output safety. Whenever I’m testing stuff I like to put some HTML into my username because it makes it really obvious where we’ve forgotten to do it.
And when I first did this these horizontal rules started appearing all over the place which let me know we weren’t doing the output safety correctly. If you’re not sure why a horizontal rule isn’t important just imagine I’ve written a script tag instead. That opens you up to all kinds of cross-site scripting attacks.
A bit of history in Rails 3. By default nothing was escaped. So if you just put something into your view template it would show it straight away. You needed to figure out every time you were using anything that was user controlled like their own display name you had to escape it with the h function.
And obviously that leads to loads of places where that gets missed. And so in Rails 4 they switched it around and they created a thing called safe buffers, which means everything is escaped by default. If you have a helper that outputs HTML then it ends up double escaped.
And to help with this migration they created two little get out of jail free cards. One of which was calling html_safe on a string and then that would be escaped again, or the other one is calling raw. Now like I said before because our developers are often inexperienced and these are quite powerful tools, it leads to situations where they can easily make mistakes - especially if they’re copying what they see in one place and using it in another.
So I took a lot of time to go through all of our legacy code and refactor it to get rid of these html_safe safe and raws. And for the most part this was straightforward. The Rails translation system lets you mark translations as “we expect to see HTML in here” and therefore you don’t need to escape them you can get rid of that raw tag.
You can get rid of html_safe from your helpers by using certain things like there’s a safe_join which is aware of how SafeBuffers work and all the Rails ActionView tag helpers do that too.
The final challenge and the hardest one to deal with was flash messages because ActionDispatch is not, the flash system is not aware of SafeBuffers. It only takes Strings, Arrays or Hashes, and so if you have a flash message which has some HTML in it you need to come up with something - you can’t just commit it to flash because it will only take a string and it will think that it’s an unescaped string.
So if you go on to Stack Overflow or anywhere and say how do I put HTML in a flash message they all give something along these lines which is just put it as a string and then when it comes to the view call html_safe on it. Jobs a good’un.
But the problem with this is that html_safe does not escape anything. It’s a declaration by the developer that what you’re handling is definitely HTML safe. There is no possible code injection in this and in this case that’s fine because it’s a hard-coded string. There’s no user information there.
But we had a problem where one of our flashes said that we were blocking users so whenever a moderator blocked a user… Well one day one of the users had some script tags in their name, and the scripts ran. So despite this being what everybody says to do, no don’t do that.
We need another way of doing it which means we are using SafeBuffers and because we can’t do them from the current action to the future action, we can’t pass the safe buffers across, we need to think about using the SafeBuffer approach after the event.
So this is a method I found somewhere else I want to share with you guys which is to make a partial with whatever HTML you want in it. So we have a couple paragraph tags here. We take advantage of the fact that Flash can take hashes and say which partial we want, any locals that we want to pass it to, we can store that hash in the Flash and then a small helper at the end which detects if we are using a hash, calls that ActionView render on that hash and it will do exactly what you want. If you’re just passing strings that’s fine, just use the Flash message as is.
I think this is a really nice technique and I have yet to see it documented anywhere. I found it by digging through some MySociety code base, when they’d had the same problem and I thought - yeah, that works.
Internationalization, as I said before, is quite important for our project and one of the interesting quirks we came across recently actually shows some of the limitations of the Rails internationalization framework which I wasn’t previously aware of so I thought I could share that with you tonight as well.
Many of you have seen this before which is you can use, you can leverage Rails in order to choose what translation you want depending on how many things you have.
So if you have one cat, it will use this translation name that’s a singular. If you have multiple in English, then you can… it’s all the same… ‘cats’.
This is different in different languages. I lived in Poland a couple of years ago and so I learned in Poland they have three different plural forms. You have one for one, you can see that two, three and four have the same pluralization but five changes to a different pluralization. And this can be handled by the translation system, that’s not a problem. It is worth noticing though that not only two “koty” but also 22, 33, these also count as few in Polish.
So it’s more like the number behaves like few. Most things that end in a two, three or four count as few, most things that don’t count as many. So it’s not strictly one, few, many.
And this is important for all the situations where you want some special text for when you have nothing or something. So we have it for comments like there is no comment yet or I need a cat. When we were doing this, I have there are no comments as a separate translation until one day one of our translators popped up and said I can’t translate this into Latvian and that’s because this is not zero as in nothing, this is zero as in numbers that behave like a zero and in Latvian that’s zero but it’s also 10, 20, 30, 100, 140 and so on.
So they couldn’t have a translation which worked for both the special case of zero and also for 10, 15 and so on. And so we dug into this and in the Ruby internationalization framework, they have the standard six translation keys that work with the system - zero, one, two, few, many and other. And we saw most of these already.
But the more comprehensive approach is to have specifically zero and specifically one as translation keys and Ruby internationalization just treats these as aliases of zero and one and that blows up the whole approach to translating in Latvian. So if you want your translations to work, if you want a special message for the zero case, then you have to have a separate translation and then you can do the rest for counts.
Sounds pretty obscure, right? But we don’t want to make the same problem again and we don’t expect any of our developers to know this or understand it. So we made a test and this runs during commits and it checks the en.yml file for any translation keys that start with zero and then it can warn the developer, no, don’t do this because it doesn’t work and then we don’t have to think about it every time we’re emerging pull requests.
Our developers will only be told about this when they come across it. It’s not like something they have to read in the documentation and so again that kind of targeted feedback is super interesting.
The final thing is going to combine those two topics of output safety and internationalization.
When I was looking at this one day and I thought that looks fair enough. We’re using the safe buffer-based approach. We’ve got a translation that we’re expecting html because we want that name to be the link but this is safe buffer aware so it’s doing the escaping.
Everything looks good. It’s hard to understand is there an output safety problem here and then I was looking at a translation. It wasn’t Polish but I was looking at that and I thought wait a minute if we’re permitting html what happens if it’s the translator who puts the dodgy html in there.
Because we have almost 2,000 translators and 200,000 translations. We’re not inspecting these manually. They’re just being automatically brought over.
So I had a quick check and it turned out- it was fine. Nobody else had figured this out before me.
But it did mean that I wanted to make sure that there was no dodgy html in any of the translations and that there never would be. And the easiest way to do that is to make sure there’s no html in any of them at all.
And that was an opportunity to do a lot of refactoring and pull out any of these translation strings where we have html in them to break them down into their parts, have the translations with no html and then use the views to build it up.
And to a certain extent this is a better way of doing it anyway. You’ll find these creep into projects but it’s a separation of concerns issue. You don’t want to have to go hunting in your translations to change paragraph tags or things like that. Of course we added a test.
So this test does two things. One it warns our developers if they ever think “oh I don’t know how to do this properly I’m just going to put a bold tag or a link directly into the translation”. It will fail and it’ll do that.
It’ll also give us a warning if any of our translators are trying something they shouldn’t do, because the build will fail when all the translations are imported.
So that’s about it for me. There was a view on things that we do in our project to deal with our main challenges.
But this is where you guys come in.
I would like your help. Read our contributing, help review our PRs, make new PRs, think about other ways that you guys know about how we can improve the developer experience, other techniques that we could be using.
If you don’t fancy coding on your time off I totally understand that. We could still do with help with issue triage or join our enormous team of translators.
Here’s some LRUG special things because this is the most technical audience I’ve ever talked about this stuff.
We’re part way through replacing the JavaScript with Hotwire. If you’re a Hotwire expert please come and tell me how to do it more easily. Restful controller renaming, super important to make the code clean and to meet developer expectations but I really struggle to come up with good names for certain controllers and certain actions on controllers.
We currently use a C++ utility to speed up our XML and JSON APIs because we can’t get the performance we need out of the Ruby view system. If you include the same partial 10,000 times it takes dozens of seconds to run. If you know how to do XML, big XML, big JSON APIs at scale come and talk to me, or any of your more typical Railsy type things like we want to build a whole notification system, models, database things, it’s a lot. Your help would be very much welcome.
So that’s it for me. Thanks very much. Any questions?
Q [This person is asking about using fake streets (known as trap streets) to understand if people are stealing your map data]
Yeah we do.
So for the benefit of the recording this is about trap streets and map data to see if people are violating the copyright. We deliberately don’t do anything wrong but you can still spot OpenStreetMap data quite easily because it’s never complete. So if you look at a map and it has some of the buildings mapped or some of the buildings mapped accurately and some of the buildings mapped not so accurately, you can easily compare that with OpenStreetMap data and then you know they’re using OpenStreetMap data.
And we can also go back through the history because we record the history of everything and publish the history of everything. We have services which will show you if they took the data six months ago or nine months ago you can do comparisons there as well.
Q [This person is asking which Rails version the OpenStreetMap app runs on]
It’s 7.1.
Yeah the latest version and we test on Ruby 3.0 and newer and that’s so that we have quite a wide range of supported Ruby versions so that a typical developer who might just have whatever the latest Ubuntu LTS is on their laptop can get started straight away. So we try and be as accommodating as possible with Ruby versions, Postgres versions, all that kind of stuff because these are the technical barriers that we don’t want to get in the way of people who might not be that technical to start with.
Q If I wanted to contribute is there a list of like we really want seasoned Ruby developer help on these issues? Like I know a lot of open source projects have this would be good for a first-time contributor type issues so that you want this would be good for a first-time experienced contributor. Is it easy for me who’s never looked at a good app before to identify those?
Yeah it’s really hard to get started with finding the right issues. We’ve tried the good first issue approach which is often not for super experienced people but it tends to attract people who are not actually that interested in OpenStreetMap or the development and more kind of just clicking somewhere and GitHub that points you to the repos. We get an awful lot of people saying could you assign the to me please and then we never hear from them.
The more major projects are not very well highlighted but I would look for the refactor tag on issues because these are generally things where I know there’s some work that needs to be done and it’s a kind of long-term thing it’s not not just you know two-line fix for a bug. But otherwise I would encourage you just to have a start having a look at things and something will pique your interest pretty quickly with 500 different things to work on and I’m sure one of them will be interesting.
Q [This person is asking two questions: 1 - How customised is the OpenStreetMap app vs standard rails? 2 - What is the hardest thing you’ve had to fix
Was the second one the hardest single thing or the hardest thing I had to fix? Well I’ll answer the second one first.
The good thing with the hardest things to fix is that Tom does all the hardest things and I do the easier ones. So when it comes to trying to get OAuth 1.0a to work happily with OAuth 2.0 at the same time it’s beyond me. Tom takes care of that.
The hardest thing that I had to work on was probably either one of those things like the flash messages took a lot of looking around.
Actually trying to import a data set which is published as a node package and pull that in. I eventually found Frozen Record which is a great way of having static models that are not database backed but in a way that still feels quite like Active Record. So again these… The hardest things are usually ones where it seems pretty trivial and then I spend an entire day searching for… “Come on there must be an easier way of doing this”.
And the first one was how much customization stuff. Well a lot less than there used to be, put it that way. I think there’s a few things where like controller naming or the routes - they’re not, it’s not heavily customized it’s just really basic and kind of like ideas that were popular back in Rails 1.0. So we have like a thousand routes which would just get this target and we’ve been slowly trying to make them into resourceful routing instead of just every path being individually handled.
That’s the least like in all the Rails app we’re doing. But apart from that most of the stuff in the app is crud things. Adding users, adding user blocks, this that and the other.
In the wider sense, one of the most interesting things that I haven’t covered, here one of the most custom things is to do with the minutely publishing of the data from the database. To do that consistently every 60 seconds with in-flight transactions and being able to handle the Postgres transactions in mid-flight whilst still publishing all the data.
That’s been something which has been super interesting and lots of different approaches over the last 20 years to keep that working as Postgres versions change.
Q [This person is asking how much upstream contribution OpenStreetMap has made to Rails]
No not big ones usually just small bug requests on things that are upstream that haven’t worked. So the multi-database handling stuff that’s gone into Rails 7 didn’t work for us up until the most recent version of Rails because nobody expects to have 20 years of migrations, and so connecting to multiple databases and old migrations and things that stuff wasn’t working.
I don’t know - Tom have we pushed anything else up?
Yeah we used to be heavily involved in the composite primary keys gem and then Rails now has composite primary keys and so that was great to step away from that. Tom had to do a lot of work upstream with that. I saw another hand a minute ago.
Q: [This person is asking how much data is in the OpenStreetMap database]
How much data is it? Tom, how much data is it?
Yeah, yeah, many terabytes! As far as we’re aware, it’s the largest open source dataset that’s using Postgres, because anything measured in terabytes is generally either just autogenerated data, or commercial. So there has been a few cases where we’ve had Postgres consultants who are interested in what we’re doing. And our lack of horizontal scalability, things like that, it’s still… big machine gets you much further than you might think.
Q [This person is asking about which postgresql extensions the OpenStreetMap database uses]
Yeah, curiously, despite being heavily map based we don’t use PostGIS which is the geospatial extension for Postgres. We don’t use that at all. Partly for historical reasons, because PostGIS was nowhere near as good as it is now, 20 years ago, partly to do with rounding errors, and not storing stuff in floating points which PostGIS likes to do. We have our way of storing coordinates which uses integers. Yeah, just straight Postgres.
[Tom mentions another reason for not using PostGIS - topologically aware data]
Oh yeah, and the other reason for not using PostGIS is that we have a topologically aware data model. And that doesn’t map neatly onto what are called OGC standards for data representation. This is like a whole other talk that I’d need to do.
Alright, I think we’ll leave it there. Thanks very much.
This post was posted on 19 October 2024 and tagged OpenStreetMap, development