New OpenCycleMap tileserver
It's been about a month since I launched with the new OpenCycleMap tileserver. The transition was actually much more hectic than I expected (I'll describe the EC2 failure some other time) but things have stabilised a bit since and are humming along nicely.
So what's new? Almost everything, actually, although mostly things you don't get to see. New hosting, for a start, on a dedicated server with both huge SATA disks (mirrored 1.5Tb) and also a pair of blindingly fast 128Gb striped SSD drives. The postgis database is on the fast disks, everything else being on the SATAs. The speed of the SSD drives really comes into its own while rebuilding the indexes during hourly updates, rather than being directly related to rendering.
I've also upgraded all the software behind the scenes - new versions of mod_tile (with much better monitoring), renderd, mapnik, osm2pgsl. The biggest change that you'll all notice is the tiles are now much smaller - around a third of the size - due to using palletised .png files. The new version of mapnik gives me the raster-handling needed to keep the background colouring at all zoom levels, which is a huge improvement. The biggest change I notice is that we're now using osm2pgsql in slim mode - allowing incremental updates to the database instead of weekly drop/reload - and all of the fancy route-relation handling is now built into osm2pgsql instead of being post-reload SQL queries. It's all much easier to deal with.
The whole upgrade project was delayed somewhat by the initial new server not working out. I'd specced out what I though was an appropriate machine, and everything worked fine - loading the database, testing rendering, the works. Unfortunately the diff updates were, on average, taking around "equal time" - an hour and a bit for an hours-worth of OSM changes, about 20 hours for a days worth of changes - and this with the machine doing nothing else! Disaster. So I scrambled to find a new provider with better hardware (at a reasonable price) and started again with the setup. Hat tip to Hetzner.de for their server offerings, and of course my ex-employers CloudMade for their continued sponsorship of the hardware.
Finally, I managed to sneak in some cartography changes while I was waiting for various things to complete (importing contours for the whole planet takes a few days!). Most of them were fairly subtle, and a fair few have gone unnoticed. Buildings are my new favourite, along with finally making new shield icons to replace the (ugh) nasty old ones (there's a story behind them, as with most things). Can you spot any of the dozen other changes?
What's next? I'd like to say more cartography updates, since there's lots more I'd like to change. But cartography always seems to take a backseat to server administration, and that's, as usual, starting to cause headaches. The huge numbers of people using the opencyclemap tileserver means that there's many millions of tiles to render, and the server is only - just about - keeping up with refreshing them as the underlying data changes. Something needs to be rethought about this, and I'm hoping to find some speed improvements from somewhere to render the tiles faster and let the server keep its head above water. The server troubles are massively exacerbated by thoughtless people out there (abetted by thoughtless software developers) who are bulk-downloading, or scraping, the tileserver for tens of thousands of tiles at a time - approximately none of which they ever view. Plans are afoot to deal with them though!
This post was posted on 8 September 2010 and tagged General