I'm just finishing spinning up OpenStreetMap servers in EC2 as one of the last stages of our testing, and they look great. We almost went with Mapbox, which is probably a pretty good alternative. But as part of that move I decided to try OSM tiles and I had them up and running in an afternoon via a docker image, created an Ansible role for deploying virtual machines that evening, and had tiles for our region built the next night after tuning some CartoCSS for our needs.
We already have our own aerial tiles, built off the free NAIP imagery, and I just went through rebuilding the 2017 NAIP tiles last week to make them higher quality and fix some seam issues.
This combination is pretty good and lets us take control of our map tile destiny.
I totally recommend hosting your own tile-server using open street map if you have the resources. Creating your own tile-server is not as difficult as it may sound however it is really resource intensive, especially if you want to cover large areas. For a single EU country it shouldn't be that resource intensive.
I have set it up on a Centos 7 server using more or less the instructions from here https://switch2osm.org/manually-building-a-tile-server-16-04... (yes they are for Ubuntu but you'll get the idea) and everything works great. Even if you don't really need it, I recommend trying it to understand how it works; it has some very intuitive ideas.
Beyond the tile server I am also proposing GeoServer (http://geoserver.org) for hosting the geo points on the maps (it can integrate with PostGIS and various other datasources and output the geo points in various different formats). You can then use Leaflet (https://leafletjs.com) to actually display the map and points!
The fastest way, if you don't have resources and don't mind being up to a few months behind, is to use OpenMapTiles vector tiles [1].
The download is an SQLite database with a standard-ish structure (MBTiles), you can either write a small server or use theirs.
I needed vector tiles in other projections — WGS84 equirectangualar, an Arctic one, and an Antarctic one. Generating those took many hours on a cluster.
That's correct — I'd forgotten that, as my project has the open data use license.
However, as an example, the one-time $40 for Colorado is likely to be much cheaper than generating them yourself, and is probably adequate if you don't need to keep up to date.
We are looking at moving to Vector tiles after the deployment of openstreetmap raster tiles is completed. We need to look at the performance closely though because on some devices it was slower with Vector tiles. I think we still want to go that direction we just have to quantify some of the performance.
You can render raster on the fly from a vector based tileserver, and optionally use vector maps for clients where it makes sense (mobile for example).
I've been running a large OSM raster rendering stack for a while, for the whole planet. Moving on to a vector based stack right now, which is what you should do if you are starting today. The reason being the server, with vector tiles pre-rendered, can very quickly render raster tiles on the fly. It's fast enough that for most cases, there is no need to cache them. Once I get the new stack fully burned in, we'll swap our raster rendering stack out for the new vector stuff and do raster on the fly. I might cache the rasters depending on throughput, but from what I can tell it should be basically a non-issue.
The difference between raster tiles and vector tiles is that in raster tiles the rendering (conversion to actual images) is done on your server (either when each tile is loaded for the first time or you may initialize a bunch of zoom levels for your area yourself; this may take hours to days depending on your area size) while for vector tiles you just send the descriptions/shapes and the inages are created on the client side. It is expected that the client performance won't be as good as with raster tiles and slow clients like mobile phones will struggle while trying to render the images.
One interesting idea would be to have both raster and vector tiles; then in your client side code you could detect if the client is a mobile device or a desktop pc (I am not sure if you can detect more things about how speedy the device would be) and serve the corresponding tileset (ie the vector tileset if the device is desktop or the raster one if it is mobile). This way you will definitely decrease the load on your server and most clients won't have performance problems!
In any case, the clients will get faster as the time passes so after some years vector tiles would probably be the norm because they have the huge advantage that it is very easy to style them on-the-fly in any way you like (while, to style raster images you need to re-create them all from scratch).
This results in having a tile server that generates & serves raster tiles using:
* carto to generate a proper stylesheet
* mapnik to create the tiles
* apache with the mod_tile module to serve the tiles
* renderd to be used as a bridge between apache/mod_tile and mapnik
* Postgres/PostGIS as a database to host the data
The actual data is an OpenStreetMap pbf file downloaded from http://download.geofabrik.de/ which is inserted to the database through osm2pgsql.
The above are used for the raster tile server only.
In addition to this, I am using geoserver (http://geoserver.org/) which is a Java web app (runs in tomcat) that is used to view/edit and (mainly) serve geospatial data. What I usually do is that when I need to display a new shape (shp) file I use shp2pgsql to insert it to a table in my PostGIS database (using something like this shp2pgsql -s SRID SHAPEFILE.shp SCHEMA.TABLE | psql -h HOST -d DATABASE -U USER) also another good tool is ogr2ogr which can convert between various formats (it also supports kml, for example ogr2ogr -f "PGDump" koko.sql test.kml).
After the shp file is inserted to the database, I can create a datasource for this dataset from GeoServer and publish this dataset to the web (i.e the dataset will have a URL of the form http://1.2.3.4:8080/geoserver/Dataset/wms); it can then be consumed by client applications (in javascipt) using leaflet or openlayers. Notice that GeoServer can easily publish various file formats for example an SHP directly (you don't need to insert it to the PostGIS; this is something I do for better control of my datasources).
PostGIS is the component that supports the vector tiles (http://docs.geoserver.org/latest/en/user/extensions/vectorti...) so I want to integrate it the OpenStreetMap data that is hosted in my PostGIS (which is used for the raster tiles); I haven't found the time yet though.
I think it's a great setup and it works great for the needs of the organization I work for; actually, I like it so much that I want (for a long time) to write a rather comprehensive tutorial on how to set ip up along with some notes on how to do various PostGIS things but I don't know when (and if) I'll find the time for something so large :/
My apologies, I'm out-of-date with the commercial offerings of OpenMapTiles — I work on an open data project, eligible for OpenMapTiles' free license which has old (currently one year old) tiles.
With a commercial license, the tiles are from 2 July.
I assume the commercial tiles are a few days behind as it's just not worth most people's effort to stay up-to-the-hour — different map services can offer that, this company is making easy self-hosted map tile packages.
The free license is presumably using older data to encourage an upgrade.
Tile generation wasn't THAT bad. We only cover Colorado and Wyoming, and generating the tiles for those at the zoom levels 5-18 takes around 7 hours to pre-render (using "render_list_geo.pl"). The end result is around 25GB of rendered meta-tiles.
Much better than the NAIP aerial imagery, which takes around 3TB for the source data, 280GB for the resulting tiles, and around a full day to generate.
Yes it wasn't bad for a single area (we cover only Greece and it was quick and easy for such a small country). However I couldn't imagine of the resources needed to cover the whole world!
If you use vector tiles, and render them client side you could do this. For raster tiles, where you're storing images, there are too many images to do this practically.
And then you need to think of how to do data updates. OSM updates minutely.
Data updates are not required for many applications. You probably don't want to be google maps! For example I haven't updated the OSM data for more than a year and I don't think that there are any problems with the data being stale. It all depends on the application of course.
One major advantage of not updating your data very often is that the tile-images won't need to be re-generated thus you save a lot of (CPU intensive) resources! I think that another commenter mentioned that he generated the tiles in a CPU intensive system and then just off-loaded the images to not-so-fast system for serving.
You can also prerender the tiles on a powerful machine and host them on a cheap instance (assuming you don't need very large regions). A while ago I created an OSM server (https://github.com/seemk/TileSweep) to do this for me, you can select regions you want to render via a polygon and you get a SQLite database of the tiles. It can also run as an alternative for mod_tile + renderd.
I rendered all of Estonia from zoom levels 0 to 18 (~16.5M tiles) which is around 20GB and host the tiles on a cheap DigitalOcean instance.
* How is the workflow for updating OSM data with this project?
* What is the recommended procedure for backup + restore?
I find a lot of projects in this space make you go through an manual copy-paste setup process like if it was a one-time thing - it is not - updating OSM data is a crucial part in this process and should be part of it.
I haven't investigated updating of data using anything clever. I know the database load script can write out a list of tiles that need to be rerendered, which can be fed into the renderd process. Renderd has multiple priority channel so bulk renders are a lower priority than user views, so a live system can update the tilecache without much easier impact.
I haven't investigated it because we are probably moving to Vector tiles over the next month or two, and we don't need up to the minute map updates. On one of our sites, we have been rendering navteq tiles and we only get updates for those quarterly. It was only our public site that we were using Google Maps for.
Backup and restore we are just doing via normal backup programs. The tilecache is stored in meta tiles, which is a d duplicated set of 64 tiles in a single file. So we are only dealing with about a million files in a directory structure totaling 25 gigabytes.
If we needed to restore or update our tiles, we would probably just spin up a new box and rerun ansible to deploy it. I have it all encoded Within a Playbook.
I was wondering too why they skipped OpenStreetMap so quickly.
Living in Germany, I find the map quality of OSM much better than Google Maps. Sure, it lacks the integration with the other Google services like a good search and live traffic, but speaking about the pure map quality, Google is lacking a lot of detail here.
Some options we could reject quickly for various reasons. OpenStreetMap is not supposed to be directly used by commercial sites. Apple Maps, even though we wouldn't mind having Apple logo on our site somewhere, was just released as beta and requires Apple developer account to test properly."
The OpenStreetMap.org website doesn't ban commerical sites (per se). The volunteer sysadmins who manage the donated hardware that runs the OpenStreetMap.org website don't offer any SLA and reserve the right to ban people/apps which put too much load on it. If you don't like it, they're prepared to give you a full refund.
I read that, but compared to the effort they put into analysing the whole issue, at least an estimate about the cost involved in hosting their own server would be a good decision.
It seems like they rejected OSM because they couldn't directly use the OSM site to serve tiles, but would instead need to host their own infrastructure for it. I understand that, but for us we decided to go ahead and own it.
I think if the maps service is part of your core product, it is good considering a self hosted solution. Yes, it costs time you want to save as a startup, but at least you control your own product and can manage your cost directly.
We found some references saying that each box could probably handle around 10,000 map tiles per second. Our full tileset is under 30GB, so it is easily cached. One of our machines can generate Tiles at a rate of 2,500/sec, another is 1,300.
They are both virtual machines, 32GB and 32 CPUs for the 1,300/sec and 20CPUs for the 2,500/sec. The 32 cores are on a host machine with two "Intel(R) Xeon(R) Silver 4108 CPU @ 1.80GHz", the 20 cores are "Intel(R) Xeon(R) CPU E5-2637 v2 @ 3.50GHz". You know, 7,001.21 bogomips (TM). :-)
How does Mapbox stack up against what you've built? Does Mapbox provide any advantages or is are the two not very comparable? How difficult would it be to spin up a Mapbox competitor?
How does Mapbox stack up against what we've built? Well, they have way more people working on the problem, so there's going to be more support and more eyes on the problem. I didn't directly compare map tiles, that was a couple other people on my team. We decided to try moving forward with our own OSM-based tiles, so the conclusion I've come to is that it compares favorably.
How difficult would it be to build a competitor to Mapbox? Well, if all you care about are map tiles, it's no big deal. If you want to build a business around it, that's always work. You're going to need to build the tiles, have infrastructure to serve it, monitoring, staff to manage it, billing and charging infrastructure, etc...
Mapbox also offers geocoding and navigation, which are quite a bit more difficult than just setting up a server that can host OSM tiles. Obviously, if all you need are maps, that's fine. But if you need maps with search, which I suspect most people do, rolling your own is tougher (though projects like pelias are pretty good).
I set up a pelias system a while ago but it never got any traction. We have another geocoder already set up using Experian QAS data, I think we've had that set up for a decade or more, so that isn't one of our concerns. Good point though.
Uh, while I upvote OSM comments a lot because I often think they are good comments, blindly upvoting based on positive mentions of OSM does not contribute to the conversation...
I only came to the comments to upvote whoever mentioned OpenStreetMaps. It needs our love.
This is the platform Maps.me is based on, correct? I ask because I downloaded Maps.me the other day, after reading HN, and have found it to be missing most businesses that I've attempted to search for. In addition, there seems to be no obvious way to suggest missing destinations.
I too want Google/Apple maps alternatives to succeed, but my limited trial so far has not been promising.
> I ask because I downloaded Maps.me the other day, after reading HN, and have found it to be missing most businesses that I've attempted to search for. In addition, there seems to be no obvious way to suggest missing destinations.
>there seems to be no obvious way to suggest missing destinations.
Unless Maps.Me use their own database for business listings, you just edit the map directly from openstreetmap.org or one of the many fat client map editors that are out there.
We already have our own aerial tiles, built off the free NAIP imagery, and I just went through rebuilding the 2017 NAIP tiles last week to make them higher quality and fix some seam issues.
This combination is pretty good and lets us take control of our map tile destiny.
References I used in setting things up:
https://github.com/zavpyj/osm-tiles-docker https://switch2osm.org/ https://ircama.github.io/osm-carto-tutorials/tile-server-ubu...