Ouch! Is there any plan to reduce the amount of ram or it's impossible?
I'm currently building a tile server with Europe's map, the PBF is 28GB, I have 32GB ram.
Also you could try switching from the default --nodemap-type=sortedtable to --nodemap-type=sparsearray. Sorted table uses 12 bytes per node and is more compact for extracts, sparse array uses 8 bytes per node but wastes some storage when there are gaps in the ID space.
There is also an osm.pbf extension that embeds node locations in ways, but you need just as much RAM to convert a file to that format: https://docs.osmcode.org/osmium/latest/osmium-add-locations-...
Once you generate the mbtiles file, you can serve from a much smaller machine.
I’m open-sourcing Flatmap today. Let me know what you think, or if you have any suggestions!
[1] https://github.com/openmaptiles/openmaptiles/issues/654#issu...
Or could I even achieve this via the config file? https://github.com/onthegomap/flatmap/blob/main/config-examp...
I maintain an openmaptiles pipeline, and we've got our planet runtimes down to about ~4 days on AWS.
Would be excited to see if we could use this tool, but we really heavily depend on PostGIS data manipulation to ensure that the data we want ends up in our output tiles. How difficult do you think migrating this SQL to your tool would be? It seems like you can support modifying the data fairly heavily, both before and after MBtile creation.
That second call lets you manipulate vector features on each tile using JTS geometry utilities (i.e. merge nearby polygons or lines with the same tags). PostGIS uses GEOS which is ported from JTS so you have access to pretty much the same geometry utilities - often with the same name.
It's definitely not as flexible as a PostGIS based solution - especially if you join faraway features that don't appear on the same tile, but in practice it was enough to port the entire OpenMapTiles schema.
See the basemap layers package: https://github.com/onthegomap/flatmap/tree/main/flatmap-base... for example see the landcover layer https://github.com/onthegomap/flatmap/blob/main/flatmap-base... which was ported from the SQL contained in https://github.com/openmaptiles/openmaptiles/tree/master/lay...
Also, this OpenMapTiles PR might help improve your existing pipeline after it gets merged: https://github.com/openmaptiles/openmaptiles-tools/pull/383
Edit: formatting
There's an ongoing effort to research vector tile solutions for openstreetmap.org. I wonder how your software stack compares to tilemaker, which has seen some optimization lately.
Edit: benchmark link https://github.com/systemed/tilemaker/issues/315#issue-99432...
Googling "flatmap" gives a bunch of bad results. There is a value in having unique names.
with z13 building merge (extra 14 CPU hours of processing):
- $4.68 on Digital Ocean 128GB 16 CPU instance (4 hours @ $1.17/hr)
- $2.75 on AWS EC2 c5ad.16xlarge instance (1 hour @ $2.75/hr)
- $2.88 on Linode 128GB 50 dedicated CPU instance (2 hours @ $1.44/hr)
Edit: formatting
I hacked around with OpenMapTiles to generate vector tiles in Arctic, Antarctic and WGS84 projections (the latter with 2×1 tiles at zoom 0) [1]. The time spent waiting for PostGIS to do reprojections etc made development very slow, and I've put off updating the tiles.
This tool is reprojects from WGS84 to Web Mercator while reading the tiles, so optimistically changing that would be much of what's needed.
[1] See the map at the bottom of gbif.org, there's a button to change the projection