Learning About the GeoTIFF File Format with a Hex Editor (2018) (opens in new tab)

(medium.com)

93 pointsoldpatricka4y ago18 comments

18 comments

This is a good hex editor overview, but also the part about Cloud-Optimized GeoTIFFs is quite good. In the past couple of years COGs have really revolutionized the satellite imagery industry. Even USGS has seen this is the future and publishes the official Landsat data in Cloud-Optimized GeoTIFF format [0].

I've been leveraging COGs recently to quickly bring satellite imagery into the browser for analysis [1].

[0]: https://www.usgs.gov/core-science-systems/nli/landsat/landsa...

[1]: https://www.unfolded.ai/blog/2021-04-28-raster-layer/

alleycat50004y ago

We use COGs at work with mapserver (and GDAL's /vsis3/ magic) to serve up imagery internally for qa/qc:

https://github.com/pedros007/mapserver-docker

Literally saves hundreds of thousands of dollars a year by avoiding having to stage the imagery locally before serving it out!

punnerud4y ago

Could we build a static “Google Maps” using really big GeoTIFF files and downloading part of them with the section of a file functionality.

Just like the SQLite example: https://news.ycombinator.com/item?id=27025829

gregsadetsky4y ago

I think that you meant to link to https://news.ycombinator.com/item?id=27016630 instead which was the recently posted SQLite/SQL.js/HTTP range library

And there was a specific sub-thread in the conversation that mentioned using this for maps too!

https://news.ycombinator.com/item?id=27020372

punnerud4y ago

Thank you, a bit too fast there

kylebarron4y ago

You could, you'd just have to be wary of how large the header metadata would be. If you had a single Cloud-Optimized GeoTIFF of the world with internal tiles up to zoom 14, the header metadata could be a few MB, which wouldn't be ideal to load in every client.

simonw4y ago

Sad to say it but these days loading a few MB to a client is par for the course. I don't think it's a reason not to try and get this working - the range header trick does sound particularly relevant to this format.

UPDATE: Just read https://kylebarron.dev/blog/cog-mosaic/overview which is excellent and clearly you're already very on top of the range request mechanism!

gregsadetsky4y ago

The posts on this topic on your site are really fascinating!

https://kylebarron.dev/blog/cog-mosaic/overview

https://kylebarron.dev/blog/cog-mosaic/naip

1 more reply

kylebarron4y ago

bdon4y ago

Yes, though for zooming in to sufficient detail you will need recursive indexes if you don't want the initial fetches to be really huge.

https://github.com/protomaps/pmtiles#specification

darekkay4y ago

I recently did something similar with Foobar 2000's playlist format [1]. It was a fun learning experience :)

[1] https://darekkay.com/blog/foobar2000-playlist-index-format/

oldpatrickaOP4y ago

Ah wow this is great! Very clear explanation. I particularly liked that there are fields that you couldn’t identify but you were still able to get the information you needed!

ivanhoe4y ago

speaking of this, what hex editor would community recommend for examining (potentially quite large) binaries?

gspr4y ago

I haven't gotten my hands dirty with it myself yet, but Poke looks super interesting!

https://www.jemarch.net/poke

https://media.ccc.de/v/ASG2019-127-gnu-poke-an-extensible-ed...

Actually, it sounds like a fun weekend task to redo this GeoTIFF writeup in Poke :-)

1vuio0pswjnm74y ago

I typed .doc without a TOPIC and poke segfaulted. Oh well.

The hex editor I rely on for multi-GB files is ired.

MontagFTB4y ago

On the Mac, I cannot recommend Hex Fiend enough: https://hexfiend.com/

redrobein4y ago

ImHex

j / k navigate · click thread line to collapse

18 comments

kylebarron4y ago

I've been leveraging COGs recently to quickly bring satellite imagery into the browser for analysis [1].

[0]: https://www.usgs.gov/core-science-systems/nli/landsat/landsa...

[1]: https://www.unfolded.ai/blog/2021-04-28-raster-layer/

alleycat50004y ago

We use COGs at work with mapserver (and GDAL's /vsis3/ magic) to serve up imagery internally for qa/qc:

https://github.com/pedros007/mapserver-docker

Literally saves hundreds of thousands of dollars a year by avoiding having to stage the imagery locally before serving it out!

punnerud4y ago

Could we build a static “Google Maps” using really big GeoTIFF files and downloading part of them with the section of a file functionality.

Just like the SQLite example: https://news.ycombinator.com/item?id=27025829

gregsadetsky4y ago

I think that you meant to link to https://news.ycombinator.com/item?id=27016630 instead which was the recently posted SQLite/SQL.js/HTTP range library

And there was a specific sub-thread in the conversation that mentioned using this for maps too!

https://news.ycombinator.com/item?id=27020372

punnerud4y ago

Thank you, a bit too fast there

kylebarron4y ago

simonw4y ago

UPDATE: Just read https://kylebarron.dev/blog/cog-mosaic/overview which is excellent and clearly you're already very on top of the range request mechanism!

gregsadetsky4y ago

The posts on this topic on your site are really fascinating!

https://kylebarron.dev/blog/cog-mosaic/overview

https://kylebarron.dev/blog/cog-mosaic/naip

1 more reply

kylebarron4y ago

bdon4y ago

Yes, though for zooming in to sufficient detail you will need recursive indexes if you don't want the initial fetches to be really huge.

https://github.com/protomaps/pmtiles#specification

darekkay4y ago

I recently did something similar with Foobar 2000's playlist format [1]. It was a fun learning experience :)

[1] https://darekkay.com/blog/foobar2000-playlist-index-format/

oldpatrickaOP4y ago

Ah wow this is great! Very clear explanation. I particularly liked that there are fields that you couldn’t identify but you were still able to get the information you needed!

ivanhoe4y ago

speaking of this, what hex editor would community recommend for examining (potentially quite large) binaries?

gspr4y ago

I haven't gotten my hands dirty with it myself yet, but Poke looks super interesting!

https://www.jemarch.net/poke

https://media.ccc.de/v/ASG2019-127-gnu-poke-an-extensible-ed...

Actually, it sounds like a fun weekend task to redo this GeoTIFF writeup in Poke :-)

1vuio0pswjnm74y ago

I typed .doc without a TOPIC and poke segfaulted. Oh well.

The hex editor I rely on for multi-GB files is ired.

MontagFTB4y ago

On the Mac, I cannot recommend Hex Fiend enough: https://hexfiend.com/

redrobein4y ago

ImHex

j / k navigate · click thread line to collapse