Hydrogen leaks in an oxygen atmosphere (like the one here on earth) are pretty dangerous, as they are effectively invisible in daylight, due to almost all the energy going directly to infrared. Due to that NASA test engineers used to walk with a broom in front of them in areas where hydrogen leaks could happen. If the broom suddenly caught fire, there was a burning hydrogen leak in front of the engineer! A picture of this can be seen here:
https://spinoff.nasa.gov/spinoff1997/ps1.html
Not just leaks, hydrogen can destroy the structure of many metals and make them brittle:
https://en.wikipedia.org/wiki/Hydrogen_embrittlement
And another issue is the very low density - you van see it with hydrolox rockets, how big they are & how their liquid hydrogen tanks are compared to the LOX tanks. This adds up in tank weight, removing some of the energy/density benefits. This is one of the reasons many next gen rockets are opting for liquid methane instead of liquid hydrogen.