Thanks for this feedback @RBR! It is still not perfect but much better indeed.
I did not test that as actually generating an optimized png instead of a jpeg (as done currently on your model) did not lead to detail being washed out. I'll have to dig a bit to understand why the sRGB to linear/linear to sRGB conversion would be so destructive for jpeg. My quick guess would be that maybe the 'empty' pixels in your map could be the reason but that will need investigation.