I had the same problem . I resolved this by turning off contrast mode and the dynamic contrast on my Philips TV.

You probably have brightness set too high in SDR, so when HDR goes back to correct brightness levels, it looks darker.

That's how they do HDR just keep it super dark so a bright scene looks brighter. /s