Update: I tried resetting the device, but the issue comes back. So far the following workaround is working for me, that doesn't require restart: Go to
chromecast settings >>display and sound >>dynamic range and color format>>change format preference to HDR. The TV flashes black and the sound will be on. So far, no more sound issues. If the issue comes back again switching to Dolby vision should fix it. At least, there is no need to restart the chromecast.