Hmm I can disable that feature without a hassle.
Go to your Google Home app on your phone. Click the most left tab (little house). Click on your SHIELD. Then click the settings icon (gauge) in the upper right corner. The "Let others control your cast media" toggle is right there for me.
Late to the party but I'm also having this issue and have for some time. Fix it Google/Nvidia!
Anyone got any updates on this? I'm having the same issue with my Sony Bravia Android TV -_- Really annoying!!
It isn't a part of the Google Cast Receiver yet. Hopefully Google will add support eventually.