Is there any way I can block casting from phone to my TV, or at least set a whitelist so only my device allows it? Sometimes people on my wifi cast to my TV accidentally, turning it on from standby and playing youtube on it, and I can't seem to find a way to block this from happening.
If you're okay with not being able to cast to your TV at all, you can try Disabling the Chromecast Built-in app.
Isn't there settings under chromecast to disable "let others control your cast media option"?
Try putting your tv on one band (5ghz) and the rest on 2.4ghz or vice versa.
See if your router allows you to setup a "guest network". Normally these are isolated so they have internet access but not access to other local devices.
One hack that comes to mind would be to add a network for your TV & any cast-permitted device and a separate one for any other device you don't want to access said TV. Maybe via a wifi repeater or something?