So just to be clear are you casting it using the actual Google cast feature rather than some other app?

Make sure your TV has the newest version of Google Cast Receiver . I had to manually update it on my Bravia for some reason.

Next, if your TV is not wired-in with Ethernet, I highly recommend it, but it's not required. My Nexus Player is wireless and works just fine, but my Bravia is wired in and I've never had an issue.

Lastly, make sure you are not casting a tab in your browser and instead are casting the video itself using the video's web player Google/Chrome Cast button. I've noticed framerate issues when casting a tab or casting my entire device and playing videos, but using the web player's built-in casting button doesn't cause these issues at all on my Bravia.