I hope that just by bringing this up you will know what I mean.
I moved from Chromecast to an NVIDIA Shield about 3 months ago. I still really enjoy finding content on my phone and casting, but I also like the advantage of lean-back content choosing using the remote and all the other advantages of the Android TV platform. What I don't like is having to remember how I got the content playing. While watching Youtube for example I'd like to be able to cast from my phone, then later press back on the remote and find myself in the android tv shield menu, then later load another video using the remote and add to the up next list using my phone. That would be intuitive to me. As it stands I have to remember how I got the content on the screen and stay in my lane.
If you use the built-in YouTube app instead of your phone when finding content, it will work as you want.
I agree with you that when you cast from your phone the TV should be intelligent enough and stream the video you wanted in the appropriate app instead of in the cast receiver.
Have same problem with Spotify. Making TV play from the phone can be done in either Spotify connect mode (requires that Spotify is running on the TV) or in Chromecast mode, where only basic information is shown on screen while streaming the music)
Would be nice if the TV could detect the incoming cast stream, and have a setup option where the user could decide the action depending on the incoming stream