There used to be a nice Chrome addon called Chromemote which worked perfectly with Google TV. I'm not sure if it works with Android TV, but it might.
Unfortunately, it uses NPAPI which is no longer supported by Chrome, so the developer has stopped working on the project. I suppose you could try running an old version of Chromium specifically for this task, 44 being the last one with NPAPI support.
Someday, I'd like to take a look at Robert Dale Smith's source code and see if I can get his HTTP server implementation running, but for now I don't have time to commit to that task.
FYI: Google has a TV Remote on play store.
Not currently. At Google I/O rumors say that Chrome/Chrome OS will support Android apps. That would make it possible to run the Android TV remove on Chrome. This might solve your issue.
Teamviewer Quicksupport on box and teamviewer on PC?
If you chromecast everything, yes.