For the longest time I would be unable to connect to several of my Nexus Players with the stock AndroidTV remote app. You would select Manage Connection and then the appropriate player and it would just go straight back to the list of players.
I finally figured out how to fix it. On the nexus player, go to settings, Apps and the find the Android Remote Service under system apps and click on it. Select Clear Data and then try to link the remote again.
Thank you kind sir. I lost my remote recently and was able to use a wireless mouse to delete android remote data and now the android tv app is working. Thank you very much