Trying using a Bluetooth mouse, or install an app in both devices called "Zank Android Remote" I think. With this app you have a lot of functionality, but the main one is controlling your android tv with a mouse!

How did you managed to get it work? On my phone it does work, but not on the tv. The system not letting turn on the notifications access. Well, it actually does but then automatically turning off.