The closest I got is using localcast app. However the way it worked for me is by streaming from my phone to the AndroidTV, chromecast, or xboxone and localcast then gives me the option to play the audio to my phone and not the TV.

This won't be any help probably, but the NVIDIA SHIELD Android TV / Pro have headphone jacks on the remote and game controller, no lag. It's pretty cool when you want to watch something on the big screen and not disturb others if someone is studying or working.

I don't know the answer to the question specifically but:

There is an app in the play store called air audio. It's used for streaming all audio from a phone/tablet to an airplay/dnla/Chromecast device. It requires root and there is a noticeable delay of probably a few seconds.

Depending on your use case you could throw kodi on your phone. Although I'm not sure if air audio would install on Android TV.

It's your TV audio really just that awful or are you trying to recreate that old school drive in theatre experience?

I would think just buying BT headphones and plugging them into your TV's audio output would be the easiest solution. There may be a way to have your phone act as a BT audio receiver through some app, but I have no idea. There's definitely no way to just "cast" audio from ATV to Android. Keeping things in sync just cannot happen right now through the Google Cast protocol. Someone else mentioned casting to their TV but keeping audio on their casting device, which the closest thing to this feature involving Android but it still loses sync and doesn't work flawlessly. /r/Chromecast has been begging for this feature for years, posting pretty much once a week asking if it can be done. I think your best best is BT headphones, or just ponying up for that Shield TV sooner rather than later. Or jump to Roku.