This feature won't work unfortunately via Bluetooth. Taken from the support page: https://support.xbox.com/en-AU/help/hardware-network/controller/connect-xbox-wireless-controller-to-pc

Controller attachments such as headsets, chatpads, and the Xbox Stereo Adaptor will not work with your controller while connected via Bluetooth.

As others have said, you can't do it if you connect via Bluetooth. You can do it if you connect the controller wired, but the sound quality isn't going to be the best. This isn't an Android-specific limitation either: if you connect the controller to anything but an Xbox (PC, whatever) it has this same limitation.

Personally I'd suggest you buy a separate Bluetooth headphone adapter anyway. There are plenty of good, cheap options, and for a variety of reasons it's better to have your TV addressing a distinct and separate device for sound playback rather than trying to route it all through one thing.

I know the auto/pcm/pass through setting in the tv audio settings is meant for output from the tv but did you try that just for ****s and giggles?

I constantly go back and forth daily on my bedroom TV as I have a 5.1 surround that is unbelievable system on HDMI ARC for movies or optical for tv.

But also use a little 8 inch soundbar (meant to mount under a computer monitor) hidden behind my headboard for nighttime use to not wake the kids, etc and I don't want loud sound as we fall asleep with the tv on with sleep timer, bad habit I know but ive had sleep issues all my life and it really helps. That soundbar is connected to the headphone out jack on the tv (just bought a 50 foot I believe was the length 3.5mm extension to run up into the attic and down the wall to behind the headboard)

Are you using the controller for games or just as a remote?

I do have a USB Bluetooth mouse/keyboard/touchpad remote which also has a head phone jack incorporated and it does work just like the Bluetooth Roku remotes with the headphone jack.

So you do have other options and it might be possible to get sound from the Xbox controller (as someone mentioned in another comment) before going a different direction with what I mentioned I'm happy to help feel free to message me.

Does not use Bluetooth, but I have found that a USB 2 "soundcard" connected to my Fire TV 4K MAX stick or Nvidia Shield TV Pro, works very well for wired headphones.

USB 3 soundcards, tend to have lower gain, due to expecting to get more power from the USB port, than is typically available.