Looks like a recent patch broke digital audio through SPDIF cable. It used to work a couple months ago. Well it never worked with Plex but it worked with VLC. AAC and AC3 both worked in VLC but DTS has always been broken.
Now AAC doesn't work in VLC. AC3 still works fine. DTS still doesn't work. Hopefully nougat update will fix all of this including Plex.
Just to clarify, I still hear audio using VLC and playing AAC or DTS but it's not digital audio. It's converting to analog. I know this because my receiver does a loud clicking sound when it switches from digital to analog or vice versa. It doesn't click when playing AAC and DTS but I still hear audio so it's converting to analog somehow inside VLC.
Can you tell me if it worked with netflix before it broke?
I wonder because I recently bought the device and noticed the 5.1 logo on netflix doesn't appear even though the audio is set to SPDIF and since this device has dolby audio built-in, it should transcode the dolby digital plus bitstream into standard dolby digital and output that through this optical audio output connection, I was really disappointed but now that I see your post I wanted to know if it worked before.