Does your Nexus Player or Shield TV search for anything you say or type in the search function or it only works for media? On my Sony TV there is nothing other than media from Youtube and even that isn't accurate I think when they launched it they showed that common queries would be answered.
With Apple TV out and the competition rising I thought Google would start focusing again but they haven't yet...
I've asked my NP how old a certain celeb is and what the weather will be like today. It returns an answer. (at least when done from the main screen, which makes sense)
I think they were smart to have it return answers relating to "watching" something, but it can still do more than just find videos.