What is the best way to do it? I can obviously use Chromecast/screen mirroring, but then the pictures are limited to my phones resolution with visible pixels and aliasing on the TV. I have my pictures synced with Google Fotos, but there seems no way to access that backup on the android TV.
Am I missing something, or is the only option to copy my pictures to USB if i want to see them in full resolution?
Either i didn't got the problem you're having or you don't understand how to cast from Google photos?
If its the later part... Just press cast icon on photos and they will show on TV with maximum resolution.. Remember you can't zoom like that and you need to screen cast if you want to zoom your pictures too.
Chromecast generally reduces the quality/resolution in my experience as well.
What you need is a UPnP renderer like Kodi on your TV and a DLNA server app on your phone. Just enable the server on the phone and browse any media (photos, movies and music) in Kodi at full resolution.
Kodi: https://play.google.com/store/apps/details?id=org.xbmc.kodi
DLNA server: https://play.google.com/store/apps/details?id=com.al.dlnaserver
OR
Bubble UPnP: https://play.google.com/store/apps/details?id=com.bubblesoft.android.bubbleupnp
but then the pictures are limited to my phones resolution with visible pixels and aliasing on the TV
That is NOT true, when you cast, whatever it's been casting is sent from the server to your TV, in Google photo's case, the pictures are sent directly from the server to your Chromecast or Chromecast built-in and the quality of the pictures you see on the TV depends directly on how you took them and also if you selected the free storage tier which reduces the quality of the pictures at the expense of being free.
You can use the new Google Photos option in "Ambient Mode" setting of Google Home app to select the folders you want to use to be displayed in the screensaver.
you can cast at resolution dont cast screen cast google photos or sideload google photos app and use a mouse (recommended)