I've done both and the only major reason I used jellyfin was to track watching on that server rather than Infuse as for some reason Infuse tends to reset stats on me.

Of course transcoding on the fly and stuff is there but I just want to stream the direct file to the tv so none of that matters.

*edit: For anyone else reading down here the parent's solution was better for me vs Plex. Simply streaming through infuse via SMB removes all the issues I don't want to deal with with Plex as it has gotten worse over the years and has invaded my purely local setup more and more. As long as it is encoded in a way Infuse can read it works great. I didn't have a file Infuse couldn't play.