I found the device I was associating in my memory with a controller/joystick. It was actually a cable box remote from the 80s and looked like this.
https://images.app.goo.gl/pYNEKB8wnZwFZFAF6
When I later got a NES with the super mario/duck hunt combo. I used to just press the gun against the screen instead of being a normal distance away.