So without the USB harddrive enclosure plugged in, the remote works as normal. When I have the harddrive plugged in, its doing this reset loop that causes things not to work right. So definitely there is some kind of interaction with the drive that is the problem.
Is there anything I can do to debug this further? I would like to have both a functional remote, and a harddrive attached to the vero