He might be holding back for some reason. I was holding back because I didn't think it was appropriate for me to be attracted to him, but I was into him pretty much as soon as I saw him appear before me again.

But you're definitely underestimating how much close contact fighting requires. You can tell him you want to spar and take that as an excuse to see how he reacts to your touch.