I had a similar experience in college...I went rushing fraternities with a guy on my floor in the dorm and one was a Jewish fraternity. When we were leaving, he asked me, "Was that a predominantly Jewish fraternity?" I said yes and he said, "do you think I'm Jewish?" I said, "well you are wearing a chai" [starofd.com]
He said, "oh, my girlfriend bought that for me, she said it looks like a monkey."