You can read the details here:
Romney says he was “joking” but I think the issue is more serious than that.
The problem is that Romney has publicly associated himself with so many birthers, like Donald Trump, it looks like he’s “playing to the birther crowd.” Hinting that Obama is “different.”
Indeed, in the crowd, there were many people who were interviewed afterwards who said they are convinced Obama’s birth certificate is a fake.
It’s de facto racism.