But don’t hold your breath.
Now, here’s the tricky part—one that ideally shouldn’t need to be addressed, but a world populated with Professor Gateses obligates it—there are certain segments of certain segments of certain segments of the population that are a mite bit sensitive on matters of skin color, sexual orientation, party affiliation, you name it. You can count on these drama queens to accuse you of racism, sexism, conservatism, you name it at the drop of a hat.
Should he actually accuse you of being a racist—thereby implicitly confirming his own racist beliefs—ask him why he feels this way.
If your state has a statute on the books that allows you to arrest any moron who follows you out of the house, yelling in your inflamed ear, disturbing the peace, and interfering your ability to do your job, arrest his ass. Make sure that you have an audio recording of the incident. Accurately transcribe what the moron has said and avoid generalizations such as “he was yelling.” Direct quotes such as “I’LL SPEAK WITH YOUR MAMA OUTSIDE!!” work much better.
Do everything you can to prevent charges from being dropped. Otherwise, it rewards bad behavior and wastes your time.