Things get misconstrued in writing, feelings get hurt*. I'd go over to them (bring muffins or something if it makes you feel better) and ask them to not come on your property again, that you appreciate it, but you still don't want someone on your property all the same.
If he comes on your property again, then send a letter certified.
In the meantime, look up your local laws for trespassing. Be prepared to call the cops if he won't give it a rest.
*I'm HOA president and just had to settle a little drama between 2 residents. One resident left another one a strongly worded letter telling her that her dog was barking sun up to sun down and she was going to call the cops if she didn't put the dog in. The recipient came over to me as president and demanded to know what our HOA rules were, yada yada. The recipient said that had she known how the other lady felt, she gladly would have brought the dog in, that she didn't know it was barking, but now she's all pissy toward the other neighbor. It all could have been solved by ringing the doorbell and politely saying, "would you mind?"