Why feel guilty? You child will still have relationships with his grandparents, aunts and uncles, and cousins. Really, there's only so many people the baby needs to form a relationship with. I like my second cousins and great aunts and uncles etc., but my life is still plenty full without them and so will your childs life.
Plus, if your H has your back, you have nothing to worry about. You're an adult. You control the people you have in your life and the way you allow people to treat you. If they can't get their act together, then tough cookies - they don't make the cut. You're allowed to be selective - diluted blood relations don't give people the right to treat you like crap.