I would say if you and your husband want kids, it's sort of now or never. Sit down and have a blunt discussion about it.
If you do decide to move forward, just make sure you take every precaution and visit your doctor regularly. The likelihood of birth defects increases every year after 30 (or something along those lines), including down syndrome (not an expert on this, consult a doctor).
The real question is, do you actually want kids or do you just feel outside pressure OF "it's now or never".