Maybe I'm the only one who feels this way, and it may be that I don't have kids, but I think it's ridiculous to expect them to stay! Yes, your lives are going to include children, but this is one time when I would expect them to go somewhere else. That would be like having your live-in mother in law stay in your house on your wedding night. That's weird.
I also think it's a matter of preference. If you feel like it's important to have that alone time, then you should stick to your guns. Honestly, the children would understand at that age, and they need to know that you are part of their father's life too - and that you will occasionally require alone time. Plus, you've already had a family vacation. This seems like something you should approach your FI about and tell him how important it is to you. Compromise might be just the thing here.