I love Sag Harbor, Also the Northfork is very nice for the wineries. The Peconic bay has such calming clear water if you not interested in the big waves that line the south shore.
It is all pretty great I have to say.
Montauk is quite a drive but gurneys has a nice spa, and the town is charming.