Three centuries plus some ago, about the time JS Bach was born, James II was King, and Salem was getting ready to try "witches", this town was organizing and it was because, for fifty years before, this harbor and this particular spot was protected from the North winds and was deep enough so the larger ships could navigate in. It is or was a fluke of natural conditions.
Protected harbors, in an era of boats and wagons were the big deal. Transportation and trade being obviously in a basic state, the luxury of a southern facing harbors was a big deal and when you look at Long Island this is the only one for a long way. It explains why the south shore of Connecticut and Rhode Island, with their trade cities were built at the end of rivers (New London for instance) or just protected bays further east prospered to the extent they did. Of course that is a simplified explanation but if you just look at a map of this region and are seeking a safe (from the weather) harbor - your options are limited.
On one hand it is great that modernizations "neatening up" the places like this occur. On the other, the reason that anyone is here to enjoy it has been fixed up so the natural harbor picture is covered up for good. The value that was here the brought people here and established the place are just painted over.
Progress I guess but at a loss of perspective.