This one of a kind pair of Second Empire style buildings, by D.T. Atwood and dating to 1831 and 1832 respectively, was, as local legend has it, built by a sea captain for his two feuding daughters, separated by a garden with hopes that it would lead to reconciliation. Records, however, show that it was built for Peter Huyler, a milkman. The roofs date to the early 1870's, added by a Huyler descendant.
Like Bank Street, Commerce Street got its name during an early 19th-century ellow fever epidemic when many business from downtown moved to the Village.