The building entire, or almost so. The library actually fills two houses: the John Tucker Daland House, seen here, (1851), and Plummer Hall, (out of sight to the left). The Daland House was built for a wealthy Salem merchant. Designed by Boston architect Gridley J. F. Bryant, it is one of the finest Italianate houses in New England and was among the last great townhouses built in Salem. Both of these buildings have the tall, narrow windows and heavy brackets supporting the roof that are typical of the Italianate style. Plummer Hall and the Daland House are owned by the Peabody Essex Museum.
For more see:
en.wikipedia.org/wiki/John_Tucker_Daland_House
This picture has been added to the gallery of images of Phillips Library on Wikipedia, and not by me! Thanks, someone.
I was lucky enough to do research in this building some years ago on George Francis Dow, director of the Essex Institute at the turn of the 20th century. The building still contains the period rooms illustrating early 19th century life that he installed and for which he has earned repute. In a recent upgrade they were saved as examples of museum education a hundred years ago.