The house was built circa 1818 by a Mr. Potter. He sold it to lawyer Phineas Ashmun. The first Brooks Post Office opened in the house January 14, 1819. Mr. Ashmun’s pay as postmaster was $8.24 annually. The house was a gathering place for leading citizens for discussion of important matters. Mr. Ashmun also held classes in winter evenings on rhetoric, arithmetic and writing.

His successor, Woodbury Davis, appointed in 1843, was paid $41.73 annually. Mr. Ashmun sold the house to Mr. Davis in 1846. Davis read law in Belfast and was admitted to the Waldo County Bar in 1847. Davis began his practice in Portland and was appointed State Treasurer in 1855. Davis was appointed an Associate Justice of the Supreme Judicial Court by Governor Morrill on October 10, 1855. He was removed by address on April 11, 1856, but was subsequently re-appointed on February 25, 1857. Davis resigned from the Court in 1865, and in 1866 was appointed Postmaster of Portland.

Mr. Davis sold the house to Hiram Pilley in 1863. Hiram Pilley married Mary Rose. Their children were all born in the Pilley House according to Bob Elliott:
Elnora “Nora” Pilley (Boody)
Bertha “Birdie” Pilley (Brackett)
Elizabeth “Lizzie” Pilley (Chase)
Albert “Al” Pilley (born February 4, 1874)
Bertha & Charles Brackett lived for a time in the apartment on the north side. Both were teachers. Daughter Gail (Bob Elliott’s mother) was born in the house in 1892.
Al Pilley married Esther Colson on February 10, 1904. He had previously established his drug store. They lived in the north side apartment. Louise was born March 1905, likely in the Pilley House.
Hiram died in 1914 and Mary in 1921.
Esther may have become a mistress of the house before Mary’s death. Louise told Betty Littlefield (past Brooks Historical Society president) that her mother remodeled the kitchen. When Esther died, Louise gave up her career in Boston and returned to Brooks to care for her father.
After Al Pilley’s death in 1969, Louise continued to live in the family home until she retired to Florida. She died there in February 2005, just shy of her 100th birthday.
The Pilley House was deeded to the Brooks Historical Society by Robert and Lee Elliott on June 2, 2005. Robert was Louise’s first cousin, once removed.
Robert “Bob” Elliott of ‘The Bob and Ray Show’ was a celebrated radio performer for over thirty years. ‘The Bob and Ray Show’ was inducted into the Radio Hall of Fame in 1995. Chris Elliott inherited his father’s talent for comedy, appearing on the tv sitcom ‘Everybody Loves Raymond,’ as well as appearing in many movies, including several with his father. He and his father both cherish the memories of happy summers spent at the Pilley House. Bob passed away here in Maine on February 2, 2016.