The United States Postal Service (USPS) has closed a small office in Virginia following concerns of the location being housed inside of a historic train depot museum that once served as a segregated institute, NPR reports.

The museum, which is located near Former President James Madison’s Montpelier estate, still includes signage above its outside doors that are labeled “White” and “Colored.” According to a Postal Service spokesman, the agency closed the Montpelier Station office during the first week of June because of its ties to the formerly racially segregated establishment.

“Service at Montpelier Station was suspended after it was determined the display at the site was unacceptable to the Postal Service,” said USPS spokesman Philip Bogenberger in an official statement. Per the organization, the “Postal Service management considered that some customers may associate the racially-based, segregated entrances with the current operations of the Post Office and thereby draw negative associations between those operation and the painful legacy of discrimination and segregation.”

The office, which was located in the region of the state where the nation’s fourth president and his wife once lived on a plantation, operated four hours daily with one employee and served roughly 100 people before closing in June.

While the decision to close the edifice was due to concerns about the signage, some people have voiced opposition to shutting the doors to the postal office, noting that the museum will remain open to the public. “Montpelier owns the Train Depot building and the exhibition will remain open,” said Christy Moriarty, The Montpelier Foundation communications director. “We call upon the USPS to reverse the decision and reopen this historic facility that has served this community for over a century.”

In a letter to the agency’s Virginia district manager, U.S. Representative Abigail Spanberger raised concerns about the closure. She currently serves the Montpellier region of the state.