An election in the fall of 1894 resulted in the relocation of the Matagorda county seat from the city of Matagorda to Bay Prairie (now Bay City). D. P. Moore, the postmaster at the nearby small town of Elliott, owned property in the new town and moved his dry goods store to Bay City in 1894. His nephew, Joseph D. Moore, became Bay City's first postmaster. In 1912 D. P. Moore sold his property to the United States government for a new post office building.
Contractor W. B. Lovell broke ground for the post office in 1917 and the building opened to the public on May 15, 1918. An addition built in 1958 provided extra working space and a loading dock at the rear of the structure. This building continued to serve the Bay City community as a post office until 1989. During that time it also served as a town meeting place, the site of the Selective Service Board during World War II, a Civil Defense fallout shelter, and a place of refuge from storms and hurricanes.
On October 30, 1990, after several months of negotiations with the United States Postal Service, the Matagorda County Museum Association purchased the building to house the Matagorda County Museum.