Included in a land grant awarded in 1824 to Martin Allen, one of Stephen F. Austin's "Old Three Hundred" colonists, this property has had a long and varied history. In the 1840s Allen's heirs sold most of his land to Albert Clinton Horton, Wharton County planter, Texas revolution veteran, and the state's first lieutenant governor. Horton's heirs sold the property in 1930 to a group of investors who planned to develop the area into a county fairground, and several structures were built on the site. Fair organizer sold the land to the county in 1942.
During World War II, over 50,000 German prisoners of war were interned in camps throughout Texas. Camp Swift in Bastrop County operated a number of branch camps along the Texas Coast, and in 1943 Camp Wharton was established at this site. The old county fair buildings were converted for use by the prisoners and U. S. Army, and the prisoners worked as farm laborers throughout Wharton and Colorado counties.