In 1886 Frank Stephenson, a boat pilot and assistant Aransas lighthouse keeper, opened an inn at this site in an old barracks. He called the facility "Tarpon Inn" for the abundant trophy fish in nearby gulf waters. The Inn served as a landmark for sailors, and Port Aransas was known for a time as "Tarpon". In 1897 Mary Cotter and her son J. E. Bought the two story inn from Stephenson. After the building burned in 1900, two new structures were built in 1904.
When the 1919 hurricane destroyed the main structure, the dining facility was used until it was sold in 1923 to James M. Ellis and his wife. Ellis soon rebuilt this inn to resemble the old barracks. He placed 20-foot poles in 16 feet of concrete with pilings at the corner of each room to reinforce it against future hurricanes. For a time guests could reach the inn only by boat. It became a tradition to sign and date a Tarpon scale and place it on the wall in the front room.
Among the famous patrons was president Franklin D. Roosevelt who fished here in 1937. Duncan Hines spent his honeymoon here and recommended the food for the next 25 years. The inn has housed many area residents during storms and served as headquarters for the Red Cross, Salvation Army and Military units.