Originally named Normanton the town was renamed Port Elgin when it was incorporated in 1874, after James Bruce, 8th Earl of Elgin, a former Governor General of the Province of Canada.

In 1998, Southampton, Port Elgin and Saugeen Township, all along the shores of Lake Huron, were amalgamated to form the Town of Port Elgin-Saugeen-Southampton.