The son of a Swedish diplomat, Oldenburg was born in 1929 in Stockholm. When he was an infant, the family moved to the United States, settling for a time in New York but eventually moving to Chicago.