The son of a doctor, Brown graduated from Stanford University and was accepted in the Group Theatre in New York in 1938. Despite its critical success, the Group Theatre folded in 1941, and Brown moved to Los Angeles to seek work in film. Along with other former Group Theatre members, he formed the Actor's Laboratory, which produced critically acclaimed works in Hollywood. At the theater, he directed plays by Arthur Miller, Nikolai Gogol and Arthur Laurent.

Although he denied being a Communist, Brown was blacklisted during the McCarthy period and moved to London with his family in 1948 where he played opposite Helen Hayes in "The Glass Menagerie." He lived in London for 40 years.

Returning briefly to Hollywood in 1949, he found work as a director and two years later finished his first feature film, "The Harlem Globetrotters," starring Dorothy Dandridge and members of the famous basketball team.

In London, he found work on stage and in such films as "Tropic of Cancer" (1970) and "Twilight's Last Gleaming" (1977). In the mid-1970s, George Lucas was filming interior scenes for "Star Wars" at a London sound stage and needed an actor with a strong American accent. Brown got the role of Luke Skywalker's Uncle Owen, the most notable role of his life.