Lee entered the primary school division of La Salle College at the age of 12. In around 1956, due to poor academic performance (or possibly poor conduct as well), he was transferred to St. Francis Xavier's College (high school) where he would be mentored by Brother Edward, a teacher and coach of the school boxing team.
In the spring of 1959, Lee got into yet another street fight and the police were called. Until his late teens, Lee's street fights became more frequent and included beating the son of a feared triad family.Eventually, Lee's father decided his son should leave Hong Kong to pursue a safer and healthier life in the United States. His parents confirmed the police's fear that this time Lee's opponent had an organised crime background, and there was the possibility that a contract was out for his life.
In April 1959, Lee's parents decided to send him to the United States to stay with his older sister, Agnes Lee (李秋鳳), who was already living with family friends in San Francisco.