Simon Brown was part of the team of five that represented Australia at the IX Winter Olympics in Alpine skiing. At Innsbruck 1964 Simon competed in the Giant Slalom event, placing 51th, and in the Slalom - which he did not finish.
 
He was also one of the two Australian athletes who participated in the Olympic Downhill, the opening Alpine event contested after the tragic accident that cost teammate Ross Milne his life. Reserve Peter Wenzel replaced Peter Brockhoff after the latter decided not to compete on a course where he had lost one of his best friends. Brown finished the Downhill in 61st place.