I think the two main factors in play here are the classic "nature vs nurture".
Nature: Using pre-historic artwork and sculptures as my reference, I would say that breasts are seen as something sexual and feminine. Pre-historic sculptures have females with exaggerated breast size and such, which, as my philosophy teacher said: "appeal to human senses". If you look at it from an artistic point of view, breasts can symbolize fertility, motherhood, maturity, etc. So all in all, breasts are ultimately sexual, and related to female puberty.
Nurture: Society and our culture ultimately shapes us to be the people we are today. So the way we deal with reacting to seeing bare breasts would be based on how we are conditioned to react to it.
That's what I think.