The truth is that moderate temperature exchanges will not affect the flavor of your beer, like taking un-opened bottles from the fridge to the counter and back again. Really extreme temperature changes are another matter, but we're guessing that you don't take your beer from boiling to freezing very often.
What does actually affect the flavor of beer is exposure to ultra-violet light. Under exposure to the UV rays in sunlight and store display lights, the hop-derived molecules in the beer will break apart and recombine with a sulfuric compound. These "light struck" beers have a distinctive wet cardboard flavor that's not very appealing to drink!
Brown-tinted bottles are mostly effective at blocking out UV rays, so we almost never have to worry about beer in these bottles getting light-struck. However, beer packaged in clear or green bottles is much more vulnerable. This has led to another beer myth that these beers are automatically skunked and should be avoided, but there's not usually an issue if these beers have been stored correctly.
What's been your experience with skunked beer?