The only downside to leather is hot, sunny days. I didn't get leather for that very reason (well, that, and the fact that I actually think the Interlagos trim looks the 'bees knees' :) )
I've been in many a leather-trimmed car, and none have ever failed to attempt to give your arse third degree burns if left in the sun for a reasonable period of time. I'd imagine if you were wearing shorts, it could actually be quite painful.