I think it was because he was so calm looking while he was driving. I can't see Sherlock as a calm driver. He's not a calm anything. If John had been making faces or hanging on for dear life, I would have found it more believable.

Btw, that thought you had about Mycroft having taught Sherlock to drive? It is totally becoming a story. Well, a plot bunny at least. If you don't mind?