Steven Moffat, who produced both Sherlock and Doctor Who, told The Sun that kooky Smith was more suited to the lead role of Holmes.
He said: "We'd already cast Benedict Cumberbatch as Holmes and the very first person we saw for Dr Watson was Matt, who came in and gave a very good audition."
"But he didn't have a chance in hell of getting it because he was clearly more of a Sherlock Holmes than a Dr Watson."
"There was also something a bit barmy about him - and you don't actually want that for Dr Watson. You want somebody a bit straighter."
"But he gave a very good account of himself and you could feel the effort in him clamping down on his barminess in order to do the audition."
Soon after Smith was cast as the 11th Time Lord in Doctor Who, replacing David Tennant in the iconic show.
Cumberbatch also revealed recently that he was offered the role as Doctor Who before Smith - but turned it down.
The 34-year-old actor said he was being considered to replace Tennant and said: ‘David and I talked about it but I thought it would have to be radically different.'
‘And anyway, I didn’t really like the whole package – being on school lunch boxes.'