Longshot: TempTable has different datatype for the columns and implicit CONVERT is involved. That's not going to take seconds on 20 rows though ...
I would look at the Query Plan (in text form!) for the Query on its own, and then for the INSERT version. Perhaps SQL is, for some reason, choosing different indexes? You don't have a PKey on the @TempTable, but if you did I can imagine that SQL might use a different strategy to get the data "in PKey order" so-to-speak, so maybe something like that is going on.
SET SHOWPLAN_TEXT ON
GO
-- ... put query here ...
SET SHOWPLAN_TEXT OFF
GO