N64 games are not particularly large or memory-intensive compared to native Wii titles. This is clearly a loading process of some kind - but what I can't figure out is why.
Closing the menu results in about another 5-7 second wait. If the Home button on the remote is pressed after the game has started, the screen blacks out before the home menu appears. When starting, say, Mario Party 2, there is about a 10-15 second delay during which the N64 logo (the N cube) spins on the screen before the game starts. I recently purchased a used first gen Wii (the one with Gamecube controller ports), and a few VC titles were already installed. It appears as if there's a significant loading delay on some N64 VC games.