Ruy Lopez Move Order

My question is simple, what's the downside of playing 7...0-0 instead of 7...d6 in the main line of the Ruy Lopez? Black is likely going to castle kingside anyway, white has no immediate treat, so why not keep the flexibility of the d-pawn? I know 7...d6 kinda forces 8.c3, but that's a move white wants to play anyway. Let's look at some common continuations after 7...0-0:

8.c3 - Black now can enter the Marshall with 8...d5 or go back to the main lines with 8...d6, so black move order is justified due to the flexibility.

8.h3 - Black can return to the main lines after 8...d6 9.c3, avoiding some variations where white might not play h3, specially the 9.d4 variation. White can play 9.d3 here, but I don't think black has to worry about it, it can even take the light-square bishop with 9...Na5 if he wants to. Black also has the possibility of playing 8...Bb7, keeping white outside of the main lines, so once again black has the flexibility, justifying the move order.

8.a4 - One of the critical lines, but white could've played this move even if black played 7...d6 and I don't really see how black is punished by not having the pawn on d6. The thing is, if white wanted to play 8.a4, he would play anyway, so black avoided the main lines without any downside.

8.d4 - Maybe the most critical line, but after 8...d6 9.c3, the most common continuation, we reach the Yates Variation and a similar point arises, if white wants to play this, he would play it anyway. So once again black avoided the main lines without any downside.

8.d3 - The last move we're going to see, but the ideia is the same, black can play 8...d6, and that avoids the main lines.

I'm not really saying the main lines are better then the lines above, but at a first glance it seems like black has nothing to lose for committing to the Marshall move order, in every line it gives black flexibility and/or remove some flexibility from white. The only reason someone would play the classical move order is to lure the opponent into a chigorin or a breyer, but white can always avoid it.

My final thoughts are that the Marshall move order seems objectively better for black in comparison to the 7...d6 move order, so, am I missing something? How can white "punish" black's move order?

You ask a very interesting question for which I don't have an answer. I note that in opening explorer there are over 17,000 games with d6 and nearly 10,000 with 0-0. The latter leads to marginally more wins for both players without much being in it.

(If I can ask a question of my own why does black not play b5 before Be7 so much ?

EDIT - note to self : I think it's because white hasn't played Re1 and can play d4 if black follows b5 with Be7, where as Be7 first obliges white to play Re1...I may be wrong. Apologies to OP for sidetracking the topic a little)

I don't think there is a downside. Some people just play it to make their opponent think about the Marshall gambit even if they're not going to play it.

Well, there might be a more direct reason, but for one I think the bishop is in a better diagonal when it retreats and b5 is weakening the queen side a bit, so black only wants to do it when white has a concrete threat of capturing the knight. Also in some lines black may prefer to defend the pawn with d6 instead of b5 and f7 may be a bit vulnerable, but I'm too sure about it.

One point is that Black isn't really free to play Na5 molesting the b3-Bishop until he has guarded his e-Pawn with d6.

That's true, black is giving up on the flexibility of the d-pawn to have more flexibility for it's bishop, that's cool, thanks man.

Interesting.

I asked this very question on the forums a couple years ago, and it just boils down to what mn79 stated in post #5.  If you're not a Marshall player there's really no point in delaying ...d6, you might as well play it on move 7 and give yourself more flexibility if white plays the a4 sidelines.