it takes away the e5 square for one, and sometimes when you get the square structure with g7 g6 f7 f6 you can advance the front pawns without weakening the king. it's not that the pawns are doubled that makes them inferior. it's that white has the queenside majority that can create a passed pawn while black's kingside majority can't, as has been known from the exchange ruy pawn skeleton. but you're not going to trade down to a king and pawn endgame so that should not be an issue.
i don't think 4..nf6 is better than 4..bf5. it may just be a different try with less theory so far. and exchanging one pair of knights doesn't really simplify the position that much. i mean, in the bf5 line the bishops get traded on d3 but it remains complicated.
i played 4..bf5 as my main reply before i switched to 4..nf6. i even played the sharper 0-0 lines just to get a fight vs 0-0-0 by white. it's sharp but also worked out in detail very deep already, and as for the 4..nf6 line it's relatively new, not the move but how black plays it.
What do you think is black’s compensation for the doubled pawns? Is it better than just, say, 4…Bf5? Can the pawns ever give you an advantage? Is it just that 4…Nf6 simplifies the position?