This is highly subjective. I look out for attacking opportunities as well as king safety.
In general, you are probably looking for highly brutal attacks on both wings (White attacking Black's kingside while Black does the same on White's queenside) if Black opts to castle kingside and you are looking into a slower, more positional game by castling queenside.
If you feel that castling is a little too early or bad in a given turn, you can choose to postpone the castling or even not castle at all.
Again, it depends on the specifics of each game.
A member of my chess club posed a question, “If white castle on queen’s side, which side will be better for black to castle on?” To add on to his question, when if at all should you castle along with which side after white castles queen’s side?