Typically the bird goes 1.f4, 2.Nf3, 3.b3, 4.Bb2, 5.e3... So you deviated by foregoing the queenside fiancetto, which is crucial to the strategy, and by playing e4 instead of e3, leaving the pawn on f4 undefended and the g1-a7 diagnol to your king open.
I have played the bird many times myself, and haved transformed by playing e4 on certain occasions, but this usually dictates castleing queenside so as not to leave the king vulnerable. Also, whenever you open with f4 beware From's Gambit and black playing 3.Bx(N)f3, both of which can lead to a very quick loss of you don't know how to play them.
Here's a game just completed which I won with the Bird Opening. All things considered I think I did fairly well. I know I had a couple blunders here and there but overall it was a solid game. Several variations in the endgame, so check the move list for possible variants.