Don't understand why removing my pawn on the A file is a bad move here.

MinorBelief vs. xVicissitudex | Analysis - Chess.com

I'm talking about move 5 here which solver says is a mistake. My idea is that I prevent a Queen move to A4, and if white takes the pawn, I can take with pawn on A6 freeing up my rook on the A file. 
The solver thinks it's a bad idea to take with the A4 file. 

So it it just a bad idea to remove the pawn on the A and H file in general? Can somebody explain to me the general concept is it to keep the King safe from attacks in that direction?

I think because it loses a pawn. b5 was attacked twice by the c-pawn and bishop but defended once by the a-pawn.

That is not why is this move bad (it is not a problem to remove a pawn from a file, it can be sometimes beneficial as it opens up your rook). First of all, you are wasting time in the opening with many pawn moves when you should be developing your pieces. The other reason is that b5 loses a pawn:

cxb5 axb5 and then Bxb5.

After that you are moving your queen too early . He could responded with a simple Nc3 and he would have a huge lead in development. Your kingside where you want to castle, as queenside pawn structure is shattered, is completely undeveloped and if played correctly, your position will probably become unholdable.

whites Bishop on d3 is terrible and that is where the computer is giving black positional points for. Then in the next 2 moves black plays to free whites Bishop and make it better for white. and the computer takes blacks points back.