It's because pawns used to be able to move only one square. En passant was created when they were allowed to move two squares, sort of pretending that it only moved one square and is why you can only do it immediately after the first pawn move, kind of where the pawn "should" be.