From the documentation:
yields an expression in which the ith part of expr is replaced by new.
Note that does not say
stores the result of an expression in which the ith part of expr is replaced by new back into the variable expr.
Try storing the result, which is what I think you want.
j=0; While[j<n, mat=ReplacePart[mat,j+1->j+1]; j++];