The potential energy for a positive charge increases when it moves against an electric field
And decreases when it moves with the electric field;
The opposite is true for a negative charge. Unless the unit charge crosses a changing magnetic field, its potential at any given point does not depend on the path taken.