Pompey twice let a lead slip before exiting the Carabao Cup at Newport.
A 3-2 defeat in south Wales ensured it would be the League Two outfit who booked their place in round three.
Ronan Curtis opened the scoring for Danny Cowley's side and restored their advantage after Will Evans had levelled.