Pompey had to battle from behind before securing their spot in the second round of the Carabao Cup.
Goals from Elliot List and Charlie Carter gave hosts Stevenage an early lead and although Ronan Curtis reduced the deficit for Kenny Jackett's side, Scott Cuthbert hit back for the League Two outfit.
Gareth Evans reduced the deficit from the spot on the stroke of half-time, while John Marquis levelled after the break to take the tie to penalties.
Pompey scored all of their efforts and Craig MacGillivray made three fine saves to secure a 3-1 shoot-out victory.
Fans can watch the key moments (top video), with extended highlights (middle video) and the full match (bottom video) available for iFollow subscribers.