Pompey suffered their first home defeat since March, as Rotherham left Fratton Park with all three points.
Jon Taylor netted the only goal during the first half of a televised contest that improved as it went on.
The Blues were much better after the break, but were unable to convert any of the numerous chances they carved out.
Kenny Jackett made four changes from the side that drew at Wigan the previous week.
He handed debuts to Stuart O’Keefe and Oliver Hawkins, while fellow deadline day signings Damien McCrory and Matty Kennedy had to be content with a place on the bench.
Matt Clarke returned from the groin injury that had kept him sidelined since pre-season and there was also a start for Conor Chaplin.
That saw a switch to a 3-4-1-2 system, with Brandon Haunstrup and Jamal Lowe occupying the wide roles, and skipper Brett Pitman playing behind Chaplin and Hawkins.
The rain was falling heavily when the game got underway and the action on the pitch was doing little to brighten the mood.
It was a fairly lethargic opening to proceedings, with both sides struggling to keep hold of the ball, perhaps hindered by the wet conditions.
The first chance fell for the hosts, though, and Pitman ran at the Rotherham defence, beat his man and fired narrowly over the bar.
But the Millers soon threatened themselves, with Michael Ihiekwe failing to find the target before O’Keefe bravely blocked Lee Frecklington’s effort.
Kieffer Moore than tried his luck from the edge of the box, but could not make a clean connection and the ball bobbled safely into Luke McGee's arms.
There were not too many clear-cut openings, but the Blues did have opportunites to break the deadlock.
Haunstrup saw a shot deflected past the post after meeting Adam May’s delivery, while Hawkins’ long-range attempt only flew narrowly wide.
Instead it was Rotherham who took the lead on 36 minutes, as Taylor latched onto a ball down the right and lashed home a powerful strike.
Pompey immediately set about trying to find a leveller and Clarke failed to divert Chaplin’s corner on target.
A superb cross Haunstrup cross from wide on the left was then headed over by Lowe, as the hosts went in behind at the break.
Half Time: Pompey 0 Rotherham 1
Jackett made a change of personnel and system for the start of the second half, with a return to four at the back.
Jack Whatmough and Chaplin both failed to emerge from the tunnel and were replaced by debutants Nathan Thompson and Kennedy.
And the latter made in instant impression with a cross that Millers keeper Richard O’Donnell could not get to.
The ball landed at the feet of Lowe at the far post and he took a touch before sending in a low drive that was deflected behind.
Pompey fans were baying for a penalty when Hawkins took a boot to the face on 57 minutes, although referee Gavin Ward was not interested.
There was then a lighter moment just past the mark, as Pitman and Kennedy both tried to keep the ball in play and the former ended up taking out the latter!
The hosts were looking much more dangerous than before the break and when a bandaged Hawkins met May’s free-kick, his header had to be cleared from the line by Darren Potter.
Jackett made a final throw of the dice on 70 minutes, with Hawkins given a generous ovation as he made way for Kal Naismith.
But the Scottish forward was getting patched up on the sidelines following a challenge with O’Donnell when the Blues came close to levelling on 79 minutes.
It was a dangerous low ball into the box from a slightly out-of-position Clarke, but a stretching Lowe could not quite make a connection at the far post.
Pompey poured forward in the closing stages, which were played out almost entirely in Rotherham’s half.
And the display of six additional minutes gave the Fratton faithful hope of salvaging something, but the visitors held on for victory.
Pompey (3-4-1-2): McGee; Burgess, Whatmough (Thompson 46), Clarke; Lowe, May, O’Keefe, Haunstrup; Pitman (c); Chaplin (Kennedy 46), Hawkins (Naismith 70)
Booked: Burgess
Subs not used: Bass, McCrory, Close, Donohue
Rotherham (4-2-3-1): O’Donnell; Cummings, Ihiekwe, Wood, Purrington; Potter, Vaulks; Williams (Forde 54), Frecklington (c) (Towell 90+1), Taylor (Newell 85); Moore
Goals: Taylor 36
Booked: Potter, Purrington, Vaulks
Subs not used: Bilboe, Ajayi, Clarke-Harris, Ball
Referee: Gavin Ward
Attendance: 17,118 (436 away fans)