The key moment arrived in the first half, when a clearly offside Chuks Aneke met Adam Matthews’ cross.
But the flag stayed down and although his header was saved by Gavin Bazunu, Conor Washington tucked home the loose ball.
Charlton went on to double their advantage at Fratton Park through a Clark Robertson own-goal, with substitute Michael Jacobs’ late strike ultimately proving no more than a consolation.
Danny Cowley refused to blame the poor decision for the defeat, though, and has demanded improvements from his side.
The Blues boss said: “I don’t think that we played well enough for long enough after what was a good opening to the game.
“We created some promising chances, but couldn’t take them and that sounds very familiar – it’s happened too many times.
"There were a lot of things I didn’t like about us overall and our decision making was erratic, so we have to go away and reflect."
“After conceding the first goal we didn’t respond well enough and it’s been the story of our home form in recent times.
“It was offside, but I’d rather focus on our performance. There have been bits that have gone against us, but we can’t control that.
“When the substitutes came on they gave us a lift and it was a strong finish, but we’re a proud football club and have to do better.
“Michael did well, but has been isolating all week and so hasn’t been available for us in training, so it wasn’t possible for him to start.
“There were a lot of things I didn’t like about us overall and our decision making was erratic, so we have to go away and reflect.”
Shaun Williams was substituted after hurting his back in a nasty collision with Robertson and left the ground in a wheelchair.
Cowley said: “He’s in a really bad way and the St John Ambulance guys were in with him and the doctor afterwards to take him to hospital.”