Capoue's volley through a crowded box following a corner in the first minute of stoppage time allowed the Premier League side to see off the second-tier hosts.
In the pick of Saturday's fixtures, Manchester City goes to fourth-tier side Newport in a culture shock for the wealthy defending Premier League champions.
"We've watched as many games as possible to understand what they do. We respect them a lot," City manager Pep Guardiola said Friday. "We are going to suffer because they are taller and stronger than us. In some areas, they are better than us."
There are 82 places separating Newport and City in the league pyramid. While City's Etihad Stadium has a capacity of 55,000, Rodney Parade fits less than 8,000 fans.
Newport has already knocked out 2016 Premier League champion Leicester. For all their success, City has recent experience of a cup upset after losing at this stage last season to then third-tier side Wigan.
The fifth round will be completed on Monday when holder Chelsea hosts Manchester United.