Aston Villa, Leeds secure quarters place
A Latics side featuring nine changes from Saturday's Premier League victory over West Ham created a host of opportunities but could not make a breakthrough during the 90 minutes or in extra-time as their spirited npower League Two opponents held on for spot-kicks.
Shaun Maloney then blazed his penalty over and Jordi Gomez saw his saved as the visitors advanced to the last eight of the competition.
Scott McDonald earned Middlesbrough a quarterfinal place with a hard-fought 1-0 victory over Sunderland in their Wear-Tees derby at the Stadium of Light.
The striker, who was handed a start by Boro boss Tony Mowbray after he scored both of the goals in last weekend's 2-1 npower Championship win over Bolton from the bench, scrambled the ball home from six yards at the near post late in the first half.
It was enough for Middlesbrough to progress as Sunderland lacked the cutting edge that has seen them draw six of their eight Barclays Premier League games so far this season.
Leeds secured another top-flight scalp as they brushed aside Southampton 3-0 at Elland Road.
Saints manager Nigel Adkins' decision to make 11 changes backfired on him as goals from Michael Tonge, El-Hadji Diouf and Luciano Becchio saw the home side through.
It took until the 70th minute for Southampton, who have been struggling in the Premier League this season, to register a shot at goal and it will be Leeds, third-round conquerors of Everton, who enter the hat on Wednesday night for a first quarter-final appearance since 1996 - a year they made the final.
Aston Villa are through to the quarter-finals for the third time in four seasons after Christian Benteke scored a late winner in a 3-2 victory over Swindon at the County Ground.
Villa looked to be cruising to victory as goals from Benteke - a £7million summer signing from Genk - and Gabriel Agbonlahor earned them a 2-0 half-time lead.
But the npower League One side, who had already overcome Stoke in this season's competition, fought back through two goals from substitute Miles Storey before Benteke had the final say.
It was his third goal in two games and Villa boss Paul Lambert will hope his side can build on this success after their worst start to a league campaign in 43 years.