The SPL side dominated throughout but were thwarted by brilliant Ayr keeper Kevin Cuthbert who made great saves in the second half from Gary Harkins, Shiels, Manuel Pascali and Garry Hay.
However, Cuthbert's stubborn resistance was broken in the 109th minute when Shiels, who had recovered from illness to play, drove low into the corner to set up a final appearance against either Celtic or Falkirk who play in the second semi-final at the national stadium on Sunday.
It was a deserved win for Kenny Shiels' side but they certainly made hard work of a game they should have won easily on the balance of possession and chances created.
However, that will not bother the Killie fans who look forward to their first final appearance in the competition in five years.
Kilmarnock boss Shiels made two changes following their 3-0 home defeat to Dunfermline in the SPL last week.
Central defenders Mohamadou Sissoko and Zdenek Kroca dropped out, the latter starting on the bench, to be replaced by debutant Michael Nelson and midfielder James Dayton, with skipper Pascali moving back into defence.
For the Irn-Bru First Division side, midfielder Alan Trouten returned following a hamstring injury and striker Gareth Wardlaw came back in, with Mark Roberts and Adam Dodd starting on the bench.
Brian Reid's part-timers had the scalps of three Scottish Premier League sides - Inverness, Hearts and St Mirren - already under their belt but there was little in a dull first half to suggest they would collect number four.
With Wardlaw playing as a lone striker, it appeared that the Somerset Park side were looking for success on the counter-attack.
The Rugby Park men had the first effort on goal in the sixth minute with Shiels sending an unconvincing drive from 25 yards wide of the target and they dominated possession thereafter.
In the 23rd minute Ayr captain John Robertson had to head clear a chip from Killie midfielder Hay from close to his own goal and that was the closest the first half came to a breakthrough for either side.
Three minutes later Hay ended another spell of Rugby Park pressure with an effort from distance which Cuthbert clutched to his chest.
Ayr's first effort on goal came in the 33rd minute when Michael McGowan curled a free-kick from 25 yards over the Killie bar, before Shiels again drove wide at the other end.
In the 38th minute Cuthbert made a decent close-range save from Ben Gordon after the Killie defender had raced into the United box to confirm that the first half would remain goalless.
Killie forced another corner within a minute of the restart after Gordon ended another fine overlapping run into the box with his cut-back blocked but again it came to nothing.
Five minutes later on the other wing, Dayton went past Ayr defender Eddie Malone and burrowed his way into the United box but could not pick out a team-mate and another opportunity was lost.
In the 55th minute the increasingly influential Shiels drove at the Ayr defence before laying the ball off for the supporting Gordon but his side-footed effort from the edge of the box was easily gathered by Cuthbert, who made another save from Hay's long-distance effort moments later.
Ayr keeper Cuthbert then came into his own. In the 66th minute he parried Dayton's 25-yard drove to the feet of Harkins and stuck his leg out to deny the Killie striker's close-range strike which looked certain to give the Rugby Park side the breakthrough.
In the 79th minute Cuthbert stretched to tip Shiels' 30-yard drive over the bar and from the corner knocked Pascali's downward header past for another corner which Ayr survived.
Killie keeper Cammy Bell made his first save of the game in the 82nd minute from Andy Geggan's drive before Cuthbert tipped Hay's powerful drive over the bar for yet another corner which Killie squandered.
In the second minute of extra-time Killie substitute David Silva, on earlier for Dayton, screwed an angled drive wide of the far post.
In the 109th minute Shiels eventually got the breakthrough against a desperately unlucky Cuthbert.
The Northern Irishman's first drive was parried by the former St Johnstone keeper, who then saved from Paul Heffernan but when the ball came out again Shiels rifled it low into the net.
Finally, Ayr responded, throwing everything forward but Killie held out for a merited but unnecessarily tough victory.
