Swansea were a penalty in the 16th minute of an absorbing Liberty Stadium encounter after Nathan Dyer went down under Aaron Ramsey's challenge.
It looked a clear-cut penalty in real time - but replays suggested that it was the Swansea winger who had made contact with the Arsenal midfielder and not the other way around.
Scott Sinclair made the most of the opportunity to maintain his 100% record from the spot and score his fourth penalty of the season to level the scores at 1-1, Robin van Persie's 22nd goal of the season having fired the Gunners into a fifth-minute lead.
Dyer went on to put Swansea ahead, but Arsenal looked set to secure at least a point when Theo Walcott dinked over Michel Vorm with 21 minutes to go, only for a lack of concentration to allow Danny Graham to nip in behind Laurent Koscielny and fire the winner across Wojciech Szczesny less than a minute later.
And while Wenger acknowledged that defensive errors had proved costly, with Per Mertesacker and Koscielny never looking assured, he was far from happy with Oliver's failure to spot what he felt was a clear dive by Dyer.
The Frenchman said: "The game was difficult as Swansea played well, but the game was decided by some strange decisions, the first by the referee was the penalty which was a complete imagination and it was a good dive by the player.
"From then on there were other turning points like when we got back to 2-2 and made a defensive mistake, we showed a lack of appreciation for the ball and the defensive performance was not good enough.
"At 2-2 I thought we had a good chance to win as we looked dangerous and the keeper did well to keep them in the game and made some great saves, and we missed some unbelievable chances."
The defeat, their third in their last four away games, leaves the Gunners four points adrift of the top four and 12 behind leaders Manchester City, with their already slim title hopes hanging by a thread.
For Swansea the result moves them into the top 10, nine points clear of the drop zone.
The club have rightly won many plaudits for an easy on the eye passing style, and manager Brendan Rodgers had called on England manager Fabio Capello to pay a visit to south Wales to cast his eye over the likes of Dyer, Sinclair and Graham.
The Northern Irishman got his wish with the Italian in attendance for an enthralling contest, and believes the trio had made their point in no uncertain terms.
"He would have been surprised at the level they played at," he said. "The front three were a massive threat all afternoon.
"Nathan scored a terrific goal and it was a wonderful finish from Danny Graham, and I'm sure it gave him something to think about."
The win saw Swansea claim their first major scalp of their maiden Premier League campaign, and the result is arguably their most noteworthy since their last spell in the top flight under John Toshack's leadership in the early 1980s.
And Rodgers hopes seeing off one of the Premier League's elite mean his side to on to bigger and better things, although survival understandably remains their priority.
He said: "It was a wonderful victory and performance up against a top club, with a world class manager, so to come out with a win and perform how we performed is incredible, it's a wonderful day for the people in the city of Swansea.
"When you beat a top club it adds a new dimension and brings out another level, but I said to the players we enjoy our success but we have to keep fighting.
"We are not just a team of style and possession, we have resolve and to be successful you need that and we demonstrated that today."
