Rovers scored three stunning goals through Chris Brown, David Cotterill and Kyle Bennett to knock the stuffing out of a young Saddlers side who played good football but lacked quality in the final third.
"I think that second goal was the killer for us," Smith said. "We played some fantastic stuff in the first half, but they scored after good work by Paynter and then hit two more wonderful goals.
"We won't get too downbeat. We are disappointed to lose 3-0 but we have work in progress and only time will tell where we end up. We shall build the players up again after this and won't let it affect where we want to go."
Smith signed former Doncaster defender James Chambers on a short-term contract the day before the game but he could do little to change the result against his former employers.
Asked if Smith felt people would look on Walsall as relegation material after the heavy opening-day defeat, the Saddlers boss replied: "I don't care what they think, we shall get on with the job and see where we go."
