Levy believes it is imperative for Spurs to find a new home if the club wants to develop and that protests against any move by "a very, very small group" of supporters will not change his mind.
Spurs are going head-to-head with West Ham in the race to move to Stratford and a decision on who should get the ground that was originally planned for Friday.
"I think the problem with the current situation that we are in now at White Hart Lane is that the project is currently not viable so we would have to go back to the drawing board and that would obviously mean looking at other locations again," Levy told Sky Sports.
"I'm totally confident that we would have no problem selling out a 60,000-seater stadium [at Stratford]. As far as the fans are concerned, I have to say that it is a very, very small group of individuals that are making it clear that whatever happens - even if we couldn't build a bigger stadium - that they would want us to stay here.
"There is always going to be competition between London clubs - we are all London clubs - so wherever we are located, there is still going to be rivalry."
Former Spurs chairman Lord Alan Sugar believes the supporters will soon get over the disappointment if it takes the club forward.
"The next important thing is that the club is ready to move to the next level," he said.
"Regretfully, moving to the next level might not be possible in Haringey due to a lot of things that have come out as far as the financial issues are concerned. It could put the club into a lot of financial problems."
