Great Britain coach Stuart Pearce is currently sounding out the individuals who may be available for the London Games.
But with the Olympic football tournament not finishing until August 11 Moyes, who prefers to run a small squad, does not think he can afford to give any additional time off to anyone involved.
"Quite a lot of our players have had letters of invitation," said Moyes, who will not apply the same criteria to players involved in Euro 2012.
"Am I worried? I am because there's a lot of them got all sorts of things going on over the summer so I am a bit worried about the whole thing.
"It's a real difficult one. It would be hard to say to players not to but in the same breath, this (Everton) is their bread and butter and this is where they get paid.
"They do need holidays and they can't then be asking for spare time off that doesn't fit in.
"If they say 'I'm playing in the Olympics then I might need a week off to recover', there can never be anything like that."
Moyes added in the Liverpool Echo: "We accept international tournaments. What we're willing to do to those boys is give them a bit of extra time off because the players definitely need a holiday.
"But I think the players going to the Olympics will have to consider their own holiday time and it couldn't be eaten into the club's time."
