With the loss, the Browns drop to 7-6 overall and host the Bengals next week at FirstEnergy Stadium.
The Browns trail the division leading Bengals who are 8-4-1 by 1½ games and the Ravens (8-5) and Steelers (8-5) by one game. The Browns play the Bengals, Panthers and Ravens to close out the season.
Pettine said the season isn't over, but the Browns have to win out to get in the playoffs.
"No, it's an opportunity we...it's a kick in the gut," he said. "It's an opportunity that we let go. We know our margin for error is now zero, that we've got to win our last three or we're done."
Thieves: The Browns took the ball away four times from the Colts and converted two of the turnovers into touchdowns. On the season, the Browns are a +8 in turnover ratio.
CB Justin Gilbert registered his first NFL interception and returned it for a 23-yard touchdown, which put the Browns up 21-7 with 10:21 to play in the third quarter.
"Obviously, need to shout out to Justin Gilbert," Pettine said. "I thought that was...for him to come off the sideline cold when K'Waun Williams got hurt and play at a high level, I think that's a great boost for his confidence. I thought the pick six was a play that he missed an interception on a couple of weeks ago on the same call and found it.
"Then, he obviously showed his skills finding the end zone," he said. "Very proud of Justin. It's a tough week. I thought he had a great week of practice and it showed in the game."
Cameron's Return: TE Jordan Cameron returned for the first time in 5 games. He had four catches for 41 yards, including a 20-yard catch.
Bouncing Back: The loss to the Colts marked the first time the Browns have lost back-to-back games this season.
"We have a division opponent next week against Cincinnati, so we have to be prepared," Joe Haden said. "This was definitely the worst loss that we had this year, but we have to forget about it and get ready for Cincinnati coming into our house next week.
"We are going to come in (Monday), study the film, and figure out what we need to do better," he said. "The best way to move forward is by beating Cincinnati."
Kruger's Day: LB Paul Kruger had a forced fumble, recovered a fumble and registered half a sack. After totaling 4.5 sacks in 2013, Kruger has 8.5 to lead the team this year. DL Desmond Bryant shot up the middle on third and 10 and hit QB Andrew Luck, along with LB Paul Kruger that forced a fumble in the end zone recovered by LB Craig Robertson to give the Browns a 7-0 lead with 40 seconds to play in the first quarter.
It was the first of two defensive touchdowns as CB Justin Gilbert intercepted Luck for a 23-yard touchdown return.
Later, Kruger had a sack taken away after dropping Luck and was assessed a personal foul for hitting the quarterback in the head.
Leonhard's game: DB Jim Leonhard, filling in for the injured DB Tashaun Gipson had an interception to go with a sack and six tackles. The Browns were able to convert the interception into just a field goal with 8:34 to play after Leonhard returned the pick to the Colts' 23.
Bad Day for Billy: K Billy Cundiff missed a 40 yard field goal, which was the fifth-straight game he missed a field goal. He did convert on a 39-yard attempt later in the game.
"It's obviously concerning," Pettine said. "We get field goal attempts, we get a chance to put points on the board against a good team and you've got to be able to convert."
Gordon Slowed: WR Josh Gordon was absent on third down during the game on several occasions. He finished with just two receptions for 15 yards on seven targets.
Pettine said the plan was to rotate the receivers more.
"Just they rotation that they talked about," he said. "I think we wanted to get Josh's reps down a little bit and kind of balance it out a little more, get Travis Benjamin and Taylor Gabriel, get their reps back up a little bit."
Mingo sack: LB Barkevious Mingo sacked Luck and finished with two tackles.
Running Game: The Browns rushed for 11 yards on 31 carries for a 3.6 average with one touchdown. RB Isaiah Crowell and RB Terrance West each had 54 yards rushing on 14 and 15 carries, respectively. Crowell scored on a 9 yard run to give the Browns a 14-7 lead with 1:14 to go in the first half.
Williams hurt: DB K'Waun Williams injured his hamstring in the second quarter and did not return.
Old Friends: LB D'Qwell Jackson, KR Josh Cribbs, DB Mike Adams and RB Trent Richardson are all former Browns playing for the Colts. Former coach Rob Chudzinski, Chuck Pagano, Hal Hunter and Brant Boyer are all coaches on the Colts staff.
Cribbs and Jackson were captains for the Browns. Cribbs had five kickoff returns for 141 yards (28.2 avg.) with a long of 41 yards. He also had three punt returns for 4 yards.
Jackson tied for the team lead with 9 tackles. He also had a tackle for loss and a quarterback hurry.
Adams intercepted Hoyer in the end zone to stop the Browns first scoring opportunity. Adams had four tackles, as well.
Richardson had seven carries for just 30 yards, including a long run of 8 yards.
