Sauron cut the forest near Isengard, but the Ents atrophied of their own accord, having lost the Entwives - and it wasn't presented as though there had been terribly many to begin with. There were only about 40 depicted in the movie, you may recall, but more than 50 marched on Isengard, with Treebeard leaving after Gandalf's visit in the night, apparently leading the Huorns (tree creatures similar to, but not quite Ents, but that followed the commands of Treebeard) that went to Helm's Deep and wiped out the fleeing forces of Isengard - more than 10,000 had marched on Helm's deep, and it didn't seem that more than 1,000 had been killed off before they got there and then swallowed the fleeing army as Theoden and Aragorn rode out against them. Merry and Pippen told them the stories that led to the Entmoot, but the Ents themselves chose to fight - the scene depicted in the movie, where the Ents decide against war, is pure bunk - they chose war on the first vote. BUUUULLLLSHIT! The Ents already knew that Saruman was destroying the forest!