I agree that after humans were enslaved, they would have gone after Andalites in revenge, and taken their homeworld back.
Still though, sure they'd have enough humans for everyone, but in all honesty, humans suck. We're physically weak, slow, need a relatively varied as well as large diet to maintain health, etc. We consume too much based on our physical abilities. Hork-Bajir just need bark. That's basically it. And plants can be dried and compacted, and can be easily reconstituted. That's just with simple human technology. I'm assuming they'd want a race that is more physically abled for combat, as well having a simple mind, and lesser need for resources. Hork-Bajir are great hosts, as well as Garatrons for their speed.
I'm sure after they are done with humans and Andalites, they'd continue hunting for the perfect host race, conquering everything on the way. Even if they found the perfect race, they'd continue on because it's all they'd be able to do at that point. A race warring so generations just doesn't decide to stop. Also include the greed of leaders like our friend Visser Three/One.
I see the peace faction as a relatively small proportion, and eventually they'd drill out sympathy for hosts through propaganda. Remember how Russians were EVIL in the 60's? And how they thought the same of us? Same deal.
Assuming they succeeded in these endeavours, the only way I see for them to fall would be internal conflict. The only way for internal conflict to really tear them down would be something like another Visser battle; two powerful leaders with similar resources fighting it out for dominance, possibly with Tom-like traitors seeking hierarchy. When this fault or civil war happened, a powerful race would need to take them down, because eventually one of the Vissers would win, and rebuild.
