I think that without the Ellimist everything would be the same up until a certain point. As soon as you throw the Time Matrix into the mix, you have problems.
No Time Matrix would simply mean that Elfangor's interaction with the humans would have ended as originally scheduled (probably), with Loren and Chapman being dropped off on Earth with memories erased and he would go on his way. Maybe he would become great, maybe not. But with no Time Matrix there would be no need to go to the Taxxon world with the humans, which would mean that Esplin 9466 does not infest Chapman and alert an old Skrit Na outpost of his findings, which means that Edriss does not abandon her post in search of Earth to begin the infestation.
Perhaps eventually Earth would have been found, but it would have been delayed.
Keeping this in mind, I would add that this would allow the Yeerks to concentrate their forces on Leera and in the Anati system. The Animorphs would obviously never exist as we knew them (Tobias wouldn't exist at all) and therefore they would not tip the balance of that war. Leera would fall, and possibly the Anati system as well since there would be more resources and host bodies to spare after a victory on Leera. Essentially, the Yeerks would become even more unstoppable. In the future, if they did discover Earth, they would have widespread Leeran and Anati hosts in addition to Hork-Bajir, Taxxons and Gedds, which would make the situation even more dismal for Earth.
Oh, and Visser Three never would have infested Alloran because he would have never BEEN on the Taxxon world to become infested. Maybe Esplin would work his way up the ranks, maybe not. Likely Edriss and Esplin would both struggle to make it to the top without their token victories.