Plus Alloran himself was already more than slightly off his rocker, which Visser Three probably absorbed by being in his body for years at a time. Take a look at the Visser Three in the Hork-Bajir Chronicles and then the Visser Three we got in the series. It was definitely Alloran's insanity that drove Visser Three over the edge, IMO.