Your taxation affects your loyalty.
Your loyalty is a reflection of your current taxation and the amount of townspeople you can have at any time is based on your current loyalty. Your number of townspeople will go up or down based on your loyalty.
Loyalty is basically the opposite of taxation.
If you have 20% taxation you can have 80 Loyalty. (100 – 20 = 80)
If you have 50% taxation you can have 50 Loyalty and therefore your current population will be half (50%) of your total population.
If you have 0% taxation you can have 100 Loyalty and therefore your current population will be 100% of your total population.
Also, the lower you have your taxation, and the higher your loyalty, the faster population rebuilds after queing troops.