The number of fans you gain or lose is determined by your performance on (and duration of) jobs, dates, and parties, and also by your rank.
A project completed with four or more stars will receive positive reviews and grant you more fans.
Conversely, a three-star rating or lower will incur negative reviews and cause you to lose fans.
A-List stars will get longer projects and earn more fans for a perfect completion than stars on the lower lists (like C-List).