Our follower graph does not only show you the number of followers gained, but also the number of followers lost. For example, if you've gained 45 new followers on saturday, but lost 4 followers that same day, the total would be 41 (45-4 = 41). This also allows for accounts with negative growth, in other words, losing followers.
A second thing to take into account is the moment that your account was updated and the time zone settings we are using. We might update your account at 8pm on a saturday, missing the last 4 hours. When you update your stats the next day these last 4 hours are also counted, because we count against the previous update. Also, 8pm server time, might be 1am your time, or 1pm your time.
We take our numbers directly from Twitter, so there is no error margin on the numbers. The only actual difference could be noted in when your stats were updated and the difference between your time zone and our server's.