Besides gaining new followers it's very likely that you're also losing followers. Your numbers are probably decreasing because more users are un-following you than following you.
The numbers we show are the numbers we get through the Twitter API. These numbers are accurate. If you're unsure at any given time the numbers are not accurate you can click on the "Update stats now" link at the top of your profile page to update the stats.