If that's the case I would hold myself out primary for income tax compliance services if I was you....not for bookkeeping and payroll.
Make that your bread and butter, and up-sell tax consulting where you can.
Depending on what kind of business tax returns you did, that experience may or may not help. For example, if you did primarily C Corp returns, you may find that most small businesses are Sch C (1040), partnership or S Corp.
I agree with nova that your geographical location matters. Locating yourself in a town/city that has higher disposable income than the surrounding area, and a growing population and economy matters, a lot.
If your boss will allow you to work on the side, that's the safer route at first. That's not always an option though.
Your biggest problem initially, and the one you need to work on, is generating leads.
You'll also need savings to cover yourself during the start-up period. Remember the motto: everything in business takes twice as long and costs twice as much as initially thought.
Good luck.
Might be opportunity for CFO services as well given your background.