Hourly Pay

I am a CPA with 20 years experience for large companies doing tax provision work. I do have tax return experience but it was at the beginning of my career. I am looking to pick up a few hours a week (6-10) in a local CPA firm to gain experience in the small business sector. I want to learn everything (even things a CPA might delegate to others) as my end goal is to be able to pick up clients of my own to do from my house. My question is, what is a typical hourly rate for a CPA in a small firm in Atlanta. Some of the suggestions online seemed quite low so wanted to double check here.

Atlanta is like many markets where you are going to have a large range of hourly fees even among small firms. $125 on low end, probably up to $350 on higher end from what I have seen.

In most cases, you're not going to make much money on taxes if you bill straight hourly. Utilize value pricing and establish minimum fees.

I assume you mean an hourly rate to work for someone else, not the rate you (or your employer) would bill out for your work.
I'd suggest checking the Robert Half salary guide. Your experience sounds limited in this type of work, and you want to learn everything from say junior/associate level to a senior level. Bookkeeping might be billed at 50-80 an hour and might pay 15-20 as an hourly rate, while tax prep for a junior might be in the 75-125 an hour billable rate so maybe 20-40ish an hour?
Keep in mind that you want to learn, and they want/need to make money from having you work there. Once you realize that dynamic and you figure out what they can bill for your time, you realize what you might expect to get paid.
I'm not sure how your tax provision work will translate to a normal small firm/tax practice.

Bookkeeping might be billed at 50-80 an hour and might pay 15-20 as an hourly rate, while tax prep for a junior might be in the 75-125 an hour billable rate so maybe 20-40ish an hour?

That agrees with our rule of thumb - pay rate = 1/4 of billing rate. Assuming a normal amount of non-chargable time and downward realization adjustments, you should expect total compensation to equal 40% of realized billings.

You can search indeed.com and narrow your search to the Atlanta area. They will publish job openings and hourly rates. You can also do a search for a remote/contract accountant and that may give you additional insight.