I've got a bit of experience, and it can be a bit of a pain depending on the configuration, and your needs.
The limiting factor, primarily, is the Server Admin tools, which only allow the alteration of a fraction of the settings actually available in most services. If you modify the actual config files too much, Server Admin may stop working with that service entirely.
I've been running OS X Server as my personal public-facing email/web/whatever server for a few years now, and it's pretty stable on the whole. If I was starting again, I'd do it with something like CentOS, which is much more versatile than OS X Server.
What it boils down to, in my 2c is...
If you're happy with Google Apps, I'd suggest sticking with it. The anti-spam is unparalleled, and downtime is pretty rare.
WARNING: If you see links to ads in the above post, blame the cheesy ad-linking software used by the owners of this website. I do not endorse these ad links. Don't click on them.
iMac 27" 3.4GHz Core i7, 20GB RAM, 256GB SSD, 1TB HDD • 27" LED Cinema Display
MacBook Pro 13" 2.8GHz Core i7, 16GB RAM, 120GB SSD
iPhone 5 32GB • iPad 2 WiFi + 3G 64GB
Mac mini 2.3GHz Core i5, 16GB RAM, 128GB SSD (XBMC)
Synology DS1512+ NAS • DX513 Expansion • 20TB