.223 Remington is an ideal varminting chambering. It's alot softer on barrels, has a WIDE selection of ammo, and can be used for other game as well.
.204 Ruger has about 3 - 4" of drop out to 300Ys. It is a death ray caliber, just the same as .220 Swift and the light .22-250. Either one is going to suck beyond 300Y and have alot of wind drift.
If I had to choose, I'd go .204 Ruger, but .223 Remington is a close second.
Don't get me wrong .204 is a badass round and a fast lil SOB but up here in north Texas it got pushed around alot most of the guys I shoot with use either .223 as you stated or .22-250.
In my experience the .204 has slightly less wind drift than the heavier .224 caliber bullets.