[Blackhawk]

Since you say that you are using 1x4 trim, it sounds like maybe you have made you own trim. Most purchased door trim will have a wide groove milled along the length of the trim on the back side. This groove helps to absorb any sheet rock edges that are past flush. If you don't have this groove milled in your trim, use a router and mill in at least a 1" wide x 1/8" deep groove that will line up with the sheetrock to jamb joint.

If you sheetrock is wavy in the vertical, your easiest option is to fill up the gaps with caulk. Of course, this only works with painted trim. For stain grade trim, I see 3 options.
1) rip down the sheetrock and straighten or replace the crooked studs
2) live with the gaps
3) lay the trim against the wall and scribe the contour of the uneven surface to the trim, then use a belt sander to remove material on the back of the trim until the trim matches the wall