Missing a turkey doesn't mean you are using the wrong shot size. Any of the legal shot sizes will kill gobblers dead if the gun is patterned well. If you havn't already, get yourself a big piece of cardboard and shoot your current load and choke combination at a dot and turkey head and neck outline in the center at different distances (20, 30, and 40 yards are good ones to use). You may have some holes in your pattern that you really didn't know about. You need at least 12 to 14 pellets in the kill zone to be really certain of your combination. Yes, it only takes one pellet in the brain or spine to kill a bird, but you really should have plenty more. Play around with different chokes if you have them as well. Size 4's are killers....as long as they are patterned well and the range is good. Many people think that because they are shooting bigger sized shot, that they can take longer shots...this can be a costly mistake. A good thing to remember is: the tighter the choke (such as a XXX full), the smaller the shot size should be. For example, many times a high velocity load of 6's will pattern MUCH better than a load of 4's or 5's out of that tight of a choke. Personally I use high velocity loads of 6's out of a .665 choke. If smaller sizes are used, such as 6's..I would highly recommend the higher velocity loads. They really knock 'em dead.
Know your gun, know your shot pattern and know your effective kill range. Hope this helps!