Just my oppion:
I think common sense would tell us that bucks that drop later would have acclerated antler growth. If you think about it, lets say we got a buck that doesnt drop till end of march and compare him to a buck that dropped the first of January. So we have a 3-4 month period of difference between two bucks. Now i think it can be agreed that all bucks shed there velvet within a month of eachother. I usually find that most are within two weeks. So this would lead me to one of two conclusions: 1. The buck that dropped later experienced accelerated horn growth, or 2. The buck that dropped later never got as big as compared to the buck that dropped earlier. I think its pretty simple to see that the second is just stupid. It would be the same arguement if you were to compare birth date of fawns. To say that there is not some kind of accelerated growth through regulation of gene expression, is saying that the later born fawns will never get as big.
When I say regulation of gene expression, what i mean is that every gene in a deer produces a protein. The most abundant protein in deer antler is collagen. So when there is a need for collegen the gene is turned on it and it starts producing antler.
The same idea holds true for the other component of antler, that is minerals. Deer deposit minerals in there skeletons throughout the year, and when as shredder said before, the daylight hours increase and we see a increase in testosterone and a decrease in melatonin, we also see a transport of minerals from the skeleton to the antler. Its like a form of osteoporosis, simply the transport of minerals through bone.
Another point that i think is worth making is that antler growth comes second to body growth. A bucks body wont focus on growing antler until the needs of its body are met first.
All of this together just kinda of shows that "growth" is not a linear or constant process. What I have said is somewhat based on science, and some of it is based on my opinion. Deer are not exactly a highly studied organism and the exact mechanism of antler growth Im sure is not fully understood. Sorry for the long anwser but I love talking about stuff like this. By no means am i trying to say i know everything, just a few ideas I thought would apply to your question.