Etymology
From rag + horn, describing the animals as non-prime trophy specimens, i.e. "rags". Another explanation is that it is descriptive of how the new antlers have a velvety covering that sloughs off each year and can hang like rags.
Definitions & examples from Wiktionary (CC BY-SA 3.0).