Chew the rag, although its original origin is unknown was first in print in the 1875 printing of “The Random House Historical Printing of American Slang”, and then in 1885 appeared in the same sentence in a book “Life in the Ranks of the British Army in India”.
Some speculate that the term relates too cloth and ladies sewing circles or women gossiping while quilting.
Source: Wikipedia