As a political term, Tory was an insult (derived from the Middle Irish word tóraidhe, modern Irish tóraí, meaning "outlaw", "robber", from the Irish word tóir, meaning "pursuit" since outlaws were "pursued men") that entered English politics during the Exclusion Bill crisis of 1678–1681.
I have seen it said that the description ‘toe-rag’ has the same etymology. That’s not commonly suggested. I can easily see that it is possible, especially given the use made of it in Scotland.
Who would have thought it?