The Donation of Constantine from the False Decretals of Isidore is not likely the origin of clerical colors, but it does justify the practice by identifying clerical attire as the Church's inheritance from the Roman Empire a la Constantine. The Donation of Constantine recounts how the Roman emperor Constantine granted the Western Empire to the Bishop of Rome and endowed his cardinals and bishops with imperial honors.
Here's the text concerning clerical colors from the "Donation of Constantine" so-called:
and to all the pontiffs his successors, who until the end of the world shall be about to sit in the seat of St. Peter: we concede and, by this present, do confer, our imperial Lateran palace, which is preferred to, and ranks above, all the palaces in the whole world; then a diadem, that is, the crown of our head, and at the same time the tiara; and, also, the shoulder band,-that is, the collar that usually surrounds our imperial neck; and also the purple mantle, and crimson tunic, and all the imperial raiment; and the same rank as those presiding over the imperial cavalry; conferring also the imperial sceptres, and, at the same time, the spears and standards; also the banners and different imperial ornaments, and all the advantage of our high imperial position, and the glory of our power.The Donation also states that Pope Sylvester refused to wear the imperial crown given to him by Constantine. Instead, Constantine invested Sylvester with the high white cap ("phrygium").
And we decree, as to those most reverend men, the clergy who serve, in different orders, that same holy Roman church [these are the Roman "cardinals"], that they shall have the same advantage, distinction, power and excellence by the glory of which our most illustrious senate is adorned; that is, that they shall be made patricians and consuls, - we commanding that they shall also be decorated with the other imperial dignities. And even as the imperial soldiery, so, we decree, shall the clergy of the holy Roman church be adorned.
The Donation's mention of the pope's white phrygium actually gives away the document as a forgery. This hat was worn by the pope in the 8th-9th century - thus proving that the document is fake. Here's a picture of Boniface VIII's white "phrygium":
By the time of Boniface VIII, a gold circlet was added to the bottom of the phrygium.
Lorenzo Valla proved in his 1440 treatise De falso credita et ementita Constantini donatione that the Donation of Constantine was a forged document based on Latin linguistic grounds. He was, of course, correct.