Praise be to Allah.

If that was due to ignorance on your part, then you do not have to do anything. If you were not sure whether it was saffron or not, then you do not have to do anything. If you were certain that it was saffron and you knew that it is not permissible for the muhrim to drink coffee in which there is saffron, then if there was a fragrance, then he did something wrong, but if there was nothing but the colour, then there is no sin on him for that.

Shaykh Muhammad ibn ‘Uthaymeen (may Allaah have mercy on him)

http://www.ibnothaimeen.com/all/noor/article_6040.shtml

Shaykh ‘Abd al-‘Azeez ibn Baaz (may Allaah have mercy on him) was asked: A woman who was in ihraam for ‘Umrah drank coffee with saffron before completing ‘Umrah. Is saffron a type of perfume? Did it spoil the ‘Umrah or not?

He replied:

The muhrim who drinks coffee containing saffron is doing wrong, because saffron is a kind of perfume and should not be used in coffee by the muhrim, just as he should not use it on his clothes or body when he is in ihraam. If the man or woman in ihraam does that out of ignorance or by mistake, then they do not have to do anything, but if they do that deliberately, knowing that he is in ihraam, then it is not permissible. He (the muhrim) should give charity, feeding six poor persons, giving each one half a saa’ of dates or wheat, or fast for three days, or sacrifice a sheep. End quote.

Majmoo’ Fataawa al-Shaykh Ibn Baaz (17/129).