Jatinder Singh, from Smethwick, was summoned to serve as a juror at Birmingham Crown Court on Monday
But, he said, a security guard refused him entry at the court over his kirpan, the sword carried by all Sikhs.
The Ministry of Justice (MoJ) said Mr Singh was released from his duties as there was a surplus of required jurors.
Meanwhile, His Majesty's Courts and Tribunals Service (HMCTS) has apologised to Mr Singh.
Khalsa Sikhs carry the five Ks with them at all times, as a symbol of their faith.
These include Kesh or uncut hair; Kara which is a a steel bracelet; the Kanga, a wooden comb; Kacca or cotton underwear and the Kirpan
Mr Singh, who has served as president at Guru Nanak Gurdwara in Smethwick and as secretary general of the Sikh Council UK, said this was the second time he has been summoned for jury service, the first passing with no issue.
On this occasion, he entered the morning session with no problems, but when he tried to return to the court after lunch was pulled aside by a security guard and told he could not go in.
"The security guard said I could take [my kirpan] off and leave it with him and collect it at the end of the day," he said.
"I felt like a child who has gone to school and taken something they shouldn't and had it confiscated.
"To have that happen to me, I felt embarrassed, I felt discriminated against, I didn't expect it to happen to me."
He called for the (MoJ) to work with Sikh and other religious organisations to create easily accessible guidelines that can be provided to staff.
Dabinderjit Singh, the principal adviser to the Sikh Federation UK said it had written to Justice Minister Alex Chalk asking him to condemn the treatment of Mr Singh.
The MoJ said members of the Sikh community wishing to enter a court building could bring in a Kirpan which was not more than six inches long (15cm) and with a blade no more than five inches (12cm) in length - which Mr Singh said his was.
A spokesperson for HMCTS added: "We have apologised to Mr Singh for any distress caused and have reminded our contracted security officers of the correct steps to take to prevent this incident from happening again."
This is an interesting problem. On the one hand, the rules were clearly defined and he followed them. On the other, I feel that religious exemptions are a slippery slope that could enable poor behavior. Though his behavior was not poor, others could use that as an opportunity to act poorly.
Ideally I'd love a policy that meets our needs when applied uniformly, but in this case I don't see an alternative to individual review of religious exemption requests. But who reviews the reviewers? It is a laborious bootstrap problem and would ultimately need a certifying body like we have NIST for measurement standards.