Well, technically atheist extremists would uphold Soviet-style "state atheism" where religious groups are repressed violently and religious affiliation is outlawed. Killing and repressing people for being Christian or Buddhist or whatever would be just as bad as doing the same thing to people for being atheist. Of course, unless you live somewhere like Xinjiang Province or North Korea, you're very unlikely to encounter any significant organisation which seeks to actively force people to abandon their religions.
Basically, unless someone is running a scam like Scientology, promoting a violent extremist sect like Wahhabi Islam, shunning "apostates" like Mormons or just running a flat-out doomsday cult or something, people should be allowed to practice a religion, own a holy book and convene in a designated place of worship with peers of their faith. They just shouldn't be allowed to compel others to join that faith, or enjoy privileges from the state such as a blanket tax-free status.