Did your doctor suggest that you stop taking the mirtazapine after three weeks? If so, did he or she say why? From what I can find on the web, mirtazapine is an antidepressant that can take several weeks to work, so I'm not sure why your doc would take you off of it after just three weeks, particularly if it was working for you. If I were you, I'd talk to your doctor and tell him or her what's happening, and he or she might put you back on the drug (if it was his or her idea to stop it in the first place).
When I first got T, my psychiatrist prescribed trazodone (for sleep), clonazepam (as needed for anxiety), and an antidepressant. I decided on my own to reduce the dose of trazodone and clonazepam, which was a bad idea; my anxiety came roaring back. After six weeks, my anxiety was greatly reduced, and I was able to taper off the clonazepam with no problem. Everyone has to make their own decision about the drugs, but if it's between being able to function on the drugs vs. being a complete mess without them, then give me the drugs.