I had a similar issue recently, turns out the compressor was completely shot. My car is an 02 with 230,000ish on the clock, when the compressor was running you could feel a rhythmic vibration through the entire car and the engine would idle rough. Eventually, there was a loud squealing from the compressor and a nasty smell of rubber.
I replaced it with a re-manufactured part for £95, plus a new belt and idler. I also changed the condenser and a worn A/C pipe. Nearly 4 weeks later and all is well, nice cold A/C and no vibration.
It might be worth checking your A/C pipes for wear in the common places - behind the rad where the PAS pipe sits, top of the engine where it can rub on the bonnet and next to the washer bottle.