I don't know about XQuartz and Yosemite but I can let you know if I I try it. I will mostly just be using X11/xterm and Inkscape though.
Jim: I have XQuartz on my other computers, that I downloaded a while ago. I just wanted to confirm that macosforge doesn't allow malicious activity on its site like sourceforge (and other download sites). It's probably fine.
As for sourceforge, it seems they sometimes bundle honest software with adware/bloatware using their installer (perhaps this is for Windows only though). They also allow malicious ads.