[ken_yap]

You just quoted some numbers for sidux and Fedora without saying how you measured them. You should be looking at the line:

-/+ buffers/cache

and without any apps running in the GUI. Also on the exact same hardware, because the footprint of the X server depends on what drivers are used. Also some of the video memory that's mapped in is also counted in the X server's footprint. So unless you are careful in what you are measuring, that 100MB may be significant, or not.