You would.
Some info on why 0mrad is not possible, and theoretically cannot exist. (Because I like this sort of theoretical physics stuff.)
Chaos theory comes into this too, essentially that if something is perfect, something will interfere with it and destroy the perfection. Something like that, I'm sure someone will correct me.
It is possible, theoretically, with very good optics to achieve very low divergence, <0.001mRad or something. But, if you could get 0mrad, how would you know it was? How would you know it was 0mrad, rather than 0.00000000001mRad or something? You simply cannot measure things carefully enough to know stuff like that.
Also consider this. No optical system is 100% perfect/ loss free. You simply cannot create an optical system to perfection because anything you do has losses in it. This is why perpetual motion is not possible. You simply cannot get 100% of the energy out of something that you put into it. If this was possible, optical trains would almost be like superconductors. (BTW, same thing. They may be very efficient but not 100%. No such thing as 100% efficiency.) And if you could get more energy out of an object than you put in it, well, you can't. There is a set amount of energy in the universe; it just gets converted into different forms.
When you lose photons in an optical system, they most likely hit something and excite its molecules to produce a tiny amount of vibration (heat).