Whether you should apply crop factor to aperture depends on whether you're interested depth of field, or exposure. You may or may not be able to adjust ISO to get comparable results at 50mm f/3.4 depending on which cameras you're comparing.
Noise is decided by the quality of the sensor, and the solid angle of light that hits each point on the sensor. The solid angle as it happens also decides depth of field.
So it is always true that you should apply crop factor to aperture. ISO is just not the correct factor to describe gain because an inverse crop factor is applied to it in effect.
So at the same exposure and shutter speed, at f/3.4 on FF you will get the same image with the same amount of noise as you would at f/1.7 on M4/3.
You'll have to set ISO higher to get the same image at f/3.4 as you would at f/1.7 on a smaller sensor because the intensity of the light is reduced.
I'm interested to learn more about the effect of light angle on noise. Do you have a source that talks about this? I didn't find anything useful in 2 minutes of searching the web.
No, the intensity of light is not reduced. ISO is not gain. In other words, ISO 200 on full frame is the same as ISO 100 on M4/3 in terms of what actually happens inside the sensor. This is because of how the ISO standard is written.
As for light angle, it's a consequence of the second law of thermodynamics, that translates in optics as the conservation of étendue.
Basically what this means it that the amount of light captured from a light source is equivalent to the solid angle of light from that light source onto the entrance pupil - whose size is the focal length divided by the f number.
So basically, what determines in the same scene how much light is received by the sensor depends on the size of the aperture in milimiters.
So on a sensor of the same efficiency, it's the physical size of the aperture that determines how much light is captured.
The intensity of the light is reduced because the aperture is the same diameter (50/3.4 = 25/1.7), but the light has to cover four times the area. To get the same exposure with double the f number and the same shutter speed will require four times the ISO.
> what determines in the same scene how much light is received by the sensor depends on the size of the aperture in milimiters
That's true, but the amount of light is not the intensity of light.