
This post is a written explanation of my video tutorial (4 minutes).
This covers just one material function implementation to handle material masking of meshes. I implemented the material function in Unreal but the concept can be transferred to all game engines.
Overview
There are multiple ways to implement the visuals of the player looking through a gun's scope during its aim-down-sight (ADS) animation. The most popular being Picture-in-Picture (PiP) and flat-image scopes. Like every implementation of features in games, there's pros and cons to these.

PiP is the most physically accurate and realistic portrayal of image, however it has an extreme performance and optimization cost with using render targets. This method does look great but the performance impact stops it from feeling great

Flat-image scopes is the simplest and easiest implementation, but it limits player feel and immersion. It feels like you're looking into a microscope instead of an actual scope
In this post, I will go over a new method that outperforms the PiP method while still maintaining a realistic image.
Concept

In this method two factors are in play that bring out the scoped ADS illusion.
1. FOV shrinking
2. Material masking of gun and player hand mesh visible in scope view.
The second part is the most important and technical. We need to make a material function to determine if the pixel is within the scope's view and needs to be masked out. This is surprisingly very simple to figure out using some trigonometry.
Given a pixel world location anywhere on the gun, we can determine if we are in the scope's view by comparing the ScopeAngle with the PixelAngle.
Finding out the PixelAngle is done through the dot product. If the PixelAngle is lower than the ScopeAngle, then we are within the scope's view so we need to occlude this pixel.
Implementation
The implementation is fairly straightforward with the dot product section being handled with a native material function. Writing the implementation in a material function allows for to reuse in multiple materials. I quickly learned that I needed to apply this function to my character material as the hand would occasionally pop in.
Because this is a visual effect only seen by the locally controlling client, we can use a material parameter collection to update the location of the scope without having to update it directly to each relevant material.

Once you have the material function, set up the material to be Masked and hook up the material function to the Opacity Mask.

In code you want to update the location of the scope every tick.
I highly recommend setting the event tick group to Post Update Work as it will have a better value of world location to render at that time.

This plane holds the material for the crosshair and other effects to complete further the effect of ADS.
Notice how this plane is just barely in front of the optic lens. This allows us to see through it with the occluder material function.
From here you should have your gun meshes occluding within the scope angle. Some more tuning and polish can be done to finish the effect and make it your own unique style.
For example I used the inverse result of the occlusion material function to blur the screen outside of the scope, furthering the ADS effect.
Conclusion
GOOD:
What makes this method better than PiP is that it performs much better and allows for a much higher level of immersion compared to the flat-image implementation. The ADS transition can be done seamlessly without any pop-in of images or drop in performance. The workflow isn't anything complex as the material code works on every optic.
BAD:
The only drawback I have with this method is that it relies of the FOV being changed to create the zoom effect. I know some gamers prefer the non-FOV change in ADS of PiP implementations. If you want no FOV change during ADS then this method won't work for you.
This isn't exactly "realistic" as the image is still from the viewpoint of the camera instead of at the end of the scope like in PiP. This negative effect still exists in flat-image scope and gameplay still feels fine so it's not a major concern.
Thanks for making it this far!
I spent a lot of time figuring out this method and seeing it through. I'm extremely happy with the results and plan to use it in my FPS project.
How to remove the muzzle and player hand from view within scopes in a more efficient and performant matter.