There are certain cases where the projection of a sensor footprint may not intersect accurately with the central body terrain. In the example images below, you can see that the intersection footprint passes through the terrain surface and is drawn inside the body.
Projection with Terrain
Projection with terrain visually turned off:
In this example, the inaccuracy is due to the very small size of this custom central body and the fact that, relative to its size, the sensor is very far away (>13 times the central body radius). STK has some tolerances that look at these ratios to determine, in part, how accurately to detect terrain crossings. Ordinarily, these tolerances may result in small errors that would be hard to notice, but on the scale of this central body, the errors are quite visible. This issue can be resolved by setting an option in the .sn file of your sensor in the scenario folder. Search for the 'DetailedTerrainHorizon' setting and change it to "Yes".
Example: DetailedTerrainHorizon Yes
This option will make rendering of the scenario slower, but STK will detect the right crossings. The result should now look something like the image below.
Projection with the DetailedTerrainHorizon option selected
This feature is on by default as of STK 11.