Caveat: I'm a horrible player for a horror game.
But there was a thread on RPG.Net a couple months ago, [Horror] What scares you?, and while there are a lot of good, standard answers (body horror, loss, dis-empowerment), one of the posters actually broke horror, in an RPG context, into three large categories that I liked: Dark, Creepy, and (as I summarized it) Suspense. I'd add in Helpless trait as well, even if (because) it just irks me.
Dark is horror based around the lack of good choices - do we kill the children or release the disease into the larger population. The choices of nastily horrible and horribly nasty. This is my favorite to have subtly done, and best done by not hammering the horrible choices, but letting the PCs be PCs and then having an outsider note how this is really, really fuckin' weird. Such as the incident that caused us to coin "Gunteel" as a portmanteau of "gun" and "genteel".
Creepy is horror based around atmosphere. Your name etched in glass. Body distortion and twisted. Mold and and random holes in the walls where you realized you've been watched. The creepy little girl, the off insectoid voice that I used for the tutor demon in Exalted, all examples of creepy ambiance.
Suspense is horror based around unknown and the eventual surprise. It is the most time delineated RPG horror trope because so much of games are seeking answers, and when you find them shooting it repeatedly. For example in Deadlands up until we had the ritual to stop the end of the world there was a sort of futile desperation setting in as we just kept trying to move forward and floundered. But the moment we had a solution we turned back into our highly lethal selves again.
Helplessness is horror based around being ineffectual. Nothing you do matters because it will have no grander effect. For me, games around about "doing things" so I will always turn and fight or resist, sitting there moping isn't all that fun for me. However, used in a limited sense, it can inspire fear/trepidation and cause players to start scrambling for a solution to end that helplessness. But over used and it turns into "Really? Another thing we can't hit until you give us the solution? Let me know when we can do something again."