Jim Sterling over on The Escapist did an excellent little piece on what's need for games to be scary, and while I more or less agree with his points I think that there's something else to it. In order for horror games to work they need to have an element of unpredictability to them.
Take Doom 3 or even Dead Space for example, both are a blending of sci-fi and horror which put the player in a survival situation. Both revolve around a series of jump scares and set piece moments designed to fill the player with dread. The problem is the pattern becomes predictable at some point and players quickly find themselves getting immunized, or worse yet annoyed, by each new "gotcha" surprise. Don't get me wrong especially when it comes to atmosphere these games are oozing with it, foreboding architecture mixed with ominously lit environments and an eerie sound score make for a great setting to cause some player anxiety. The problem is said players know where all the monsters are hiding. Couple that with lots of guns and it's not really scary, rather it's simply a dark action game. So, how to solve this problem? I think the best solution would be to take some advice from one of the masters, H.P. Lovecraft. In effect he's famous said that it's fear of the unknown that really gets under people's skin. To apply this to video games two things need to be done.
First, don't let players know the full extent of what their getting into. Doom 3 failed to frighten many players simply because the story and enemies were a rehash of the first game. Dead Space sufferes from a slightly different problem in that the main threat "Necromorphs" are exactly what they sound like. No twists like they are the byproduct of nanobots gone wrong or some kind alien collective of micro-organisms...sorry turns out they're just a bunch of mutant space zombies. If you watched the high production value trailer then you pretty much already know everything about your foe which kills a lot of the suspense.
Second, there has to be a kind of randomization to the scares. Not knowing where and when the monsters will appear increases the dread quite a bit. Condemned: Criminal Origins succeeded to some degree with this by using non-scripted AI routines. But to be honest dragon attacks in Skyrim are probably some of the best examples of how to keep players on their toes. Plus it avoids the monster closet or worse yet the teleporting behind you trick which gets especially irritating in subsequent playthroughs. Utilizing such a system also plays to the strengths of video games since other forms of horror media can really only get you the first time you experience it.
I should conclude by saying that I'm a big survival horror fan, and have played everything from Resident Evil and Silent Hill to Fatal Frame and Clock Tower. But I have to say I have never played a horror title that keeps the player in the dark knowledge-wise while fully embracing emergent gameplay. What would such a game be like? I don't know for sure, but I bet it would be pretty damn freaky.