There are 5 categories that I use when I critique horror movies. Those categories are: Plot, Characters, VFX, Setting and Music / Scoring. I give each category a rating out of 10 (10 being the best), then average those scores into a final rating. It's also very important to note that I soley base my decision off of the movies. I will not take the book(s) into account at all. Below I will go into more detail on how I grade each category.
Plot: This probably the most imporatant categories of them all. What I look for in the plot is a solid foundation. Good plots will put the characters into a position that doesn't allow them to make the most realistic choices. For example, The Conjuring has a very solid plot. It's mentioned that the family had to move into that haunted house otherwise they would be homeless. This was a great way to pit the characters into a situation that would allow the rest of the story to make sense.
Characters: Characters is a tough category to grade because it depends on the type of movie. Their personalities are going to be different in a slasher movie versus a supernatural one. What I look for is that the characters are not air-heads. I want to see them making rational decisions given the situation they are in. Not every character has to have a complicated arc, but I don't want them to be just another body in the movie. An example of great characters would be Get Out. Almost every person in that movie served a role. There were a few filler characters, but overall there were no loose ends.
VFX: There's not much that I look for in VFX. I just like to see that the FX used are matching the background / scenery. Sometimes movies will add too much and make the object look completely unrealistic. A great example would be The Thing. The movie had great, practical effects that blended well with the scenery.
Setting: What I like in the setting is isolation. If there's a haunted house or a killer on the loose; I like to know that the main characters can't just run next door and the plot would be ruined. Although, this also depends on the type of movie. Halloween has always been set in a town, but the writing always makes it work. So it really depends on the story itself.
Scoring: More often than not, movies will start playing music or score to keep the flow going. Eventually, it becomes too much and it starts to interfere with the events in the movie. The Witch, The Conjuring, and Insidious are great examples of using scoring, or lack of scoring, to either make the audience feel they have space or become claustrophobic. All I'm looking for is music that isn't distracting to the rest of the movie.