To encourage player to continue, I think the best options are:
-Make it so the death isn't unfair
the player has to know beforehand that something can be dangerous, I know it seems stupid that I say this, but players don't know what to expect in some cases, if a new sort of trap appears, make it so the player see it working once, so he can learn from that and thus, will forgive you if he died because of that.
-control often, and not only by yourselves, the difficulty curve of your game
a game doesn't have to be easy or difficult, but it has to start quite easy, then increasing in difficulty without having a huge "OMFG What is this Impossibru thing I fought 2 goombas the level before and now there are 10 hammer brothers coming for me O_O"' "
make it so the player can do a mistake without being punished too much (An unfair exemple: In ninja gaiden, If the player lose a life in the level 6-3, he come back at the begginning of 6-3, when reaching 6-4, the first phase of the last boss, the health isn't refill, and if he lose one life, he don't go back to 6-4, or 6-3 no, but 6-1, that is quite an exemple of unforgiving checkpoint madness combined with no health refill when needed)
-If possible, have a difficulty setting that works, so if people find it too hard, you can make it less horrible to some extend, and harder for people that seek challenge.
-Don't do "Game over" if your game is long, losing your last life to the final boss stage because of one stupid mistake is the worst thing that can happen
-Maybe letting some clues to the players on what they did wrong, or how to do it right when they lose a certain amount of times.
But mostly the player should lose because of himself, If I lose because of bad controls, I'll be mad and will end up turning off the game, if it is because of a random glitch, I'll be mad but whatever, If because of unfairness, I'll be real mad, and will really have a hate relationship with the said game, but If It because of myself, well, gotta get better, and Finally I'll be able to finish a game I love