you probably did - on destroyed - add some number to score.
what you should do is move that action (add some number to score) to event that checks if button was pressed in correct time.
then when object is destroyed on that event of pressing button in the right time - you get the score
otherwise you don't.
I'm not quite sure how to do that though, because of the way the game is played.
Basically a spawner to the right is spawning the objects and they're scrolling to the left, this is one object with 7 different frames of different colors, and when its spawned it chooses a random color to be. Now when player clicks on a color, any oof that object thats the same color will be destroyed, leaving the rest.
heres a refference photo if it helps