First, let us examine the scriptures concerning your question about why Cain was rejected by God. We are first informed that Abel (the younger of the two men) was a shepherd of sheep while his brother was a farmer (Genesis 4:2).
When the time came to make an offering to God, Abel chooses to bring and kill a firstborn animal from his flock. His brother, however, offered some of his crops toward his worship of God (verses 3 and 4). The response both received clearly conveyed whether what they did was pleasing or not.
"And the Lord respected Abel and his offering, but he DID NOT RESPECT CAIN AND HIS OFFERING . . ." (Genesis 4:2 - 5).
Three factors
Many have speculated as to why the offerings were not respected. Three factors, however, can be safely assumed as contributing to the displeasure of God. The first is the attitude of mind in regard to the offerings. The second is the quality of the offerings. The third is the worship that was not done in faith.
When one considers the jealousy of Cain toward his brother that drove him to murder him, a poor attitude in giving offerings to God is very plausible. This attitude was likely a factor in offering to him poor quality products rather than the very best.
The Bible does reveal God's response, in general, to offerings that are not of the quality He commands.
You offer defiled bread upon My altar. And you say, 'In what way do we defile You?' . . . And if you offer the blind for sacrifice, is it not evil? And if you offer the lame and the sick, is it not evil? (Malachi 1:7, HBFV).
The mental attitude of Cain was a key factor in displeasing God. Our Father knows man's inner thoughts. The Apostle Paul's writings attest to the fact that while He may like a person who is generous, he loves those who do so out of a heart of cheerfulness and not grudgingly (2Corinthians 9:7).
Faith in God
In the well-known faith chapter of the New Testament, Hebrews 11, the apostle Paul makes it abundantly clear that Abel had faith in God but that, by implication, his brother lacked it.
By faith Abel offered to God a more excellent sacrifice than Cain, by which he obtained witness that he was righteous, God testifying of his gifts; and through it, though he died, he is yet speaking (Hebrews 11:4, HBFV).
Our human nature not only can think evil and hinder us to do good works, but it can also dispose us toward those who reject evil and are doers of good works like Abel. Jesus himself attested to his righteousness when he was roundly chastising the Pharisees for their attitude and behavior.
You serpents, you offspring of vipers, how shall you escape the judgment of Gehenna? Because of this, behold, I send to you prophets . . . So that upon you may come all the righteous blood poured out upon the earth, from the blood of Abel the righteous . . . (Matthew 23:33 - 35).
We can acquire more of the mind of Christ, and not have our heavenly Father reject us like Cain, by constantly studying His word. If we do not do this, our minds can easily be led to being deceived by the devil who is "the god of this age . . ." (2Corinthians 4:4).