The Warning
Jude, writing to Christians in general, warns of false Christians and false teachers infiltrating the church of God (Jude 1:4). These deceivers, who secretly attach themselves to believers, promote doctrines contrary to what Jesus and the apostles taught - the "faith once delivered" (verse 3). Jude then offers three historical examples of the errors committed by these people, the first of which is the way of Cain.
Woe to them! For they have walked in the way of Cain; and for gain, they have wholly given themselves up to Balaam's delusion ("error of Balaam" in the King James), and have perished in the rebellion ("gainsaying" in the King James) of Korah (Jude 1:10 - 11, HBFV throughout).
How serious is Jude's denunciation of Cain's way? The word "woe" is used to draw special attention to something that will cause grief, trials and trouble. Outside of the gospel accounts, it is used only in Jude, 1Corinthians 9:16 and the book of Revelation (Revelation 8:13, 9:12, 11:14, 12:12).
Apostle Paul's Testimony
The Apostle Paul adds some revealing information about Cain's way.
By faith Abel offered to God a more excellent sacrifice than Cain . . . (Hebrews 11:4).
Faith is based on knowing what God wants, what he promises, and so on (Romans 10:17). The next step is trusting that he is telling the truth and that we should follow it even if it is contrary to what we may think or do.
Both Cain and Abel had at least a basic understanding of what was required in a sacrifice to God. The Lord's comments after Cain's unacceptable sacrifice (Genesis 4:5 - 7) would make little if any sense if his error was simply that he wrongly guessed what was acceptable.
The fact that the Lord was willing to directly converse with Cain bolsters the belief that both him and Abel were informed on what was required.
God, by virtue of his power and ability to carry out anything he wills, has the right to determine (and enforce) how all others worship him. He has the right to determine what is and is not acceptable, the symbols or ceremonies to be used in worship and so on. Abel, according to Paul, believed what God said regarding what was an acceptable sacrifice. Cain's way, however, was to reject the Lord's will and replace it with his own.
John's Teaching
The Apostle John offers us a bit more information about the way of Cain.
For this is the message that you heard from the beginning - that we should love one another;
Not as Cain, who was of the wicked one, and murdered his own brother. And what was the reason that he murdered him? Because his own works were wicked, but his brother’s works were righteous (1John 3:11 - 12).
God's response to Cain's offering (Genesis 4:5) was meant to teach him that what he did was not acceptable. The rational response would have been to reflect on his actions, take responsibility for what he did, and ask for forgiveness. Given God's willingness to directly interact with him, he could have even asked the Lord to explain his rejection and what could be done to correct the situation!
Cain, however, self-righteously felt the offering He determined to give should have been accepted.
Make no mistake, Cain was first and foremost enraged at the Lord (Genesis 4:5). He, of course, knew he did not have the power to force the Lord's acceptance. He chose, instead, to direct his unholy wrath toward Abel whose righteous behavior revealed that his own was wicked.
Conclusion
Cain's way is determining for oneself, apart from God, what is right and wrong, good and evil. The false teachers spoken about by Jude promoted doctrines that seemed to be right but were wrong. Their teachings came from the imaginations of men who felt, just like Cain, that their own views were greater than the Lord's.
The way of Cain has continued throughout history to the present day, especially in the world's "Christian" churches. Decisions on how to best worship God, such as creating holidays (e.g. Christmas, Easter, etc.) with symbols not endorsed in the Bible, or declaring a different Sabbath day than the one promoted in Scripture, follow down the same path as Adam's firstborn son.