ANSWER: It is amazing how human we are and how we all generally think the same. As you know, the apostle Paul clearly said we all delight in the law of God after the inward man (Romans 7:22). The problem is we also have the law of sin, the same law that led to adultery, in our members as well. Let us take a look at what Paul the apostle said about his (and our) struggles against sin.
14 We know that the Law is spiritual; but I am a mortal (other translations have 'carnal'), sold as a slave to sin. 15 I do not understand what I do; for I don’t do what I would like to do, but instead I do what I hate. 18 I know that good does not live in me . . . (Romans 7:14 - 15, 18)
We ALL sin according to the Bible, but when we commit a sin which is OUR worst sin we cannot seem to forgive ourselves (Romans 7:23).
Even though God does not grade sin, we humans tend to do so. Christ said He forgives ALL manner of sin! The answer to Apostle Paul's question as to WHO will save him (and us) from a body that wants to lead us to death is Jesus Christ our Savior (Romans 7:25)!
Overcoming guilt is a matter of faith in Christ. The very fact you said you confessed your sin to God, your wife, pastors and the congregation and you said God forgives you. The reality is since you feel guilt means you doubt God forgives you. Intellectually, you know God forgives you from a Biblical point of view. Your problem is that in your heart you believe the sin was TOO evil. You are discriminating sin. You are grading sin which God does not do.
Since God does not grade sin, we should feel guilty about all our sins. But this is the only sin you feel guilty about. That, my friend, is being self-righteous!
Paul said the law of God was given that we might realize how much sin we commit (Romans 5:20). Either this is your only sin bad enough for you to feel guilty or you do not understand how sinful you really are. That is self-righteousness. All sin is equal, and therefore, we should continually see our guilt and sinful nature and know we look to Christ to forgive us. My suggestion is you need to pray for deeper repentance and have the faith that God forgives us regardless of the type of sin.
Until you forgive yourself for your adultery, you really do not believe Christ's blood has paid for that sin. My friend, some extra Bible study, deeper repentance and faith will remove the guilt!