Even though God doesn't grade sin, we humans tend to do so. Christ said He forgives ALL manner of sin and the answer to Paul's question "who shall deliver me from the body of this death?" is Jesus Christ our Savior (Romans 7:25)!
Overcoming guilt is a matter of faith in Christ. The very fact you said you confessed your sin to God, your wife, pastors and the congregation and you said God forgives you. The reality is since you feel guilt means you doubt God forgives you. Intellectually, you know God forgives you from a Biblical point of view. Your problem is that in your heart you believe the sin was TOO evil. You are discriminating sin. You are grading sin which God doesn't do.
You are also saying aside from this sin, you haven't sinned. You and I both know better. We all sin! If you believe we all continue to sin (besides the one you mention) then you should feel guilty about them all! Since only THIS sin makes you guilty you evidently are having a very human problem of SELF-RIGHTEOUSNESS.
Since God doesn't grade sin, we should feel guilty about all our sins. But this is the only sin you feel guilty about. That, my friend, is being self-righteous!
Paul said the law of God was given that we might realize how much sin we commit (Romans 5:20). Either this is your only sin bad enough for you to feel guilty or you don't understand how sinful you really are. That is self-righteousness. All sin is equal, and therefore, we should continually see our guilt and sinful nature and know we look to Christ to forgive us. My suggestion is you need to pray for deeper repentance and have the faith that God forgives us regardless of the type of sin.
Until you forgive yourself for your sin (sins), you really do not believe Christ's blood has paid for that sin. My friend, deeper repentance and faith will remove the guilt!