ANSWER: The exact Bible verse your pastor is referred to is verse 5 of Deuteronomy 22.
5 Women are not to wear men’s clothing (the NKJV has 'anything that pertains to a man'), and men are not to wear women’s clothing; the Lord your God hates people who do such things. (Deuteronomy 22:5)
I understand how you feel, quite confused as I was some time ago when some of our ministers also taught that women should not wear pants - but only dresses. This teaching forced me to do some research on the subject.
In the first place, what were the ladies wearing in the days that the Levitical priesthood was set up and the Law of God was brought down from the mountain of God? Does anyone have a picture to show me what they looked like? Obviously, there had to be some differences between a man's robe and woman's robe. Do real feminine women want to look like their masculine males? I don't really think so. So if both wore robes wouldn't one look just a bit more feminine than the other? I have difficulty understanding men who want to dress up in frilly female garb (our perverted society accepts this) in the first place and ask if this is what most men want to wear? No, I think not. Most men would not be seen in public dressed in a nice looking pantsuit with ruffles and bows and other trim.
My husband does not have a good sense of color or a great imagination to even appreciate the subtle nuances of fabric or color or shape or any of those things that females seem to enjoy and feel comfortable wearing. He would not care if he had stripes and plaids on in the same outfit or if orange and red were worn in the same ensemble. I do not know about your preacher but I think most men seem to be in this category. Most women that I know not only do the shopping but in some cases arrange the daily dress for their men to wear.
There should be none of this transvestite stuff of cross dressing as this is an abomination that the Bible says God hates. He gave each of the genders their characteristics and it was VERY GOOD when he created them (Genesis 1:31). Why is it so difficult to maintain those beautiful characteristics as He made them? When we try to be any different we are in effect telling God he did not know what he was doing when he separated the genders in the way he did.
I wear nice looking slacks and pant suits when out in public and find them to be not only feminine and attractive but more modest than the sometimes quite short skirts that some women wear. I am an older, retired woman and comfort is more important than what others might think. When I am home alone I get out the even more comfortable sweat pants and T-shirts. So if a woman is never to wear pants (by the way this was how I was brought up) then I am sure to go to the lake of fire and vanish, but my God is most merciful.
I hope this helps to make you feel less guilty in what you wear and what you don't. You are free to wear dresses, slacks or any combination thereof. As a note of caution though, when attending church where your preacher would take note of your attire (and to avoid embarrassment, his or yours) you may wish to wear clothes that would be considered acceptable.
Even if you feel comfortable in a nice looking pantsuit or pair of slacks and lovely blouse to match it (definitely not meant for men to wear) it might not be edifying to flaunt it in this environment. There are many things with which we might have to hold ourselves in check until God teaches the other men and women the same lesson.