ANSWER: The exact Bible verse your pastor is referred to, in regard to women only allowed to wear dresses and not pants, is Deuteronomy 22:5. It states, "Anything that pertains to a man shall not be worn by a woman, neither shall a man put a woman’s garment on. For all that do so are abominable to the Lord . . ." (Deuteronomy 22:5, HBFV)
As a woman I understand how you feel. I was also quite confused a while ago when the church I attended also taught that women should only wear dresses and not pants. This teaching forced me to do some research on the subject.
For starters, what were the women wearing in the days that the Levitical priesthood was set up and the Law of God was written? Obviously, there had to be some differences between a man's robe and that of a woman. Do real feminine women want to look like men? I don't think so. So if both wore robes wouldn't one look just a bit more feminine than the other?
One of the basic principles of Bible study is to note what a verse says and what it does not say. Deuteronomy 22:5 does not give us any details regarding how clothes should cover the male or female body in Moses' time or throughout history. It does not say men should wear short sleeves and women longer ones. It does not state how long clothes for each sex should be or even if patterns on the cloth are acceptable for one sex and not another.
Implict in God's Deuteronomy command is that the sexes should be distinguished from each other in regard to how they dress. The details of how this distinction is manifested is based on the commonly accepted norms of the society and culture a person lives in.
It should be noted that the difference between the dress of men and women, in Moses day, did not revolve around whether cloth was used to distinguish each person's legs (pants) or not (dresses). It centered around something a bit different. The book "Manners and Customs of Bible Lands" (Fred Wight) states this distinction (emphasis added).
"The dress of women was different in DETAIL rather in kind. They (women) ALSO (like men) wore a tunic and cloak . . . Doubtless they wore longer tunics, larger mantles than their menfolk."
Women were also sometimes distinguised in that they, at times, wore veils (see Genesis 24:65) while men did not. There were also other visible distinctions of sex such as females wearing makeup and generally placing on their bodies more ornamentation than males.
In most western societies women have been wearing slacks (pants, jeans, etc.) for many years. This type of clothing is commonly accepted for both sexes. While some ladies may prefer to wear dresses, not doing so is not prohibited in the Bible.