Herod's temple was not the first in Jerusalem, but the second (or third, depending how you view it) building dedicated to worshipping the true God. The first edifice came as an idea to King David (2Samuel 7:1 - 2). Although he could not build the temple himself because he was a "bloody man" (1Chronicles 22:8 - 10), he nevertheless collected materials for its construction.
David's son Solomon began work on the first temple sometime between 966 to 965 B.C. He built it on Mount Moriah, which has links to Abraham (Genesis 22:1 - 14). He finished it around 959 B.C.
In 586, still many years before Herod's arrival on the scene, the temple was completely destroyed by Babylonian King Nebuchadnezzar. He carried all of its treasures to Babylon. In 539 B.C., Cyrus the Great of Persia allowed captive Jews to return to Jerusalem and begin the rebuilding process (2Chronicles 36:23, Ezra 1:1 - 4). With the aid of the prophets Haggai and Zechariah the temple was finally completed in 516 (Ezra 5, 6).
Herod's link to the temple begins in 40 B.C. when Rome makes him 'king of the Jews' (king of Judea). In 20 B.C., he proposes, partly as a gesture to gain the favor of pious Jews he often enraged, to take on the task of rebuilding God's house of prayer. Although such a project was an incredibly ambitious, bold and expensive undertaking, it perfectly fit Herod's giant ego.
At the time of Herod's proposal to rebuild the temple, the edifice had existed for roughly 500 years and was in fairly bad shape. Over the years, it had experienced neglect, a tremendous amount of natural decay, and some destruction as various armies sought to conquer it and the city. The first century Jewish historian Josephus wrote about the monumental task that was carried out in his work entitled "Antiquities of the Jews" (Book 15, Chapter 11).
According to the 1906 Jewish Encyclopedia, the existing structure was torn completely down and the temple was rebuilt from the ground up. Although he died around 4 B.C., the rebuilding effort he started continued long past his death. In fact, work was still being performed during Jesus' entire ministry (John 2:16, 19 - 21) and continued as the New Testament church expanded into many parts of the Roman world. It did not completely finish until 65 A.D.
Sadly, only a few years after renovations were completed, Herod's temple was destroyed. This event was prophesied by Christ (see Matthew 24). In 70 A.D., Roman legions stormed Jerusalem, and notwithstanding the efforts of General (later Emperor) Titus to preserve it, his soldiers set fire to the temple in several locations.
The total destruction of God's house of prayer began on the 9th of Ab, the same day its first destruction occurred. Its demise was completed on Ab 10, which corresponds to Sunday, August 5th in 70 A.D.
Ultimately, the need for an earthly sanctuary in Jerusalem was fulfilled and brought to completion by the sacrifice of Jesus Christ, which ended the Old Covenant and the need for a physical priesthood. Under the New Covenant, the true temple of God is in heaven where Jesus Christ is High Priest and head of His church.