There are three main types of DNA test that can help you with your family tree research. Which test is best for you very much depends on what question you want the answer to. In brief, if you want to research a particular surname, then a Y-DNA test might be the best; if you want to see if two people are related on your direct maternal line, then a mitochondrial DNA test might be best; and if you are interested in connecting with cousins who might be able to help you break through that Brick Wall in your research, then an autosomal DNA test might be best.
With Y-DNA testing we are only looking at one particular line of an individual's ancestors - specifically the direct male line of the participant, their father's father's father etc. Y-DNA testing is only done on men because they are the only ones to pass on the Y-chromosome.
Because the surname is also passed on along a direct male line, the Y-DNA test is used extensively in projects devoted to researching a particular surname, and helps to answer questions like which bearers of the surname are related to each other? where did it come from? and what variants of the surname are associated with each other. It is also useful for verifying and validating surname genealogies.
The Y-DNA test can also tell us what route our patrilineal ancestors took out of Africa, what mutations occurred and when, and where they might have stopped along the way. These migratory patterns of the different Y subgroups (haplogroups) go back to the common male ancestor of all modern humans, an African man who lived sometime between 200,000 to 330,000 years ago.
Another type of DNA test can trace the direct female line, the mother's mother's mother, etc.
The mitochondrial DNA test can be taken by both males and females. It is useful for answering a targeted question such as: is this particular ancestor on my direct female line related to this other particular ancestor on your direct female line? In this situation, two people would have to take the test (you and the other descendant).
Mitochondrial DNA testing has elucidated migratory patterns for the different mitochondrial subgroups, going back to the first common female ancestor of all modern humans, an African woman who lived some 200,000 years ago.
This third type of test is the most useful and the most popular among genealogists. It can be taken by both males and females, and can help identify more recent relatives (usually up to 5th cousins). The autosomal DNA test looks at how much genetic material is shared between two individuals and calculates if they are likely to share a common ancestor. However the test is only accurate back about 7 generations or so (i.e. to your 4th great grandparents). This is because with each generation the contribution of an ancestor becomes more and more diluted.
Put another way, you share 50% (1/2) of your DNA with each of your parents, approximately 25% (1/4) with any one of your 4 grandparents, 12.5% with a great grandparent (1/8) , and only 1.6% with a 4th great grandparent (1/64). Thus, after 7 generations, the contribution of any single ancestor is likely to be so diluted as to be virtually undetectable. Siblings will share roughly 50% of the same DNA, first cousins 12.5%, third cousins, 0.78%, and fifth cousins 0.05%. However, random recombination means that two cousins could theoretically share no genetic material at all.
The autosomal DNA test will detect 99% of your first and second cousins, 90% of your third cousins (common great great grandparent), but only 50% of your fourth cousins, and a mere 10% of your 5th cousins.
Connecting with cousins allows you to collaborate on solving family mysteries. They may have the information you need that solves a puzzle that has been in your family for decades. They may also have photos or even personal letters that have been passed down in their family from generation to generation. And this new information may help you push back a particular ancestral line an extra generation, or it may open up a whole new wealth of ancestral material that you never would have expected.