James Madison, Jr. was born at Belle Grove Plantation near Port Conway, Virginia on March 16, 1751, (March 5, 1751, Old Style, Julian calendar), where his mother had returned to her parents’ home to give birth. He grew up as the oldest of twelve children. Nelly and James Sr. had seven more boys and four girls. Three of James Jr.’s brothers died as infants, including one who was stillborn. In the summer of 1775, his sister Elizabeth (age 7) and his brother Reuben (age 3) died in a dysentery epidemic that swept through Orange County because of contaminated water.
Sources: http://en.wikipedia.org/wiki/James_Madison