The earliest use of the phrase is unknown, but in his autobiography, Mark Twain (1835–1910) observed that it was in general use even in his childhood. Twain refers to an episode from 1847, when he was working as a printer's apprentice; Smith (1994) tells the tale thus:
[Twain] recounts a practical joke a friend played on a revival preacher when Twain was an apprentice in a printing shop that Alexander Campbell, a famous evangelist then visiting Hannibal, hired to print a pamphlet of his sermon. While checking the galleys, Twain's fellow apprentice, Wales McCormick, found he had to make room for some dropped words, which he managed by shortening Jesus Christ on the same line to J. C. As soon as Campbell had read the proofs, he swept indignantly into the shop and commanded McCormick, "So long as you live, don't you ever diminish the Savior's name again. Put it all in." The puckish McCormick obeyed, and then some: he set Jesus H. Christ and printed up all the pamphlets.
Smith suggests (1994:331-2) that "Jesus H. Christ" is a specifically American profanity, and indicates that at least in his experience it is uttered primarily by men. Quinion (2009), a British author, likewise specifies the phrase as belonging to American English.
Also, the term "Christ" is not a name, it is a title (if it were a name does that mean his father was Joseph Christ or Mary Christ)? Correctly spoken it should be Jesus the Christ. Too bad something so simple eludes the Christians.