James Madison Porter was born near Norristown, Pennsylvania, one of eight children born to Andrew Porter and his second wife, Elizabeth Parker. He studied law and was admitted to the bar in 1812. He moved to Easton in 1818 where he served as Deputy Attorney General and later head of the bar of the county. Porter had a successful law practice in the eastern part of Pennsylvania for more than forty years. In 1838 he was elected a member of the convention to reform the Constitution of Pennsylvania.

More than any other person, Porter is responsible for the founding of Lafayette College in 1826. He promoted the idea of having a college in Easton and he chose the name after meeting the Marquis during his triumphal return to the country. Porter served as President of the Board of Trustees for twenty-five years, 1826 – 1852, and as Professor of Jurisprudence and Political Economy for twenty-four, 1837 – 1852 and performed these duties without pay. In addition to his law practice and teaching responsibilities, he was President of the Delaware Bridge Company, the Belvidere and Delaware Railroad, and Lehigh Valley Railroad.

In 1821 Porter married Eliza Michler, member of one of the oldest and most respected families in Easton. They had seven children. Their grandson, James Madison Porter III, taught Civil Engineering at Lafayette and designed the Free Bridge across the Delaware River.