Born in Washington, D.C. as a descendant of free slaves, Holly was baptized and confirmed a Roman Catholic. His early years were spent in D.C. and Brooklyn where he connected with Frederick Douglass and other black abolitionists. After a dispute over the ordination of black clergy, Holly left the Catholic Church and joined the Episcopal Church in 1851. As an Episcopalian, Holly attended the first National Emigration Convention as a delegate and was selected commissioner for the newly formed National Emigration Board. Under the auspices of this office, he traveled to Haiti in order to negotiate an emigration treaty. While there, he explored the possibility of establishing a mission for the Episcopal Church. Upon Holly’s return to the United States, he requested that he be sent to Haiti to serve as a missionary, a request denied by the Board of Missions of the Episcopal Church.
In 1856 Holly has ordained a priest in New Haven, Connecticut. That same year he co-founded the Protestant Episcopal Society for Promoting the Extension of the Church Among Colored People, which challenged the Church to take a position at General Convention against slavery. Holly served as a rector at St Luke’s Church in New Haven, Connecticut until 1861 during which time he traveled and extolled the benefits of emigration to Haiti, a country where slaves had led a successful revolt and overthrown their European forces. He viewed Haiti as an opportunity for blacks to bind together and establish a black nation in the Western hemisphere. Holly believed that bringing Anglicanism to Haiti would help to stabilize and develop the country more expeditiously.
Upon Holly’s resignation from his position at St. Luke’s in 1861, he led a group of 110 people to Haiti. Many in the group were congregants of St. Luke’s and subscribed to Holly’s vision for Haiti. Despite the obstacles, they succeeded in the establishment of a mission in Haiti. The disease and poor living conditions plaguing the settlers (forty-three members died of yellow fever and malaria during the first year, including his mother, wife, and two of his children), prompted many of them to return to the United States. Holly remained with only the most dedicated followers to establish schools, a church, and programs in pastoral training and countryside medicine.
It was not until 1865 and after many denied requests, that Holly finally began to receive limited sponsorship from the Board of Missions for his work in Haiti. After serving as consul for Liberia at Port-au-Prince from 1864 to 1874, Holly was consecrated missionary bishop to Haiti at Grace Episcopal Church in New York City, becoming the first African American bishop in the Episcopal Church. As bishop, Holly served as a delegate to the Lambeth conference in England and received a doctoral degree from Howard University and an honorary law degree from Liberia College, Monrovia. He continued to live and work in Haiti, returning rarely to the United States, until his death in 1911.