It links Ethiopia's capital, Addis Ababa, to the Red Sea port of Djibouti - a stretch of more than 750km (466 miles).
Travelling at 120km/h, the new service cuts the journey time down from three days by road to about 12 hours.
The $3.4bn (£2.7bn) project was built with the help of funding from a Chinese bank and will have Chinese staff.