Nope, very dumb. I, as a Somali, don't want non-muslim Africans to have free movement In my land. The EU countries have a similar culture, through centuries of cultural overlap (trade, war, related royals, Christianity, Rome, literally the same alphabet...)
Africa is a whole different gravy. Somalis have nothing to do with South Africans, South Sudanese (except muslims), Ugandans etc
Why should I join such a synthetic union, when my biggest enemy is other Africans taking my land?
Africans are not, and never have been, anything like Europe. What they have is organic, what you propose is a desperate attempt spawned from an inferiority complex(not saying you in particular, but pan-africanism stems from jealousy of Europeans)
A muslim, or even arab, Union is certainly more preferable