I think we are largely arguing semantics when I look at it carefully. Your point is that this wasn't truly an absolutely rigid and set caste system like in India and that people, depending on the times and circumstances could shift in and out of these roles? I'm inclined to see your point and not necessarily argue about that. Even the European writers who make the quick Hindu caste comparison probably would concede this. Somalia, Arabia, the Horn and the Maghreb do not truly have a rigid, age-old, immutable caste system with no social mobility and that literally correlates with the genetics of the populations. Not really for the most part, anyway.
My only point, however, is that the tribes and peoples not engaged in this sort of work usually did in fact look down on it and feel "superior" to it and ostracize the people practicing it at that time, whoever they were and force them to form their own "in-group", regardless of how fluid that in-group could be from time to time. This clearly seems an established tradition. You can call it a bondsman system or a pseudo-caste system but at that point we're arguing semantics, no?
