A lot of those areas would be either difficult to expand into due to geographical barriers or pointless because they had less economic value.
Neither Yemeni, Omani Arabs, or Somalis tried to take control of the islands of Socotra, the land is pretty useless from a historical point of view, for farmland or grazing other than fishing. There was also no precious metals or stones to be found like Sofala.
Madagascar was seen as unattractive place it was to distant and isolated from the major trade routes that connected the rest of the East Africa to Arabia. It also lacked immediate resource wealth.
When they expanded to areas of southern portion of East Africa like the Swahili coast what prevented them from expanding into the interior was the geography of thick forests, mountains, rift valley and tetsy fly etc . If you notice Islam is concentrated on the coast unlike the interior the way it's spread out amongst Somalis, geography has to facilitate conquest and expansion.