I agree with your point.Because they meet the standards of saheeh, the people in the isnad are considered trustworthy, reliable and with good memory. Still, given the time gap and distance from Makkah nobody can be accused of being a munafiq or kaafir for not considering it authoritative. Later scholars (i.e after 230 years) agreeing isn’t like scientists agreeing because they’re even further removed from the Prophet (saw), are less likely to determine what he said and whether the people in the isnad are reliable and trustworthy.