You must understand it's a social construct, so yes they're brown. It doesn't care for the connections or differences. Barack Obama is black, despite knowing his identity which is half-Kenyan, and half-English, with his mom being really white. Racial classifications which is a social construct wouldn't care either way. He's black because he looks black.
So if you have a kid with a white girl, your kid would be black or white, depending on how light his/her skin were. People like to pigeonhole other people and don't care for the specifics.