LaVar Ball could be right in his analysis of LeBron James' kids -- 'cause Austin Rivers says he's been "getting s**t" for being Doc Rivers' son since he was 6 ... but it never fazed him.

We got Austin leaving Catch in L.A. where he APPLAUDED James for clapping back at LaVar -- who recently said Bron's kids could get crap from other kids just because he has a famous father.

LeBron told LaVar to NEVER talk about his kids ever again.

But when we asked Rivers what it was like for him growing up with an NBA star for a dad ... this was his response:

"I've been getting that since I was 6 years old just for being the son of Doc. I've been getting s**t since I was a kid ... and it's never fazed me. I'm here now and I'm only getting started."