Bob McNair's "inmates" comments could serve as free agent repellent ... scaring off top NFL players who simply don't want to play for the Houston Texans owner ... so says Kenny Smith.

The former Rockets star is a legend in Houston and says McNair's comments at a recent NFL owner's meeting -- where he said, "Can't have the inmates running the prison" -- didn't fall on deaf ears.

"It will be difficult to attract great free agents," Smith says ... noting that McNair has to "really make a big stance" to prove he's not the racially insensitive jerk some of his own players think he is.