James Worthy ain't backing down from Kevin Durant -- saying the NBA star was not only WRONG when he trashed him on a recent podcast, but claims KD's underhanded allegations could seriously damage his career.

Worthy is upset about a story Durant told on "The Bill Simmons Podcast" -- in which KD says he met Worthy last year and they had a very cool conversation about Durant leaving OKC for Golden State.

But Kevin says he learned a short time later that Worthy trashed him about the move during a TV interview. Durant felt betrayed and called James "shady" and "corny."

But Worthy says Durant's version of the story is simply untrue -- and he was so upset, he tracked down the TV interview in question to prove he's not a two-faced liar.