He shoved him aside knowing it was an official he was touching. Williams appears to have not even seen that it was a ref next to him, he never turns his head.
And honestly, in the Smith case, I think an argument can be made that the official didn't have to have his foreman jabbed into his chest shoving Smith back. I think officials (in all sports) can get a bit too big for their britches. That doesn't mean players should be able to hit officials, but I can see why Smith would do what he did there.
I get it that some experts say it's a no brainer call and there is no room for interpretation, but I think there should be. Because what Williams did is no different to me than a player angrily knocking a ref over after he bounces up from a fight on the ground.