Thanks for the reply.
I too believe that physical port to be removed from the port-channel, but is there any vmware article or KB to confirm this?
I found this William's post https://www.virtuallyghetto.com/2017/11/moving-esxi-hosts-with-lacplag-between-vcenter-servers.html that says it can be moved just like that to VSS without removing physical port from port channel, but this never works.
Interested to know what is the VMware supported way of doing this.