Easy way to clean up this folder is actually deleting it completely but extensions are also in that folder.
In my case, I have extracted the extension list to txt file,
remove the folder completely
install the extensions from extension list txt file
Extracting extensions list to txt file
Run below command in vscode terminal directly
code --list-extensions > remote_extensions.txt
Remove .vscode-server folder
close remote connection of vscode to server
close vscode
connect to server via ssh
rm -rf ~/.vscode-server
Install extensions from txt file
Open vscode and connect to remote server first
run below command in vscode terminal
cat remote_extensions.txt | xargs -L 1 code --install-extension