I have a windows vps with a server that I want to protect from DDoS and hide from outside world. Is there a way by using PHP IIS webserver on another vps to somehow whitelist ips or redirect only clean traffic to my windows vps?

Run the ip through cloudflare. They have ddos protection

Do you control the router? No? Than you cannot.
If you don't care about bandwith, but CPU usage indirectly resulting from the traffic, optimize the protocol (e.g. authenticated sequence numbers in each packets, and no answer on unauthenticated packets, never answer with more data than received).

Reverse nginx proxy with rate limiting maybe? No clue how to set that up on windows, though.

Careful with whitelists.. Many ISPs outside yout homeland do rotate IPs quite often. And people tend to use services not only at home but also at work and at friends'.

This problem is two-fold.
1. if your user gets a new IP he will most likely won't be in the list. So no access. Bad for business.
2. If user gets his IP rotated, jis previous IP might be assigned to someone who might consider ddos'ing you. And that IP is in your whitelist, sooo...

Also, a bonus note: whitelisted IPs could refer to generic VPNs, proxies or so, meaning masses of other computers will have the same external IP. Who's to say some of them will not fall victims to a botnet?

All I'm saying.. careful with whitelists