The PS4 Pro has an HDMI 2.0 output connector, which allows it to use an HDMI 2.0 capable 4k display at that resolution, at 60 Hz, in RGB mode (which means essentially a completely lossless picture). Also if the 4k display supports HDR, it requires HDMI 2.0.
The PSVR, however, degrades this scenario to HDMI 1.4, because the processing unit box that sits between the console and the display has only HDMI 1.4 connectors. This means that if the PSVR processing unit is in use (which is necessary for PSVR to work), then the 4k display cannot be used in RGB mode, nor using HDR. The display will still be used on 60 Hz, but in YUV420 mode (if the display supports it), which has degraded color quality (essentially it's a lossy image, where there is less color information, causing less vibrant colors, and some color artifacts).
The following diagram illustrates the exact situation (click the image for a larger version):
The HDMI cable connections marked with blue lines indicate the setup when using the PSVR. The connection marked with the green line denotes a setup where the PS4 Pro is connected directly to the display.
These two setups are mutually exclusive, because the PS4 Pro has only one HDMI output. Both the blue and green connections cannot be made at the same time. This moreso if the display itself has only one HDMI in connector (which is quite common, even with 4k displays).
This situation is very hard to fix even with HDMI switcher boxes (especially if the display has only one HDMI connector).
For starters, most HDMI switchers are designed to have two or more input HDMI connections, and one output connection (because their idea is that you can choose to display one of multiple picture sources). This might theoretically help with half of the problem in the diagram above, on the display end (where the input signal may come from either the console or the processing unit). However, it still doesn't help with the fact that the console needs to output to either the display directly, or to the processing unit.
This would require a quite complex HDMI switcher box, which is able to take the input from the PS4 and redirect it either to the display directly, or to the processing unit, and in the latter case it also needs to get the input from the processing unit and redirect it to the display. So in essence it requires redirecting input A to output A, and at the same time redirect input B to output B. And, as an alternative, it needs to be able to redirect input A to output B (and ignore input B). I'm unsure that such HDMI switches even exist.
You could try to get away with it by having two HDMI switches, one on the PS4 side, and another on the display side. However, the one on the PS4 side would still need to be of the type that takes one input and is able to redirect it to two optional outputs. As far as I know, this is a rarer form of HDMI switching.
Secondly, the switcher(s) needs to support HDMI 2.0. The PS4 Pro is really picky about this (probably because of technical reasons rather than deliberately). If you connect an old cheap HDMI switcher between it and the 4k display, it just won't work. The display won't receive any signal. While HDMI 2.0 supporting switchers may be becoming more common, in my experience they still tend to be quite rare. Also some research reveals that people who have tried to solve this situation with switchers have found out that it, indeed, simply doesn't work. The PS4 Pro seems to be really picky about something being between it and the display (even if that something supports HDCP.)
The only practical solution is to manually switch connections every time you want to switch between PSVR games and other games. That is, in the diagram above, switch the green connection to the blue one both on the PS4 side and the display side (if your display has only one HDMI port).
A slight alleviation can be achieved by having a display with two HDMI ports, or using two displays (in which case only the connection on the PS4 side needs to be manually changed).
