On modern hardware, who handles what bit of hardware state can be unclear

Suppose that you have a fancy modern server with some fancy modern GPUs, and you're of the opinion that the fans on those GPUs aren't running fast enough when the GPUs are under load. You might be quite interested in the question of who is responsible for handling the fan speed, and unfortunately the answer for this sort of question is rather unclear.

Let's list some options, ranging from extremely unlikely to less unlikely but possibly more alarming:

• the GPU card directly drives the fan speed based on onboard temperature and power sensors using a fixed, hard-coded fan speed profile. Your fan speed problem is local to the GPU and probably a hardware fault.
• the GPU card directly drives the fan, as above, but using a fan profile that's stored in NVRAM. Its stored fan profile may be incorrect or corrupted, and reloading the profile will probably require a special vendor tool.
• the GPU card directly drives the fan but using a fan profile in volatile RAM that's installed either when the system starts (by the firmware) or when your OS's hardware driver loads (or both). Before a profile is installed, the fans run (hopefully briefly) at full power, producing the typical server full volume fan howl during system POST.

  Your situation may be because the profile being uploaded by the firmware or the driver is wrong, or perhaps it's not uploading a profile at all and the GPU is falling back to a built in one that isn't right for your circumstances.

• the GPU fan speed is controlled by the system firmware (through some management mode that runs even when the host OS has started) or by the OS hardware driver, although using temperature and power data read from the GPU card. One or the other are doing this wrong, or have the wrong profile information, or simply isn't doing this at all, leaving the GPU at some default fan setting (or basic profile) that's wrong.

• the GPU fan speed is actually controlled by the server BMC, because in practice the BMC is what has the best access to the sensor and power information for the rest of the system (and perhaps the server was designed from the ground up as a GPU server and the GPUs were designed to be put into such a server, where they could assume that there was a BMC to do this). The BMC controls GPU fan speed based on profiles and other settings and it has the wrong profile, or is getting misleading data from other sensors that lead it to think (for example) that the ambient air flowing through the case is substantially cooler than it actually is.

  (Alternately, the GPU card controls fan speed but adjusts its calculated speed based on an offset that the BMC periodically sends it, with the BMC calculating the offset based on its sensor information and other factors. This offset could be off.)

  If the server BMC directly controls the GPU fan speed, it might be using different GPU sensors than the ones the card exposes to the host OS, or reading the existing sensors in a different way (one that's special and specific to the BMC's connection to the GPU card). This could result in the BMC using different temperature and power data to decide fan speeds than the data you see from host-side GPU hardware monitoring.

• the GPU card controls the fan speed but it uses a profile that's loaded by the server BMC, not the system firmware or the OS GPU driver. This profile is effectively invisible to the host system and might not be visible or changeable even in the BMC's web interface, because the server vendor considers it an internal thing for vendor use only. The GPU fan speed could then be off because this profile itself is off.

  These profiles could even be custom-generated at the factory for the specific system configuration (based on things like number and type of GPUs, PSUs, processors, RAM, and so on, and actual performance under test load). This provides all sorts of ways to have the custom profile be off, for instance if the hardware changed. Modern BMCs are already somewhat individually programmed at the factory (to load specific serial numbers, set initial passwords, and so on).

You can of course combine these. Perhaps the GPU has onboard fan control based on its own sensors that acts as a fallback, but the BMC normally directly controls fan speed. Or vice versa, where the GPU normally controls fan speed but if something 'seems' to be wrong some combination of system firmware, OS drivers, and the server BMC can more or less directly control the fan speed (either setting it to a fixed value or by following their own profile). I put 'seems' in quotes because perhaps the drivers are mistaken, and their override is doing more harm than good. There's also the possibility that everything involved mistakenly thinks that good GPU fan control is someone else's responsibility. The GPU card thinks the firmware, the driver, or the BMC is in charge of the fan, the BMC thinks it's supposed to be hands off, the driver hasn't had its fan control turned on, and so on and so forth. This could be especially fun if the setting for 'the BMC (or the firmware) is or isn't in charge of GPU fans' is hidden inside the BMC and isn't exposed to you through things like the BMC's web interface.

(Such a setting might exist for vendor diagnostic purposes and be undocumented but persistent because it's for those said diagnostic purposes. There are a lot of deliberately undocumented things in server BMCs and host server firmware ('BIOS').)

Modern server GPUs generally can report things like various temperatures and power draws. When the GPU directly controls its own fan speed, it's presumably using similar information to make fan speed decisions, but you (we) have no idea if it's using the same sensors we see or if it's deciding based on at least some internal sensors that we can't see readings for (I believe this is very common in CPUs, which may have far more internal thermal sensors than they report). It would also make sense for a modern GPU (and its firmware) to make power and cooling decisions in the same way that modern CPUs do, namely based on what's actually running or going to be running. If the GPU knows it's about to draw a lot of power, it could preemptively spin the fan RPMs up.

Since consumer GPU cards get put into random machines with quite varying capabilities, I assume that they have to be relatively robust and standalone for things like fan control (although OS drivers can clearly get involved). This also applies to other PCIe cards, power supplies, and so on. Server and datacenter focused GPUs and other hardware isn't necessarily so restricted and may require more from the system it's in, such as having a BMC that can drive things. This might be designed into the hardware or it might be custom firmware for the GPU, network card, or whatever (or both; maybe the hardware got a minor tweak to expose some things over eg I2C).

(Big server vendors have in the past theoretically had custom firmware in their branded disks. You could buy more or less the same underlying HDDs or SSDs from the real drive vendor, but they wouldn't come with the special firmware sauce. Did the server vendor disk firmware matter? Maybe.)