If you are only looking at the dynamics of a quantum harmonic oscillator, you can go ahead and drop this term. However, the universe isn't a single particle trapped in a $1$ dimensional harmonic oscillator. Thus we very, very frequently are looking at systems like these:
- There are many harmonic oscillators with some interaction.
- A harmonic oscillator in one dimension and something else is happening in other dimensions.
- Something that's only approximated as a harmonic oscillator but at higher energies the potential deviates from $x^2$.
- A harmonic oscillator whose coefficient varies with time.
In such more complicated situations, often the groundstate energy of the harmonic oscillator will have physical consequences. Consider the following real-world examples:
Adiabatic release from optical lattices
Using counterpropagating laser beams detuned from atomic transitions, we can create potentials that trap atoms proportional to $\cos(2kx)$, where $k$ is the angular wavenumber of the laser. Then near the bottoms of these potentials, they can be approximated as a harmonic oscillator. (picture from the NIST website)

Many atomic physics experiments will use something like RSC or EIT cooling too cool atoms to the groundstate of these localized potentials. Those experiments typically want the atoms to be as cold as possible. If you suddenly turn off the laser beams, the atoms in free space are left with roughly the groundstate energy of the harmonic oscillator they were trapped in (there's one physical consequence of the groundstate energy). However, if you slowly turn off the laser beams, the resulting atoms are colder. This is because you're effectively turning down $c$ over time in the potential $V=c(t)x^2/2$. Because of the adiabatic theorem, the atoms stay in the groundstate, and that groundstate energy decreases over time. Here are two papers where this is done:
- Fast quantum gas formation via electromagnetically induced transparency cooling
- Degenerate Raman sideband cooling of 39K
Molecular energy levels
Consider this diagram of molecular energy levels from hyperphysics. For very simple diatomic molecules, we can calculate an effective potential as a function of distance between the two nuclei, and the relative motion of the two atoms is governed by this effective potential (via the Born-Oppenheimer approximation). This effective potential can then be approximated as a quadratic near the equilibrium point. And the picture shows an important idea - that quadratic coefficient is different for different electronic energy levels. Sometimes in atomic physics experiments we want to drive electronic transitions in molecules or photoionize molecules. In those circumstances, the groundstate energy of the harmonic oscillator actually modifies the wavelength of light needed to drive these transitions.
So yeah, if you stayed in one harmonic oscillator, you could redefine energy to remove the groundstate energy. But if you have transitions between different harmonic oscillators or excite to a regime where the harmonic oscillator approximation is no longer valid, the groundstate energy will have physical consequences.

The $\nabla \mathbf{B}$ force in a nonuniform magnetic field
In a lot of quantum mechanics classes, shortly after learning about the harmonic oscillator you will learn about Landau levels, the quantum mechanical solution to a charged particle moving in a magnetic field. You will find that in this situation, the energy levels are the same as those arising in a 2 dimensional harmonic oscillator. Now what happens if the magnetic field varies in magnitude with position? Perhaps lets say the length scale over which the magnetic field varies is small compared to the orbital radius of the groundstate of the charged particle. In classical mechanics, we find that this charged particle doing a "cyclotron orbit" also feels an "effective force" pointing in the direction of the gradient of the magnetic field $\nabla \mathbf{B}$ (described in the "guiding center" wiki). In quantum mechanics, we will find that this effective force still occurs for atoms in the groundstate of the potential created by the local magnetic field. This is because the quadratic coefficient of the harmonic oscillator will vary with position, and because of the adiabatic theorem and the Born-Oppenheimer approximation, the particle will feel an effective force pushing it toward places where it has a lower groundstate energy.
As a less realistic but simpler toy model, consider a system which is a harmonic oscillator in $x$, but the coefficient varies with $y$: $V=c(y)x^2/2$. Let's say that the length scale of variation in $y$ is much smaller than the size of the groundstate in $x$. And maybe we assume the particle is in the groundstate in $x$. You will find that in $y$, the particle moves in an effective potential given by $\hbar\omega(y)/2$, where $\omega(y)=\sqrt{c(y)/m}$.