I mean, yeah, this won’t happen over night, it needs to be a process. According to this, cows used for dairy get slaughtered at the age of 6 years, at most. Whereas cows raised for beef tend to be slaughtered at ages 2-4. So, that’s kind of the cycle that we’re likely looking at. If we don’t want to slaughter them, they can live 15-20 years.

Reducing reproduction is going to be challenging, too, though. You do need to impregnate cows for them to produce milk.
According to this, it’s arranged that a cow gives birth to a calf ever 12-18 months, and according to this, cows can have their first pregnancy at around 23 months old. So, in the lifetime of a dairy cow, we’re probably looking at something like 2-4 calves.

So yeah, in the reality we live, it will likely only work that way, that dairy consumption falls, then more calves from dairy production get slaughtered rather than kept for future dairy production, and then after a few years, less dairy cows exist, less calves get born, less calves need to be slaughtered.

Alternative sources for milk products are coming down the pike, which helps with the dairy cow pregnancy problem. You can make actual literal milk protein with yeast now. It will probably end up being a lot cheaper.