There’s an element of truth here, in that parts of the world have a system where farm animals eat stuff humans can’t, such as wild grass, kitchen waste and straw. But modern ‘factory farms’ do use insane amounts of growth su0lements and medicines, which are just as bad.
There’s land that isn’t good enough to grow crops, but is good enough for wild plants to grow. You can, as you said, ‘manage’ it - give enough fertiliser and water to make it suitable for agriculture. But that is often unaffordable for the people living in such places, so using animals to gather and concentrate the available nutrients is the best option available to them.