Old English attests the word as docga. My hypothesis is that it’s dōc “bastard, mongrel” + -ga [diminutive suffix], roughly like calling it “the little mutt”. The vowel shortening would’ve been happened already back then, otherwise the modern form would be *doog /du:g/.

Note that Latin/Romance attests similar phenomena (depreciative word for animal becoming the default word + diminutives being ingrained into the main word). And typically when you see weird stuff going on in a language you’ll see it happening in other languages too.