Coronavirus disease-2019 (COVID-19) is associated with a wide range of post-acute sequelae. The prevalence of erectile dysfunction (ED) that developed after COVID-19 and the associated underlying factors were analyzed based on a questionnaire survey, COVID-19 Recovery Study II in Japan. A case–control study was conducted with those with or without ED one and two years hospitalized with COVID-19 between March and September 2021. Six hundred and nine Japanese men, with a median age of 48 years, were analyzed. During the study period, 116 subjects (19.0%) had erectile dysfunction. The patients with ED responded with less subjective awareness of recovery and high breathless and fatigue scores compared to those without ED. The patients with ED also showed higher Hospital Anxiety and Depression Scale-D (depression) and the EuroQol 5-dimensions 5-level scores for pain/discomfort and anxiety/depression scores compared before COVID-19 infection. Sleep disturbance was suggested to be associated with erectile dysfunction using an exploratory clustering analysis in the one-year survey. There were no associations of COVID-19 severity, reinfection, vaccination frequency, antiviral treatment for COVID-19 with the presence of erectile dysfunction. It was considered that mental support for the subject with erectile dysfunction as a long-COVID symptom is warranted.