Sexual orientation is beyond people's control, too.
Mormons tried to preach for years (they might still, I haven't followed up) that "homosexuality itself was not wrong but "homosexual behavior was" and used to encourage gay members just to marry straight members and keep their gay thoughts to themselves.
Most famously, there was Josh Weed who was tauted as a success for this method. For years (6?), he gave speaking tours, etc arguing that others should do the same. The Weeds were the poster children for this idea, which worked until it didn't.
Now Josh says that even though his wife was his best friend, they were living a lie that ultimately hurt them both (particularly her, he said it shattered her self esteem) and wasn't the loving family either of them needed or their children needed.
Here is more on them. It's an interesting and sad story:
https://www.patheos.com/blogs/daylightatheism/2018/01/strange-sad-story-josh-weed/
Anyway, since we are not barring old widows (let's use widows so we don't have to have a divorce argument) from getting re-married, it must not actually be the fertility argument (getting re-married as an old widow is a choice after all and one under the widow's control). So it must be the "re-shapes" society argument, but that same argument was used to try to prevent interracial marriage, too.
I realize you were giving a "devil's advocate" view and not necessarily your own but I have heard people try to give logical arguments to what really boils down to "I don't want to see that and I don't want my kids to either" which is actually homophobia whether they realize it or not.