Penn Badgley knows you love to look back at his past romance with Blake Lively. 

However, the Gossip Girl alum—who now shares son James, 4, and is currently expecting twins with wife Domino Kirke—recently admitted it was “the struggle” to have a romance that was so closely interwoven with their television characters. 

“When you do a television show, you’re doing it constantly because of the nature of that show,” Penn told Alex Cooper on the April 23 episode of Call Her Daddy. “You feel like you’re constantly, even though no one may be actually asking you to do this, you feel like you’re constantly needing to prove yourself in some manner. Otherwise, what are you doing? I mean, you know, what is the space?”

Of course, Penn’s difficulties went beyond simply the difference between his relationship with Blake and his character Dan’s relationship with Serena, he also spoke to how his lifestyle in general was impacted by the public’s perception of his character.