Kody Brown, best known from the reality TV series Sister Wives, was unaware that his actions on Special Forces: World’s Toughest Test were a topic of conversation among his peers. Despite nearly reaching the end of the Season 4 selection process, Brown frequently faced criticism from fellow participants who took issue with his teamwork and unique personality. Reflecting on the show, Brown admitted to Decider that hearing the remarks made him feel “embarrassed” and “bad.”

“I didn’t realize I was that annoying,” he confessed. “I was just trying to stay optimistic throughout the experience, but apparently, I rubbed a lot of people the wrong way.”

Famous for his life with four wives and 18 children on TLC’s Sister Wives, Kody shared that even his wife, Robyn Brown, felt “embarrassed” by others’ reactions on the show. “She’s very protective,” he explained. “She knows me well and loves me, so she wasn’t fond of how others reacted.”

Currently, Kody is dedicated to a monogamous life with just one wife, as he revealed to his fellow participants on Special Forces. He expressed no intention of expanding his marriage any further.

“I don’t feel a spiritual or religious obligation to have more than one wife anymore because I’ve already experienced it,” he stated. “I feel like I’ve fulfilled that law. In the end, I consider myself a failure. But it’s fine,” he added, emphasizing that he doesn’t see any benefit in bringing another woman into his “loving” relationship with Robyn.

When Kody stopped by our studio, he talked more about his withdrawal from the course, why Brianna “Chickenfry” LaPaglia told him he “stunk,” and how he’s making “efforts” to rebuild his relationship with his estranged children. Check out the full interview below.

DECIDER: I was not surprised to see you make it to the end. You were very intense and locked-in through the whole process. Is that how you always are? Or did you have to turn it on specifically for this show?

KODY BROWN: Oh, I’m always intense. I have a condition and I have to really fight to focus, so you see me a lot of times with my eyes really wide. I think they compared me to the doctor from Back to the Future, Dr. Emmett Brown. I was very intense, I tried to be locked in, but I was just trying to hyper focus. 

Now that you’ve seen the show, I’m wondering: were you aware of how much you got under your fellow recruits’ skin in the moment? What’s been your reaction to seeing what they’ve said behind your back?

Not only was it embarrassing, it made me feel bad. I’m sitting there and being optimistic the entire time. I think the optimism was carrying me and the enthusiasm. It’s kind of a weird place, just the excitement of being there, because every time we would accomplish a challenge, if we weren’t ridiculed too much after we were done, you feel elation, this power of having done something super hard and lived to tell the tale. No, I didn’t know I was that irritating. I’m out there, I’m trying to be optimistic the entire time, and I just rubbed a lot of people the wrong way. 

Even the DS had some fun poking at you! One of them called you a big Chucky doll. How did you feel about that?

Well, I’m having a first-person experience the entire time. I got these 18 hour days of just running, running, running. And then I’m watching the third person and I’m hearing what other people are saying about me. In the world that I live in, it is what other people say about you that develops your character or who you are. I didn’t see myself that way. It sucked to hear them talk about me like I was a crazy person that was only successful because I was too nuts to realize I was in pain. I was just optimistic. 

Well, I thought you brought a sense of comedy and levity! The quirkiness was fun.

Thank you, I’ll tell my wife. Other people like me. Not because she doesn’t, don’t get me wrong. Watching the show embarrassed her.

What was your wife’s reaction to everything?

I don’t want to out her in any way, but she’s very protective. She’s very loving. She’s very caring. She’s very protective. You don’t want your wife to have to protect you. But when you’re kind of all out naked in front of the whole world, just doing your thing and you’re quirky or whatever I am, she knows me and loves me and basically just didn’t like the reactions that other people or the treatment that people had. So I had to leave her home because I didn’t want her to get on anybody’s case about it. 

Well, it’s always good to have a protective wife.

I should be protecting her, though. That’s the problem. When I got a big mouth and I’m running it all the time, she has to sort of say, “He’s really a nice guy.”  She’s like, with everybody, “He’s not abusive. He’s really a nice guy.” I’m just talking constantly.

In your mirror room scene, we saw you break down over the state of your relationship with your kids – and finally admit to your part in the estrangement. 

I took all credit for it at the time, because in the military, I just don’t think you do things half measures. I can’t say, “Well, I’m partly to blame.” So I’m like, “Yeah, I could have done better, as everybody in the situation we were in with the divorce.” My children were victims of a very nasty multiple-layer divorce. They, of course, were closer to their mothers. I had a close relationship, or I thought a good relationship, with all of my children until the divorces. And then there was just this polarization that started to happen and a lot of blame game and a lot of things that happened. There’s a lot of water under the bridge, but I could have done better and I know that I could have done better. And of course, I’m making an effort and I have been making an effort to be in contact with my children to go through that hard work of fixing relationships that are really strained. 

How’s that been going? What has their response been to see you talk about this on the show and to you making these efforts?

It’s really a tough thing because when you’re in reality TV or you’re doing podcasting or anything like that, what you say goes on record and other people start to judge you by that. My opinion of myself won’t matter. Your opinion of me, what you tell the public is going to matter. So when you have this happen with your children and your children go, “My dad did this or my dad did that,” you really have no defense of it. I can’t throw my children under the bus, or I’ve made the mistake of saying something that I shouldn’t have about my children, and this hurts relationships. So reality television can be really hard on relationships. 

Special Forces is this type of situation where you’re going to trauma bond with each other. I mean, in the end, they’re pitting us against each other. Like, “She says you’re creepy.” And I’m like, “Well, I am kind of creepy. Have you seen my hair?” I think she was just looking for a word to protect me. She’s like, “He’s creepy,” in order to say, “I don’t know him. We’re not familiar like you might want to try and tell us we are.” So you just have to be careful in these environments not to be offended or be hurt by what is said. You just go, “OK, this is reality television and it is a form of a circus.” I was just the circus clown. 

Have you heard from any of your ex-wives?

Not at all. I’ve heard nothing. I’m not surfing social media or very much media. I don’t know if they’ve talked about me. I don’t know if they’ve said anything. I know that my oldest son, who’s talked to me a little bit about it, is watching, and then I watch it with all the children at home. My oldest son of my wife and I comes home to watch it with us. 

It seems like a handful of people you were there with were just finding out about your lifestyle. What kind of reaction do you typically get when you tell people about your multiple wives and many children?

Well, the problem is now the polygamy thing is all in my past. The 18 children, of course, are not in my past. So I used to say, “Well, I have four wives,” to introduce myself to people that were strangers, that we’re going to find out anyway. Then I started introducing with the divorces, [saying], “Oh, I have 18 children.” That’s kind of a segue into a conversation. 

When you were opening up to the other recruits, you said Robyn is your final wife and that you are done getting married. Why is that? And are you sure?

Mostly just because I don’t feel like there’s a spiritual obligation, or a religious obligation to having more than one wife now because I’ve already done it. I feel like I’ve fulfilled the law in that sense. I came out as a failure in the end. But whatever. I don’t know what to do with it now. And mostly because  there’s a synergy and a trust that she and I have. [We have] a really good, loving relationship. I just don’t see any value in having another woman be part of that. I don’t want that. I really just want quality time with you. This quantity over quality should have never been part of my life. I should have always been seeking quality. I have quality now, and that’s what I want to stick to. As I move forward with the relationship with all my children, I’m hoping for more quality in that relationship. But we have to do a lot of work. All of us have to do a lot of work with regaining trust. 

You made it to the end, but unfortunately, you failed the final task and were removed from the course. How did you feel about that? And do you think Gia and Shawn’s win was well-deserved?

Oh, I think watching the finale, watching them manage the capture and everything, yeah. They really struggled in the very beginning. And from a perspective of being in condition and ready to be there to perform, I was maybe a little bit arrogant and I was like, “You ladies don’t really belong here.” Not Shawn. Shawn came in shape, I believe. They did so well in a place where I couldn’t. I watched the final episode and I’m like, “You guys earned that.” I could run, I could keep up with the pro ballers. I pulled my hamstring on day three. I tore it. [I had a] big purple bruise behind my knee. But I just managed to keep up with those things. But in the end, I just really lost my head. And I got to the point where I knew myself. I’m going to talk, and the first person that’s going to talk is going to be me unless I just stop talking. So, I stopped talking and got into a place where, because I wasn’t talking, I couldn’t protect anybody or help anybody or serve anybody besides just not talk. When I open my mouth, you don’t know if I’m going to be valuable or invaluable or destructive. So I just shut up. I didn’t know what to do or say. Those girls managed that really well. To be fair, I’m disappointed for Bri because she really deserved to be on that podium. And I think the panic hitter finally with the coffin. Who wants to lay down a mud coffin? But I think she could have done it. And I think she deserved to be on the podium. 

The DS are interesting guys. Did you have a favorite while you were on the course? And did you have a least favorite, or one in particular you avoided?

In the other shows that I’ve seen, the first thing I thought was that Q was a sadist. I’m like, “He’s mean.” But I saw their humanity in all of them eventually. Then talking with Q after it was all done at a media event, it was fun to get to know Q. Of course, Rudy seems like a guy with a heart of gold for real. Rudy has a talent of making you still feel loved while he is beating you with a stick. He’s just really good that way because you feel his humanity, that he’s trying to push you. My favorite to start with, from watching other shows, was Billy. When Billy grabbed my head and stuffed my face in the mud, I giggled because Eddie Van Halen was stringing my guitar. I giggled and choked on the mud water. So I was coughing up brown stuff for a couple days. 

I was going to ask if there were any moments you wished made it to air that didn’t, but you were just telling me about Brianna making you wear deodorant. Tell me more about that.

Having watched other shows, I just didn’t know we were going to be able to shower. I was taking supplements I’d need to be able to just manage myself. I took foot care mostly. I just didn’t know what we were going to have. We were allowed to shower, but I’m the oldest guy and the last one in the shower and the last thing I wanted to do–and I take long showers because I’m exhausted–was get called out by the DS or have to go out and get smoked, so I just didn’t shower for a couple days. Bri LaPaglia was just like, “Kody, you stink.” And I was like, “I’m so sorry I haven’t been showering.” She said, “Well, why don’t you take a shower?” I didn’t even bring deodorant because I didn’t think we’d be able to shower. So I take a shower. Andrew East is a great guy, a man’s man. [I said], “Andrew, can another guy use your deodorant?” [He said], “Sure, brother. Here you go.” It didn’t help, I still smelled. I couldn’t smell anybody but myself.

This interview has been edited for length and clarity.

Special Forces: Worlds Toughest Test Season 4 is currently streaming on Hulu.