James Comey's wife warned him: 'Don't be the torture guy'

When your job is to handle the nation's most classified information, it makes it difficult to vent about your day at work when you come home.

That's something former FBI director James Comey knows well.

"It adds to your level of stress," Comey said.

"The rules are, if you're dealing with a classified matter, you may only discuss it with someone who has a need to know it, a work related need, and the appropriate clearance. Well, your spouse has neither," he said.

So when it comes to talking to his wife of more than three decades, Patrice, she's predictably cut out.

"As much as I loved her and as important as an advisor she's been to me my whole life, she didn't have a need to know anything about the classified topics I worked on. And she didn't have the appropriate clearance. She's an extraordinarily trustworthy person, but she doesn't have the appropriate clearance," he said.

During an exclusive interview with ABC News chief anchor George Stephanopoulos ahead of the April 17 release of his book, "A Higher Loyalty: Truth, Lies, and Leadership," Comey talked about one time when he got spooked by something Patrice said about his job.

During the George W. Bush administration, when Comey worked as the deputy attorney general, one of the issues Comey was working on was the American government's use of interrogation techniques in dealing with terrorism that some considered torture. It was garnering "a great deal of news and debate," Comey said.

"She knew that, and she also knew I was under some sort of great stress," Comey said of his wife.

"And she said to me one evening, 'Don't be the torture guy.' And I said, 'Wha -- you know I can't talk to you about that kinda stuff.' And she said, 'I don't wanna talk about it. Just don't be the torture guy.' And she repeated that periodically thereafter," he said.Comey told Stephanopoulos that "in the moment, it was irritating" to hear her say that because he wanted to counter with an explanation that he couldn't give because of the rules about sharing information.

"I wanted to say, 'You have no idea how hard these legal issues are. You have no idea that Congress defined torture in American criminal law differently than you and I would understand it,'" he told Stephanopoulos.

"So saying, 'Don't be the torture guy,' I don't wanna be. But my job as a lawyer is to say, 'Here's what the statute means,'" he added. "And there's a whole lot that would pass muster under the statute, that I would think that any normal person would think is torture."

Comey said his wife's warning "echoed around my head an awful lot," and he interpreted her message to mean: "Rise above and remember, someday you're gonna explain to your grandchildren how you conducted yourself."

For the most part, however, Patrice Comey told Stephanopoulos that her husband can turn off work when he's at home.

"I think that's one of the things he enjoys about being home, is he's kind of forced to turn it off, because that won't, it won't fly," she said with a laugh. "And he doesn't wanna be that at home. He really is interested in the kids and what I'm doing."

James and Patrice Comey have five children, having lost one son, Collin, who died at a very early age. The couple's four daughters, after being disappointed that Hillary Clinton didn't win, have been politically active in the wake of the 2016 election.

"I wanted a woman president really badly, and I supported Hillary Clinton. A lot of my friends worked for her. And I was devastated when she lost," Patrice Comey told Stephanopoulos.

Patrice Comey said that, like many, she was surprised about the letter her husband wrote 11 days before the election to announce that the FBI was going to look into more Clinton emails in former Congressman Anthony Weiner's laptop.

"Here's the thing: I knew that if there were emails that pertained to this investigation on Anthony Weiner's laptop, the FBI had to go get them. They had to," Patrice Comey said.

"And as a supporter of Hillary Clinton, I wanted her to say, 'That's what I want the FBI to do. Of course they have to do that. I don't want them on Anthony Weiner's computer. And I'm sure it's innocent, but that's what they have to do,'" she said.

"And then I'd want her to say -- this is what I was dying for her to say -- was, 'We need the FBI. I'm glad they're doing it. I trust them. I didn't do anything wrong," Patrice Comey added.

"Instead, I felt like she attacked the institution and my husband, and I was disappointed," Patrice Comey continued. "I still wanted her to win. And I have a bias. I know I have a bias, 'cause I love my husband and I trust him."

When asked about how Clinton wrote in her book, "What Happened," that she believed James Comey "shivved" her, Patrice Comey said, "That made me so sad."

"I just can't even imagine the pain she's gone through. I cried Election Day. I went on the Women's March the day after the inauguration. And I'm hoping for a woman in 2020," Patrice Comey said of the next presidential election.

Her husband didn't share any specific political hopes about the 2020 election, but laid out his working plan for the future.

"I'm gonna teach. I'm gonna travel around and speak about leadership," James Comey said.

"I wanna be useful, especially to young people. It's really fun to engage with young people. Talk about people who will poke and prod you. But I wanna offer them a vision of 'here's what it should look like.'

He went on to share the message he has for young people.

"Don't withdraw. Get into the public square. Be part of the life of this country 'cause we need you so badly," James Comey said. "And when you do it, think about what matters most of all. And that is the values involved in your institutions."