The thought first hit him as a broom he was pushing fell apart and clattered down the metal steps at the Dunkin Donuts Center in Providence in the wee hours of the morning. These were the same steps thousands of people once walked down to find their seat before his fights.

Five times people had paid their hard-earned cash to watch him in this cavernous building. Now he was being paid to clean it up behind fans of other performers. It was a job though and Peter Manfredo, Jr. was glad to have it but he missed being on the other side of that broom. He missed being the performer, the man for which a broom after he finished working was necessary and that’s how it all started again.

Manfredo returned to boxing in April after announcing his retirement following a stunning, third round knockout loss last Nov. 13 to Sakio Bika in the same building where he now worked as a laborer. It was honest work. No shame in it. Yet something was missing. Prize fighting was missing.

Every professional boxer goes through this kind of sad, shocking withdrawal when he retires. Barry McGuigan recently told the BBC it is the absence of something you knew so well in your youth but now must accept can never be replaced. Manfredo began to understand that as the head of his broom clattered down the steps only hours after he’d watched Manny Pacquiao, a fighter with whom he once trained at Freddie Roach’s Wild Card Gym in Hollywood, dominate and defeat Oscar De La Hoya.

The juxtaposition of what he was doing at that moment and what he knew was going on for Pacquiao in Las Vegas began to make the 28-year-old former super middleweight title contender think and the conclusion he came to slowly became clear.

“I came to my grandmother’s house after working four or five hours on a set up crew at the Dunkin Donuts Center and I had to go back for a third shift sweep for another eight hours so I watched the De La Hoya-Pacquiao fight there,’’ Manfredo recalled. “A few hours later I’m sweeping the stands and I’m thinking about Manny and Freddie (Pacquiao’s trainer) Freddie Roach celebrating after winning a big fight and here I am pushing a broom around in the building where I used to fight.

“The broom breaks and I start swearing and some of the guys hollered ‘Maybe this ain’t for you!’ It was tough. That’s when I first started thinking about fighting again.

“I love my job with the laborer’s union. I like going to work but it’s going to be tough that day when I can’t walk into the ring any more. Right now I know I still can so that’s why I came back.

“Hopefully, I’ll know when it’s time to stop. To be honest with you, the whole goal now is to pay off my house. Then I can just go to work and my wife can stay home and raise the kids. I’m an old-fashioned Italian that way but you need three incomes to raise a family today.’’

Manfredo’s return was a spectacular one, knocking out Walid Smichet in Smichet’s adopted hometown of Montreal in seven rounds last month, dropping him in the third, staggering him in the sixth and then finishing him off with a booming left hook at the end of the seventh round.

Trying to build off that victory, Manfredo was set to go again June 20 in Minnesota against Matt Vanda, a fight he was pursuing for the same reason he agreed to face Smichet – because Peter Manfredo is stalking John Duddy.

The Vanda fight fell through, however, so he continues to look for the kind of matches that can keep both his career and his search for Duddy alive while also reducing his mortgage as fast as he can. To do the latter, he feels, John Duddy is what he needs. Getting to him has always been the hard part.

“He lost his last fight (to journeyman Billy Lyell in a surprising upset that actually might make a Manfredo match more viable in the odd way the business of boxing is conducted these days) but I still want to fight him,’’ Manfredo (32-6, 17 KO) said. “It would still be a big money fight. He’s got one loss and I got six. What’s a loss?

“It’s an Irish guy vs. an Italian guy. It’s two guys with names people recognize. Obviously I don’t got the best defense in the world and neither does he. We both come to fight. Now that I’m fighting at 160 and not 168 it’s a big difference when you get hit. Smichet hit me and I didn’t even feel it. Bika hit me and, wow!’’

What if Duddy (26-1, 17 KO) hits him? Manfredo is unconcerned about that because A) he intends to hit him back and B) the reward is well worth the risk for both of them if they’re honest about it.

Once each was in a position to dictate terms – Duddy when he was the hottest Irish boxer in the U.S. and Manfredo when he was coming off a popular run on “The Contender” reality TV series. But Manfredo has now lost two shots at some form of the super middleweight title and Duddy not only has now been beaten by a guy with seven losses (Lyell) but was also life-and-death with Smichet, whom Manfredo destroyed.

So what’s the holdup? In Manfredo’s opinion it’s Duddy. Or, more likely, the new team around him that already got him beat only months into their handling of his affairs.

“If Duddy beat Lyell he wouldn’t ever fight me,’’ Manfredo said. “He’s been avoiding me for two years. If he’d won he would have gotten a shot at (middleweight champion Kelly) Pavlik and got knocked out. But he didn’t.

“Now he needs me as much as I need him to get that title shot. It’s a perfect fight for New England. It’s a perfect fight for both of us and we should be able to get paid for it. The winner gets back into contention. The loser? I don’t know what Duddy’s gonna do.’’

Manfredo is once again being trained by his father, Peter, Sr., with whom he had a difficult falling out in the long tradition of fathers and sons in boxing. The difficulties that developed leading up to the Bika fight resulted from the father trying to wear three hats – parent, trainer and manager –and the son chafing at all those familial and business entanglements crossing wires.

Over time the fences were mended however, and the latter hat now belongs to family friend Larry Army, who also manages rising middleweight prospect Edwin Rodriguez. In the opinion of the younger Manfredo, life is now back to normal and he and his father are in complete agreement on one very important thing. They both see John Duddy as opportunity knocking.

“I like boxing when my father is the way he is now,” Manfredo said. “I got to give him a lot of credit. He’s my trainer now but at the end of the day he’s my father, too. He had such high expectations for me. I understand that. It probably had to go the way it did for us to get to this point again. I like coming to the gym and training with him the way it is now.

“Larry’s my father’s friend. He helped us with this. It’s so much easier now. It’s a job for my dad. He can just concentrate on the training and so can I. He treats me like the rest of his fighters in the gym. Outside of it we don’t talk about boxing.

“We both agreed when I decided to come back that I’d fight at 160. It’s hard to make because I’m 28 and I like to eat but 168 isn’t my weight. I needed the Bika fight to really realize that. He blew me out in three rounds. I didn’t belong in there with him. He was too strong. I’m done fighting those big guys. It’s a struggle to make 160 but now I have a reason to do it.’’

To a large extent that reason is Duddy, a natural opponent against whom he might already have fought several times if this was still boxing’s glory days of the 1940s, ‘50s and ‘60s when ethnic fights of this ilk were made regularly because they sold.

Manfredo believes this one still will and frankly he may well be right. Ideally he would prefer another fight or two first but if Duddy finally wants to fight Manfredo he understands there’s no reason to wait any longer.

“This is my last shot at it,’’ Manfredo said. “I got the opportunity to fight Smichet in his hometown and I took it. It was a confidence boost for me taking him out like that when Duddy struggled with him and nearly lost.

“They underestimated me. They wanted to get a name at the right time on their resume but it kind of backfired. Now Duddy and I are in the same boat. We need each other to move ahead. That’s what this business is. I don’t care how good you are, in boxing timing is everything.’’

Peter Manfredo, Jr. has been willing to make time for John Duddy for the past year or two. The only question now is what time is it for Duddy?