Will Ferrell admits he was a bit of a cotton-headed ninnymuggins. 

The Elf star shared that his late costar James Caan often got fed up with his jokes on the set of the 2003 holiday film.

As Will explained to Christina Applegate and Jamie-Lynn Sigler during an appearance on the July 9 episode of their MeSsy podcast, "I literally drove him crazy in that movie."

And James—who died due to heart complications in 2022 at age 82—apparently wasn't shy about expressing his frustration with the 56-year-old. In fact, Will recalled the Godfather actor teasing him about his comedy bits "with love."

“In between setups, he’d be like, ‘I don’t get you. You’re not funny,’" the Anchorman actor explained. "And I’m like, ‘I know. I’m not Robin Williams.’"

But that dynamic wound up working out for the film, considering James played the grumpy biological father to Will's Buddy the Elf character. 

“I love that the whole time, he’s not acting,” he continued. “He’s truly annoyed with me. He’s like, ‘Can this guy shut the f--k up?’"

Once the movie was released, James had a change of heart about Will's comedic skills after seeing an early screening of the Jon Favreau-directed classic, which also starred Zooey Deschanel, Peter Dinklage and Mary Steenburgen. 

“He was like, ‘I’ve got to tell you, I thought everything you were doing while we were filming was way too over the top. Now that I see it in the movie, it’s brilliant,’" Will remembered. “That was so funny."

He added, "He’s walking out, shaking his head and going, ‘Great job.’"

Keep reading to uncover more secrets from the Elf movie set...

1. The first script for Elf was actually written in 1993, with Jim Carrey in mind to play the titular Santa's helper. Because it took over a decade for the film to finally be made, the Ace Ventura: Pet Detective star never ended up taking on the role, , which proved to be the ultimate Christmas gift for Will Ferrell.

2. Elf was also initially intended to be a bit more sinister, with director Jon Favreau revealing to Rolling Stone, "He was a darker character."

3. Before he was asked to come in to rewrite the script and transform the movie into a PG friendly-family affair, Favreau initially turned down the job.

"I took a look at the script, and I wasn't particularly interested," he told Rolling Stone. "It was a much darker version of the film. I liked the notion of being involved with Will in his first solo movie after SNL, but it wasn't quite there."

4. Buddy's iconic costume was modeled off of the elves from the 1964 Rankin/Bass film Rudolph the Red-Nosed Reindeer.

"The costume was important," Favreau told ABC News. "It looks almost like he's a puppet and the tights and the boots and the hat and he was extremely brave wearing that costume." 

5. Costume designer Laura Jean Shannon, tasked with finding the perfect look, revealed it wasn't accommodating Ferrell's 6-foot-3 frame that was the hardest part of her job. 

"It was less a challenge dressing Will due to his size and more a challenge to make sure nothing seemed off-putting or in bad taste." she explained to Feel Christmassy. "After all, I had to dress a grown man in tights and a cutaway coat. Needless to say, we did have a fair amount of fittings to be sure we struck the right balance between absurd and adorable."

6. Ferrell's Buddy costume was so captivating that the actor actually caused a few traffic incidents while filming in New York City.

"When we had Will in the Lincoln Tunnel, the tunnel was open. Same thing with the 59th Street Bridge," Favreau told Rolling Stone. "Whenever he was out there in his suit, we'd hear screeches and fender-benders and lights smashing. People would be looking at him walking on the side and that would cause a few minor traffic accidents."

7. Buddy's infamous 12-second long burp wasn't actually performed by Ferrell. Voice actor Maurice LaMarche, best known for giving life to The Brain on Pinky and the Brain, provided the dinner table belch. And, yes, we heard that. 

8. The cotton balls Buddy can't stop eating in the doctor's office were actually pieces of cotton candy that had yet to be dyed.

 9. Buddy's all-sugar diet also became Ferrell's daily food intake while filming as he actually scarfed down all of that maple syrup-coated spaghetti and candy during scenes. 

"I ingested a lot of sugar in this movie and I didn't get a lot of sleep," Ferrell told The Sun. "I constantly stayed up. But anything for the movie, I'm there. If it takes eating a lot of maple syrup, then I will, if that's what the job calls for."

15. Some of Buddy's most iconic lines—"You smell like beef and cheese!" and "You sit on a throne of lies!"—were improvised by Ferrell in the moment.

16. The big fight scene between Buddy and the fake Santa, played by comedian Artie Lange, was filmed in one take because it took the art department weeks to decorate Gimbels.

"We had one take to destroy it," Lange told ABC News, "So Favreau said, 'Just go nuts!'"

17. While most people know Favreau had a small role as the pediatrician who examines Buddy and confirms he's Walter's son, the director also had another part: He voiced the Narwhal who infamously said, "Bye Buddy, hope you find your dad!"

18. Ferrell turned down a reported $29 million paycheck to reprise the role of Buddy for a sequel, telling The Guardian in 2006 that it "wasn't difficult at all" to pass on the opportunity."I remember asking myself: could I withstand the criticism when it's bad and they say, 'He did the sequel for the money?'" he explained. "I decided I wouldn't be able to. I didn't want to wander into an area that could erase all the good work I've done—but you watch, I'll do some sequel in the future that's crap."

19. But Caan alleged it was tension between Ferrell and Favreau that possibly squashed any plans for a follow-up.

"We were gonna do it, and I thought, 'Oh my God, I finally have a franchise movie. I can make some money, let my kids do what the hell they want to do,'" he said on 92.3 The Fan in Cleveland. "The director and Will didn't get along very well. Will wanted to do it, and he didn't want the director. He had it in his contract. It was one of those things."

20. While a sequel won't be found in anyone's stockings, fans were gifted with a Broadway musical inspired by the film in 2010. The show ran through 2013 and traveled overseas to the West End in 2015. And a revival is set to London for the 2023 holiday season.

For the latest breaking news updates, click here to download the E! News App