PEOPLE understands that William and Kate will not attend the Buckingham Palace lunch because they’re already at Anmer Hall in Norfolk for the holidays

King Charles is kicking off the Christmas celebrations at his annual pre-Christmas Palace lunch, but not everyone will be in attendance.

PEOPLE understands that Kate Middleton, Prince William and their three children, Prince George, 11, Princess Charlotte, 9, and Prince Louis, 6, will not be present at this year’s festivities.

The Prince and Princess of Wales were never expected to attend the lunch at Buckingham Palace on Thursday, Dec. 19. The couple and their children are already in Norfolk, England, more than 100 miles away from London, ahead of the royal family’s private Christmas celebrations at the Sandringham Estate.

PEOPLE understands that William and Kate are looking forward to catching up with family at Sandringham over Christmas as usual.

Prince William, Prince George, Princess Charlotte, Prince Louis and Kate Middleton at the 2023 Christmas carol service.Aaron Chown – WPA Pool/Getty

While Kate and William may not have joined in the festivities, Charles, 76, still hosted several members of the royal family for Christmas lunch, including his younger brother Prince Edward and his wife Sophie, the Duchess of Edinburgh, who were seen arriving at Buckingham Palace on Thursday.

Charles’ brother Prince Andrew and Sarah Ferguson canceled their appearance at the Christmas luncheon in the wake of backlash around the Duke of York’s suspected connection to an alleged Chinese “spy.”

Prince Andrew and Sarah Ferguson at the Easter Mattins Service at Windsor Castle on March 31, 2024.Samir Hussein/WireImage

PEOPLE understands that the Duke of York, 64, reached the decision after discussing the matter with his ex-wife, the Duchess of York, who convinced him that this was the right move. The same is true of Prince Andrew’s decision to skip the royal family’s Christmas gathering at Sandringham next week on Dec. 25.

Royal biographer Ingrid Seward tells PEOPLE that Andrew will have also spoken things over with his daughters, Princess Beatrice and Princess Eugenie, who are protective and loyal towards him.

“Despite [the Duchess’] missteps in the past, she does know how things will play out socially. She would tell him it’s best to stay away. And Beatrice would have backed her up in that,” says Seward.

Prince Andrew and Ferguson, who is popularly known as Fergie, divorced in 1996 after a decade of marriage, but remain close today and continue to live at Royal Lodge in Windsor.

The royals’ Christmas lunch is a private family gathering, and a tradition King Charles has continued from his late mother Queen Elizabeth’s reign. Dozens of family members convene about a week before Christmas for the upbeat holiday meal at Buckingham Palace in London. From there, the late Queen would head to Sandringham House, her private estate in the Norfolk countryside, and step out for church at St. Mary Magdalene on Christmas Day with her family in tow.

King Charles and Queen Camilla at Sandringham for Christmas with Prince George, Princess Charlotte, and Prince Louis in 2022.Samir Hussein/WireImage

The royal family’s Christmas plans have been under unusual scrutiny this year amid fallout around Andrew’s alleged links to the “spy” scandal. Charles’ younger brother has been under harsh spotlight since Dec. 13, when it emerged that he reportedly had close professional ties to a Chinese national, alleged to be a spy. The story made headlines when a British court upheld an existing decision to ban the man from the U.K. on national security grounds following a special immigration investigation, drawing attention also to his reported relationship with Andrew.

U.K. newspaper The Times reported that the businessman, 50, had been appointed as a “business adviser” by Andrew and was promised access to the highest levels of the royal family “in an apparently serious breach of national security.”

The businessman — initially codenamed only as H6, and later named on Dec. 16 as Yang Tengbo (also known as Chris Yang) — became a close confidant of King Charles’ brother and allegedly earned “an unusual degree of trust” from the royal, the BBC reported.

Yang denied any wrongdoing in a statement shared by the BBC, and Buckingham Palace didn’t comment as Andrew is not a working royal.

The Duke of York took a forced step back from public duties following his bombshell Newsnight interview in November 2019 about his speculated connection to convicted sex offender Jeffrey Epstein, who died in prison while awaiting trial.

Prince Andrew, Sarah Ferguson, Sir Timothy Laurence and more royals at the Thanksgiving Service for King Constantine of Greece at St. George’s Chapel at Windsor Castle on Feb. 27, 2024.Andrew Matthews – WPA Pool/Getty

Andrew has repeatedly denied any wrongdoing, and his transition away from royal responsibilities was made official when Queen Elizabeth stripped him of his military titles and patronages in January 2022 amid Virginia Roberts Giuffre’s civil sexual assault lawsuit. An out-of-court settlement for an undisclosed sum was reached that February.

Looking ahead to the royal family’s Christmas plans on Dec. 25, PEOPLE understands that Prince Andrew, Fergie, Princess Beatrice, Princess Eugenie, and their respective families will not join King Charles at Sandringham because of the spy scandal. Though Andrew hasn’t held a working royal role for years, he continues to join his family for major ceremonial events and holiday celebrations.

The York family had been widely expected to stay at Wood Farm on the Sandringham estate for Christmas like they did last year, but the plans have apparently pivoted.

PEOPLE understands that Beatrice, 36, and Eugenie, 34, had invitations to spend Christmas with their respective in-laws some time ago and have decided to do so. Their attendance at Sandringham had been in question, an insider says, while their parents will likely keep a low profile at Royal Lodge, their home on grounds of Windsor Castle.