THERE’S A SAYING in Hong Kong property circles that if the city’s richest man, Li Ka-Shing, is selling, you don’t want to be the buyer.

Now, a group of investors who paid $5.2 billion for Mr. Li’s stake in The Center almost three years ago — making it the world’s most expensive skyscraper — is finding out why. After initially making quick profits flipping floors in the 73-story tower, the combination of anti-government protests, the coronavirus pandemic and escalating US-China tensions has seen vacancies surge, rents drop and dealmaking dry up.

Just one sale has been made this year — at a 35% discount to early 2019 prices, according to property-data provider Real Capital Analytics. Almost one-fifth of the building is empty — one of the highest vacancy rates in Hong Kong’s sought-after central business district — and rents are down about 20% from a year ago.

“It was a reasonable investment decision back then,” said Thomas Lam, an executive director at Knight Frank LLP. Market prices were higher than the average cost the group paid, and flipping floors seemed easy, he said. “But now, as rental yields and office demand decline amid the worsening economy, buyers are much more reserved.”

When a group of local investors with colorful nicknames like “Minibus King” and “Queen of Shells” banded together to buy Mr. Li’s 75% stake in late 2017, Hong Kong’s office property market was riding high. Prices in Central had risen 20% in just under a year, according to Savills Plc, and the office vacancy rate in the district was just 2%. (CK Asset Holdings Ltd., Mr. Li’s property arm, sold the other 25% of the tower in the years after it opened in 1998.)

After the deal closed in mid-2018, the group quickly divvied up the 47 floors, 402 parking spaces, office suites and retail outlets and started flipping them. Within a year, they had offloaded more than eight floors and a dozen office suites for about $1.3 billion, reaping hundreds of millions of dollars profit.

Then in June 2019, the city was rocked by the first of a double-whammy of calamities that has sent the economy into its deepest-ever recession, with the eruption of anti-government protests that grew increasingly violent and disruptive. The unrest ran into the New Year, when the coronavirus pandemic took hold, while worsening tension between China and the US also chilled the outlook for the future of Asia’s financial hub.

All that has virtually put an end to dealmaking at The Center. One sale that was in contract when the protests broke out was eventually terminated by the end of 2019, with the buyer forfeiting a $1.1 million deposit, according to Real Capital Analytics. And just the one sale has been struck this year, despite three floors being on the market.

Hong Kong CBD office rents have slumped since protests erupted in mid-2019

“As they can’t sell at a good price right now, they would want to offload just one or two floors for some cash and keep most of their portfolio for rent until the market turns around,” said James Mak, a district sales director in Midland IC&I Ltd. “These tycoons from the last generation are not willing to lose money.”

The dearth of activity augurs poorly for the broader office market. Office valuations in the city may slump as much as 20% this year, according to Jones Lang LaSalle, Inc. And with vacancies at a 16-year high, prime office rents may fall a further 5% over the rest of this year, according to Bloomberg Intelligence.

Keeping floors for rental income isn’t a money-spinner either.

The change in ownership from one of Hong Kong’s biggest developers to a group of individual owners with a history of flipping property has deterred tenants who favor stability in ownership and management. The Center’s vacancy rate was 19% in August, compared with 5.2% in the rest of Central, according to Centaline Property Agency Ltd.

To make things worse, leases signed this year at The Center are fetching an average of just HK$69 ($8.90) per square foot a month, 20% lower than a year ago, according to Bloomberg calculations from data provided by Midland IC&I.

All that has put the buyers in a hole.

“These guys were hoping to flip the properties at a 30% gain straight away, but they’ve been caught out by other factors,” said Phillip Zhong, a real estate analyst at Morningstar Investment Service. Rental income may not cover interest payments on loans to finance the deal, meaning even selling at the initial cost price would “mean taking a big hit overall,” he said.

The background of the investors behind the record-breaking deal drew as much interest as the transaction itself. Instead of a listed developer or big private-equity fund ubiquitous in large property deals, the consortium brought together a disparate group of local entrepreneurs who have capitalized on Hong Kong’s ever-rising property prices to turbo-charge their fortunes.

The most high-profile members of the group are Ma Ah-muk and Pollyanna Chu, who initially took 13 and seven floors respectively.

Mr. Ma began building a serious fortune in 1977 when he founded Yan Yan Motors Ltd., which operates Hong Kong’s minibus routes, earning him the moniker of the “Minibus King.” Mr. Ma’s fleet of green vehicles serve areas standard buses can’t reach, acting both as a last-mile option for commuters as well as a way to avoid the crowded metro trains.

Ms. Chu is known as the “Queen of Shells” for financing small-cap companies. She began her career investing in San Francisco real estate before returning to Hong Kong in 1992. Her Kingston Financial Group Ltd. has carved out a niche offering underwriting and margin services for small companies going public.

Ms. Chu had a lengthy spell as Hong Kong’s richest woman, but despite her family empire now spanning hotels and watchmakers, her net worth today is a fraction of what it used to be. Kingston shares are down 18% this year and her exposure to Macau’s casinos during pandemic-induced shutdowns has caused further headaches. Ms. Chu’s wealth now stands at about $603 million — far behind today’s richest Hong Kong woman, Vivien Chen of real estate developer Nan Fung Group at $4.8 billion, according to the Bloomberg Billionaires Index.

Representatives for Mr. Ma and Ms. Chu didn’t reply to e-mails and calls requesting comment.

After selling The Center, Li’s CK Asset used part of the proceeds to acquire 5 Broadgate — UBS Group AG’s London headquarters — for $1.3 billion in June 2018. The Swiss bank is committed to the building until 2035 and the rental yield for the London property was almost double that of The Center, Ming Pao reported in 2018, citing company management.

Once again, Mr. Li made the better deal. — Bloomberg