Backstory: Southampton's Amazing Port of Missing Men Mansion

If you drive up North Sea Road to where it ends amidst the sand dunes of Sebonac, you might notice a private driveway that goes off to your left heading for a mansion far off by the bay. Linden trees lining the drive will prevent you seeing the building. But it’s there, a splendid mansion called “The Port of Missing Men,” built in 1924 on a parcel of land that, at that time, exceeded 2,000 acres.

This was the second home that the wealthy New York City Rogers family had built in Southampton. The first, called “The Beach House,” was built oceanfront in the estate section of town in 1891. (It was bulldozed down in 1932.)

The Rogers fortune for the Beach House had been made in the 1880s by Henry Hudelson Rogers, the co-founder of Standard Oil. His son Henry Hudelson Rogers II, who inherited a $100 million fortune when his father passed away, built the Port of Missing Men so far away north of the highway as a counterpoint to the beach house.

And women were not allowed. There were no telephones, no contact with the main house. There was hunting and fishing, card playing, a bar (this was during prohibition), kennels, pool tables, a smoking room and a living room lined with a library of books. There was also an indoor swimming pool, a reception hall, dining room, many bedrooms, a full staff and down the beach a landing dock suitable for late night rumrunning, and, it was said, visits to a cathouse nearby.

Perhaps the most interesting feature was not at the house but offshore in the bay. Duck blinds had been set up, and they were wired directly to Wall Street so that as stocks were bought and sold, the “lost” men could keep track of their investments minute by minute.

Henry Huddleston Rogers II had said upon receiving his inheritance that he did not intend to invest it, he intended to spend it. And spend it he did.

Among his children was Mary Millicent Abigal Rogers, born in 1902, who grew up beautiful and became one of the great socialites of that era. Rogers II gave Millicent the name of some eligible American bachelors she might marry but she’d have none of it. At age 20, she eloped with Count Ludwig von Salm-Hoogstraeten and moved to Europe. Salm-Hoogstraeten was part of royalty but with no money and eventually Rogers II paid him $600,000 to end the marriage.  A son, Peter Salm, was born of this union. Millicent then married Argentine aristocrat Arturo Peralta-Ramos. Her third marriage was in Vienna to an American stockbroker, Ronald Balcom, and that lasted another five years.

Millicent then took up with actor Cary Grant who she said broke her heart, and then settled in New York City and Southampton where as an heiress she became a high-society fashion favorite at parties, well photographed in Vogue and Vanity Fair.

In 1938, learning of the plight of the Jews in Europe, she used much of her fortune to bribe border guards in Switzerland to bring refugees out.

Millicent died in 1953 when she was just 50 years old. Her son, Peter, inherited the Port of Missing Men and it is his widow – Peter passed away in 1994 – Countess Wiltraud von Salm-Hoogstraeten, now 86, who lives there today.

Sometimes, working with non-profits, she gives tours of this fabulous mansion much of which is restored with original furniture.

Though the mansion has remained in the family since it was built 110 years ago, it now sits on 600 acres. In the 1990s, about 1,400 acres of the property – some waterfront dunes – was sold to Louis Moore Bacon III, the billionaire owner of Robins Island, which sits out in Peconic Bay to the west. Bacon agreed never to develop it. He says he will keep it wild so when he is on his island he can look out and see it as nature has created it.

Have a East End real estate story? Want to share? Text us at 516-527-3566. We’ll call you back, and then write it up for this weekly column. –Dan