When the Topp family abandoned the Gothic mansion, they determined to take away everything that could be moved and rented a freight car for that purpose.  All the furnishings (including the Coxe pieces that had been left in the house), the painting of the estate commissioned by William Henry Coxe, even the chandeliers were packed for shipping.  The project was ill-fated from the start.  The cast-iron arch (with the name Coxe inscribed at the top), which stood at the entrance to the front walk, was pulled from the ground and loaded onto a wagon.  On the way to the depot, the wagon tipped over and the arch was broken into pieces.  Then, enroute to Louisville,  a fire broke out in the freight car destroying most of its contents.

The now bare mansion had also suffered decades of neglect. Photographs of the entrance gates and the house taken by Lem Johnson in 1901 show that the structure had lost its porches.   Holly Springs historian John Mickle recalls that, in the lean years of the nineties during the  hard times of five-cent cotton,  the great houses of Holly Springs went begging, and William Logan Walker bought the Topp place for a song.  On March 2, 1896, Walker paid $2,000 for the mansion and the diminished estate of 4 acres, a project  that had originally cost $40,000 (DB 60:281).

Walker's wife was Sallie Trotter, daughter of Judge James Fisher Trotter (1801-1866) and his second wife Elizabeth Wynne.  One of the countians chosen for the Mississippi Hall of Fame, Trotter served as United States Senator, justice of the state supreme court, and professor of law at the University of Mississippi.  His wife, Elizabeth Wynne, was the sister of Joel Echols Wynne, who built the mansion now known as Wakefield,  east on Salem from Airliewood.  In 1850, Judge Trotter purchased ten acres across the street from the Coxe estate.  Shortly afterward Trotter built on the highest part of the tract (near the intersection of Salem and Chesterman and said to be the highest point in the town) a substantial two-story frame house with a two-story gallery across the front supported by six columns.  In May of 1870, Trotter s widow deeded the tract and house to her three daughters, one being Sallie Trotter Walker (DB 30:531).  In 1871, Logan Walker bought out his wife's sisters (DB 32: 121, 273), and he and his family occupied the Trotter house until it burned shortly before he bought the Topp mansion in 1896. 

An enterprising man, Logan Walker (1840-1897), who had grown up south of Waterford, moved to Holly Springs after the Civil War.  He is listed in the 1870 census as an insurance agent, but in September of 1875, the Holly Springs South announced that  W. Logan Walker has opened a dry goods store next door to his grocery in the southwest block of the square, and Mr. Mickle tells us that  after the fever [in 1878},  Walker was associated with Col. John Calhoun as business manager and advertising solicitor of the other newspaper, the Reporter.  Apparently he continued in this position until his death.

Logan Walker died little more than a year after he had purchased the Topp mansion.  At Sallie Trotter Walker's death in 1901, her two sons (named for their maternal and paternal grandfathers) divided the estate (DB 65:223). James Trotter Walker kept the large Trotter lot on Salem, and William Wright Walker purchased his brother s interest in the Topp estate.

In succeeding years, Laura Anderson Walker, widow of James Trotter Walker and daughter of John E. Anderson, first president of the Merchants and Farmers Bank and one of the wealthiest men in the county, sold a number of small lots from the old Trotter holding.  William Wright Walker and his wife, Parrie Totten Goodrum, an Arkansan and the granddaughter of James Lockhart Totten, early representative from Marshall County and Speaker of the Mississippi House, continued to live in the Topp mansion with their daughters.

At the time of the 1901 division with his brother,  William Wright Walker (1875-1908) was facing the bankruptcy of his dry goods establishment on the east side of the square, and he was forced to pledge all his possessions, including  the Coxe or Topp property against his indebtedness (DB 65:236).  Four months later, on April 2, 1902, the property was lost to his creditors, and the Gothic mansion was purchased for the first time by people from a distance, a family from Michigan (DB 65:599).