In 1743 William Crichton-Dalrymple, 5th Earl of Dumfries, resigned his army commission and retired to his Ayrshire Estate. He was made a member of the Most Ancient and Most Noble Order of the Thistle in 1752, a very prestigious award granted directly by the Sovereign. Desirous of a grander house, the Earl asked the Adam brothers, Robert, John and James, to submit a design. Their completed drawings were presented in 1754 and a contract was agreed to build a new house to be called Leifnorris House. It was only when the foundation stone was laid on the 18th of July 1754 that his lordship decided the name should be changed to Dumfries House in line with his title. The house was completed on time and on budget in 1759.

A year after the build had begun, Lord Dumfries' wife, Lady Anne Gordon, died. He mourned deeply but did not abandon the project. He set out to furnish the house, drawing largely on the rococo style, with a vision of an elegant interior and spent considerable sums to realise his plan. In addition to a large commission from the talented Scottish wrights Alexander Peter, Francis Brodie and William Mathie, the Earl hand-picked the finest furniture from the workshop of Thomas Chippendale, thereby creating one of the most treasured interiors of the Scottish Enlightenment. He was hopeful that his new, richly furnished Palladian house may help attract a new wife and produce his much-desired heir. The Earl's plan appeared to work, as in June 1762 he married Anne Duff, a distant cousin. However, the marriage was not a great success and six years later the 5th Earl died without an heir. His widow left Dumfries House for Edinburgh and later remarried.

Following the death of the 5th Earl, the estate passed to his nephew, Patrick Macdouall (1726-1803), who became Lord Dumfries and 6th Earl. In 1771 he married Margaret Crauford and for the next 35 years they lived at Dumfries House. It is Margaret who is credited with making improvements to the productivity of the land on the estate. They demolished the original house, Leifnorris Tower, which still stood nearby and in November 1772 the couple's daughter, Lady Elisabeth Penelope, was born; she was the mother of the 2nd Marquess of Bute.

One of the most important figures in our story is John Patrick Crichton-Stuart (1847-1900), 3rd Marquess of Bute, who inherited the estate at the age of six months. He so loved Dumfries House that at one time he called it the "homeliest of all his homes". In 1889 he became aware of a Scottish architect, Robert Weir Shultz, who would go on to add sympathetic extensions to the east and west wings of Dumfries House. By building upwards and backwards from the original Georgian pavilion, the extensions sit harmoniously with the original building. Sadly, the 3rd Marquess died at Dumfries House, never having seen the extensions completed.

Dumfries House remained a family home from 1760-1993, when the last full-time occupant, Lady Eileen, Dowager Marchioness of Bute, passed away. The house remained in the hands of the 7th Marquess of Bute who ensured that it was maintained, although not used as a primary residence.

The house was subsequently offered for sale and two auctions dates were set aside in July 2007. However, a consortium led by HRH The Prince of Wales succeeded in purchasing the house and contents and, in doing so, saved Dumfries House and one of the most important collections of Georgian Scottish and English furniture for the nation.

Under the auspices of The Great Steward of Scotland's Dumfries House Trust, and since 2018 The Prince's Foundation, improvements have been made to the grounds and gardens, historic outbuildings have been brought back to use and the estate is a focus for a range of training opportunities for young people and traditional skills and crafts. The transformation of Dumfries House has brought many employment opportunities to the local area, which was greatly affected by the demise of the coal mining industry, and stands as a prime example of heritage-led regeneration.