Fraser was the second son of William Fraser of Park, near Fraserburgh (a kinsman of George Fraser, 15th Lord Saltoun), and his wife, Katherine née Kinellar. On 29 September 1778, Fraser was appointed a lieutenant in the 73rd Highlanders, afterwards the 71st Highland Light Infantry. He was later on board Rodney’s fleet with a second battalion (afterwards disbanded) of this regiment during the actions with the Spanish Caraccas fleet under Juan de Lángara and at the relief of Gibraltar. He served at the defence of Gibraltar in 1780–82, until the loss of his right leg, his second wound during the defence, compelled him to return home.
Fraser was captain of a garrison invalid company at Hull in 1785–93, and at the outbreak of the French Revolutionary War, raised men for an independent company. He became a major on 28 August 1794, and a lieutenant colonel, royal garrison battalion, on 1 September 1795. He served at Gibraltar in 1796–8, part of the time as acting judge advocate and civil judge. On 1 January 1800 he was appointed colonel of the Royal African Corps, composed of military offenders from various regiments pardoned on condition of life-service in Africa and the West Indies. With this corps, he served on the west coast of Africa in 1801–4, and made a brave but unsuccessful defence of Gorée in Senegal, against a superior French force from Cayenne. Gorée was compelled to surrender on 18 January 1804, but not before the enemy’s loss exceeded the total strength of the defenders at the outset.
After his exchange, Fraser was appointed to command an expedition against Senegal, which never started. He became a major general in 1808, served in Guernsey in 1808–9, and in the latter year he was appointed to the staff at Gibraltar. Fraser commanded that garrison until the arrival of General Campbell. He was then sent to negotiate for the admission of British troops into the Spanish fortress of Ceuta (on the Moroccan coast opposite Gibraltar); afterwards he commanded the British garrison there until his return to England on promotion to lieutenant general in 1813. In 1809, in recognition of its distinguished conduct in the West Indies, the Royal African Corps was reorganized as the Royal York Rangers, another Royal African Corps being formed in its place. Fraser retained the colonelcy of the rangers until the regiment was disbanded after the peace. He was made Lieutenant-Governor of Chester Castle in 1828, GCH in 1832, and was a member of the consolidated board of general officers. He became a general in 1838.
Fraser married, first, on 15 April 1790, Everilda, daughter of James Hamer of Hamer Hall, Lancashire, and they had one son and two daughters, one of whom, Everilda (d. November 1840), married General Francis Rawdon Chesney (1789–1872). His second marriage, about three years before his death, was to Miss A’Court. Fraser was described by his kinsman, Lord Saltoun, as a brave, chivalrous, upright old soldier. He died at Campden Hill, Kensington, London, on 14 November 1843 and was buried at St Barnabas’ Church, West Kensington, London.