They used the asylum to house the "deviants" they found in Amn who were caught practicing magic without an official license. There was a portal to the Underdark in a secret room on the main floor of the building. The building itself was connected to a magical dungeon, which was used in exceedingly rare cases to test inmates to see if they were sane enough to leave.
The cursed elven mage, Jon Irenicus, as well as the Bhaalspawn, Imoen, were sentenced to Spellhold for casting illegal arcane magic within the confines of Athkatla. While being imprisoned, Irenicus was too powerful to contain and killed all of the Cowled Wizards who resided there, taking over the facility to perform his own experiments. Irenicus's vampire sister, Bodhi, used the lower floor dungeons as a hunting ground to toy with those who made it there.
It was eventually abandoned after a mass prisoner escape. Several adventuring parties attempted to loot Spellhold of its valuables but were never heard from again. It became known as an anathema to the otherwise ordered nation of Amn.