The site was probably the location of a pre-existing castro that was Romanized during the Roman epoch. It is considered the oldest constructed Romanesque castle located in the Beira Interior region. The rectangular tower reveals a typology typical of the Roman military architecture, due to its modest dimensions, placement in the center of the courtyard and support by the granite massif. Eighty years later, the first Romanesque Portuguese castles were constructed by the Templars who appropriated the older fortifications that were located near the borders with the Kingdom of León and disputed lands in the Côa Valley.
After a period of abandon, it was re-populated sometime in the 12th century, during the reign of King Sancho. During the reign of his successor (Sancho II), following the model of Évora the settlement was conceded a Foral and a formal castle was built in 1228. It is likely that the walls were erected after the construction of the keep, around the 14th century during the sequence of the reforms initiated by King D. Denis (or later by King D. Fernando, during his battles with the Kingdom of Castille). The fact that the town was issued a new foral (under the reign of King D. Sancho II) and received a fair charter (under King D. Denis) proves the regional importance of the locality.
In the 15th century, the alcalde of the castle was Manuel Sardinha, who was succeeded by Pêro Zuzarte. By 1422, there was a reference to approximately 2130 inhabitants living on the hilltop, from the Rol dos Besteiros. But, by the end of that century (1496), the Inquirição identified less than 144 residents on the site. The sites importance was reversed in 1510, with the renovation of the foral by King D. Manuel, who initiated a campaign of public works at the castle, that included the placement of a Manueline emblem over the main gate. In 1522, the alcadery was given to Garcia Zuzarte, then D. Manuel's custodian.
In 1527, the town was elevated to seat of royal county under the administration of King D. John III, headed by Luís da Silveira, his custodian. The Numeramento (royal census) indicated 383 habitantes. The site's importance continued decline, with its military structures repurposed for other military roles and its northwest gate closed-off and the battlements were reconstructed.
Starting in 1940, the DGEMN began work to upgrade and maintain the castle and its walls; upgrades to the site included reinforcing the merlons and supporting the toweres and corner walls with reinforced concrete, in addition to re-construction and consolidation of the doorways. As part of the nationalist efforts, between 1942 and 1945, repairs continued with the consolidation of the walls, repair of pavement stone in the cistern, and repairs to the woodframe and doors. These repairs continued in 1951 and 1952, and then later in 1984. As part of the 2000-2001 program to recuperate historic villages, then rombos were reconstructed and pavement along the battlements along the castle perimeter were repaired.
The isolated castle is located in an urban area, implanted on a hilltop between the Serra da Malcata and Serra da Pena, at approximately 786 metres (2,579 ft) above sea level. The building overlooks the small town of Sortelha and the valley, with its southern flank comprising inaccessible cliffs. It is integrated into the urban area, totally encircled by a line of fortified walls and situated on the southeastern elevation. Most of the landscape is surrounded by rocky hills, dominated by large granite boulders and medium-elevation shrubbery.
To the south of the castle is Cova da Beira, to the west is the Serra da Estrela and Castle of Belmonte, while to the east is a large plateau that demarcated the Côa Valley and Sabugal. In front of the castle is the Manuelino pillory.
The medieval courtyard is defined by an irregular oval, marked by principal gates, protected by towers, similar to many of the Gothic defenses of the time. During the Manueline constructions, the fortifications transitioned into design supporting canons. The Varanda do Juiz, also known as the Varanda de Pilatos, erected over the northwest gate, came from the Manueline campaign and marked by the royal arms, placed over the arch and varanda. Also, the beginning of the residence was begun (or reformulated) at the time.