The palace of the viscounts formed one single building with the parish church which was later separated from it when the parish of Blanes was constituted in 1319.

In the year 1114 the castle-palace then belonged to the lords of the Blanes family and in 1378 was bought by the Cabrera family. In 1623 the estate was sold by judicial decree and acquired by the merchant Esteve Alemany. The palace of Cabrera was used as a barracks for the troops which were then moving around the region and the careless behaviour of the soldiers was the reason why it was not restored as it deteriorated. The wars against France in the same century destroyed a great part of the palace.

Of the artistic and architectural aspect that remains at present, the bell tower and the 15th century facade are worth note. The rest was built after the war.