Olimje castle is a symmetrically designed Renaissance building with four wings and prominent round towers. In the castle the Pauline order later found refuge, which is why the single-nave church of the Assumption of Mary and the chapel of St Francis were added to the building. The monks converted the lower rooms of the western castle tower into a pharmacy, which was among the first in this part of Europe.
The Pauline pharmacy, which can still be seen today, is painted with scenes connected with healing. Since 1990 Minorite fathers have lived in the monastery. Besides the church and the pharmacy, in the rooms of the castle it is also possible to see a permanent exhibition about Bishop A. M. Slomšek and to obtain medicinal preparations from the old pharmacy.