The Olimje Monastery has an interesting history. On the ground of today’s monastery stood a fort owned by the Counts of Pilstien, most probably built around 1015. One of the owners was also Ema of Pilstein, otherwise known St. Ema of Gurk. The fort was then remodelled into a country mansion. In 1550, the subsequent owner Count Tattenbach enlarged the building and renewed it in renaissance style, transforming the mansion into a castle.
Baron Ivan Zakmardy de Dyankoch bought the property in 1657 and donated it to the Pauline monks in Lepoglav, Croatia. 1663 saw the arrival of the first six Pauline monks. Their immediate efforts were given towards reorganizing the castle to meet the needs of a monastery as well as building a pilgrimage church. Thus the castle became a monastery.
The monks stayed in Olimje until 1782, when the emperor Joseph II dismissed the monastery. They returned to Croatia and the first parish was established in Olimje. In 1805, the monastery was put up for auction and bought by count Attems and the monastery consequently became a castle again, according to ownership.
Two years later, two wings of the castle were pulled down and the third one became a residence for the parish priest on the second floor, and for the keeper of the count’s forest on the first floor. In 1945, the partisans killed count Ferdinand Attems, the castle’s owner at the time and the building was nationalized. Shortly before the downfall of Yugoslavia, efforts were made to restore the property as a monastery. The diocese of Maribor took the castle on hire and handed it over to the Conventual Friars. In 1999, the Slovenian Conventual Province founded the friary in Olimje, which took its name after blessed Anton Martin Slomšek. Thus the castle became a religious house again. The Olimje Monastery has an interesting history. On the ground of today's monastery stood a fort owned by the Counts of Pilstien, most probably built around 1015. One of the owners was also Ema of Pilstein, otherwise known St. Ema of Gurk. The fort was then remodelled into a country mansion. In 1550, the subsequent owner Count Tattenbach enlarged the building and renewed it in renaissance style, transforming the mansion into a castle.

Baron Ivan Zakmardy de Dyankoch bought the property in 1657 and donated it to the Pauline monks in Lepoglav, Croatia. 1663 saw the arrival of the first six Pauline monks. Their immediate efforts were given towards reorganizing the castle to meet the needs of a monastery as well as building a pilgrimage church. Thus the castle became a monastery.

The monks stayed in Olimje until 1782, when the emperor Joseph II dismissed the monastery. They returned to Croatia and the first parish was established in Olimje. In 1805, the monastery was put up for auction and bought by count Attems and the monastery consequently became a castle again, according to ownership.

Two years later, two wings of the castle were pulled down and the third one became a residence for the parish priest on the second floor, and for the keeper of the count's forest on the first floor. In 1945, the partisans killed count Ferdinand Attems, the castle's owner at the time and the building was nationalized. Shortly before the downfall of Yugoslavia, efforts were made to restore the property as a monastery. The diocese of Maribor took the castle on hire and handed it over to the Conventual Friars. In 1999, the Slovenian Conventual Province founded the friary in Olimje, which took its name after blessed Anton Martin Slomšek. Thus the castle became a religious house again.

Olimski samostan ima zanimivo zgodovino. Na mestu sedanjega samostana je nekoč stal stolp, ki je bil verjetno zgrajen okrog leta 1015 in je pripadal Pilsteinskim grofom. Lastnica je bila tudi sv. Ema s Pilštanja, bolj poznana kot Hema Krška. Pozneje so stolp preuredili v podeželski dvorec. Leta 1550 ga je grof Tattenbach povečal ter obnovil v renesančnem stilu. Iz dvorca je nastal grad.

Leta 1657 ga je kupil baron Ivan Zakmardy de Dyankoch in ga podaril pavlinskim menihom v Lepoglavi na Hrvaškem. Leta 1663 je prišlo v Olimje 6 patrov. Njihova takojšnja skrb je bila preureditev gradu za samostanske potrebe in graditev božjepotne cerkve. Tako se je grad spremenil v samostan.

V Olimju so ostali do leta 1782. Takrat je cesar Jožef II. razpustil samostan. Menihi so se vrnili v hrvaške samostane, v Olimju pa je bila ustanovljena župnija. Samostan je bil leta 1805 prodan na dražbi. Kupil ga je grof Attems in tako, vsaj po lastništvu, samostan postane zopet grad.

Dve leti pozneje so podrli dva trakta gradu v tretjem pa je dobil stanovanje vsakokratni župnik in grofov gozdar. Leta 1945 so partizani v Slovenski Bistrici ubili takratnega lastnika gradu grofa Ferdinanda Attemsa in stavbo nacionalizirali. Tik pred razpadom Jugoslavije so se začela prizadevanja za ponovno oživitev samostana. Grad je vzela v najem mariborska škofija in ga izročila redovnikom minoritom. Leta 1999 je slovenska minoritska provinca ustanovila v Olimju samostan, ki so ga po razglasitvi Slomška za blaženega poimenovali samostan bl. Antona Martina Slomška. Tako je olimski grad zopet postal samostan.