In the 12-th century there was already the castle with a square tower, palace and palisade on the volcanic top of a basalt rock 65m high.
The origin of the castle is connected with the tale about a shepherd called Filek, and his dog Fules, as they found treasure, from which a castle was built.
The first written record comes from the year 1242, where it is written, that the castle survived the Tartar incursion.
Another record says, that a robber knight called Fulko was the owner of the castle as long as king Belo IV confiscated it in 1246. Then the castle was owned by several owners: knight Hartwig (1283), Master Egyed, Matúš Csák (1311-1321), Blažej Fónyi (1322.) In the years 1348 and 1385 it is mentioned as a royal property. In the year 1435 the castle was given as a mortgage to the Perényi family.

In the year 1440 queen Elizabeth, the owner of the castle, accredited Ján Jiskra to ensure the domain. In the year 1483 the castle was conquered by Matej Korvín who expelled Štefan Perényi from the castle.

From the year 1490 captain Raškaj is authorized to administer the castle. His daughter married František Bebek and the castle was given to him as a marriage portion.

As the Turkish danger rose, the castle was rebuilt in 1551 according to the plans of an Italian architect Alessandro Da Vedano into a massive renaissance fortress. It was enlarged and stronged. The middle and lower castle was built with two pentagonal bastions, walls, entrance corridor and cannon positions. In the same time the town got the wall with the gates and bastions as well.

The castle was a component of the defence line against the Turkish expansion in the 16-th century. In 1554 the Turks captured the castle and had been governing in and around Filakovo for 39 years.

The territorial integrity of Novohrad has been violated by the Turks in the second half of the 16th century. They were successful in subduing almost the whole territory of the county (they conquered also the castles of Novohrad, Modrý Kamen, Divín, Šomoška, Salgó ant others). From Filakovo beg Hamza and after him other begs (Velidžan (1562-1564), Arslan beg (1564), Mehemed (Mahmat 1575), Hasan (1576), Mahmud (1579), Korkud (1579-1590), Ali (1591) a Jusuf beg (1593)) ruled to the large part of southern Slovakia, called Filakovo sanjak.

Counts Nicholas Pállfy and Christooph Tiefenbach with the royal army liberated the castle and its surroundings in 1593. There was a fire, which ruined a part of the castle and town in 1615. Stephanus Koháry I got the castle from the emperor in 1657. His son, Stephanus Koháry II ordered to rebuild the ruined parts in 1667.

Emerich Thököly, the reigning prince of Transsylvania with the help of the big Turkish army captured the castle in 1682. The castle was destroyed and since this year remained in ruins. Thököly was proclaimed the Hungarian king here. Stephanus Koháry II after his liberation from Thököly's confinement ordered to build the new church and monastery in 1725.
The last owners of the castle were the Coburgs.
From the year 1972 conservation of the castle ruins is realised. The castle is opened for public since 1993.