It was built in a strategic location in the middle of the Netherlands, where the Maas and Waal rivers come together (just west of current day villages Poederoijen and Brakel, in the municipality of Zaltbommel, in Gelderland). At first it was a simple square brick building, used to charge toll from trading vessels using the rivers. In the 16th century (around 1575, orders given by William the Silent) it was expanded to a larger fortress surrounded by earthen fortifications with two (later three) stone bastions on the northern side, two moats, an arsenal, and housing for a commander and soldiers.
It changed hands twice between the Northern Dutch and the Spanish (December 9, 1570 it was taken by the Geuzen, ten days later Spanish again, and from June 25, 1572 Dutch till this day), the warring parties of the day. The castle soon (from 1619) became a prison for political prisoners. One famous inmate was the eminent lawyer, poet and politician Hugo de Groot (Hugo Grotius) often presented as the "father of modern international law", who was serving a controversially imposed life sentence from 1619. In 1621 Hugo de Groot managed to pull off a daring escape in a book chest. The idea for this escape came from his wife Maria van Reigersberg (also living in the castle). He subsequently became the Swedish Ambassador to France for 10 years. Another high profile inmate was the English Vice-Admiral George Ayscue.