Just to the west of St. Illtyd's Church stands a large, steep-sided, flat-topped mound. This is a well preserved castle motte. This probably dates to the 11th or 12th century, though it may incorporate an earlier, prehistoric mound.
Originally the motte would have been surmounted by a defensive wooden tower and palisade and may have been surrounded by an enclosed outer bailey that would have contained a hall, stables and other associated out-buildings. It is possible that the bailey once extended as far as the site of the stone castle on the other side of the churchyard. It is not known whether the motte was built by Norman invaders or by the native Welsh princes of Gwynllwg (the Welsh kingdom of which Llanhilleth formed a part) to defend their northern territories.