It might be that you could do this:
oldHistory = $HistoryLength
$HistoryLength = 0
$HistoryLength = oldHistory
which removes all the stored history for the notebook and would clear up memory that way.
Or maybe you could do:
Out[(*whatever output the memory-hogging one is*)] =.
which clears the specified line (as accessed by the character %)