To reduce memory use in the kernel:
* Use a setting such as $HistoryLength=1 to limit the number of
input and output lines stored. (Usually the kernel saves all inputs and
results, in case they are wanted again later. If you keep
the intermediate results you want to save in variables, you don't
need to store the In and Out data.)
* Use Remove [vars] to remove unneeded variables.
* Use the command Share periodically.
You can try smaller versions of your code and see how MaxMemoryUsed grows with the size of the problem.
If your code is plain text, please feel free to include it in a Comment post. Plain or simple, a notebook (.nb) file with it can be attached. That would help diagnosis a lot.