You are correct this this question is a little outside the usual scope of the Mathematica for Students Support Forum. I do not know them, but I am confident there are better groups for finding numerical algorithms.
The publicly available information on NSolve internals is at
* For sparse linear systems, Solve and NSolve use several efficient numerical methods, mostly based on Gauss factoring with Markowitz products (approximately 250 pages of code).
* For systems of algebraic equations, NSolve computes a numerical Gröbner basis using an efficient monomial ordering, then uses eigensystem methods to extract numerical roots.
The library work to learn about Markowitz products and Gröbner bases is an exercise for the reader.
Breaking open the encoded or compiled files to get at the code is a big no-no.