So I finally got the Mie scattering calculations to work even for large k, and size parameter. I used formalism which did not involve a derivative expression in the calculation of the coefficients (a, b).
for ex, instead of using f'[x], where f[x_]:=some Bessel functions[x], I actually symbolically derived what f' is and put that in the expression to compute a and b coeff. (ie, no more ' prime expression)
That seemed to do the trick...now, what is is about how mathemtica treats derivative evaluation that could have made my computations unstable with them in it vs without them?
[This appears to be related to http://forums.wolfram.com/student-support/topics/23626. - Forum Moderator]