Mathematica 9 is now available
Student Support Forum
-----
Student Support Forum: 'Rounding errors?' topicStudent Support Forum > General > Archives > "Rounding errors?"

Next Comment >Help | Reply To Topic
Author Comment/Response
Rick Bischoff
02/11/02 06:42am

Hi,

I was evaluating

(256^3 - 1)
-----------
256^2

Mathematica gives the correct fraction for this:

16777215
--------
65536

But, when I issue the N[%] command to get the decimal approximation, it incorrectly returns

256

Instead of 255.99....

I tried N[%, 10] thru N[%,20] and finally got it to show the right answer at 20... Is there any to set the default precision on N[%] operations (and other functions that return real numbers, etc) to 20+ so I don't get confused anymore??

Thanks,
Rick


URL: ,

Subject (listing for 'Rounding errors?')
Author Date Posted
Rounding errors? Rick Bischoff 02/11/02 06:42am
Default DaveP 02/14/02 4:36pm
Next Comment >Help | Reply To Topic