strange rounding result
- To: mathgroup at smc.vnet.net
- Subject: [mg80918] strange rounding result
- From: "Chris Chiasson" <chris at chiasson.name>
- Date: Wed, 5 Sep 2007 02:59:41 -0400 (EDT)
Round[9995,10] Round[999.5,1] Round[99.95,1/10] Round[9.995,1/100] Round[.9995,1/1000] The results of these commands are decreasing powers of ten, except the second to last one, which gives 999/100 on my machine. I assume it has something to do with the storage of floating point numbers in binary rather than decimal form, but I don't know. Anyway, it just seemed strange to me and I wanted to point it out. Maybe it has something to do with the "round to even" behavior? Meh. -- http://chris.chiasson.name/
- Follow-Ups:
- Re: strange rounding result
- From: Curtis Osterhoudt <cfo@lanl.gov>
- Re: strange rounding result
- From: Syd Geraghty <sydgeraghty@mac.com>
- Re: strange rounding result
- From: Carl Woll <carlw@wolfram.com>
- Re: strange rounding result