[Date Index] [Thread Index] [Author Index]

strange rounding result

• To: mathgroup at smc.vnet.net
• Subject: [mg80918] strange rounding result
• From: "Chris Chiasson" <chris at chiasson.name>
• Date: Wed, 5 Sep 2007 02:59:41 -0400 (EDT)

Round[9995,10]
Round[999.5,1]
Round[99.95,1/10]
Round[9.995,1/100]
Round[.9995,1/1000]

The results of these commands are decreasing powers of ten, except the
second to last one, which gives 999/100 on my machine. I assume it has
something to do with the storage of floating point numbers in binary
rather than decimal form, but I don't know. Anyway, it just seemed
strange to me and I wanted to point it out.

Maybe it has something to do with the "round to even" behavior? Meh.

-- 
http://chris.chiasson.name/

• Follow-Ups:
  • Re: strange rounding result
    • From: Curtis Osterhoudt <cfo@lanl.gov>
  • Re: strange rounding result
    • From: Syd Geraghty <sydgeraghty@mac.com>
  • Re: strange rounding result
    • From: Carl Woll <carlw@wolfram.com>