 |
|
| Computers Forum Index » Computer - Databases - Sybase » Improper rounding of float value by using round... |
|
Page 1 of 1 |
|
| Author |
Message |
| Subind... |
 Posted: Thu Jun 04, 2009 12:32 pm |
|
|
|
Guest
|
Hi,
I'm facing issue with rounding of any float value. These are the two
ways I tried, both gave different results.
The method 1 doesn't do the rounding. why is that so?
Can anybody please throw some light on this issue?
Method 1
------------------------
declare at (no spam) b float
select at (no spam) b=2.135
select round( at (no spam) b,2)
Result:2.13
Method 2
------------------------
select round(2.135,2)
Result:2.14
------------------------ |
|
|
| Back to top |
|
|
|
| David Kerber... |
| Posted: Thu Jun 04, 2009 5:16 pm |
|
|
|
Guest
|
In article <e3f37b75-fa66-422e-b582-b847ba9f13c8 at (no spam)
21g2000vbk.googlegroups.com>, subind123 at (no spam) gmail.com says...
Quote: Hi,
I'm facing issue with rounding of any float value. These are the two
ways I tried, both gave different results.
The method 1 doesn't do the rounding. why is that so?
Can anybody please throw some light on this issue?
Method 1
------------------------
declare at (no spam) b float
select at (no spam) b=2.135
select round( at (no spam) b,2)
Result:2.13
Method 2
------------------------
select round(2.135,2)
Result:2.14
------------------------
Most likely, when 2.135 is stored into variable b, it can't be exactly
represented in binary floating point, so its internal representation is
equal to something like 2.134999999999999, which of course rounds to
2.13. If you declare b as a DECIMAL or equivalent type, it will
probably work.
In the second case, 2.135 is probably representated as a Decimal (or
other exact) data type, and therefore rounds correctly.
--
/~\ The ASCII
\ / Ribbon Campaign
X Against HTML
/ \ Email!
Remove the ns_ from if replying by e-mail (but keep posts in the
newsgroups if possible). |
|
|
| Back to top |
|
|
|
|
|
All times are GMT
The time now is Wed Nov 25, 2009 3:36 am
|
|