Main Page | Report this Page
Computers Forum Index  »  Computer - Databases - Informix  »  unload creating duplicate versions of edited records...
Page 1 of 1    

unload creating duplicate versions of edited records...

Author Message
Bones...
Posted: Tue Oct 20, 2009 12:51 am
Guest
Hi,

We have a 4GL program that extracts data from a table using an
unload. If it encounters a record that is been edited it then creates
2 versions of this record, one pre-edit and one post edit. We don't
need a particular version of the record, as long as we only get one
version.

I have never heard nor seen this behaviour before and am at a loss as
to what causes it and how to solve the problem.

Thanks for any suggestions.

We are using 4GL 7.31.FC1, Informix Dynamic Server 7.31.UD3 on Solaris
8.
 
Bones...
Posted: Tue Oct 20, 2009 2:31 am
Guest
On Oct 20, 1:00 pm, Andrew Clarke <acla... at (no spam) civica.com.au> wrote:
Quote:
Hi,

We have a 4GL program that extracts data from a table using an
unload.  If it encounters a record that is been edited it then creates
2 versions of this record, one pre-edit and one post edit.  We don't
need a particular version of the record, as long as we only get one
version.

I have never heard nor seen this behaviour before and am at a loss as
to what causes it and how to solve the problem.

The program has probably set DIRTY READ ISOLATION.  You want COMMITTED READ
or CURSOR STABILITY isolation and to SET LOCK MODE TO WAIT <nseconds> to
avoid getting lockout errors from brief transient locks.

Art

This would presuppose that the row is being rewritten to a different position
in the table for it to be picked up again, and that implies in-place alter has
happened to the table, and consequently a row rewrite is taking place?

The updates that happen to the table are straight updates. As far as
I understand it, there is no re-write. The code in the program where
the proble is happening is:
UNLOAD TO m_unl_file_name
SELECT zzdb040.* FROM zzdb040
WHERE z040_shno IN
(SELECT i710_shno from irdb710
WHERE i710_coy = m_coy)
UNION
SELECT zzdb040.* FROM zzdb040
WHERE z040_shno IN (SELECT i060_shno FROM irdb060)

Hope this helps a bit.
 
Bones...
Posted: Tue Oct 20, 2009 3:43 am
Guest
On Oct 20, 2:31 pm, Jonathan Leffler <jleff... at (no spam) earthlink.net> wrote:
Quote:
Bones wrote:
On Oct 20, 1:00 pm, Andrew Clarke <acla... at (no spam) civica.com.au> wrote:
Hi,
We have a 4GL program that extracts data from a table using an
unload.  If it encounters a record that is been edited it then creates
2 versions of this record, one pre-edit and one post edit.  We don't
need a particular version of the record, as long as we only get one
version.
I have never heard nor seen this behaviour before and am at a loss as
to what causes it and how to solve the problem.
The program has probably set DIRTY READ ISOLATION.  You want COMMITTED READ
or CURSOR STABILITY isolation and to SET LOCK MODE TO WAIT <nseconds> to
avoid getting lockout errors from brief transient locks.
Art
This would presuppose that the row is being rewritten to a different position
in the table for it to be picked up again, and that implies in-place alter has
happened to the table, and consequently a row rewrite is taking place?

The updates that happen to the table are straight updates.  As far as
I understand it, there is no re-write.  The code in the program where
the proble is happening is:
UNLOAD TO m_unl_file_name
    SELECT zzdb040.* FROM zzdb040
        WHERE z040_shno IN
            (SELECT i710_shno from irdb710
            WHERE i710_coy  = m_coy)
    UNION
    SELECT zzdb040.* FROM zzdb040
        WHERE z040_shno IN (SELECT i060_shno FROM irdb060)

Do any of the irdb710.i710_shno values also appear in irdb060.i060_shno?
  Although UNION eliminates duplicate rows, if the row has changed while
the UNLOAD is occurring, then it might explain why you see two different
values.

Have you considered locking the zzdb040 table in SHARE mode to prevent
others from updating it while the UNLOAD proceeds?  Or, as others have
suggested, use a more stringent isolation level - I think repeatable
read is likely to be the most reliable choice, but it will also apply a
lot of locks.

-=JL=-

No, none of the i710_shno values also appear in irdb060.i060_shno, so
this is not the problem.

I had thought of SHARE mode, unfortunately the table does need to be
available for update while been unloaded. I have set the isolation
level to "COMMITTED READ", assuming the level it currently it is set
to is "ISOLATION" read. I did think that "COMMITTED READ" was the
default and we have no code changing the the ISOLATION level, but from
the discussion it seems likely that this may be the problem.

Thanks.
 
Bones...
Posted: Tue Oct 20, 2009 4:15 am
Guest
On Oct 20, 3:04 pm, Andrew Clarke <acla... at (no spam) civica.com.au> wrote:
Quote:
Not necessarily, it would also happen if the query were using an index and
that index key was being modified by some users so that the row shows up
again as the unload follows the btree.

hold the phone; a plain unload doesn't follow indexes; do you have a WHERE
clause on the unload?

Yes,


There is a "WHERE". See above where I posted the unload.
 
Andrew Clarke...
Posted: Tue Oct 20, 2009 5:15 am
Guest
Quote:
Not necessarily, it would also happen if the query were using an index and
that index key was being modified by some users so that the row shows up
again as the unload follows the btree.


Oh yeah, that. Which implies COMMITTED READ or CURSOR STABILITY may not be
enough...

Really, the safest way is to unload on a quiet db. REPEATABLE READ isn't much
short of LOCK TABLE, and you can guess the effect of that on other people.

At least LOCK TABLE will guarantee a consistent unload. With REPEATABLE READ,
you can toss a coin as to whether your unload process wins or loses in a
conflict with one other process. If there are many updaters out there, the
chances of everyone suffering rises accordingly.

Since you are on a 7.3 engine, the following won't help you much; I believe
the 11 engines have a mode where you can pretend you are not seeing any
updates, similar to Oracle's slant on this. However I'm not familiar with this
mode. Probably someone else can comment.
 
Jonathan Leffler...
Posted: Tue Oct 20, 2009 5:15 am
Guest
Bones wrote:
Quote:
On Oct 20, 1:00 pm, Andrew Clarke <acla... at (no spam) civica.com.au> wrote:
Hi,
We have a 4GL program that extracts data from a table using an
unload. If it encounters a record that is been edited it then creates
2 versions of this record, one pre-edit and one post edit. We don't
need a particular version of the record, as long as we only get one
version.
I have never heard nor seen this behaviour before and am at a loss as
to what causes it and how to solve the problem.
The program has probably set DIRTY READ ISOLATION. You want COMMITTED READ
or CURSOR STABILITY isolation and to SET LOCK MODE TO WAIT <nseconds> to
avoid getting lockout errors from brief transient locks.
Art
This would presuppose that the row is being rewritten to a different position
in the table for it to be picked up again, and that implies in-place alter has
happened to the table, and consequently a row rewrite is taking place?

The updates that happen to the table are straight updates. As far as
I understand it, there is no re-write. The code in the program where
the proble is happening is:
UNLOAD TO m_unl_file_name
SELECT zzdb040.* FROM zzdb040
WHERE z040_shno IN
(SELECT i710_shno from irdb710
WHERE i710_coy = m_coy)
UNION
SELECT zzdb040.* FROM zzdb040
WHERE z040_shno IN (SELECT i060_shno FROM irdb060)


Do any of the irdb710.i710_shno values also appear in irdb060.i060_shno?
Although UNION eliminates duplicate rows, if the row has changed while
the UNLOAD is occurring, then it might explain why you see two different
values.

Have you considered locking the zzdb040 table in SHARE mode to prevent
others from updating it while the UNLOAD proceeds? Or, as others have
suggested, use a more stringent isolation level - I think repeatable
read is likely to be the most reliable choice, but it will also apply a
lot of locks.


-=JL=-
 
Andrew Clarke...
Posted: Tue Oct 20, 2009 5:15 am
Guest
Quote:
On Oct 20, 3:04?pm, Andrew Clarke <acla... at (no spam) civica.com.au> wrote:
Not necessarily, it would also happen if the query were using an index
and that index key was being modified by some users so that the row
shows up again as the unload follows the btree.

hold the phone; a plain unload doesn't follow indexes; do you have a
WHERE clause on the unload?

Yes,

There is a "WHERE". See above where I posted the unload.

oh yeah, there it is. So row drift in the index is a good candidate. This

doesn't matter too much though, the mechanism of the cause won't affect the
solution.
 
Art Kagel...
Posted: Tue Oct 20, 2009 5:15 am
Guest
The program has probably set DIRTY READ ISOLATION. You want COMMITTED READ
or CURSOR STABILITY isolation and to SET LOCK MODE TO WAIT <nseconds> to
avoid getting lockout errors from brief transient locks.

Art

Art S. Kagel
Oninit (www.oninit.com)
IIUG Board of Directors (art at (no spam) iiug.org)

Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Oninit, the IIUG, nor any other organization
with which I am associated either explicitly or implicitly. Neither do
those opinions reflect those of other individuals affiliated with any entity
with which I am affiliated nor those of the entities themselves.



On Mon, Oct 19, 2009 at 8:51 PM, Bones <wayne at (no spam) barebonesphoto.com> wrote:

Quote:
Hi,

We have a 4GL program that extracts data from a table using an
unload. If it encounters a record that is been edited it then creates
2 versions of this record, one pre-edit and one post edit. We don't
need a particular version of the record, as long as we only get one
version.

I have never heard nor seen this behaviour before and am at a loss as
to what causes it and how to solve the problem.

Thanks for any suggestions.

We are using 4GL 7.31.FC1, Informix Dynamic Server 7.31.UD3 on Solaris
8.
_______________________________________________
Informix-list mailing list
Informix-list at (no spam) iiug.org
http://www.iiug.org/mailman/listinfo/informix-list
 
Art Kagel...
Posted: Tue Oct 20, 2009 5:15 am
Guest
Not necessarily, it would also happen if the query were using an index and
that index key was being modified by some users so that the row shows up
again as the unload follows the btree.

Art

Art S. Kagel
Oninit (www.oninit.com)
IIUG Board of Directors (art at (no spam) iiug.org)

Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Oninit, the IIUG, nor any other organization
with which I am associated either explicitly or implicitly. Neither do
those opinions reflect those of other individuals affiliated with any entity
with which I am affiliated nor those of the entities themselves.



On Mon, Oct 19, 2009 at 10:00 PM, Andrew Clarke <aclarke at (no spam) civica.com.au>wrote:

Quote:
Hi,

We have a 4GL program that extracts data from a table using an
unload. If it encounters a record that is been edited it then creates
2 versions of this record, one pre-edit and one post edit. We don't
need a particular version of the record, as long as we only get one
version.

I have never heard nor seen this behaviour before and am at a loss as
to what causes it and how to solve the problem.


The program has probably set DIRTY READ ISOLATION. You want COMMITTED
READ
or CURSOR STABILITY isolation and to SET LOCK MODE TO WAIT <nseconds> to
avoid getting lockout errors from brief transient locks.

Art

This would presuppose that the row is being rewritten to a different
position
in the table for it to be picked up again, and that implies in-place alter
has
happened to the table, and consequently a row rewrite is taking place?



 
Andrew Clarke...
Posted: Tue Oct 20, 2009 5:15 am
Guest
Quote:
Not necessarily, it would also happen if the query were using an index and
that index key was being modified by some users so that the row shows up
again as the unload follows the btree.


hold the phone; a plain unload doesn't follow indexes; do you have a WHERE
clause on the unload?

Regardless of whether the rows are drifting in the index or "in-place alter"
effects are kicking in, all that's said so far is relevant.

PS - when I say "in-place alter" I mean an effect where .... well, the easiest
thing to do would be to read about it in the "Altering a table definition"
section of the "Performance guide for Informix Dynamic Server". It's what can
happen when you issue an ALTER TABLE command.
 
Andrew Clarke...
Posted: Tue Oct 20, 2009 5:15 am
Guest
Quote:
Hi,

We have a 4GL program that extracts data from a table using an
unload. If it encounters a record that is been edited it then creates
2 versions of this record, one pre-edit and one post edit. We don't
need a particular version of the record, as long as we only get one
version.

I have never heard nor seen this behaviour before and am at a loss as
to what causes it and how to solve the problem.


The program has probably set DIRTY READ ISOLATION. You want COMMITTED READ
or CURSOR STABILITY isolation and to SET LOCK MODE TO WAIT <nseconds> to
avoid getting lockout errors from brief transient locks.

Art

This would presuppose that the row is being rewritten to a different position
in the table for it to be picked up again, and that implies in-place alter has
happened to the table, and consequently a row rewrite is taking place?
 
Andrew Clarke...
Posted: Tue Oct 20, 2009 9:33 am
Guest
Approximately how many rows are there in the zzdb040 table, and approximately
how many rows pop out of the UNION?

I can think of a work-around where you firstly select all the rows into a temp
table under REPEATABLE READ or LOCK TABLE and then apply the SELECT to the
temp table. The idea is to do the quickest possible burst of locking and then
release your locks so you have a clean snapshot, but timing depends on how big
the table is. The method would rely on your SELECT INTO TEMP completing before
any other process's LOCK timeout fires.

Another trick might be to

1/ create a temp table to receive the rows; simplest way is

SELECT zzdb040.* FROM zzdb040 where 1=0 into temp thingy with no log
(check my syntax is correct)

2/ put a unique index on the temp table which is unique according to the
criteria you know identifies a particular row

3/ select the source rows in a FOREACH loop and insert into the temp, being
careful to ignore duplicate violations.

If you feel fancy and want to have the latest value of a row, you could detect
the duplicate violation and issue an UPDATE on the temp table row instead.

4/ finally unload from the temp table.
 
Fernando Nunes...
Posted: Wed Oct 21, 2009 1:50 am
Guest
Andrew Clarke wrote:
Quote:
Not necessarily, it would also happen if the query were using an index and
that index key was being modified by some users so that the row shows up
again as the unload follows the btree.


Oh yeah, that. Which implies COMMITTED READ or CURSOR STABILITY may not be
enough...

Really, the safest way is to unload on a quiet db. REPEATABLE READ isn't much
short of LOCK TABLE, and you can guess the effect of that on other people.

At least LOCK TABLE will guarantee a consistent unload. With REPEATABLE READ,
you can toss a coin as to whether your unload process wins or loses in a
conflict with one other process. If there are many updaters out there, the
chances of everyone suffering rises accordingly.

Since you are on a 7.3 engine, the following won't help you much; I believe
the 11 engines have a mode where you can pretend you are not seeing any
updates, similar to Oracle's slant on this. However I'm not familiar with this
mode. Probably someone else can comment.




That is COMMITTED READ LAST COMMITTED and it would probably not help here.
This is a known situation. It happens because we're browsing through
live data and not looking at a snapshot of data. Possible solutions:

1- shared lock on the table
2- use REPEATABLE READ (probably has the same disadvantages as above,
and it will consume more locks)
3- Access the table using an index (check notes)
4- run the query on an RSS server with delayed apply (not possible with V7)
5- Put a timestamp on the records that save the last update timestamp.
And include that on your WHERE condition
6- Avoid concurrent deletes/inserts (check notes)
7- Do the select into temp table and eliminate the duplicates (you need
to define the criteria for elimination... check 3) and 5) )

Notes:
.. Unless you have a table with pending in-place alters (and even so I
doubt it), you're probably having concurrent delete/insert of records.
This situation would be difficult, if not impossible, to reproduce with
an update. In this scenario, if you're able to control delete/insert
while doing the unload you should be fine...

Regards.
 
 
Page 1 of 1    
All times are GMT
The time now is Tue Dec 01, 2009 3:52 am