Any disastrous or amusing upgrade stories?...

Main Page | Report this Page

Computers Forum Index » Computer - Databases - Pick » Any disastrous or amusing upgrade stories?...

Page 1 of 3 Goto page 1, 2, 3 Next

Any disastrous or amusing upgrade stories?...

Author

Message

dawn...

Posted: Mon Oct 12, 2009 1:45 pm

Guest

These stories need not be pick-specific. I'm teaching a couple of
courses this semester and wanted to give a few upgrades-that-went-awry
stories to a class. I am looking specifically for anything related to
the development or run-time software environments, such as those
related to version skew or suffering new bad things in exchange for
the advantages of the upgrade, or simple upgrades that end up with
long outages for a company, or ...

Example of an upgrade pattern I'm calling NEW BUGS OR UNDESIRED SIDE-
EFFECTS. We probably all have tons of stories of this type, so much so
that I don't have many logged in my brain where I can remember them.
I've probably told this one here before. It does not qualify as a
disaster, just a little example of what we have all seen in various
ways with upgrade:

In the late 70's we upgraded the operating system on a Pr1me computer,
at which point the new COBOL compiler failed to handle addition. ADD 1
to COUNTER no longer compiled. We needed other features and/or it
would have been difficult to back off the upgrade, so we moved
forward. Any already-compiled programs were fine, but for everything
we were writing or changing and compiling again, we had to remove all
addition. Fortunately multiplication and subtraction were both working
so we could multiply by -1 and subtract. Maybe a month later we got a
patch from the vendor and changed all programs back.

I have a terrible Novell Upgrade story that classifies in the pattern
of EXTENDED DOWNTIME. There are also CORRUPTED BACKUPS stories when a
restore is in order.

One of my favorite patterns is what I will call CASCADING UPGRADES --
when you have to upgrade in order to correct one problem, and that
causes you to have to upgrade something else, which causes you to
upgrade something else, ... a cascade of upgrades to fix a little
problem, sometimes resulting in HARDWARE UPGRADE REQUIRED.

Then there is the upgrade pattern of IMPOSSIBLE VERSIONS where you
need to upgrade software A in a way that causes another upgrade to
software B, but B requires that you upgrade C to a version
incompatible with software A (or some other similar variation). I know
I have encountered that one, but the details are very fuzzy. More
often the upgrade of C would be incompatible with another piece of
software, D, that is also needed in that environment.

Other patterns might be DOWNGRADE REQUIRED. In order to upgrade or
install once piece of software, you need to downgrade another.

I have not seen a list of such "upgrade patterns" so if you have a
pattern to add to those above, that would be super.

If you have any good stories that can be written in a way that is
short enough to read to a class to illustrate the point that simple-
sounding upgrades sometimes grow into big projects, perhaps holding up
companies for days, please pass them along. If for some reason you do
not want to post a story here, feel free to email me dwolt at tincat-
group dot com. Thanks. --dawn

JJCSR...

Posted: Tue Oct 13, 2009 3:26 pm

Guest

Dawn:

This tale goes back to my pre-PICK days, back when terms like “head
crash” were synonymous with “cardiac arrest”. I had been operating
on a series of IBM System 360’s, having gone from a 360/25 to 360/30,
then to a 360/40. Due to a couple of major application installations
planned (on-line wholesale order entry, and polling of 30-store POS
system), the need for another upgrade to hardware led to replacement
of the IBM 360 system with a 370/138.

At the same time the hardware was placed on order, the implementation
of programming for the order entry system began. I contracted with a
gentleman from Cheney, Wash., to write our COBOL software. When I
told him of our hardware-upgrade, and advised that we would be running
on DOS/VS rel. 34 (which was then a public domain product), my friend,
the programmer, told me he could obtain a version, FREE of charge, and
have it shipped directly to me.

The 370/138 computer was being installed by a 3rd-party maintenance
firm. All pieces of hardware were in place, consuming much of the
18ft. x 26ft. computer room. It was time to fire up the system,
requiring the disk with the operating , the “SYSRES disk”, to be
mounted into one of the two “pizza oven style” disk cabinets (there
were 2 disks per cabinet – each disk accounting for 100MB of storage –
WOW!). I handed the disk to the tech from the maintenance firm, and
he promptly refused, telling me that it was against his company’s
policy for any technician to place a disk into a disk cabinet.

So, I was left with the honor of firing up the 370/138 for the first
time. The disk began to spin, faster and faster, until it reached
the speed where the read/write heads would extract from their
compartments, and reach out between the 10 platters. For those who
may not be aware, read/write heads never come in contact with the
surface of the platters; they float above the platter-surface (at
least, not in NORMAL conditions). But this would be one of those
ABNORMAL conditions.

Unbeknownst to me, the cover of the disk had been impregnated by a
forklift at the airport. When holding the disk to remove the cover,
the damaged portion of the platters were at the rear of the disk – I
never saw that damage. Several of the middle platters had been bent
up to about a 45-degree angle. As the drive reached speed for the
heads to make their move into the space between the platters, the bent
platters grabbed hold of the heads and immediately spewed fragments of
the heads, and the platters, along with oxide dust, completely
covering the window of the cabinet. This calamity was accompanied by
a shrill sound I haven’t heard the likes of, until a few summers ago
when I first heard the cry of a fisher cat; much akin to the sound of
a screaming woman.

This was my first, and only, “head crash” experience. I had heard
about them, but assumed I’d never have to worry about being a part of
one. And, I quickly understood why the technician refused to put the
disk into the cabinet. He was free of responsibility.

The drive was rebuilt at a cost of $4000 (cheap by today’s
standards). We were able to obtain another DOS/VS disk, which the
programmer-friend used to build another SYSRES disk. By some
miracle, or collection of same, we were up and running within 24 hours
of the incident that brought us to our knees.

And one footnote: this upgrade to the 370/138 created a need to rent
the 12’ x 20’ room, adjacent to the 18’ x 26’ computer room, to house
a 20-ton air-conditioner. A hole on the base of the wall between the
rooms was used to feed the cold air into the raised-floor computer
room. The floor beneath the raised-floor was the plenum for the
cold air to rise through strategically-placed, perforated floor-
tiles.

Oh, those were the days.

Jim Cronin
Dir. MIS
Kittery Trading Post

dawn...

Posted: Wed Oct 14, 2009 12:10 am

Guest

Good story. That has cascading upgrades (even of the physical plant
variety) as well as the damaged hardware story. I also like the policy
issue where you had to be the one to load the platters.

It reminded me of other unexpected hardware failures, such as a fire
in either a terminal or an 8088 machine (seems like I would remember
which, but...).

Thanks, Jim. --dawn

On Oct 13, 10:26 am, JJCSR <JCro... at (no spam) ktp.com> wrote:

Quote:

Peter McMurray...

Posted: Wed Oct 14, 2009 4:45 am

Guest

Hi Dawn
Given the Hitachi Sidekick debacle this week do you really need any other
stories Smile

I'll toss in a couple
Never under-estimate the ingenuity of the Company Secretary when they
descend from on high to show mere mortals how it is done. The early
Reality's came with a cartridge tape. The tape was in a thick perspex box
with a one eighth thick steel base and a cut-out for a steel locating pin.
Alwyn got it in upside down and backwards through sheer brute force. The
engineer had to take to the case with a two foot wrecking bar to get it out
.. AWA supplied half inch tape drive as a replacement and Alwyn struck
again. The unit was mounted vertically so a full tape would spill off when
you first loaded it. This did not bother Alwyn who just threaded it onto
the second spool and hit GO. Bang instant mylar boot lace. He did it so
often that the engineers gave him a pair of scissors and a batch of end of
tape stickers to do the repairs himself.
Early DEC gear was quite delicate and large. Unlike Realitys it needed
false floors, air-conditioning and much cossetting. The software engineer
arrived to finish the install after much building work just in time to see
the truckers bringing the heavy system in. Method simple. Drop six foot
high crate off truck onto tyres then roll it end for end down the corridor.
Delivery of replacement was done by alternative truckers.
Original IBM PCs were extremely badly designed electrically with a symmetric
plug for the power supply. I warned eager young son of client "never work
on the innards on your own and always use a static strap and always
disconnect the power". Eager beaver promptly came in to empty office on
Saturday afternoon and was discovered lying half conscious against the wall
with a badly blackened PC on the table. No strap, no power disconnect and
the plug in the wrong way around.
Micromax was a great multi-user PC in the early '80s but ISAM and BTREE
files were not the simplest things to recover after a power failure. I
preached always keep a daily save off site and I refused to supply without a
UPS. So this site had eight girls working flat out taking a 1000 new
customers a week and dealing with 12 trucks doing up to 50 deliveries a day
each, mostly to individual home accounts. The office was on the first floor
(second to Yanks) and the UPS was mounted outside on the wall. The works
foreman went in to get the girls to power off, however his off-sider put a
ladder up on the wall, hopped up and yanked the UPS before the foreman had
chance to get a word out. He discovered that the "ladies" had a very clear
command of some of the more basic terms in the English language. It then
transpired that they had all been too busy to do a regular backup in fact
for 7 weeks as it turned out - I had labelled tapes Moday, Tuesday... Week1
Week 2..... Month 1, Month2 and Month 3 plus I had done a full save myself
when on site a fortnight before. I was 3000 miles away at a client in South
Western Australia so I said use mine. AH! the boss had felt guilty and done
one the week end before. Which tape did he use? Mine! He loaded it and got
a tape error. Much vile language and he fired it into the waste bin fulll
of cigarette ends etc which was duly emptied into the dumpster. This all
happened before they called me again. I told him I could recover tape
errors, not easy but possible, just put the tape in and I will dial up.
That was when I got the full story. I believe the drivers still talk about
the day there was one great rugby forward bottom and two mini skirted
bottoms sticking out of the dumpster for an hour. They found it and heavens
to betsy he had put it back in the box before tossing it so it was oily and
ashy on the outside but OK. I then set about recovering thousands of
accounts over a 300 baud acoustic coupler on a country phone line.
AAAArrrrgghh!
Have Fun and always back up on equipment that you own in a separate
building. Clouds can rain and vanish Smile

Peter McMurray

Ross Ferris...

Posted: Wed Oct 14, 2009 6:40 am

Guest

Under the heading of unexpected hardware failures ...

many moons ago we were planning a migration for a new client that was
to take place on 1st July (first day of the new financial year here).
We had done a few trial data conversion runs & fine tuned everything
and were in count down mode.

The existing hardware was an early "micro" (name escapes me, but it
was the size of a decent mini-computer)

Anyway, on the 29th June I was visiting the site to clear out the old
conversion data in anticipation of the "real" stuff, and to do some
last minute checks for the new Wyse 50 terminals --> I do remember
that the system they were moving to was "R83 pick" running on a top of
the line 20Mhz Compaq 386 style system.

Anyway, whilst I was onsite, the Murphy Switch in the old server got
tripped, something shorted, and a small electrical fire broke out
inside the main system cabinet - anything the fire didn't melt, the
powder from the fire extinguisher took care of!!.... to cut a long
story short, we ended up getting them "live" that night, with data
from the previous conversion that was a week old --> if we hadn't done
the trial conversions, they would have had NOTHING, as the company
that made the old system had disappeared, and if the fire had happened
an hour or so later, one of the jobs I had planned on doing was to
clear the data files out ready to do the "real" conversion!

These days we take backups of trial conversion data 'cause you just
never know!

Art Martz...

Posted: Wed Oct 14, 2009 8:42 pm

Guest

Ross Ferris wrote:

Quote:

Under the heading of unexpected hardware failures ...

Brings to mind a Microdata hard disk drive upgrade. What the FE failed

to take into account was the power in this business park was 240V SINGLE
phase (from the olden days), and he proceeded to hook up two phases
like he always did, thinking each phase was 110v. When he flipped the
power switch, and 480V went thru the power supply, the resulting flash
was spectacular, to say the least! The FE was looking right at it at the
time, and it took a few minutes before the stars stopped swimming about
and he could see again! One new power supply on order shortly there-after.
Art

JJCSR...

Posted: Wed Oct 14, 2009 9:47 pm

Guest

Dawn:

Coming off of General Automation, back in the early 90's, and
converting to Advanced Pick, I stumbled onto two particular software
"gotcha's", and they both pertained to nuances between some of GA's
BASIC commands, vs. PICK's.

The first one I discovered was that "soundex" was not the same. In
G.A., you could create a soundex table using virtually any number of
characters, where A/P used the first letter, and up to the third
consonant in the string being "soundexed". E.G., consider the words
MADISON and MADISONVILLE. In the case of these two words just shown,
MDSN (ignore vowels) would be used to generate the soundex code -
M325. However, G.A. used all consonants (I believe double-consonants
were considered as one, I.E., "LL" = "L'). G.A.'s soundex codes for
these two words are M325 and M32514, respectively.

I discovered this problem when the first attempt was made to do
soundex-lookup on names of vendors, customers, products - we had
generated them all under G.A.'s PICK Basic, and tried to carry over
the logic to Advanced PICK. Fortunately, G.A. had the source code
for "soundex" on the "BP" file that was distributed with the system,
so I was able to copy it into a subroutine that I named,
"soundx.subrout", using two arguments (ARG1, ARG2). ARG1 was the
string being soundexed; AGR2 was the code generated by the routine.

Once I had the subroutine operational, I had to find every program
that utilized "soundex", either in the form of creating the soundex
code, or using the code to match to the lookup-string entered by the
operator.

Secondly, the BASIC command, "convert", also presented major
differences, mostly in the area of converting "multi-character"
portions of a string to single-character, or other multi-character
portions of a string. Without getting too specific, suffice to say,
another home-grown "subroutine" was needed to replace the short-
comings of Advance PICK's "convert".

These, indeed, were "downgrade" scenarios that grew from "upgrade".

Jim Cronin
Kittery Trading Post

Peter McMurray...

Posted: Thu Oct 15, 2009 2:01 am

Guest

Hi Dawn
A previous post about fire in a printer jogged the memory of how dumb some
people on help desks can be.
ICL designed a wonderful COLOUR PC that blew the minds of executives used to
sending their requests to the typing pool. Ampol bought a significant
number of them for use in petroleum distributorships around Australia.
These magnificent beasts had been designed for use in the dales of Yorkshire
however they went to places like Alice Springs and Dubbo where people
actually plan to take their holidays in a place where they hope it will
rain. My client was sitting at her desk working on our Pick AP software,
looking out of the window gazing at a 50,000 litre petrol tank and dreaming
of diving in the pool within an hour to esacpe the 45C heat. She suddenly
realised that there was smoke rising from the old ICL gear nearby. She
promptly leapt into action and ripped the plug out of the wall with some
hallooing. Ampol help desk, when called, complained bitterly "didn't you
shut down properly first" Smile

Then of course there is the RTFM only if all else fails type of operator.
National Mutual in Melbourne is built on a quite steep hill. A big IBM shop
with the computer room on the first floor in those days. IBM provided a
massive printer that took those large rolls of paper that you load with a
forklift for test. Operator put paper in and threaded the way they thought
it should go. Press load button. Oh Bother! wrong path. Full roll leaped
out of its cradle and flew across the floor. Unfortunately the building was
a high rise using a steel frame design so the wall was just a block filler
and no match for an angry quarter ton of paper. Well we always needed a
door there and it didn't hit too many cars on its way down to Flinders
Street station.
Peter McMurray

wjhonson...

Posted: Wed Oct 28, 2009 11:07 pm

Guest

My very first job was on the original Reality, which was still running
in 1983 (believe it or don't!)

I was just the night-time computer operator, but I had access to TCL
obviously because I had to run queries and such at night (you can't
run them during the day because it slows the system too much).

At any rate, I discovered that "T" was a shortcut for "Time" and I
thought I'd check out what other shortcuts might exist. So I tried A,
B, C, and so on to see what they did at TCL.

Previously, a tech had setup some kind of shortcut which would "wipe
the disk" (that's what they said) in preparation for a new install.
The shortcut was "\" and of course I found it at midnight or some such
thing, and it just threw up a comment like "Working..." or whatever.
No way to abort it and it was time to go home anyway.

The next day I got a call early to ask "Did anything odd happen last
night?"

So the moral is, don't create shortcuts that do amusing things, like
reformatting the disk.

Will Johnson

dawn...

Posted: Thu Oct 29, 2009 12:23 pm

Guest

On Oct 28, 6:07 pm, wjhonson <wjhon... at (no spam) aol.com> wrote:

Quote:

Hi Will, long time no see (or are you a different Will Johnson?)
Great story! --dawn

Kevin Powick...

Posted: Thu Oct 29, 2009 1:38 pm

Guest

On Oct 28, 7:07 pm, wjhonson <wjhon... at (no spam) aol.com> wrote:

Quote:

Previously, a tech had setup some kind of shortcut which would "wipe
the disk" (that's what they said) in preparation for a new install.
The shortcut was "\"

I worked on a system where somebody did that with the "D" key. Every
production file was cleared and re-loaded with a base set of data.
Not good.

--
Kevin Powick

frosty...

Posted: Thu Oct 29, 2009 7:51 pm

Guest

Quote:

On Oct 28, 7:07 pm, wjhonson <wjhon... at (no spam) aol.com> wrote:

Previously, a tech had setup some kind of shortcut which would "wipe
the disk" (that's what they said) in preparation for a new install.
The shortcut was "\"

Kevin Powick wrote:
I worked on a system where somebody did that with the "D" key. Every
production file was cleared and re-loaded with a base set of data.
Not good.

<Eric Idle Voice>You were lucky! On our system, any one key command
would erase the entire hard drive and set the computer room on fire!</EIV>

--
frosty

Tony Gravagno...

Posted: Thu Oct 29, 2009 8:15 pm

Guest

"frosty" wrote:

Quote:

Eric Idle Voice>You were lucky! On our system, any one key command
would erase the entire hard drive and set the computer room on fire!</EIV

<Graham Chapman>That's nothing! Our system would set us on fire if we
just thought about it. Then the system admin would feed our barbecued
remains to lions who would regurgitate us as full-time developers so
that we could be ridiculed for the rest of our days. Oh yes, and then
the hard drives would be wiped as well.</GC>

dawn...

Posted: Thu Oct 29, 2009 9:15 pm

Guest

On Oct 29, 10:51 am, "frosty" <fros... at (no spam) bogus.tld> wrote:

Quote:

LOL. Thanks. --dawn

Quote:

--
frosty

Peter McMurray...

Posted: Sat Oct 31, 2009 3:15 am

Guest

On A more Reliable note for those that remember punched cards and assembler.
A programmer at a large steel mill in Wales spent six months writing his
program and tracing a fault. Every time he ran the program it kicked off
splendidly then the machine halted (more sad memories of Open Architecture)
One day a great cheer rang around the programmers room he had found it. The
procedure he had to follow in the end was to take the deck of cards and
laboriously align every one with a ruler to check the code as punched.
Bingo! one miss punch produced a command that only the system engineers knew
about ; System Halt
Peter McMurray
"frosty" <frostyj at (no spam) bogus.tld> wrote in message
news:qtednWg3kvGMJ3TXnZ2dnUVZ_uSdnZ2d at (no spam) centurytel.net...

Quote:

Page 1 of 3 Goto page 1, 2, 3 Next

All times are GMT The time now is Fri Dec 04, 2009 1:17 pm Computers Forum Index