Day 3 - The first day of the show

I get in nice and early because, as usual, I'm expecting to find the
network lying in a smoking ruin at the bottom of a dark pit in the middle
of the NOC. Some people might say that I was being overly paranoid, but
I still can't believe that is has all gone so smoothly. Unfortunately my
stress is unwarranted and we are still forwarding packets to the outside
world. So we kick back, wait for the show to open and monitor the radio
for any trouble.

This is fortunate because I happen to overhear Show Management trying
to deal with a 'situation' and still be polite about it over the airwaves.

"...Um there seems to be someone at the front entrance who
is preaching to the attendees"

"Could you repeat that ?"

"Well, apparently there is a Guru at the front, you should
probably call security, they are, well, I'm not sure exactly, but
the report is that they are preaching or something..."

I'm sure what they really wanted to say was "There is a religious nutter
out here wearing robes and a dead animal performing all manner of street
theater and you had better get the rubber van before he hurts someone".

James decided that this was probably as good a reason as any to try
and gather some photo documentation of the days proceedings (Actually ,
i think what he really said was "Photos of booth babes", but we'll give
him his moment of denial) and ran off into the throng.

Moment of irony would have to be that the one booth that was drawing the
largest crowd, was asked to stop what they were doing by show management
because their Penthouse Pet booth babes were not allowed to handout
flyers on the show floor. There were many disappointed geeks when that
news got around.

The day thus became a quiet one. We watched the network, watched the
floor, waited for the show to close and crashed the Worldcom shmooze
and booze hour. People were met, cards were exchanged, hands were shaken
and then we realised that Extreme were going to buy us all dinner so a
restaurant booking was quickly made.

This was when things turned ugly. I'm not sure who was captain of the
SS Extreme that evening, but I'm do recall all the bills went to some
guy called BJ. He had the good taste to invite us all to the Sydney W
hotel for dinner and more drinking but beyond that it was all a bit of
a blur. The trouble is that since so much business in Australia uses
beer as a valid currency (you may laugh, but I refer the skeptics to
do their homework and look up something called the 'Rum Rebellion'),
sales and marketing folks are thus a heavy drinking bunch and any major
tradeshow experience is an exercise in liver collapse.

I vaguely recall a nice Japanese restaurant, a lot of beer and wine,
a table that kept growing ("Another table your highness, I have more
clients on the way...") and then at some point we all poured ourselves
into the lobby bar and kept falling off the funky furniture (all cleverly
designed, I'm sure, to keep the beautiful people on their toes). Bernard
eventually gave up even trying to look comfortable on the weird mushroom
shaped seat he was left to deal with, and just started dancing with
some very scary looking locals. It was very amusing to watch, and we
have the photos to prove it.

Sometime later, after drinking some very nice cognac, it was time to go
home, it was way past Beer O'Clock.

Day 2 - The Move in Begins

I am as usual, probably due to residual jetlag, the first person in the
morning. I foolishly expected to find the NOC locked, but instead the door
was wide open and there were a couple of contractors helping themselves to
the contents of the water cooler. It has apparently transpired that this
is precisely the reason the door was open in the first place and mostly
due to the fact that on leaving last night the security guard requested
that we leave things in such a way as to allow him to have a drink.

Now although I'm never one to give a security individual a hard time,
over here I'm even more inclined than usual to help them out. The security
here is much more relaxed. In the USA they'd usually shoot you before
they asked you what you were doing (Except for the fact that the average
security guard is, for various personal legal reasons that I don't really
want to think about but they tell me anyway, no longer allowed to own a
gun). Over here The guards stop you from doing things not because you
are breaking the rules, but because you have inconvenienced them by
making them go to the effort of stopping you.

Bernard was stopped from riding his bike in the center because the guard
"...Just got hassled by his boss over the radio to go and stop him
(they've got cameras everywhere)". Later we get bailed up for going
through the wrong door "Er, please don't do that again" The guard says
"Because now I have to go over to you to tell you to not do that again..."

It's not so much the security that stops you as the guilt trip they lay
on you afterwards.

I'm also in early because I eagerly await the arrival of the helpdesk
system by our sponsor 'Commander'. The system seems to be shrouded in
secrecy since it hasn't as yet managed to make it onto the show floor.
I have to assume that is is locked in a lab somewhere going through some
vigorous beta testing. Finally, around 9am one of the commander team drops
by to tell me that "There system is on its way, but it has been delayed,
Glenn is still building it". Ah the joys of the temporary life we live.

The actual presence of the helpdesk system, when it does finally arrive,
is in many ways moot. Not because the system doesn't work, because it
did, and it tracked all our tickets and who had them out at the time,
and who was going to resolve the problem and who was their first born etc
etc. However since most of the tickets related to people who were just
too damn lazy to configure their PC's, we don't really have any data
that possibly suggests our network is, in any way, disfunctional. The
damn thing just keeps working. Even with Sanjeev and Paul rewriting the
router configs every 10 minutes, we're still forwarding frames and people
are none the wiser.

Moment of irony would have to be that the most difficult ticket we had
to work on was to get the ticketing system actually connected to the
correct network.

This does lead us nicely into to today's lesson in swearing, which
comes to us via an a random ex TAC engineer, who mostly described his
customers as "Complete Fuckwits". You should be warned that as much as
Australian Society tends to breed familiarity through contempt, this
is not a way you would ever describe a friend. If you use these words,
you had better be right, and prepared for a fight.

Although I'm convinced that someone has made a pact with the devil for
all this success, I'm reasonably sure that it isn't going to be me so I
decide that we all need to go out for a long lunch. The NOC never had
phones to begin with (we all used our mobiles) so our slightly remote
absence isn't going to affect things much.

As we're being seated Bernard points out that for some strange reason
the noodle bar we're in is offering free neck and shoulder massages to
its patrons. I can only imagine that this because at some point the hippy
in the owner kicked in and they felt that they needed to offset the, as
I now discovered looking at the menu, outrageous prices. Hardly being
one to not saver all that is on offer we partake in the pleasure and,
Feeling No Guilt At All, we return some indeterminate time later relaxed,
fed and still wondering what and when it will all go horribly wrong.

I casually place all my chips on wireless, this is usually a safe bet.
We're still in setup mode and only about one third of the exhibitors are
here but a brief survey reveals that there are more than ten wireless
access points on the show floor and people haven't been clever about how
they distributing the frequencies. This is also when I discover that one
exhibitor has yet to install their 6 wireless devices but they hope to be
operational before the end of the day. I'd nonrmally panic at this point
however, although there are a lot of base stations, I can't see that there
are actually any clients connected to them so most of this infrastructure
is seemingly pointless. Since I can't really do anything about at this
time I decide that I'll just have to wait for it all to fail.

Returning from my survey I find that our lunchtime massage ("Er, 2 steaks,
a beer and a shiatsu, please") has had the desired effect and the NOC has
become very relaxed indeed. The stereo is cranking out lounge music back
beats, everyone is quietly hacking away on their pet project (today's
winner is Bernard and his code that walked every router interface we
have and is now collecting statistics on them) and we're slowly eating
away at the day feeling functional without being stressed. Of course,
at this time we're only dimly aware that our reticence is primarily due
to the hot humid weather that we're all working under and that it will
eventually bite us in the arse.

Late in the afternoon the NOC becomes unbearable. Since we're right
against the west wall of the hall, it transpires that we're been slowly
roasting as the wall heats up in the afternoon. This is compounded by
our nice black walled NOC and the the new halogen lights that just got
installed (although I now notice that someone has unplugged them). It is a
subtle process but after a while you realise that the sweat running down
your armpits at that rate is not normal and you don't really recall your
clothes sticking to you quite that badly earlier in the day. Although the
work isn't stressing us, the atmosphere is but we only need to hang around
for a few more minutes before the helpdesk closes, so the end is near.

For some reason Glenn thinks that this would be a good time to go an
play a few games of tennis. We think that it is just too damn hot,
it's Beer O'Clock.

Day 1 - Continue the reserach

This is, surprisingly, an extremely comfortable project. It's not
like we're idle, or that we don't have anything to do. but I'm this
strange twilight zone where there always seems to be someone to do
whatever needs to be done. Probably the most difficult part of the day
was organising Lunch. In Sydney, and especially in Chinatown, which is
where the convention center is mostly located, it has become a traditional
pastime to have Yum Cha (Dim Sum) for a Sunday lunch/brunch we were well
aware of this and had planned to make the day worthwhile by finding a
Good restaurant, but somewhere between overwork and apathy we didn't
actually book anywhere and instead just wandered our way into Chinatown
roughly around lunchtime looking for a feed.

Unfortunately either a misdirected desire to not offend the fellow
workers or just sheer ignorance meant that not only did we not know
where to go, but no-one really wanted to take charge and just pick a
place. The fascist in me kicked in early, but my out of date lack of
knowledge meant that our first choice was closed, so I waved my arms
and deferred tot the team. Glen, the Worldcom systems guy "Had a plan"
but somehow this reverted to 10 people squeezed into an elevator where
we quickly concluded that there was "no fucking way there is fucking
Yum Cha joint on the 5th floor of this place...". Glen admitted out of
date knowledge (or a complete lack of spatial orientation) and finally
we ended up in somewhere that actually a) sold food and b) sold beer.

Getting back to the real work for a moment, we had another productive
day. The sparkie who had promised me power the night before hadn't
followed through but assured me he would do it right now. I went back to
the NOC to check on the status of the racks and realised that we probably
needed to optimise our hardware inventory and rather than keep all these
extra UPS's in boxes we could put them out in the racks. Of course this
meant that needed to go back to the sparkie and now tell him that I needed
2 circuits and not one. Karma being what it was, as I passed one of the
rack locations I noticed that he had actually given me 2 circuits so,
as usual, the real work had actually been done.

However, with real power we were able to roll out the peds onto the
show floor and try our luck at patching the whole shooting match into a
functional network. Some time earlier Glenn and I attempted to try and
test our fibre links, but the fibre test kit we'd been given had a few
personal problems. Since we seemed to be unable to set any reference other
than "Really Dark" there was this wonderful moment of self realisation
where (after about a 15 second silence) we both concluded that we should
"Bugger the fibre test, lets just patch the bastards in and see what
happens".

I only thought of this a long time later when casually asked if we'd
patched in the peds. "Yes," was the reply, "All the links came up
just fine".

Glenn and I tried to focus and turn our addressing/patching data into some
sort of reality. Extreme, because they wanted to try this new 'Super VLAN'
technology required us to develop a stable list of IP address to Port
matchup. so I had to find some way to resolve my hacked up addressing
programme with the patching spreadsheet and still keep our sanity. This
mostly was resolved by a lot of silence by Glenn as he wrestled the data
into submission. But he emerged at the end with a coherent patching list
and sent the team off to ping the bejesus out of the network.

Our moment of irony was when the ONE booth we actually required fluke
tools for was, of course, the Fluke booth.

Extreme 1, NOC 0.

At about 4pm we re-convened for a state of the nation and decided
that since "The damn thing just works" we should deliver addressing
sheets and, thus, the network. This was when Paul (extreme) chimed in
and reminded us that he hadn't actually dropped in the final configs
(with the Super VLANS) and that would probably be in our best interests
to re-ping all the drops. I made a mental note to inform Paul about my
plans in advance and admitted that, yes, we should re-test the entire
network if we respected our handiwork.

So while the team was out pinging away and delivering drop sheets we
distracted ourselves with the finishing touches. Sanjeev explained how
the two external routers failed over:

"I couldn't get VRRP stable enough for my liking" He started "So
what you do is disconnect this yellow cable, here and re-patch it
into the second router, here..."

And Bernard decided that he couldn't wait until the morning and would
the fix our NOC door. The Actual fixing was not really the amusing part
(mostly a lot of grunting and shaking of the booth). What was more
entertaining was watching everyone work out the problem for themselves

"What's wrong with it ?" They'd ask
"Try and lock the door"
"Yeah, it has lock, I can see that"
"No," We'd all chant, "Try and LOCK the door"

This was when they finally discovered that we could very easily and
securely lock ourselves INTO the NOC, but anyone from the outside could
just turn the latch.

Safe with the knowledge that the outside world could be kept behind
locked doors, it was Beer O'Clock.

Day 0 - We get the Hall

I arrive at 8am in a slight state of panic. Because all the carpet was
laid yesterday, the decorator got a jump start and more that half the gem
structures are already built. And we were supposed to pull through all
the drops, so stressing that we have to do some emergency pit
work. However as I walk around I notice that not only have all the drops
already been pulled from the pits, but Gary has terminated tagged and
tested them.

When I bailed up Gary later for the test reports he said

"I gave them to you. "
"Where ?"
"On the fluke you loaned me, I saved all the reports on the DSP"

The NOC was so far progressed I had to get them to remove a panel so I
could get the racks in. There was, however, no power so I track down
the foreman who points me to the sparkies. We walks over to the booth
looks at the situation and says "Wait a sec" he walks over to the next
pit, grabs the distribution panel and plugs it into my pit. "Will that
work for you ?" "Yes" I sigh. For some reason I was expecting this to
be a wearisome 4 hour waiting game, but instead Everyone Here Is Your
Mate and "No worries, I'll do it in just a sec" is the mantra

At around 9:30 the team starts to arrive, Gary is there finishing up
some of the aerial runs, Worldcom are wrangling a rack from their CO to
the show floor, Glenn, Bernard and the usual suspects are wandering in,
and a small team of volunteers are also making their way towards the
NOC. I'm about to start to wonder if we have too many people, when a
truck arrives with all the APC equipment. SO I'm able to get the boys
racking and stacking while another few teams walk the floor and check
that the contractors aren't doing something stupid with my drops. We
quickly become a busy little team.

Colorful lexicon.

It is nice to be back in a country where we're not afraid to call a spade
a spade and a friend a bastard. Australia has what can only be described
as a colourful lexicon and is really not afraid to use it. Over here you
can safely interchange the word "Damn" with "Bugger" and still be
considered a member of polite society. "Bugger it", "Oh Bugger", "Bugger
him, and the horse he rode in on" and my favorite, "Full of Buggery", Are
the kinds of words that I often heard coming from the mouth of the
receptionist.

Of course, once you get onto a building site, like the show floor, that's
when the contractors and their years of hard swearing experience truly
come into form. Long working days and hard drinking nights can bring out
the poet in some people, so I was priveliged to overhear this masterpiece:

" Oh for fucks sake Harry, you've gone and fucked the fucker.
Now I'll have to get my fucking tools and fuckin un-fuck it. "

Once we get the NOC stabilised the team sits down for the mandatory 3M
fibre termination class, questionable only in the fact that the the
terminations I'm demonstrating are also the first ones I've ever done.
But we're working on the assumption that I only need one of my 3 pairs so
we can afford to make a few mistakes. The VF45 termination seems to be so
easy that we leave the last PED to a random pair of volunteers to finish
up the work and test all the links.

Worldcom have made the final install and patch - like all telco racks all
the cable is Very Neat And Tidy and every end has been polished and
attenuated to the Specified Range. But this also means that WE have
externals - blindingly fast, watching one download we were pretty sure
that what was slowing down the 60meg of data was not the pipe, but the
slow disk drive.

Paul From extreme has been madly trying to hot stage his network. He's
been patching like a man possessed and has configs coming out if his ears.
Muttering 'EAPS!' like a proclamation he emerges sometime later and
proudly presents an unlabeled clusterfuck of 3 peds, 6 routers and a
LOT of Gig backbone. "But look, the ring fails over...". We decide to
leave him with the cleanup, it's Beer O'Clock.

Day (-1) - Still Early Install

Both halls are empty and the electricians are installing power. Gary has
made some sort of pact with the devil and got ALL the cable installed
last night. "I was only able to test about 80% of them before I told the
boys to knock off and get some sleep. We'll get the rest done tonight". He
then nudges me and says, "Since the halls are empty we'll probably sneak
back here at about 9pm and finish up"

Worldcom have managed to fix their backhoe problem and have gotten me
a circuit, they really weren't sure how much bandwidth we required so
they've given me an STM-4 stub, which was, last time I checked, an OC-12
(622Mbit). Unfortunately that had to be terminated onto a breakout unit
that was lacking a few interfaces so the we were only able to peel off
a single OC3 for the NOC.

Anyway, we left the telcos to play with their special IR glasses and
OTDR and work their magic and went off to examine the floor