Sunday, March 03, 2002

Day 1 - Continue the reserach

This is, surprisingly, an extremely comfortable project. It's not
like we're idle, or that we don't have anything to do. but I'm this
strange twilight zone where there always seems to be someone to do
whatever needs to be done. Probably the most difficult part of the day
was organising Lunch. In Sydney, and especially in Chinatown, which is
where the convention center is mostly located, it has become a traditional
pastime to have Yum Cha (Dim Sum) for a Sunday lunch/brunch we were well
aware of this and had planned to make the day worthwhile by finding a
Good restaurant, but somewhere between overwork and apathy we didn't
actually book anywhere and instead just wandered our way into Chinatown
roughly around lunchtime looking for a feed.

Unfortunately either a misdirected desire to not offend the fellow
workers or just sheer ignorance meant that not only did we not know
where to go, but no-one really wanted to take charge and just pick a
place. The fascist in me kicked in early, but my out of date lack of
knowledge meant that our first choice was closed, so I waved my arms
and deferred tot the team. Glen, the Worldcom systems guy "Had a plan"
but somehow this reverted to 10 people squeezed into an elevator where
we quickly concluded that there was "no fucking way there is fucking
Yum Cha joint on the 5th floor of this place...". Glen admitted out of
date knowledge (or a complete lack of spatial orientation) and finally
we ended up in somewhere that actually a) sold food and b) sold beer.

Getting back to the real work for a moment, we had another productive
day. The sparkie who had promised me power the night before hadn't
followed through but assured me he would do it right now. I went back to
the NOC to check on the status of the racks and realised that we probably
needed to optimise our hardware inventory and rather than keep all these
extra UPS's in boxes we could put them out in the racks. Of course this
meant that needed to go back to the sparkie and now tell him that I needed
2 circuits and not one. Karma being what it was, as I passed one of the
rack locations I noticed that he had actually given me 2 circuits so,
as usual, the real work had actually been done.

However, with real power we were able to roll out the peds onto the
show floor and try our luck at patching the whole shooting match into a
functional network. Some time earlier Glenn and I attempted to try and
test our fibre links, but the fibre test kit we'd been given had a few
personal problems. Since we seemed to be unable to set any reference other
than "Really Dark" there was this wonderful moment of self realisation
where (after about a 15 second silence) we both concluded that we should
"Bugger the fibre test, lets just patch the bastards in and see what
happens".

I only thought of this a long time later when casually asked if we'd
patched in the peds. "Yes," was the reply, "All the links came up
just fine".

Glenn and I tried to focus and turn our addressing/patching data into some
sort of reality. Extreme, because they wanted to try this new 'Super VLAN'
technology required us to develop a stable list of IP address to Port
matchup. so I had to find some way to resolve my hacked up addressing
programme with the patching spreadsheet and still keep our sanity. This
mostly was resolved by a lot of silence by Glenn as he wrestled the data
into submission. But he emerged at the end with a coherent patching list
and sent the team off to ping the bejesus out of the network.

Our moment of irony was when the ONE booth we actually required fluke
tools for was, of course, the Fluke booth.

Extreme 1, NOC 0.

At about 4pm we re-convened for a state of the nation and decided
that since "The damn thing just works" we should deliver addressing
sheets and, thus, the network. This was when Paul (extreme) chimed in
and reminded us that he hadn't actually dropped in the final configs
(with the Super VLANS) and that would probably be in our best interests
to re-ping all the drops. I made a mental note to inform Paul about my
plans in advance and admitted that, yes, we should re-test the entire
network if we respected our handiwork.

So while the team was out pinging away and delivering drop sheets we
distracted ourselves with the finishing touches. Sanjeev explained how
the two external routers failed over:

"I couldn't get VRRP stable enough for my liking" He started "So
what you do is disconnect this yellow cable, here and re-patch it
into the second router, here..."

And Bernard decided that he couldn't wait until the morning and would
the fix our NOC door. The Actual fixing was not really the amusing part
(mostly a lot of grunting and shaking of the booth). What was more
entertaining was watching everyone work out the problem for themselves

"What's wrong with it ?" They'd ask
"Try and lock the door"
"Yeah, it has lock, I can see that"
"No," We'd all chant, "Try and LOCK the door"

This was when they finally discovered that we could very easily and
securely lock ourselves INTO the NOC, but anyone from the outside could
just turn the latch.

Safe with the knowledge that the outside world could be kept behind
locked doors, it was Beer O'Clock.

Saturday, March 02, 2002

Day 0 - We get the Hall

I arrive at 8am in a slight state of panic. Because all the carpet was
laid yesterday, the decorator got a jump start and more that half the gem
structures are already built. And we were supposed to pull through all
the drops, so stressing that we have to do some emergency pit
work. However as I walk around I notice that not only have all the drops
already been pulled from the pits, but Gary has terminated tagged and
tested them.

When I bailed up Gary later for the test reports he said

"I gave them to you. "
"Where ?"
"On the fluke you loaned me, I saved all the reports on the DSP"

The NOC was so far progressed I had to get them to remove a panel so I
could get the racks in. There was, however, no power so I track down
the foreman who points me to the sparkies. We walks over to the booth
looks at the situation and says "Wait a sec" he walks over to the next
pit, grabs the distribution panel and plugs it into my pit. "Will that
work for you ?" "Yes" I sigh. For some reason I was expecting this to
be a wearisome 4 hour waiting game, but instead Everyone Here Is Your
Mate and "No worries, I'll do it in just a sec" is the mantra

At around 9:30 the team starts to arrive, Gary is there finishing up
some of the aerial runs, Worldcom are wrangling a rack from their CO to
the show floor, Glenn, Bernard and the usual suspects are wandering in,
and a small team of volunteers are also making their way towards the
NOC. I'm about to start to wonder if we have too many people, when a
truck arrives with all the APC equipment. SO I'm able to get the boys
racking and stacking while another few teams walk the floor and check
that the contractors aren't doing something stupid with my drops. We
quickly become a busy little team.

Colorful lexicon.

It is nice to be back in a country where we're not afraid to call a spade
a spade and a friend a bastard. Australia has what can only be described
as a colourful lexicon and is really not afraid to use it. Over here you
can safely interchange the word "Damn" with "Bugger" and still be
considered a member of polite society. "Bugger it", "Oh Bugger", "Bugger
him, and the horse he rode in on" and my favorite, "Full of Buggery", Are
the kinds of words that I often heard coming from the mouth of the
receptionist.

Of course, once you get onto a building site, like the show floor, that's
when the contractors and their years of hard swearing experience truly
come into form. Long working days and hard drinking nights can bring out
the poet in some people, so I was priveliged to overhear this masterpiece:

" Oh for fucks sake Harry, you've gone and fucked the fucker.
Now I'll have to get my fucking tools and fuckin un-fuck it. "

Once we get the NOC stabilised the team sits down for the mandatory 3M
fibre termination class, questionable only in the fact that the the
terminations I'm demonstrating are also the first ones I've ever done.
But we're working on the assumption that I only need one of my 3 pairs so
we can afford to make a few mistakes. The VF45 termination seems to be so
easy that we leave the last PED to a random pair of volunteers to finish
up the work and test all the links.

Worldcom have made the final install and patch - like all telco racks all
the cable is Very Neat And Tidy and every end has been polished and
attenuated to the Specified Range. But this also means that WE have
externals - blindingly fast, watching one download we were pretty sure
that what was slowing down the 60meg of data was not the pipe, but the
slow disk drive.

Paul From extreme has been madly trying to hot stage his network. He's
been patching like a man possessed and has configs coming out if his ears.
Muttering 'EAPS!' like a proclamation he emerges sometime later and
proudly presents an unlabeled clusterfuck of 3 peds, 6 routers and a
LOT of Gig backbone. "But look, the ring fails over...". We decide to
leave him with the cleanup, it's Beer O'Clock.

Friday, March 01, 2002

Day (-1) - Still Early Install

Both halls are empty and the electricians are installing power. Gary has
made some sort of pact with the devil and got ALL the cable installed
last night. "I was only able to test about 80% of them before I told the
boys to knock off and get some sleep. We'll get the rest done tonight". He
then nudges me and says, "Since the halls are empty we'll probably sneak
back here at about 9pm and finish up"

Worldcom have managed to fix their backhoe problem and have gotten me
a circuit, they really weren't sure how much bandwidth we required so
they've given me an STM-4 stub, which was, last time I checked, an OC-12
(622Mbit). Unfortunately that had to be terminated onto a breakout unit
that was lacking a few interfaces so the we were only able to peel off
a single OC3 for the NOC.

Anyway, we left the telcos to play with their special IR glasses and
OTDR and work their magic and went off to examine the floor