Subversion Upgrade Complete

To everyone playing along at home, our subversion server has been upgraded, moved, vaccumed, manicured, and gotten a hair cut. It should be faster and more shiny. (Some new features aren't available yet, but will be soon.) Also coming is a new front page, with answers to all those questions you never knew you had. Let svn at perl.org know if you find anything wrong or broken.

Subversion Upgrade Soon

At some point later this week, or maybe this weekend, I will be taking the perl.org Subversion server down to move it to different hardware. This most likely won't happen until after the imminent Parrot release.

From an end user perspective, nothing should change. (Although we are upgrading to a newer version of the server and associated programs.) The server will be there, and then it will be down for a fewhours... and then it'll be back, bright, shiny and happy. More details once it is complete.

Cover opened, please panic!

You may have noticed that you didn't get any perl.org or pm.org mailing list email from 9am until 12 noon (pacific) this morning. We had a slight failure that required manual intervention. (The machine OOMed, and ssh got hosed as a side effect.)

I drove into the data center, and after a bit of fiddling to get the display working, discovered that the machine was very patiently waiting for a keypress because the cover sensor had been disturbed. That "feature" has been disabled.

No mail has been lost. Everything should have been caught up by now.

Everything is green again

Just got back from the datacenter. Everything was running again about 60-90 minutes ago.

Since the last reboot the server in question had more disks installed. The disks are split across two 3ware raid controllers (one with the new disks and one with the old). The bios decided to try booting off the controller with the new disks – of course there's no boot record on those so it just stalled.

The serial console was working, but for some reason it'd not show me anything until it had skipped past the "press button to enter bios setup" window. Robert could get into the bios remotely, but he was too lazy to check the settings so downtown I went. Grrh.

file server reboot

We have one particular server that's hosting many of our databases, NFS for our internal software distributions etc etc. We reboot it a couple of times a year to get kernel updates and such. One of those times is in a minute (so a short outage of some services will occur). Crossed fingers it'll come up alright, we don't want to go to the datacenter right now. :-)

update: eek, it's not coming back. crap.

Overload!

You may have noticed some trouble accessing some perl.org services this morning. Our network link is being saturated, because one of the DNS services we host is getting hammered. We've shut down the server on our side, but the other side is still sending. Mad props to InterNAP for being helpful in diagnosing and putting workarounds in place to get us stable again.

nntp upgrade

Tonight we rolled out the upgrade of our nntp server (nntp.perl.org) to colobus 2.1. Previously we were running a very old, somewhat hacked up version. Now we're running a new, only slightly hacked up version. (We had to make some changes for performance and quantity -- all sent back to the author of course.) You shouldn't notice any difference -- the speed seems about the same. The new version is backed by MySQL, which conveniently gives us easy access to the overview database (which contains threading information.) This will let us do cool things in the new web interface (which has been "Coming soon" for way too long.)

Full Duplex

Tonight I went to the datacenter and changed our incoming internet pipe to full duplex mode. This will give us a few more usable megabits of bandwidth, making it much less likely that we'll max out our bandwidth. It also cuts off a few milleseconds of latency.

As always, I need to thank the wonderful support staff at InterNAP for making everything so easy! I called, and within 10 minutes we had the link forced to full duplex with no interruption of service.

Why were we in half duplex mode in the first place? Good question. Their side was set to autonegotiate as was ours. Maybe it's the media converter? Maybe its our switch? Setting our switch to full duplex (and leaving theirs on auto) only made things worse. But now both sides of the link are happy, I'm happy, and our users should be happy too!

Help us at Perl.org!

If you are reading this weblog wondering how you can help, here's your chance. :-)

RT / perlbug down today

So the issue from yesterday didn't get sorted out last night. I've been working on it today and the database is happy again. Just waiting for Robert to startup RT again. :-)

RT / perlbug issues

MySQL on our main database server (we were running an old-ish 4.1.x version) crashed and corrupted some (unrelated to the crash, I think) InnoDB tables.   We did run it with some vaguely aggressive settings, but it shouldn't lead to corruption (only losing a few updates).   Grrh!

Speaking of the database server, AMD are giving us at least one of their servers; we don't really know too much about it except that a package is supposed to arrive at my house.  One of our planned uses is a new database server which should speed up RT and several other services.

From:   Robert Spier
Subject: Perlbug Email Down, Web Interface Unstable
Date: December 6, 2005 9:52:20 PM PST

Dear p5p, p6i,

We're having some database issues.

In order to make my life easier when I try and recover from them, I've disabled incoming email to the RT instance.  (It will be queued up and
delivered later.)  You may also find the web interface to be unstable
as the database decides to disappear mid-stream.

I will be working on this tomorrow, and hope to have everything back

to normal within 24 hours.

(Details: it appears that somehow, we've made innodb unhappy.

Tomorrow, I will be doing a lot of dumps and restores to try and

straighten it out.)

-R

update 1am PST: All seems well again. Unless it goes bonkers again during the night Robert is going to turn processing of the RT emails back on in the morning. - ask

Perl Foundation Weblog

Apologies if you already saw this on use.perl, my personal weblog or one of the soon bazillion other places it's been posted.

Yesterday we announced the Perl Foundation Weblog. Robert and I will post there occasionally too. I'm not sure yet if we'll just take some posts from here or if we'll split it up and leave this weblog more for just operational stuff and have news and status reports there.

CPAN / FUnet update

It's not something we (Robert, Graham & Ask) are in control of, but if you were wondering why search.cpan.org hasn't been getting new module uploads quickly lately it's because of problems with the hardware running the FUnet.fi master CPAN mirror.  It's making all of CPAN "not move" so to speak.

PAUSE is still running and taking uploads and no data is being lost anywhere, just a little delayed.

Last I heard it should get better sometime this week.