Help us at Perl.org!

If you are reading this weblog wondering how you can help, here's your chance. :-)

RT / perlbug down today

So the issue from yesterday didn't get sorted out last night. I've been working on it today and the database is happy again. Just waiting for Robert to startup RT again. :-)

RT / perlbug issues

MySQL on our main database server (we were running an old-ish 4.1.x version) crashed and corrupted some (unrelated to the crash, I think) InnoDB tables.   We did run it with some vaguely aggressive settings, but it shouldn't lead to corruption (only losing a few updates).   Grrh!

Speaking of the database server, AMD are giving us at least one of their servers; we don't really know too much about it except that a package is supposed to arrive at my house.  One of our planned uses is a new database server which should speed up RT and several other services.

From:   Robert Spier
Subject: Perlbug Email Down, Web Interface Unstable
Date: December 6, 2005 9:52:20 PM PST

Dear p5p, p6i,

We're having some database issues.

In order to make my life easier when I try and recover from them, I've disabled incoming email to the RT instance.  (It will be queued up and
delivered later.)  You may also find the web interface to be unstable
as the database decides to disappear mid-stream.

I will be working on this tomorrow, and hope to have everything back

to normal within 24 hours.

(Details: it appears that somehow, we've made innodb unhappy.

Tomorrow, I will be doing a lot of dumps and restores to try and

straighten it out.)

-R

update 1am PST: All seems well again. Unless it goes bonkers again during the night Robert is going to turn processing of the RT emails back on in the morning. - ask

Perl Foundation Weblog

Apologies if you already saw this on use.perl, my personal weblog or one of the soon bazillion other places it's been posted.

Yesterday we announced the Perl Foundation Weblog. Robert and I will post there occasionally too. I'm not sure yet if we'll just take some posts from here or if we'll split it up and leave this weblog more for just operational stuff and have news and status reports there.

CPAN / FUnet update

It's not something we (Robert, Graham & Ask) are in control of, but if you were wondering why search.cpan.org hasn't been getting new module uploads quickly lately it's because of problems with the hardware running the FUnet.fi master CPAN mirror.  It's making all of CPAN "not move" so to speak.

PAUSE is still running and taking uploads and no data is being lost anywhere, just a little delayed.

Last I heard it should get better sometime this week.

New search engine

For a long time Search CPAN has used the WAIT module to provide searching of POD documents. Today we started testing the use of Swish-e. Try it and let us know what you think

power is back...

The power is back and the console server came up. Now for trying to sort out everything else, hopefully without having to drive there.

update: alright, I got everything beat into running again except for the mailing list server. A few of the other boxes (on really really really old hardware) had a hard time getting started too, but after an extra power cycle and a manual fsck I got them going. x6, the list server, isn't as responsive though. Grrrh.

On the radio they're talking about "45 minutes traffic" from Hollywood to downtown on the 101 (when there's no traffic it takes just 10-15 minutes) so I'm hesitating a little bit running down there to look at it, but I will have to go soon as I can't be holed up down there for too long tonight.

update 2, much too much later: The mailing list mails are flowing again... (and the console redirection on that server is working again; one of the harddrives in the RAID not so much. grrh).

Power outage in Los Angeles, perl.org down

Apparently there's a Really Big power outage here in Los Angeles.

At my house the power flicked enough for the UPS to click in and for things not on the UPS to reboot, but there's power just fine now so I didn't think anything of it.

Ask: Circuit not working. Can't breathe!

Internap NOC: Los Angeles? Big power outage there. Traffic signals not working, police on "full tactical alert" and stuff.

Ask: Uh-oh! Okay.

Ask: Generators?

Internap NOC: They said they came on but shut down due to overload.

Ask: Doh.

Our servers are in the same building but on a different floor, so it's entirely likely that they are affected too. Grrh. I'll have some fun getting everything running again when the power comes back down there (Robert is traveling today).

Datacenter day - update

That went mostly well. We got two dead servers assembled into one that'll probably work, or maybe not. It was installing slower than slowly over the network. Robert set it up to redirect the BIOS to the serial port and we hooked it into the console server so he'll try installing it from home later. We also hooked the new switch up to the console. Cyclades++.

We got all the disks in the new RAID working. It'll be very very cool when we start actually using them. The hardware RAID-5 with the 120GB disks we are using now isn't performing well at all.

While taking disks out and checking the connectors Robert pulled the disk we thought was an extra extra hot spare for the existing RAID only to minutes later realize it had a swap partition on it. Ooops!

MySQL was apparently swapped a bit because it crashed with a really really long dump to the log and some corrupted MyISAM indexes. All the InnoDB tables came up fine though. When InnoDB first came out I was much more comfortable with MyISAM because how simple it was. I'm only going to use MyISAM in the future when the particular performance characteristics of it are needed.

Datacenter day

Robert and I are in the data center today (afternoon PST) replacing a bad disk or two and moving some equipment around.

Some of the services we run might be unavailable for a few moments once or twice, so don't be alarmed.

We have monitoring setup to alert if when things are not working, but if something is not working by the end of the day (~1am UTC), please let us know. :)

update: over here....

Create your own "BackPAN"

If you didn't know, "backpan" is all the PAUSE uploads but with no deletions (PAUSE, by Andreas König, is the Perl Authors Upload SErver which is the way the vast majority of content enters CPAN now).

Robert and I run a backpan at backpan.perl.org and Elaine is running an equivalent one at backpan.cpan.org.

We could use more though! It doesn't have to be public, but it'd be nice to know that we had a few more backups of old old CPAN distributions.

First mirror CPAN if you aren't already. Once a day is fine. Also once a day then run something like the following:

rsync --exclude CHECKSUMS -vrptgx ~/mirror/CPAN/authors/id/ ~/mirror/backpan/authors/id/

to copy new files into backpan. Notice there's no --delete argument to that rsync command.

CPAN Ratings updates

Spent a couple hours this afternoon on CPAN Ratings and fixed and cleaned up a bunch of little things.

One of them was adding a link to the RSS feed. I didn't realize we forgot that when we added it. Oops! :-)

Of course I might have broken something too. Let me know if you see anything not working so well.

It's great that so many people have been "rating reviews" (marking them helpful or not). Next time I work on the site I'll make it do something useful with that data (besides just showing it next to each review).

Link exchange requests

The perl.org webmaster email address gets link exchange requests from time to time. Today, we got a great one...

Hello,
I have found your website perl.org by searching Google for "greenzap scam". I think our websites has a similar theme, so I have already added your link to my website.
....

Needless to say, we don't respond.

update: check out the comment this entry got. Hilarious. (I deleted their link from it though...) - ask