|
|
|
19-01-2007, 01:06
|
#1 |
|
I'm going for a scuttle...
Join Date: Jul 2006
Posts: 2,021
|
ARGH
I got a call at about 6pm tonight - the shock radio webserver is no longer serving ANY pages (apache running on gentoo). Despite the fact that I am one of two people the chief eng can contact that knows about Linux, I am so far lacking in root access (or any shell access at all) to the box - so I get on MSN to someone who does and start the process of finding out whats wrong.
It accepted an SSH session but the box was apparrently shutting down ... hmm. Wait a few minutes and ssh back in - everything is royally buggered, services up, down, left, right who knows what. dmesg indicated a problem with the fileserver. I get my coat on.... Get to the studio, walk into the server room to find that the webserver is halfway down (refusing logins but still multi-user) with tty1 just showing its progress through shutting down... The fileserver isnt much healthier - RAID5 in degraded mode. Now, if we were running something GOOD this wouldnt be too much of an issue but we have a crap rocketraid sata controller with freenas on top. Its obviously spat at least one disk but I hear from the station manager that "it stopped working yesterday, then came back so I thought it was fine" - now FreeNAS is totally unresponsive, the web interface is down, the box wont reboot and god only knows what. The thing is, FreeNAS appears to be buggered but it can see individual drives (ad2, 4 and 6) and THEN an array ar0 rather than just seeing one array. Is it hardware raid? The raid card thinks so. Is it software raid? FreeNAS thinks so. The webserver died because for some reason, something critical is in a folder on the fileserver. Whats the point in that? WHY? It introduces so many ways to bring the web server down its unfunny. The webserver isnt the only thing that runs on that box either, so its not like it crashing only takes out our web presence! The data on this array has NEVER been backed up and is not only irreplacable but required to be kept by law. The penalty for losing this stuff is a fairly hefty fine and/or imprisonment. Quite why they felt using GOD DAMN MAXTOR DRIVES for this I dont quite know. F****** ridiculous. Going to rebuild the array tomorrow hopefully (!) and promptly make a backup. Thankfully its "only" 400Gb :/ Just needed a bit of a rant...
__________________
Last edited by Dr. Z; 19-01-2007 at 01:08. |
|
|
|
19-01-2007, 02:11
|
#2 |
|
I iz speshul
Join Date: Jun 2006
Location: Liverpool
Posts: 6,296
|
Given that it's *Nix based I only understood half of that post, but I get the general gist that things went FUBAR big time. Hope you can recover everything as quick as possible, though I'm getting the impression it won't be a quick or easy job.
__________________
 Our deepest fear is not that we are inadequate. Our deepest fear is that we are powerful beyond measure. It is our light, not our darkness, that most frightens us. Your playing small does not serve the world. There is nothing enlightened about shrinking so that other people won't feel insecure around you. We are all meant to shine as children do. It's not just in some of us; it is in everyone. And as we let our own lights shine, we unconsciously give other people permission to do the same. As we are liberated from our own fear, our presence automatically liberates others. |
|
|
|
|
19-01-2007, 10:44
|
#3 | |
|
The Stig
Join Date: Jun 2006
Location: Swad!
Posts: 10,713
|
Quote:
Depending on how that box has been setup, you might need to get familiar with dmraid. If the array is handled by dmraid (and device mapper), then you should see something other than 'control' in /dev/mapper, if it even exists. Either way, good look. Touch wood (/touches wood), I've never really had to do much recovery wise on a Linux box, most *nix admins I've worked with (if it isnt myself anyway) keep good backup procedures, so it's always been restoring rather than recovering.
__________________
apt-get moo |
|
|
|
|
|
19-01-2007, 17:05
|
#4 | |
|
I'm going for a scuttle...
Join Date: Jul 2006
Posts: 2,021
|
Quote:
 Tomorrow I will get the money for a new drive so I will try it and see. Im touching your wood daz, as I only have fake stuff to hand here
__________________
|
|
|
|
|
|
19-01-2007, 17:37
|
#5 | |
|
Good Cat
Join Date: Jun 2006
Posts: 5,550
|
Quote:
 Does Dee know about this? 
__________________
 Oooooh Cecil, what have you done? |
|
|
|
|
|
19-01-2007, 17:39
|
#6 | |||
|
The Stig
Join Date: Jun 2006
Location: Swad!
Posts: 10,713
|
Quote:
It could of course just be traditional Linux software raid, in which case you'll see /dev/md0,md1 etc (multi-disk devices). This would be easier in terms of not worrying about dmraid (it can be a bitch believe me), but I'm not sure how it'll help in recovering a RAID-5. It should be well documented though if it is this particular flavour. Assuming fakeraid, if it's entirely handled by the driver (unlikely, most 2.4 kernels and certainly 2.6 see straight through them to the raw devices) then who knows what you'll have to do. Quote:
Quote:
__________________
apt-get moo |
|||
|
|
|
|
19-01-2007, 10:47
|
#7 |
|
The Night Worker
Join Date: Jul 2006
Posts: 5,228
|
Although i read your O.P. yeterday and today i still have no idea what the **** you are on about.
That doesn't stop me feeling your pain though so good luck with sorting it all out today.
__________________
 
|
|
|
|
|
19-01-2007, 17:48
|
#8 | |
|
The Stig
Join Date: Jun 2006
Location: Swad!
Posts: 10,713
|
I'm reading there are some proprietary binary drivers kicking about for RocketRAID cards, and you may well be using them. Could do with knowing he chipset model really to find out what we're dealing with
 Quote:
http://linuxmafia.com/faq/Hardware/sata.html
__________________
apt-get moo |
|
|
|
|
|
20-01-2007, 20:05
|
#9 |
|
I'm going for a scuttle...
Join Date: Jul 2006
Posts: 2,021
|
Its a 1640.
I now have a drive to replace the failed one so I am good to go for the new suggestions!
__________________
|
|
|
|
 |
| Thread Tools | |
| Display Modes | |
|
|
Hybrid Mode
