[fastcart]

Uhh, CloudFFVII... Please read the whole thread before correcting someone. We already know how long it's gonna be because it's been changed (on the NA site) and repeated a few times now. It's not like this thread is five pages long or anything. ^^ Ok?

kthxbai.

[grunion]

From Aug. 8, 2005 from 8:00 to Aug. 9, 2005 2:00, PST

[michaeldue]

OMG!? How many people can post the same info lol.

[Tsama]

I just wish they would say what this damn maintenance is going to do.... Word on the net is it's the blm nerf, which I am waiting to see, because if it is as bad as the previous blm nerfs I know a good 100 people that are gone. I really wish SE, would fix things instead of messing more things up.

[felpa_de_osa]

As long as Square Enix don't recycle their machines from beta, cough ragnarok online cough hopefully we wont need more mainteneces or long ones like this...

[kuu]

Originally Posted by Caspian

I think they said they were changing servers or something. Which if they were doing same thing at the hospital where I work that would be like 2 weeks of downtime. And, USUALLY, stuff like that they overestimate the time needed to fix it, often it takes less than the time they give for something like this. Though this is SE who I cant recall them ever finishing an update on time, so who knows.

S-E acutally have a somewhat decent maintence estimate trackrecord, and good update on ETA notice rate.

It's the nature of MMOs, and almost all of them say 2 hrs that turn into 5 or 10 into 15.

Squeezing a big maintaince in pass the 8 hours is usually good, because some companies refuse to pay for more then normal shifts and just let it roll to the next day/shift.

People who do these things are human you know, lol.

[exario]

I think that whomever is incharge of this Maintenence must of been one of the idiot - yes men. All the WAN IT ppl I know would not of done maintenene this way... the most effiecent and accurate way to do maintenence is to:

1- copy excisting servers
2- update the these backups...and fully test them.
3- inform everyone to be off at 4am
4- unplug the RG45 plug from old one and plug into the backup server.(thus making the original now the back up....and repeat!

down time or mantenence time...would be 10 minutes max...and that would include rebooting all the servers

this is my story....I hope you enjoyed it.
E

[kuu]

Originally Posted by exario

I think that whomever is incharge of this Maintenence must of been one of the idiot - yes men. All the WAN IT ppl I know would not of done maintenene this way... the most effiecent and accurate way to do maintenence is to:

1- copy excisting servers
2- update the these backups...and fully test them.
3- inform everyone to be off at 4am
4- unplug the RG45 plug from old one and plug into the backup server.(thus making the original now the back up....and repeat!

down time or mantenence time...would be 10 minutes max...and that would include rebooting all the servers

this is my story....I hope you enjoyed it.
E

Have you done any professional major server maintence @.@???

That's like saying launching people to space is just sticking a bunch of thrusters on a plane and shoot it.

While some people don't go 100% on the job and get lazy, it's not all simple

[exario]

actually I and CCNP and have done ALOT of server maintenence...and it really is that easy to do. the only hard part is...the trouble shooting...the only true way to trouble shoot an enterprise server is to...have it up and running...but...the way that it is done now...appears to be more of pull down the only active servers...because no backups are available. work on it....then put them online again...after so many hours and still find problems....there will always be somthing forgotten or something unforseen...but to pull down a server , make changes, and then put it back up...every week is insane....with proper testing and planning the endusers should never be put out this much...

where I did my server maintenence...we would of lost our jobs for 8 hrs of downtime....lol we would of lost them for 2 hrs of DT unless it was something absolutly necessary to pull a server for so long. but weekly or even monthly maintenence should NEVER take that long everytime.

there are many varibles....agreed... but with a lot more planing on there part ....DT should be minimal


Double Post Edited:
lol......oops...that would be I am a CCNP....LOLOLOLOLOL...well it just goes to show...I would of never made it through college with out spell and grammer check....LOLOLOLOL

[kuu]

Originally Posted by exario

actually I and CCNP and have done ALOT of server maintenence...and it really is that easy to do. the only hard part is...the trouble shooting...the only true way to trouble shoot an enterprise server is to...have it up and running...but...the way that it is done now...appears to be more of pull down the only active servers...because no backups are available. work on it....then put them online again...after so many hours and still find problems....there will always be somthing forgotten or something unforseen...but to pull down a server , make changes, and then put it back up...every week is insane....with proper testing and planning the endusers should never be put out this much...

You do realize this is POL that's being updated. Which can arguably be the most important peice in the network due to all the interaction that has to be made to ever other server(Billing server, ffxi portal, terramaster, customer helpdesk, mail server, account server,....so on)

How are you going to run that through a backup and ensure any kind of QoS except take it all down?

Originally Posted by exario

where I did my server maintenence...we would of lost our jobs for 8 hrs of downtime....lol we would of lost them for 2 hrs of DT unless it was something absolutly necessary to pull a server for so long. but weekly or even monthly maintenence should NEVER take that long everytime.

This isn't a run of the mil maintence, as there is "server upgrade" whatever that maybe... could be anything to a 32k patch(not likely) to a full blown overhaul.

Reason it's stated as so long.

[Isendre]

well whatever it is we will all just have to wait for it to be done ;; no use complainin or debating a "better" way to do it /whines
i just wanna check my AH /twitches

[exario]

my original comment was just being critical of the past updating technic that is being used.

This current maintenence... I was not aware of the specifics... I do understand it would take a lot of time and effort. but are you telling me that this work could not of been done on a backup server ....properly troubleshot...streamlined... and then put online?... all of those specific billing servers ESPECIALLY billing and account DBs should ALWAYS have at least 2 backups...fulling ready and secure... with the hacker community the way it is... when it comes to money and or anything possibly minipulated for a gain...there must be backups ready at all times. IE those AH attacks a couple of months ago. that screw up FF for 6weeks. I can understand that kind of downtime and why it took them so long to couter meassure.

but it is also very easy to trace the location of an attacking computer...(understand if they have routers and switches this would be more difficult) you can easity counter the attack and fry there computer.

I can see why they would not have done it... lawsuits or what not...but many major IT companies pay hackers to destroy there competitions networks and servers. This is a cat and mouse game between hackers that has been around for a while.

[Kutch]

Originally Posted by exario

my original comment was just being critical of the past updating technic that is being used.

This current maintenence... I was not aware of the specifics... I do understand it would take a lot of time and effort. but are you telling me that this work could not of been done on a backup server

Simply moving the terabytes of data that the servers contain would probably take hours and hours on end. I'm a support manager for a software product that handles storages anywhere from four to fourty terabyte. There are two issues with these systems:

1) You can't just buy two of'em. They cost too much, and are usually built as failsafe as possible (various forms of raids and mirroring). Our smallest customer is running a 4TB storage where the backup alone takes around 5 hours to complete on an automated tape jukebox.

2) People here seem to talk about servers as if it was your local double pentium four. MMO systems are never "one" machine, they usually have one or more computers running each area wihtin one world, that's around 100 computers for each server out there. What do you folks think this is? For the POL server alone, that is handling hundreds of thousands of accounts, do you really think there's ONE server behind that? think again, I'm guessing server farms that are anything BUT easy to maintain.

I'm giving huge thumbs up, not only to SE for managing maintenance in such relatively small time (it's been posted before, normally, such large systems take large amounts of time to upgrade / perform any major maintenance on), but to all MMO's out there with their heads over the water.

Forget the "replace the RJ45 plug with the backup server" .. you're talking small business servers, these are huge enterprise farms.

[kuu]

Originally Posted by exario

my original comment was just being critical of the past updating technic that is being used.

Yea, but then reality tends to fester ineffiency more often then not. Business especially big ones has a very "Don't change it if it's not broken" mentality.

And there's more server then account servers here, besides which the more you have backups lying around the more money which is very unlikely you'll have a budget for. The whole POL idea reaks of odd and ends nitnacks..as what always happens when you try to intergrate everything.

Hopefully as a side effect of a prolong thorough maintence, a lot of spring cleaning will be done, that generally can mean less maintence in the near future itself.

Nothing really stops a determined DDoS attack except having more people on watchduty. There are tricks here and there, but in the end it's all about action and reaction... and wasting unnecessary money will get you the manager from hell.

DDoS is a very...how would you say...age old ...problem to the computer world. It's only resently that the resources to launch one has become so easy. Until the you fix zombie computers to launching one becomes costly enough, it will just get worse.

[fuz]

I'm thinking they took everything offline, then made a full backup (instead of relying on some incremental system), then upgraded whatever networks/computers/misc hardware, tested them, reboot, upgrade software, test software, then go live if nothing goes wrong (which is rare). Each of those steps probably takes ~1-2 hours, while testing and debugging can go on much longer than that. I'm not sure how much data they have stored, but suffice to say it should range easily in the multi-terabyte range. (They also seem to keep multiple backups active for things like abuse hunting or the return to vana-diel thing.)

This isn't just the FFXI servers as mentioned, but the whole online network that POL encompasses. The logicstics alone of such a project are a huge headache. However, unlike other applications, games are rarely business critical (especially since you pay monthly with FFXI, not daily or hourly). So they can take more time than allowed in other scenarios, e.g. the company e-mail server and not suffer immediate revenue loss.

I'm actually impressed they have the guts to do this all at once in such a timeframe--they seem to expect that nothing will go wrong. So they must have already been testing this for some time.