Chief Technical Officer
|
| 51 people marked this as a favorite. |
We apologize for paizo.com's recent periods of unexpected downtime. Our team is currently making significant changes to our website infrastructure that should ultimately result in faster and better performance. These changes may result in at least one additional (but much shorter) outage in the coming weeks. Advance notice of this downtime will be posted to the front page of paizo.com and Paizo's official social media channels.
In apology, we're offering you 10% off any one paizo.com order placed between now and 11:59 PM Pacific Time on Sunday, September 30. Just enter the promotion code downtime during checkout. This 10% discount is redeemable only once and does not apply to subscriptions, backorders, preorders, or downloads from companies other than Paizo, but it does stack with the Paizo Advantage discount.
Thank you for your patience as we work to improve your paizo.com experience!
| 5 people marked this as a favorite. |
Paizo, as always, shows itself to be a class act!
That's great, thank you! I hope everything's going better on the back end now.
Lovely. Apparently my order didn't qualify, despite it being sent out today.
Lovely. Apparently my order didn't qualify, despite it being sent out today.
As class act as Paizo is being about all this, I'm sure they'll be happy to work something out with you if you email customer service.
| 1 person marked this as a favorite. |
Lovely. Apparently my order didn't qualify, despite it being sent out today.
Re-read Vic's post. specifically the part where he states "Placed between now and 11:59 PM Pacific Time on Sunday, September 30." and you'll understand why an order that shipped today wouldn't be eligible for the discount. Because placed =/= shipped, you'd have likely had to cancel that order, and then place it again to use the discount but if you contact Customer Service there may be something they can do, but if it's already shipped, it's unlikely. Of course, you can always use it for your next order, since it's good through the end of September. Although, as stated in Vic's post, it doesn't apply to everything.
| 2 people marked this as a favorite. |
Aw, ya didn't have to.... but since you did...
Time to loot and pillage some modules...
Aw, ya didn't have to.... but since you did...
Time to loot and pillage some modules...
That's the spirit (makes preparations to do the same).
| 6 people marked this as a favorite. |
Too bad I have a bigger discount on everything. Would have been nice if they gave those of us with the Paizo Advantage an extra 2% or something like that. I have no incentive to place an extra order outside of my normal subscription.
... It stacks with Paizo Advantage.
Thank you for the discount and thank you for being back up.
|
| 5 people marked this as a favorite. |
Yeah guys, I know the offices and IT space over there must essentially be on fire still, but if you guys are being subject to some kind of DDOS or Corporate Sabotage it would be nice to know so we aren't actually, you know, blaming the Purple Golem itself.
If this is a specific teams responsibility and they simply screwed up, that would be good to know as well, because as it stands I'm not sure I can trust paizo to even handle Purchases or Transactions securely anymore with as many problems the Website and Store have been having.
Transparency has always been a strength for Paizo, and frankly the generic apology and a temporary discount simply cannot restore the trust lost over the last two weeks. People want some REAL answers, and I'm sure your partners are none-too happy about the situation as well.
Thank you Paizo for the discount and the free Game Play Forums but I do hope the Game Play forums stabilize soon since the outages are hurting the games I belong to.
| 4 people marked this as a favorite. |
Thanks, Paizo. That's unexpected but appreciated.
Hoping things continue to improve. The updates on your social media are helpful; please keep them coming as warranted.
It has to have been a rough time--I agree with a sentiment above, I hope site attacks haven't contributed. Sadly I wouldn't put it past a certain type of gamer to do such a thing.
| 18 people marked this as a favorite. |
If this is a specific teams responsibility and they simply screwed up, that would be good to know as well...
I am neither a "fanboi" nor a "hater", but if Paizo were to name the team and/or individuals, that would cause me to seriously consider deleting my account and walking away from here.
Two things:
1) A saying that I learned when I was an LT in the Army, "praise in public criticize in private". If this was preventable, then Lisa / Vic should ream someone out, but in private, not here on the boards.
2) I worked for a computer company for over 35 years, the last 20+ in mainframe system support. And many of those years as a "team lead" (ie. non manager). We once had an external customer demand to know who screwed up a change that created some downtime outside the normal change window, as they were going to "personally see that person get fired." My answer was the team did it, and if you want to blame anyone, as team lead, then blame me. Needless to say, I did not get fired, and that customer wasn't to happy that he was stood up to, but he learned to deal with the disappointment.
-- david
Can we trust that the site will stay up between the 26th and 30th of September? I am sure that I am not the only one who wants to include some September release PDFs in that special order.
| 7 people marked this as a favorite. |
I must admit, while it's a lovely gesture and wholeheartedly appreciated, I would personally gladly trade the discount in for a general explanation as to what has been happening.
Right now, because I have no idea what's happening, I'm assuming that the site is going to be down again soon and backing information up on characters and such stored here. I have no reason to assume that the site won't be down, as it's been down without warning at least once after this message was posted and I have no idea what's happening.
I'm worried about continuing games or joining new ones, as I don't know if this is the kind of issue that might see a days posts lost or even the loss of whole games.
I'm even a little worried about my personal date, if this is some kind of attack on Paizo's website. Is it possible I should be looking at the cards I have saved in the store in case someone has managed to get that information? Am I at risk here? I don't know. I can only trust that Paizo would tell us if that was even a potential problem.
I love you guys, and while I have no desire to see someone named and shamed, a really general idea of what the hiccups are and any potential fallout for us users would be really appreciated.
|
Chief Technical Officer
|
| 7 people marked this as a favorite. |
It would be inappropriate to air the specifics of our recent issues in public, but rest assured that data security has not been affected.
I would like to point out that we have an open job posting for a System Administrator, so if you know a qualified candidate, please encourage them to contact us.
Also, the unplanned downtime that we've had these past two days is not the additional planned short outage that I referred to in my initial post. That's still going to happen in the coming weeks; again, we'll provide advance notice of that on the front page of paizo.com and on social media.
Thank you for that assurance, at the very least. It's good to know my data is safe.
Would you be able to say if we can expect more random downtime until whatever issue is resolved?
|
...we're offering you 10% off...does not apply to subscriptions...
So, for people who subscribe to the certain products they are interested in, and the discount does not apply to subscriptions, in order to use it, we need to cancel our subscriptions, order the products to receive the discount, and then resubscribe after the order is complete? Seems like an unnecessarily inconvenient hurdle to jump through for your most dedicated customers.
| 1 person marked this as a favorite. |
Also, I'd like to point out, sometimes things happen. Sometimes hardware goes bad. Sometimes something unexpected happens. They work really hard and they do good work. Maybe don't assume the worst.
Yup. That is life in the age of technology. Heck, it was life before the age of technology as well. Everything will work out. :)
..we're offering you 10% off...does not apply to subscriptions...
So if I'm already subscribed to the products that I'm actually interested in, I get nothing, AND those things that I am interested in have been very delayed in being shipped.
It just feels weird, yeah? Especially since I originally couldn't read this post when I saw the Reddit post about it.....because the site was down for another couple days.
The lack of transparency really makes me angry. No knowing what's going on made me delete my payment methods now. Not sure if that's doing a lot of good to protect it in case of hacking ....
But it's the only thing I could think of right now.
| 8 people marked this as a favorite. |
They've already said that data security wasn't breached. I'm not sure what transparency that people believe they are entitled to. Stuff broke down. There was a problem and they are working to fix it. It is an inconvenience and that sucks, but outside of that it doesn't appear that people are being unduly affected.
If nothing else, it gives the forums time to cool off.
| 5 people marked this as a favorite. |
If nothing else, it gives the forums time to cool off.
Optimist. :p
Man, this has just be a crap month for you guys... hope everybody's hanging in there.
| 2 people marked this as a favorite. |
How many more outages should we expect on a daily basis?
Thank you for the update and gift!
They've already said that data security wasn't breached. I'm not sure what transparency that people believe they are entitled to. Stuff broke down. There was a problem and they are working to fix it. It is an inconvenience and that sucks, but outside of that it doesn't appear that people are being unduly affected.
If nothing else, it gives the forums time to cool off.
I don’t believe that I’m entitled to anything. This is a trust issue rather than an entitlement issue. Without a public postmortem on what went wrong, I cannot trust that the issues are well and truly resolved and that appropriate lessons were learned and measures being put into place to help prevent it from happening again. This is unlikely to effect any of my interactions with Paizo yet, but it may drive other people away from the forums to competing services for things like PbP or even purchasing product in general. But, if we get another week of downtime with nary a word besides “trust us”, at that point, I would no longer trust Paizo with their technical platform and will begin cutting ties as a result.
I understand that making a public postmortem is difficult as it requires being more transparent about underlying technology used and/or business processes. It does not require naming any names, and nor should you unless you’re naming people who went above and beyond their duty and deserve some praise and recognition for that. I encourage Paizo to look at postmortems posted by other companies for downtime and severe issues to get a feel for what they’re like and to spend some time drafting one themselves. Here is one such example.
Advance notice of this downtime will be posted to the front page of paizo.com and Paizo's official social media channels.
If the error page, or the page it sends you to, were to indicate the nature of the outage, it would be nice. As I generally do not hit the main page, I never see the announcement, and so am in the dark [without blindsight].
... discount ... does stack ...
Yeah! Stacking bonus.
/cevah
| 1 person marked this as a favorite. |
we'll provide advance notice of that on the front page of paizo.com and on social media.
Would it be possible to also post it as a little notice at the top of the page, as when the Paizo staff are off for a holiday? I typically go right for the messageboard and rarely see Paizo.com's front page (and one never knows what Facebook's actually going to show on one's feed). Thanks!
| 5 people marked this as a favorite. |
They've already said that data security wasn't breached. I'm not sure what transparency that people believe they are entitled to. Stuff broke down. There was a problem and they are working to fix it. It is an inconvenience and that sucks, but outside of that it doesn't appear that people are being unduly affected.
If nothing else, it gives the forums time to cool off.
I don’t believe that I’m entitled to anything. This is a trust issue rather than an entitlement issue. Without a public postmortem on what went wrong, I cannot trust that the issues are well and truly resolved and that appropriate lessons were learned and measures being put into place to help prevent it from happening again. This is unlikely to effect any of my interactions with Paizo yet, but it may drive other people away from the forums to competing services for things like PbP or even purchasing product in general. But, if we get another week of downtime with nary a word besides “trust us”, at that point, I would no longer trust Paizo with their technical platform and will begin cutting ties as a result.
I understand that making a public postmortem is difficult as it requires being more transparent about underlying technology used and/or business processes. It does not require naming any names, and nor should you unless you’re naming people who went above and beyond their duty and deserve some praise and recognition for that. I encourage Paizo to look at postmortems posted by other companies for downtime and severe issues to get a feel for what they’re like and to spend some time drafting one themselves. Here is one such example.
Outside of people really interested in the technical aspects what purpose does a postmortem provide? If you do not trust that when they say the problem is fixed that it is fixed then I doubt a breakdown of what happened is going to provide any additional assurance.
Things break down. Things happen. We don't need a multi-paragraph explanation of what, when, how and why. I'd rather they fix it and prevent it from happening and move on from it rather than explaining "what lessons were learned".
| 6 people marked this as a favorite. |
I encourage Paizo to look at postmortems posted by other companies for downtime and severe issues to get a feel for what they’re like and to spend some time drafting one themselves. Here is one such example.
Paizo is not selling its software services. It has no need to make people feel better about the quality of its software services beyond "your data are safe and there was no breach". So your example is irrelevant.
A privately held company has no need to discuss its business systems and software set ups with anyone but the owners. They don't even have stockholders that they need to inform.
I'm sure they are at least as anxious as this audience to get things straightened out and working smoothly. We don't need to know the details in order to decide whether or not we'll keep buying their *actual* product. Games, game aids, stuff like that.
| 2 people marked this as a favorite. |
My example was of a postmortem in general; the exact sphere of business is largely irrelevant. You can find postmortems from other companies who are not selling software as well.
And, I do need to know whether or not I can trust their technological platform in order to continue interacting with it. If the site continues to be down or flaky for long periods of time starting from whenever they do their next maintenance, I can no longer trust them and as a result will drop my subscriptions and no longer purchase PDFs. I might pick up a book or two from my FLGS, but this will result in a net loss of income for paizo compared to what I currently spend. Not enough to matter if I'm the only one that does it, but enough to matter if enough other people think the same way I do.
There is usually no good reason to withhold details for things with high customer impact. It takes time to write it, sure, but that is time well spent as a means of introspection into what happened and what went wrong, and how they can be better moving forward. It doesn't even need to be as long as the thing I originally linked; 4-5 paragraphs would do. This would maybe take half a day to collect the info for it, write it, and publish it. As I mentioned in my previous post, being transparent increases trust. Being opaque makes people suspect that you're trying to hide something.
| 5 people marked this as a favorite. |
My example was of a postmortem in general; the exact sphere of business is largely irrelevant.
Actually, the root cause analysis is largely irrelevant. Vic has said that data security wasn't the issue. That leaves hardware issues and software bugs.
I'm almost positive Paizo's on some clustered solution though I don't know for sure there aren't single point-of-failure hardware items involved. Still, if a switch, or PDU, or blade server, or router failed intermittently, it isn't a matter of trust.
As for software, again, it doesn't matter if it's a random div by zero error, or input-validation failure that defaults with an abnormal end, or an SQL table that allocated 3 bits to a column and it hit 8, or if someone coded the forum to assume nobody would ever use the string "goblin" more than seventeen times in a single post, or an issue in the PDF stamping process that couldn't handle a filename some 3rd-party seller provided. It's again, not a matter of trust.
The only trust violation here is with data integrity. Otherwise it's an availability issue only, and one regarding non-mission-critical services. It's not life-or-death. It's not ambulance dispatch. It's a webstore that experienced (extended) down-time.
The fact that it failed down instead of open is all I really need* to continue to trust Paizo with my money and my data. If it's not data security, it's not our business. I don't need to know it was a disgruntled intern who threw a time-bomb in the code, or a vendor with a screwed-up billing system who thought Paizo hadn't paid up their datacenter bill, or an epileptic coder who had a seizure on the job and in the rush to get them medical assistance their code got checked-in with a serious issue in it. Not our business if Vic says so.
*Though as an IT guy morbid curiosity and sympathy is a thing.
| 1 person marked this as a favorite. |
I really need to poke around more, I'm just now finding out about this!
Although it's not needed, this is why I love Paizo and will continue to support them no matter what.
Thank you!
I am glad I randomly clicked into this section of the forum and found out about this, I just ordered some pawns and bases I had been meaning to order for ages! It is good to be able to look forward to a parcel.