Experience the difference of "Elite".

About the Dec 26th Outage

  • Posted on
  • By

About the Dec 26th Outage

  • Posted on
  • By

On December 26th, OwnerRez experienced a database corruption issue with our cloud service provider.

We saw the issue right away and started working on it, but there was a domino effect that grew significantly beyond our initial expectations.  Our engineering staff has been recalled from vacation, connecting in from overseas, etc. to diagnose and resolve the problems as quickly as possible.

At present, there are no indications of any hostile activity or attacks, and no reason to believe that any previously gathered data has been lost. We are currently operating on a restored system with up-to-date database backups that occurred at the same time the issue started to develop.

We will be posting information and analysis (here in this same blog post), and holding a public webinar, as soon as we have a more thorough understanding of what went wrong and future preventive actions.

In the meantime, here is a list of common questions that have been coming in.  This list will grow over time as we add more.

Why did it happen? What caused it?

We are still investigating all the details, and there are puzzling issues we still don't fully understand, but what we can say is this...

This past Saturday, as part of a routine update, we upgraded some database servers with a new configuration that we had used extensively for many months.  The configuration was a very routine thing and, again, something that our engineers have used in their own environments (and elsewhere in testing/staging) for a while.  There was no reason to believe the update or new configuration would cause anything out of the ordinary.

From Saturday until Monday, the new configuration started to corrupt data.  We're not sure exactly when or why (this is the part still being investigated) but several database areas began to be unresponsive.  The entire platform started to go down on Monday, and our remediation/failover processes started to kick in, but there was a domino effect, and the remediation/failover processes were affected similarly as the system bounced from up to down and back again.

The investigation into the bad database configuration is ongoing.  By all accounts, it should not have happened.  Our cloud provider has service guarantees with us, so we are working with them on the investigation.

Were others affected, is this a widespread issue?

No, not that we know of.  This is isolated to something we did (updating a database configuration) to our database servers with our cloud provider.  But the obvious question is - what about other businesses that used the same database configuration?

Yes, indeed - we are wondering the same thing.  We purposefully watch to see if software updates by third parties are stable before upgrading our server configurations.  In fact, we purposefully waited on this database configuration (from back in March) because there were some known issues with it affecting servers.  We skipped over it and used a different configuration that was reported to be stable and in use by others, and we used that configuration in test/stage environments for several months before moving forward.  As mentioned above, we are very puzzled by why it happened and by all accounts, it should not have.

(And no, the Southwest Airlines debacle is not related to this. At least, we highly doubt it.)

Was any data lost? Doesn't restoring a backup mean some recent data was lost?

The short answer is: no lost data, but we are still investigating and there may be discrepancies with third parties.

The reason we say "no lost data" is because we have data redundancies in place that save data to multiple places at the same time.  When our primary systems go down, we still have access to other systems.  When we rebuilt and restored the underlying configurations, we took data from secondary systems that were up to date as of the time the issue happened.

After the recovery period finished, we went back and compared the new data with the corrupted data.  There were no discrepancies in terms of missing records.  In other words, no records were showing in the old corrupted version that didn't exist in the new recovered version.

That being said, there are many channels and 3rd parties that we connect with that also send data back and forth and could have "sent" or "received" some data that never got through during the outage.  So a channel or 3rd party might show something different than what OwnerRez does, which means the data is incorrect on one side or another (or both).  Messaging could be another example of this.  Gmail and other messaging services watch volume and throttle accordingly.  When the final recovery effort fully kicked in, a large firehose of queued messaging went out (eg. from you to guests, or from OR to you).  Gmail may have throttled or dropped some of that due to volume concerns on their side.  We have not seen signs of that, but it is possible.

Also, there were intermittent "up" periods, early in the recovering period, when you (or a third party) may have changed data that ultimately did not get saved when we did the final recovery effort.  So there too, data may be missing.

Was any sensitive data (credit card, phone number, PII) lost or taken?

No. This was not the result of a security breach nor any outside influence. Nothing was hacked or stolen.  This was a database configuration issue where a seemingly-insignificant update led to data corruption and then domino'd elsewhere as our fail-over processes started responding.

Don't you have redundancies in place to prevent this from happening?

Yes, many.  Our engineering team spends a lot of time (and on an ongoing regular basis) managing and analyzing our infrastructure.  We have lots of tooling and logging in place that gives us visibility into errors, slow responses, spikes, partner issues, background queues, and much more.  OwnerRez is large and full of many moving parts, and in each of those moving parts, we store data where it can be replicated and archived stably as it grows.

Yesterday, we saw the issue immediately and our fail-over process kicked into gear, but the database configuration affected every remediation effort that swung into place.  We were forced to pause everything completely, get to the root of the problem, and rebuild configurations before turning the firehose back on.

The bad configuration was, itself, something we tested and used extensively for months beforehand, so we are still puzzled by (and investigating) why it happened.  But we made the decision to stop the failover/automation and roll back to a configuration that we thought would make a bigger impact.  We knew it would take a couple of hours, but we believed (and still do) that it was a better trade-off for a return to stability rather than fight corrupt data in rolling cycles for what could be many days.

But we clearly have work to do. We do not want to shift blame or throw up our hands here.  We are committed to being your elite PMS and channel manager and that comes with certain expectations on your part and responsibilities on our part.  We are taking steps to learn from this and incorporate new processes for additional redundancies and better communication.  Some stark problems became very clear over the past 36 hours.

Does Airbnb automatically reactivate property listings, or must that be done manually?

During the outage, you may have received automatic emails from Airbnb stating that your listings were disabled or hidden because your software (ie. OwnerRez) was offline.  Airbnb does this as a safety precaution so that guests are not double-booking your property since Airbnb has no way of confirming with you (ie. via OwnerRez) that everything is okay.

Post-outage, Airbnb should have reactivated any hidden listings automatically, but you should check your listings and not assume this is the case.  When our system started communicating with Airbnb again, we confirmed that they were turning listings back on again by checking with specific listings and seeing that they were visible.  However, do not trust that this is the case across the board.  Check your listings so that you know for sure.  Our helpdesk has reported that some users are still seeing hidden listings.  That could be for specific reasons that don't apply to you, but everyone should still check that their listings are showing and the calendars are open and bookable.  We apologize for the inconvenience this places on you.

If you did not receive any "hidden listings" email from Airbnb, you should be fine.  Airbnb only hides listings if they can't send bookings through, not for other reasons.  However, if you're worried about it, we recommend that you still verify your listings on Airbnb to make sure they are showing and bookable.  It could be that it happened to you, but you didn't see the email or an email was never sent.

During the outage, I disconnected Airbnb. Now what?

During the outage, some users were recommending to each other to go into Airbnb and manually disconnect OwnerRez from the Airbnb side so that they could return to "platform mode" and deal with guests and bookings directly on Airbnb.  If you did this, there is no way to undo it from the Airbnb side.  You need to go into OwnerRez > Settings > API > Airbnb and reconnect the account from the OwnerRez side.  This is safe to do, and your previous connection will reactivate and merge in changes safely.

I did a full resync for Airbnb, but nothing changed.

Triggering a "full resync" for Airbnb does not happen immediately in OwnerRez, and it never has.  The request for the resync is put into the queue, along with many other sync actions, and is pulled out as the queue processes every few seconds.  Typically, in normal conditions, the queue is processed so quickly it can appear to be immediate, but it always follows the queue.

After the outage, the queues were filled with many waiting requests, so everything took time to process.  If you were clicking the "trigger full resync" option on the channel, it was put into the queue behind other things and probably took a while to process.

Post outage, we have been monitoring all of our queues to watch many types of requests process - bookings, channel syncs, messaging, and so on. If you are waiting for a message to be delivered or channel sync to go through, make sure to give it some time to catch up.  If it's been a while (ie. an hour) reach out to helpdesk, and we'll take a look.

Where are my Booking.com bookings? They aren't showing up on my calendar.

Unlike some of our other channel partners, Booking.com does not provide an automatic method to restore lost or missed data, including booking records. Our team does have access to logs and is manually working through that information to find affected accounts and properly update calendars. This should be completed by the end of today, but, note that not all booking information can be restored. However, the calendar data will be correctly entered, preventing any double bookings.

OwnerRez says my Airbnb properties are fully synced and online, but, I still can't see them on Airbnb.

We are seeing a relative handful of this situation - as in, a few dozen properties out of tens of thousands. It appears to be individual properties affected - that is, you might have 10 Airbnb properties in one API-connected account, 9 are working fine, but 1 refuses to come back online no matter what you do. We've tried several things to get those properties working again and haven't had success either - so, we've filed a Critical bug with Airbnb partner support. If you have properties in this condition and have not already reported it to us, either via the Helpdesk or in this thread, please do so - we'll add your information to the Airbnb bug report and track them all together.

**UPDATE** - All but a couple of the offline Airbnb properties have been restored, and those are expected to be fixed by the end of the day.

121 Comments (add yours)

Dec 27, 2022 10:04 AM
Rich S says:

Please share thanks to the team for abandoning their holiday to get the problems fixed!

Dec 27, 2022 10:05 AM
ISLAND VACATION says:

Greetings!  We lost a connection to Resort Cleaning which resulted in 2 scheduled cleanings being deleted for today.

Dec 27, 2022 10:05 AM
XENOS Guest-Frie says:

Thank you for the update and working to correct the issues

Dec 27, 2022 10:06 AM
Tufan B says:

My booking.com booking that happened during outage, did not synch. I did a API resynch and it still is not showing up

Dec 27, 2022 10:07 AM
Camelle R says:

Thanks for all your hard work!

Dec 27, 2022 10:08 AM
Aunger Vacation says:

Thanks very much… good efforts getting things back to normal. Appreciate it. When was the last know back up from date wise? 

Dec 27, 2022 10:12 AM
Julie R says:

Thank you for the explanation and transparency. Cheers! 

Dec 27, 2022 10:16 AM
Lisa S says:

Thanks for the transparency and for being on top of this!

Dec 27, 2022 10:18 AM
Barbara P says:

Thanks for being so proactive and committed to excellent customer service. Appreciate the sacrifice of all employees. 

God bless,

Barbara 

Dec 27, 2022 10:19 AM
Carol G says:

Thank you for addressing the issue during the holiday. It is much appreciated. Things like this happen. 

Dec 27, 2022 10:20 AM
Fenwick Vacation says:

We have a booking missing from the calendar.

Dec 27, 2022 10:24 AM
Wayne D says:

Airbnb suspended some of my listing. Do I need to reach out to Airbnb or will they be automatically restored?

Dec 27, 2022 10:29 AM
Johnny B says:

Our calendar allowed us to accept two bookings on the same dates and double book

Dec 27, 2022 10:39 AM
KAY-FAR says:

Thanks for getting us up and running.  This was scary for me as a new user of your platform and this left me realizing just how vulnerable I am should your systems go down.  They directly impact my business and while I do know that things happen, hopefully steps will be taken to ensure that we have a path forward for quick recovery should we ever be phased with a similar situation.  Life is about learning from new experiences and this was a new one as it seems for OwnerRez as well as those using your platform.  Looking forward to the public webinar where I hope you will discuss steps put in place to help us all pivot quickly.  Thanks again to the team for your proactive response.

Dec 27, 2022 10:41 AM
Julie B says:

Thanks guys!! I know last night was rough for you. You are awesome. Did the auto triggers email after the outages and catch up or should we manually send emails that may have been missed?

Dec 27, 2022 10:42 AM
KAY-FAR says:

"faced not phased"  sorry :)

Dec 27, 2022 10:44 AM
Ken T says:

Missed email triggers should have been sent, albeit late - you can confirm this in your Communication History.  As noted, you can manually resend any you're not sure of in the usual ways, but we do not believe this to be generally necessary.

Dec 27, 2022 10:44 AM
David A says:

Thanks for rolling up your sleeves and dealing with an emergent database issue in the middle of vacation. Appreciate the hard work under pressure. Hope you get some downtime now. 

Dec 27, 2022 10:48 AM
Bill and Donna B says:

Thank you for your communication and transparency.   It is very much appreciated.

Dec 27, 2022 10:49 AM
Michele W says:

As a proactive precaution. Can we disconnect our APis when Or goes down so that bookings will not get missed? AIrbnb took our properties offline during the outage. For example, can I disconnect my Airbnb and VRBO APIs then reconnect them later and all be as was prior to the disconnect? 

Dec 27, 2022 10:50 AM
Matthew H says:

I've had several years of rock solid performance from OwnerRez, so I continue to have great confidence in you.

Dec 27, 2022 10:50 AM
Ken T says:

You definitely *can not* disconnect and reconnect the Vrbo API on the fly - don't attempt that.

Regarding Airbnb, we have seen some reports of clients using this approach successfully, but have not yet confirmed that recovery from that action is smooth so are not recommending that at this time.  It does seem plausible though, and we'll know for sure in a bit.

Dec 27, 2022 10:52 AM
Ash says:

I understand things happen, BUT....

1. Will your staff be able to track "lost reservation data" and insert back into correct places

2. Was credit card data breached, along with telephone numbers and addresses

3. What can we do to back track and check reservations past, present and future to ensure all is good. Reaching out to guests and informing them of issues will not engender sympathy but distrust

4. I trust that you are taking steps to ensure this does not happen again. What steps to a non techie like me are you taking?

Appreciate everything your guys do. Thank You!

Dec 27, 2022 10:52 AM
Ken T says:

@Wayne - It does look like Airbnb automatically reactivated your property listings once service was restored.  Our belief is that this should occur for everyone, so, you should not need to take any manual action to take that happen.

Dec 27, 2022 10:53 AM
JAC Rentals says:

Thanks to the tech guys doing their thing.  

Dec 27, 2022 10:53 AM
Bill and Donna B says:

-

Dec 27, 2022 10:54 AM
Jeremy O says:

Losing access to our calendar for that long was very problematic. We had several check-ins with no idea of who would be arriving and how many people they were bringing. We also missed out on multiple bookings. Potential guests called us to inquire about booking and we weren't able to book them or even to tell if our properties were available. The winter holidays are a very busy time for the hospitality industry; we wouldn't be as annoyed if you weren't doubling your rates. 

Dec 27, 2022 10:55 AM
RNLH says:

thanks! Better customer service than Southwest!! That is a cluster melt down!

Dec 27, 2022 11:05 AM
Kylie R says:

Our listings are all still hidden on Airbnb. Can we do anything to get back up and running?

Dec 27, 2022 11:07 AM
BlueSky Getwaway says:

Please consider away that we can easily do a daily, weekly, monthly database backup ourselves for the future. 

Dec 27, 2022 11:08 AM
Oumou says:
Airbnb
Hello Omou,
We have noticed that the availabilities of one or more of your listings have not been updated. Notably, a high percentage of booking attempts for dates marked as available in your listing's calendar were declined by your software. As a result, your listings will be hidden from travelers until you resolve the issue. You will not receive bookings while listings are hidden.
The hidden ads are as follows:
Apt 1 bedroom-living room-equipped kitchen-wifi fiber s
Please contact your software provider to ensure that your availability calendar on Airbnb remains up to date. We will monitor your ads daily. As soon as your availabilities are updated, they will automatically be visible to travelers again.
Thanks,
The Airbnb team
Dec 27, 2022 11:09 AM
Evelyn E says:

What is the best way to back up all our information on a local HD?  Is something in place that we can set up so that we can at least carry on business even if it requires adding the new information to the OR platform after it is back up?  I would really appreciate guidance here.  Thank you.  

Dec 27, 2022 11:10 AM
Ken T says:

There's no need to do a full database backup yourself, but, you can certainly export all your contacts and bookings via several different methods.  We'll likely be adding more convenient or even automated ways to get this done, but for now, the simplest method is the main Bookings screen, List view, Export.  This will include all future bookings, including guest contact information.

Dec 27, 2022 11:15 AM
Giani C says:

Thank you so much

Dec 27, 2022 11:19 AM
Adrian M says:

I lost a lot of bookings from this and I still have several listing that are hidden in AirBnB so it's not possible for guests to find or book them

Dec 27, 2022 11:20 AM
Maged says:

Thanks for your hard work with fixing the issue during the holidays. OR has been more than rock dependable for me for over five years now and the team has always been extraordinary. Things happen sometimes especially when dealing with multiple external factors. Continue the good work. You guys are awesome!

Dec 27, 2022 11:21 AM
Evelyn E says:

Thank you for that info.  I appreciate the speed you have been able to solve the issue and get us back up.  It is comforting that you were so ON IT even during the holiday when everyone expected to be off work.  Thank you for your commitment to all of us who depend on you so much.  Blessings for the New Year to each of you and your families.  

Dec 27, 2022 11:22 AM
OPCD TX says:

Thanks guys! 

Dec 27, 2022 11:24 AM
Blake A says:

I have a booking from VRBO that is not showing up on OwnerRez, any ideas on getting it synced? 

Dec 27, 2022 11:25 AM
Adrian M says:

stop receiving updates

Dec 27, 2022 11:26 AM
Chip & Miao W says:

Thank you for your teams effort in getting everything back up and running.  I am sure they would have rathrr bern celebrating the holidays.

Dec 27, 2022 11:32 AM
Derrick A says:

API is also throwing error 

Dec 27, 2022 11:35 AM
Shelley S says:

I appreciate your communication during this period and understand things happen. I DON'T understand how a secondary back up server could also go down.

Unfortunately the timing resulted in me losing TWO bookings  - two people trying to book yesterday (one through my API integrated VRBO listing, couldn't book since OR was down) and one directly. 

Resulted in lost bookings and revenue for me. Very disappointing.

Dec 27, 2022 11:43 AM
Mark M says:

It's certainly frustrating on our end when we need to adjust a security deposit before it gets released but understand bad things happen in the digital world. The question I have is what exactly happened to initiate such a catastrophic outage?

Dec 27, 2022 11:50 AM
Heather L says:

Grace Vacation Rentals Maui greatly appreciate the attention paid to the issue.  

Dec 27, 2022 11:57 AM
Debra S says:

Thank you so much for working quickly to resolve the issue!  The only affect I seem to have had is with AirBnB who disabled my listing due to a calendar sync issue.  Other platforms seem to be working just fine and all are sync'd.  

Dec 27, 2022 11:59 AM
Shelley S says:

turning off notifications

Dec 27, 2022 12:02 PM
Austin D says:

I understand tech issues happen and I do appreciate your candor and explanation here.  That said, we lost three potential bookings and had airbnb suspend several of our listings.  Stuff happens and I'm not here to pile on, but this is pretty concerning.  I look forward to hearing specifically what steps will be made to try and prevent this from happening again.  Meanwhile, many of us lost revenue because of this.  I'm not asking for anything from OR -- again, I do understand stuff happens -- but maybe keep it in mind next time you plan a price increase for hosts.  Kinda hard to ask us to pay more for your services after something like this happens.  

Dec 27, 2022 12:10 PM
Samantha H says:

Thanks so much for this quick response and the transparency! One of our listings (Bissell House) is also still missing from Airbnb. Any advice on this?

Dec 27, 2022 12:11 PM
Lisa S says:

t

Dec 27, 2022 12:11 PM
Nick F says:

We had two listings on Airbnb get "hidden" because bookings did not process during the outage. I spoke with Airbnb this morning to "unhide" the listings and they said this is still an issue with OwnerRez?

Dec 27, 2022 12:14 PM
Ken T says:

We are running a full resync with Airbnb which should get all properties back online without any manual action, unless you took manual action to disconnect them yourself previously.  This process is going slower than expected, but is over half done and should be complete by the end of the day.  Please do not attempt to request a manual resync, as that will only delay the process.

Dec 27, 2022 12:21 PM
Kismet Prop says:

I'm sure it was rough night. Very unusual for this system to be done.  In fact, I have never experienced it.  Thanks for working through the day and night to get it back up and running!

Dec 27, 2022 12:25 PM
Mat W says:

My AirBnB listing has been suspended due this issue. I have several inquiries that are unable to book due to this issue. I’m having to submit an appeal with AirBnB to get my property back to an active status. I’m actively losing money because of this issue. 

Disappointed with the lack of proactive information being provided. OwnerRez users are left to find out that their listing have been suspended at no fault of their own? This product should meant to make STR management easier! Not suspend my listing! Any downtime for this product is unacceptable. 

Based on your comments, it sounds like like this was caused not by an outside bad actor or malicious activity, but by an internal change. I’m curious to see the full report, but assuming this issue was self-inflicted, what is OwnerRez doing to make this right? 

Dec 27, 2022 12:28 PM
Steve K says:

Still having synch issues with 3rd parties via API.   Airbnb messaging that occurred during outage has not come through.  

Professional Liability and Errors/Omissions policies on behalf of cloud hosting provider and OwnerRez are designed to handle these types of issues.  Plus as hosts and businness owners you have the ability to purchase insurance like Loss of Business policies to aid in these situations.

Considering backup solution in the future it would be best for it not to be in the same data center and cloud hosting provider.  Sometimes a hot backup can be co-located but you run the risk of what happened yesterday when the single point of failure is related to hosting provider.  Then you do not have redundancy real redundancy.  You may want to consider different cloud provider as a hot backup options to ensure true redundancy.  Also you should have a date time stamp on the file you used for restoral to get everyone backup up and running.  It would be helpful to provide that date time stamp with appropriate time zone so that we can go through our systems to help identify gaps in our data and try to complete restoral.

 

Dec 27, 2022 12:52 PM
Alana A says:

Our discounts are no longer working, particularly the Snowbird monthly discount.

Dec 27, 2022 12:59 PM
Bria says:

We lost several high-dollar booking opportunities. Many guests were confused during check-in/out as automated instructions were not sent, which could impact reviews. We spent a long time on the phone with support form Airbnb to Vrbo due to outage-related issues. While the fix is appreciated, it resulted in thousands of dollars worth of lost opportunities as booking inquiries / requests ended up booking elsewhere. It will take more hours going through calendars and messages to look for sync errors and unsent communications. This outage and the price increase of 2023 has soured our experience with this software. Please ensure an outage does not happen again by installing better protections and using backup servers.

Dec 27, 2022 1:02 PM
Fenwick Vacation says:

Completely agree with the last posts.  Bad experience for guests and lists revenues.  Now looking through everything making sure things are. It missing.  Now I am wondering if I need to always have a back up.  

Also poor communication from ownerrez on the outage and also what we as owners could do.  Say stop instant booking?

Dec 27, 2022 1:03 PM
Fatfatmouse says:

Thank you for acting quickly and resolving the issue. When I couldn’t log in, I panicked! Especially for this holiday peak season. But your quick response made me relieved and I had faith on your engineering team. A day later, all up and running - thank you!

Dec 27, 2022 1:06 PM
Gwen S says:

I'm seeing now how important all of our notifications are that land in the inbox of my business email, truly grateful I was able to find information there for all of our 20+ properties that I needed regarding lease agreements, messages and data I typically go straight to OwnerRez for. The system is well thought out for just such emergencies. Thanks for getting on this as quickly as you did on our second busiest day of the year, fingers crossed December 31 is better for all of us!

Dec 27, 2022 1:40 PM
CWV says:

A big thank you to OR staff and especially those who were called unexpectedly to work during their time off.

Dec 27, 2022 1:42 PM
Ruterra Apartmen says:

We're currently receiving overbookings for new years eve...

 

Dec 27, 2022 1:42 PM
Erika F says:

Oh boy. As someone who has run websites for years, I understand how frustrating and stressful this can be.

I got really lucky. A booking sneaked in and took care of their contract and security deposit during a brief interval when the site was back up. It was a last minute booking so I was really worried but it all worked out and they are safely checked in. 

Hope they are able to diagnose whatever happened for you. Happy holidays.

Dec 27, 2022 1:47 PM
Holly W says:

I am happy about the communication I am seeing here from Owner Rez. To all the owners who had "no access to their calendar" during this time- this is a highlight for you. You are responsible for your own system of backup for your calendar. Yes, Owner Rez is our "master calendar" but all businesses need to have backup systems in place. Just like I have a generator here for my home/business (and vacation rental) should the local power company "go down", I have a complete calendar for all properties, offline, at all times. Owner Rez will do their fact finding and possibly implement additional measures if needed, but bottom line is outages occur, and are not always the fault of the provider (example major widespread natural disaster) so its a good time to put your own backup measures in place. For me, this is as simple as 1) maintaining a log of all guests, 2) a copy of all contracts after signature (gives me contact info, I download the PDF to a file on my computer, which yes, is backed up - in two places) and 3) an actual "calendar" of all guests in a paper or PDF version.

Dec 27, 2022 2:11 PM
Angelo G says:

My Booking.com reservations from yesterday are still not showing in the system. What should I do?

Dec 27, 2022 2:13 PM
Ken T says:

We are aware of the issue with dropped Booking.com bookings and are investigating as to the best resolution.

Dec 27, 2022 2:20 PM
Lisa T says:

Love the OR crew!  Thanks for working through all the mirad of cascading issues and for taking time away from your vacation to set things right.  Stuff happens its what you do when the _____ hits that fan that matters.  Great example for all of us OR.  Kuddos! 

Dec 27, 2022 2:25 PM
Ruterra Apartmen says:

Wish you all the best 

 

Please fix the problem with booking.com channel and be aware of it, since we've received overbookings from that channel today

Dec 27, 2022 2:46 PM
Randy H says:

While I cannot deny this was a huge inconvenience I have worked in I.T. for over 40 years and all I can say to those who feel they were let down all I can say is, SHIT HAPPENS.

Sometimes no amount of redundancy will keep the enterprise up and running during a failure however restoring data back in time to the exact point of failure is not difficult though time consuming.  

With that said thanks for keeping us informed throughout the day.

 

Dec 27, 2022 2:52 PM
Paul W says:

Hi everyone, we updated this blog post with a FAQ section.  Please refresh and give it a read.  We covered most of the questions that have come in the most, and we'll be adding more over the next 48 hours.

Dec 27, 2022 2:57 PM
Ken T says:

A fix for the Booking.com async issue is under way and should be completed today.  More information will be added to the main post soon.

Dec 27, 2022 3:29 PM
Angela R says:

Hi ken! Mine finally shows synced, but my Airbnb is still hidden . What can we do to have our Airbnb gets restored ? This is really making us lose on bookings 

Dec 27, 2022 3:34 PM
Ken T says:

@Angela I see what you mean, we are investigating.

Dec 27, 2022 3:42 PM
Paul W says:

Two more questions have been added, so please refresh again and read.  Booking.com and triggering full resyncs for Airbnb.

Dec 27, 2022 3:50 PM
Brittany H says:

This is what Airbnb has said to me
"Even when the information looks correct in your software provider, changes to your account still need to be made through your software and not on Airbnb. We can only pull the information we’re shown by your software.

If you'd like assistance making updates, the best thing to do is reach out to your software provider. They're in the best position to help. Often, a simple re-sync of your software can fix the issue.I will be closing this message thread now, and I hope that is okay with you. We are here with you all throughout your Airbnb journey so if ever you need assistance or have any questions, please don't hesitate to reach out to us. We will be happy to help you."

 

Do we resync or just continue waiting it out?

Dec 27, 2022 4:02 PM
Erika F says:

Today I synced my Airbnb several times but rate changes and season changes are not taking effect on Airbnb yet.

Dec 27, 2022 4:06 PM
Paul W says:

@Brittany Any action made in OwnerRez (changing a photo, updating a rate, changing availability) is already queued to update on the Airbnb side. It may not have happened yet because the channel queues are running a good hour or more behind, but they are queued to happen and will happen when it catches up.

Now, if you're worried about that or just want some extra confirmation, you can certainly trigger a full resync and OwnerRez will add a "full sync" of everything to the channel queue.  However, the full resync will wait in line behind everything else.

So what will happen is all of a sudden all of your stuff will update, both the waiting changes from before and the full resync.

Unless there's a significant and pressing issue, I would wait a bit longer and check later in the evening (or overnight, depending on where you're at)  to see if the listing is still off.  Check out the Airbnb channel dashboard in OwnerRez and the Sync Actions tab.  We have a lot of transparency there on when your OwnerRez side changed and when e last updated Airbnb, including specific categories of content.

Dec 27, 2022 4:18 PM
Kylie R says:

Our listings are showing synched with Airbnb as of 3 hours ago, and our listings are still hidden on Airbnb.

Dec 27, 2022 5:23 PM
Angela R says:

Ken it is Still not showing …. Anyone else ? Been almost 48 hrs since this ownerrez maintenance snafu started and 24 hrs since this issue triggered Airbnb to hide listings… is there any new update ? This is reallt costing us a lot of bookings missed … we have people who are wanting to book and can’t… I know issues happen and we understand that but the frustratingpart here is how long it is persisting considering how mission/time critical this service is to all the users . 

Dec 27, 2022 5:57 PM
Patricia Knight says:

Thank you to the OwnerRez team for your dedication to resolving this issue and communicating updates to everyone. Greatly appreciated!  Also shows how we all become dependent on other providers for our business tech stack. This definitely highlighted the need to at least do a weekly backup of our calendars and guest information just in case. Thankfully I had template messages saved elsewhere so I was able to manually send an email to a guest that booked last minute yesterday morning during all this chaos. 

Dec 27, 2022 6:02 PM
Nicki & Florian S says:

Thank you for your efforts in fixing this issue as quickly as possible!

I just tried to download AirBnB data through Channel Bridge, and it wouldn't connect. Does this have to do with this issue as well?

Thanks again for being on top of this and for the transparency - it is very much appreciated!

Florian

Dec 27, 2022 6:16 PM
Erika F says:

Even though it says synced with Airbnb and have done several times today, the new info is not showing up on Air

Dec 27, 2022 6:22 PM
Angela R says:

Hey flagging here … mine was synced about 2 hours ago. I just checked now and my Airbnb listing was completely disconnected from ownerrez and I had to add it again.

Dec 27, 2022 7:07 PM
Ryan H says:

I hate when tested updates cause problems for no apparent reason.  Seen that before in my corporate IT days.  Never fun.  Thanks for working to get this resolved.  Might I suggest adding an out-of-band communication method to notify users?  I found and checked your twitter, for example, but there was no message there about the outage.  Maybe having a third party system like that to notify users of the outage and attempts at remediation would be useful in the future?

 

Thanks to your operations teams for coming in on vacation to get the problem fixed!

Dec 27, 2022 7:34 PM
Go Rentals Colom says:

I checked on the airbnb account and my listing says "Activated" non-the-less when I sent the link to a guest trying to book, he says he can't opened. I called airbnb and they say the listing is not active  due to problems in OR. How can I just unlink it? it is a very important weekend and I am still with no bookings.... who can help me???

Dec 27, 2022 8:30 PM
Sandra G says:

Ownerrez did an amazing recovery and quick turn around. We so appreciate the dedication and talent of your team!

Dec 27, 2022 8:47 PM
Ken T says:

For those with Airbnb API-connected properties that aren't working for no obvious reason - where OwnerRez says the property is fully synced and online but you can't actually browse it on Airbnb - please see the FAQ above.

Dec 27, 2022 8:55 PM
BlueSky Getwaway says:

My listings are visible but I still can't get a booking that canceled yesterday during outagein Airbnb off my OR calendar and I'm losing two possible bookings for tmrw because of it. I've tried everything I know other than disconnect from Airbnb so I can cancel or delete it. I can't alter, move, cancel or delete this booking. Can someone please advise, I've had a guest waiting to book since 11am this morning. And no fix for this particular issue.

Dec 27, 2022 8:58 PM
Ken T says:

@BSG That sounds like a different issue.  Please write in to the Helpdesk with full details so we can take a look.

Dec 27, 2022 9:05 PM
Angela R says:

Ken my listing is still not up.a lot of people think theirs are up because when youre logged in as host, the listing shows fine, but when you check in incognito and open your link, it still showing the error..I appreciate everyone working hard... but 48 hours of outage with no resolve to 100% of the people is just really really disappointing.... we are losing revenue by the minute as post christmas is the busiest booking time for the following year. 

Dec 27, 2022 9:15 PM
Nicki & Florian S says:

.

Dec 27, 2022 9:33 PM
BlueSky Getwaway says:

Ken, I sent another email explaining the situation. It's blocking me from booking guests for tomorrow.

Dec 27, 2022 9:36 PM
Bria says:

As others also commented, we are also not able to see some of our Airbnb listings in incognito mode and guests are unable to book. They receive "unknown server errors" in the Airbnb app and get 404 missing page errors when trying to book our listings in a browser. Prospective Vrbo guests are getting strange error messages as well. This is occurring in spite of our listings appearing to be synced and active on the OR side. Airbnb and Vrbo support have not been helpful and say the issue is not coming from their end.

Dec 27, 2022 9:38 PM
Ken T says:

@BSG - Saw your ticket, and I believe it has been fixed.

@Bria - Please write in with specific details so we can investigate.

Dec 27, 2022 10:34 PM
Bria says:

Not sure if this will help anyone else, but we got Airbnb working by disconnecting and reconnecting (several times) the API connection for properties that were invisible. Now they are working. ¯\_(ツ)_/¯

Dec 27, 2022 10:42 PM
Sehrish K says:

Hi 

My property with OwnerRez is not visible online. On Ownerrez it says Active and Synced.

Dec 27, 2022 10:55 PM
Angela R says:

Hey Bria!

 

Did you just do that just now? Afraid of doing it as might cause it to get delayed so just checking in

Dec 27, 2022 10:58 PM
Bria says:

@Angela R, yes did it very recently. There doesn't appear to be any issues or delayed re-syncing. I cannot guarantee the same results for anyone else, but the alternative is waiting for Airbnb to fix it since the FAQ for this blog says the OwnerRez team was unable to do so.

Dec 28, 2022 4:40 AM
Competa Escapes says:

Thanks for the transparency - it is appreciated! Please pass on our thanks to the team for coming forward from their holidays to fix this. 

Dec 28, 2022 5:07 AM
Chris Hynes says:

@Angela, @Bria, @Serish -- we've heard back from Airbnb. What happens is that they send 10 test availability checks throughout the day and will reactivate the listing once those availability checks return available. 

I checked your listings just now and I am seeing them showing on Airbnb now.

Dec 28, 2022 7:08 AM
Angela R says:

Thanks Chris! I’m finally online and showing up. Thank for looking into it and giving an update . Appreciate all the efforts . Happy holidays! 

Dec 28, 2022 9:06 AM
Sandra G says:

.

Dec 28, 2022 11:00 AM
Ed M says:

It's technology.  Shit happens.  The best you can do is the best you can do.  We all know you wanted the system up as fast as we did.  Thanks for being on top of it.  Non-IT type people don't understand that even in redundant environments a database problem CAN replicate to redundant systems.  Just look at RAID puncture and you'll see that all the failover, fault tolerant replications in the world won't save you from it as the bad block iteself replicates its state.

  

Dec 28, 2022 11:23 AM
Val Rogers says:

I am thankful to the staff at OwnerRez, you all are the best! I'm so sorry this situation affected your team's holiday but am grateful for your dedication and support. I have full confidence in you and your handling of this situation and continue to champion and support your product and your team.

Dec 28, 2022 2:39 PM
Stacie S says:

Thank you to the entire crew for getting this resolved as quickly as possible and so sorry vacations had to be disrupted!

Dec 28, 2022 5:19 PM
Beach Resort Ser says:

Thank you and your entire team.  At first I was panic stricken but soon realized your team was on it and working hard to resolve the issues.  We have 100% faith and confidence in OR.  Wishing you many blessings and no hacks in 2023!  LOL!!

Dec 28, 2022 9:06 PM
Tony R says:

Thanks for sorting this out! I’m taking screen shots from now on every two or three days. Was the outage on Amazons side?

Dec 28, 2022 9:09 PM
Tony R says:

Taking screen shots of the monthly calendars.

Dec 28, 2022 11:41 PM
Yavuz says:

Thank you. We appreciate the effort to recover as well as the transparency, and time spent to write such detailed explanations.  

Dec 29, 2022 9:08 AM
Jonelle Kop-Agen says:

I still have one apartment, summer sea 230, where I changed the calendar to show February available but that is not reflected in Airbnb or VRBO. I have set full sinc a few times but no change. Please help! 

Dec 29, 2022 10:16 AM
Jonelle Kop-Agen says:

I just double checked add none of my properties are no won AIRBNB...

Dec 29, 2022 10:18 AM
Ken T says:

@Jonelle - I just checked and I can see them on Airbnb just fine.  But you have a lot and I didn't check them all.  Please write in to the Helpdesk with specific examples of properties that are not appearing on Airbnb so we can investigate.

Dec 29, 2022 10:28 AM
Jonelle K says:

Apartment 230 is available for the month of February. But when i search, it does not come up. When I search with no dates, and look by the map, none show up... 

Dec 29, 2022 12:01 PM
Cole W says:

I'm a software engineer and have been on the other side of outages like this. I want to commend you guys on your quick response, your diligence to make sure there weren't any data corruption issues, your ownership of the outage, and the transparency of this write up. Genuinely, I see few public postmortems that describe as thoroughly what went wrong and right from a technical perspective, and I think you should be commended for it. Few end users truly understand the fragility of large, complex software systems and how difficult it is to build true resiliency. I am sure you will take your learnings from this issue and incorporate them into an even more resilient system.

And kudos to all of those who responded to the outage. Getting a page about a primary db corruption during a work holiday does not sound like a good time to me.

Dec 29, 2022 9:30 PM
Ella says:

Still seeing on Airbnb:

Calendar sync

We’re having trouble syncing some of your calendars. Click the sync icon to try again. If you continue having problems, remove the calendar and re-import it.

OwnerREzSync failed

Dec 30, 2022 5:27 PM
Dori123 says:

Thanks for your hard work on this difficult problem, guys. For your log -- I saw Ken had commented: "Missed email triggers should have been sent, albeit late - you can confirm this in your Communication History.  As noted, you can manually resend any you're not sure of in the usual ways, but we do not believe this to be generally necessary."

I can confirm that several triggers were missed and never sent until I sent them manually yesterday (12/29). Neither the original triggers were sent  nor those that had the option "try to send a missed trigger up until the booking has arrived."

Dec 30, 2022 5:49 PM
Chris Hynes says:

@Ella -- Have you removed the calendar and re-imported it as Airbnb suggests?

@Dori123 -- I see a manual send of a keycodes trigger in your account, but the reason that trigger didn't fire is that it wasn't applicable to the booking's listing site. If there was a different trigger that didn't fire, please write in to the helpdesk with the details and we'll trace the reason.

Dec 30, 2022 5:51 PM
Ella says:

It's working now 

Dec 30, 2022 6:21 PM
Kylie R says:

....

Jan 1, 2023 9:09 PM
Jason W says:

It's technology, it happens!  We appreciate your team taking action to get everything back up and running.  Here's to a great 2023!

Jan 3, 2023 2:09 PM
KAY-FAR says:

HAPPY 2023 OwnerRez and OwnerRez users!

Thanks OwnerRez for working extra hard to get us back online and crossing into 2023 without any more outage.  As mentioned in my previous comment, I am still fairly new to your platform but I must say I am impressed by the quick move to fix.  Your Support team also continued to do a great job in responding to messages eventually letting us know what happened.  As seemingly the "Goliath" in this space, I hope that your team will continue to invest time and brain power in actively searching for threats to your platform and jumping on them as you did this one to help minimize disruption to operations.  THANKS AGAIN FOR THE TREMENDOUS EFFORTS THAT WENT INTO THIS.  Oops sorry, didn't mean to yell with the all caps n' all! ;).   

Happy 2023!

Add your comment