PDF generation failing for all content

Resolved

Major outage

Started over 3 years agoNovember 30, 2022Lasted about 10 hoursNovember 30, 20221:06:0010:47:24 AMUTC

Affected

Inform

Inform Admin

Updates

Resolved
November 30, 2022 at 10:47:24 AMUTC
Resolved
November 30, 2022 at 10:47:24 AMUTC
The corrupt acknowledgements file has been replaced and has resolved the issue.

Replication has been reactivated, and we are continuing to investigate why the acknowledgements file continues to cause problems.
Identified
November 30, 2022 at 10:41:57 AMUTC
Identified
November 30, 2022 at 10:41:57 AMUTC
We have identified the issue as being caused by a corrupt acknowledgements file.

We're restoring a known-good backup version.
Investigating
November 30, 2022 at 10:22:38 AMUTC
Investigating
November 30, 2022 at 10:22:38 AMUTC
We have rolled back the problematic Australian instance, and have reindexed and run a consistency check on the admin database but this has not resolved the issue.

We're continuing to investigate possible causes for this failure.
Identified
November 30, 2022 at 9:12:53 AMUTC
Identified
November 30, 2022 at 9:12:53 AMUTC
We are running a consistency check on the Ireland server which will mean it will be unavailable for around 30-40 mins
Investigating
November 30, 2022 at 1:06:00 AMUTC
Investigating
November 30, 2022 at 1:06:00 AMUTC
We are currently investigating this incident.

EIDO Systems International - PDF generation failing for all content – Incident details