EIDO Systems International - PDF generation failing for all content – Incident details

PDF generation failing for all content

Resolved
Major outage
Started over 2 years agoLasted about 10 hours

Affected

Inform

Major outage from 1:06 AM to 10:47 AM

Inform Admin

Major outage from 1:06 AM to 10:47 AM

Updates
  • Resolved
    Resolved

    The corrupt acknowledgements file has been replaced and has resolved the issue.

    Replication has been reactivated, and we are continuing to investigate why the acknowledgements file continues to cause problems.

  • Identified
    Identified

    We have identified the issue as being caused by a corrupt acknowledgements file.

    We're restoring a known-good backup version.

  • Investigating
    Investigating

    We have rolled back the problematic Australian instance, and have reindexed and run a consistency check on the admin database but this has not resolved the issue.

    We're continuing to investigate possible causes for this failure.

  • Identified
    Identified

    We are running a consistency check on the Ireland server which will mean it will be unavailable for around 30-40 mins

  • Investigating
    Investigating

    We are currently investigating this incident.