Skip to content

Comment on Bank CEOs, NSA Tell Congress That AI-Driven Patches Are Behind Outages

You are viewing one comment and its replies. View all comments · View the comment above

Comments

  • “Pushing too many patches”? What is that supposed to mean? It’s not like they are sending each patch separately.

    I’ll have to throw in a “yeah, but.” Nobody in their right mind would bundle (say) 50 patches together into a single upgrade. The chances of regression are too great, and if it happens, it makes it harder to isolate which patch caused the problem. You can mitigate that by proper testing, but in complex systems, that takes significant time and resources, and there’s countervailing time pressure to immediately deal with zero-days.

    But they were also talking about scheduled maintenance, which might suggest that they still make provision to shut the system down to apply changes, rather than having spent the money and technical effort to allow them to maintain system operations when applying changes (for example, by blue/green deployments). At very least, it’s good practice to minimize downtime even when you can’t maintain operational continuity, by applying changes in the smallest possible increments.

    Minimizing downtime might matter less in banks, where back-end systems often still run in batch mode, due to the banks’ reluctance to invest properly in IT (as you mentioned) and because some of the external systems they have to interface with also run in batch (e.g., overnight clearing, which is instantaneous in most of the rest of the developed world).

    Tech debt from the mainframe era is the world’s biggest submerged iceberg.

    The NSA outage sounded like it was a failed comms upgrade, which is probably for different causes.

    4
    • Yes, but the exact quote is “They’re trying to push through a lot of patches all at once — more than the system was designed for.”

      The process can have bottlenecks and can not handle enough patches properly (as you describe, issues with testing, regressions, downtimes etc.). The system doesn’t handle patches like that. It doesn’t process each patch separately. It’s not like every dev or AI agent send a patch at once and the system fails to process them.

      -2

Sign in to comment. Sign in