Skip to content

Comment on Software is breaking down at a systemic level, by Jimmy Breck-McKye

You are viewing one comment and its replies. View all comments · View the comment above

Comments

  • I think metrics like code coverage are a big part of the problem where devs are encouraged to stop short of fully testing something or write low-quality tests that prove nothing to satisfy the metric.

    The classic simple example is a divide function that takes a dividend and divisor and returns a quotient. One test that passes 6 and 3 and asserts 2 yields 100% code coverage, but what what about divide by zero errors? Does it work with negative numbers? Etc., etc. You’re definitely not done with tests, despite the metric saying you are, and a slop generator (human or AI) could certainly write a fundamentally broken implementation that passes the tests.

    Now combine that with a tendency to focus on unit tests that test the system in a manner that hardly reflects how it works in production, and you have a system that passes all the tests, yet doesn’t actually work.

    12
    • This is generally solvable via mutation testing. Sadly, almost no developer knows what mutation testing even is.

      7
    • One test that passes 6 and 3 and asserts 2 yields 100% code coverage, but what what about divide by zero errors? Does it work with negative numbers?

      Or NaN values.

      There was once a bug in, I think, FORTRAN code, which converted a decimal string to double. It had a bug that became relevant when the input was near the precision limit, and led to an infinite loop. That bug was found in quite a number of runtime implementations like PHP.

      1

Sign in to comment. Sign in