128
What was the silliest issue you discovered during debugging?
A while back I was developing a workflow scheduler. However the run duration kept coming back negative. The runs kept finishing like 15 seconds before they started. Eventually I figured out why; the company had configured a policy to use their custom NTP server, but it was 16.7 seconds off. So my Windows laptop’s time was in the future, but the WSL VM (which ran the program) used another NTP server and was correct. In the end I just added a bodge that made the program request the time from an external server on startup to calculate an offset.
Comments
Back when I was learning c++ on Windows, I naively jumped right into game dev (as one does) before understanding the purpose of a debugger. I spent 3 months chasing down a period that should have been a comma. Due to function overloading, this was not an error or warning. It’s just silently calling the wrong function and crashing the program.
Thus began my distrust of magic compiler features. To save typing a handful of characters, languages add new ways to blow off your own leg. Every time you blow off your leg, the time you lose is orders of magnitude greater than just being explicit. Just read this thread for more examples. Today, I program almost exclusively in C. I have no desire to use more feature-rich/higher level languages for anything.
Forgot to save. You ever spend 45 minutes looking for a bug you already fixed?
One time I was struggling to understand why a code branch I wrote was being unexpectedly taken, and I found that I had written something like
if pet == 'cat' || 'dog'. Since'dog'was a string, the condition was always true. I laughed out loud at myself.Hidden: you blocked this person.
Goddamn case sensitive POS JavaScript.
ECU (the computer in your car) only came online on the bus at every 3rd restart. No memory was written, yet the bastard somehow knew it anyway.
Turns out the transciever’s enable pin was left unconfigured and supposedly some charge accumulated during two startups to set it.
Love this
I fixed a 5 year old issue that the previous guy claimed was “unfixable”. Turns out it was because someone copy-pasted a value from an Outlook email…and it had a single dash in it…which outlook converted to a double dash.
That’s why I was so confused about the AI emdash connection. MS Office has been replacing dash with emdash for 20+ years.
I have had issues of having the same characters but in Unicode failing. But if I type it manually or paste it into Notepad++ forcing it to ASCII first, it works.
That I’m not cut out for coding. lol
I can’t quite remember what I was debugging exactly, some constant was different in runtime compared to what was in code. Someone added a leading 0 to a const int to make the number align with others in terms of character count just to make it pretty I guess. That is when I learned about octal literals in C. I think that was the only time I’ve seen one in many years and it was used unintentionally. To save you a search leading 0 in C makes number to be base 8 same as 0x makes it hex aka base 16, so that inconspicuous 01000 is not a 1000 it’s a 512.
Haha, I always love that one when I debug some other codebase
This was dumb, and it was on me.
Couldn’t figure out why two obscure software packages were erroring in a very particular way. Thinking there was no way in hell a search would help, I plugged it in Google. There was user on Spiceworks outlining my exact situation and fix. Who is this mad genius?!
It was me. I fixed it two years previously and went back to note so in the forum.
The one and only denvercoder?
Person before flipped a bunch of if checks to put the value before the variable (if 6==x rather then if x==6). They messed up a bunch of the <= and >= checks by forgetting to reverse them. I improved performance and fixed numerous bugs by tracking those down.
Ah yes, yoda conditionals. Because guarding against accidentally typoing
if(x=6)is worth destroying your code’s readability.This is not never anymore thanks to compiler warnings
Tell that to my colleague who still insists on writing if conditions like that. Maybe it’s just a PHP thing.
Internet Explorer didn’t automatically have “console.log” defined until you opened the console window. So there was an error breaking the web page that would disappear when you tried to open the console to see the error. The error was that console.log wasn’t defined. Fucking IE.
/thread, holy shit lmao
I was hired to work on a CMS designed for schools. I was tasked with debugging.
I started off with the newest issues and worked my way back.
Well, there was a bug where teachers would upload a video but it wouldn’t play on the page. It was plaguing a team of 8 for YEARS.
It was mentioned in our morning standup, and I decided to fast track it.
The dude who wrote the code was shrinking the display size in half: well how do you think that would work when a dimension of the video isn’t divisible by 2?
So, remember, I am the new guy, quickly fixed it using an if/then and the % modulus operator, and lo and behold, the next morning standup I had to explain the fix in front of the project manager, his boss, and the entire team…
fucking awkward. “You see, there’s this thing in division called a remainder.” This was the first thing in the morning.
But because the bug had been so ever present, that’s why the boss’s boss attended the standup. God, I just wanted to die.
I don’t think I understand what the issue was or what the fix was. Was it a language or system that cannot perform the division with truncation?
Let’s say height and width are 320x214.
(these numbers are arbitrarily chosen)
The code was sizing it down by /2, so now it is 160 by 107.
Because the width being odd, it would display.
But:
if ($width%2) { $width = --$width; }
Hastily written out on the phone, but that was the fix. This was back with php 5.
Sorry, but this comment still doesn’t make sense (or has typos).
For a strong typed language, an integer division should not cast to any other data type. Even assigning a float to an integer should not be allowed (that’s why C is not really strongly typed). I think from the syntax it was PHP. If any of the dimensions were not divisible by 2, then the number changed from integer to float and it could cause an issue somewhere downstream the video processing pipeline.
OP didn’t say that the problem was with the change in type. I think the interesting part is the downstream issue: what failed when a float was given while an integer was expected?
Dude, this reminds me of my coworker who was setting up a network and was trying to figure out a subnet mask, and he said, “I am trying to find a subnet mask between /22 and /23.”
He spent the entire day on it. I finally had to ask, and once he told me his goal, I of course asked, “how the fuck do you get half of a bit?”
Oh, the blood drained from his face. “Yeah, bro, that’s CIDR.”
How does this not make sense? I am spoon feeding this shit. Are you just vibe coding?
I think there are some versions of Python and/or PHP where the normal division operator silently promotes to IEEE floats, and you have to explicitly cast back to truncate/floor the value.
Back in the 90’s I had to debug a printer routine in a reporting tool. It would occasionally crash. Only on Wednesdays. Only on Wednesdays in September. Only on Wednesdays in September after September 9th.
Code was written in c.
What was the fix. I’m a dumb data scientist, I don’t know C
As an even dumber data analyst, something about this comment makes me feel seen. Thank you. <3
Might be the character count. Wednesday has many, September has many, after the 9th has two.
Program filling the date into a form field of width 1 less than was possible with actual dates.
c requires you to do things like define the maximum length of a string. In fact a string in c is literally defined as an array of characters. There are no protections in c, so you can define a string to be 10 characters long and then copy 20 characters to it. The result is most often a stack overflow that immediately crashes the program.
This was legacy code that was likely written in the early days of DOS before things like Microsoft Windows existed, so memory usage was still pretty important. Whoever wrote this code tried to calculate the maximum size of a string used to print a header in this reporting package as a way to optimize memory usage. But they were off by one character. So when the code created a header with a date containing the longest month name, the longest date name, and a two digit day of the month, then that string overflowed and crashed the program.
two days for an
=instead of an==“Yoda conditionals” are a must in languages where bindings/references are mutable by default. E.g.: “if (0 == size)” or “if (0x4269 == magic)”. If you write ‘=’ instead of ‘==’ you get a compiler error, so always put the constant first.
Hope you never have to use a language that lets you redefine numbers.
Wasn’t there a Python version where True and False were mutable by accident?
I tend to use languages where immutability is the default these days like Haskell, Idris, or Purescript. But, even when using Java and Javascript, I tossed around a lot more final / const than my peers. :)
Redefining the number 2 in Python
The biggest const-by-default lang is probably Rust now isn’t it?
Maybe? I don’t really keep up with lang popularity. Got through the Rust tutorial and did one project with it, but I honestly was missing HKTs and (mostly) isolating effects based on type signature. Of course, Rust may have grown those features since then, it was last decade when I played with it.
I once spent several hours debugging a performance issue in the Linux builds of a project. The service worked fine on Windows but couldn’t keep up with the traffic when deployed on Linux. Interestingly, CPU load was actually higher on Windows, and no sampling profiler showed any suspicious hot spots.
It turned out the problem was in the program’s main loop, which was OS-specific and called something like
sleep(float seconds)(yes, I know…). The parameter was set to1e-3on Windows and10e-3on Linux. I stared at that line several times before I noticed the difference.Another time, all services of a distributed system were regularly reporting a loss of communication at 4 AM. The servers were supposed to receive data continuously, so even a few seconds of silence was enough to raise an alert.
Half of the team investigated all kinds of possible locking scenarios (the applications were multi-threaded). Then we noticed that all the logs in the system (including the kernel’s) had the same gap in timestamps.
Then someone remembered, that the customer had a cron job which did some maintenance at night. They confirmed that it halted the entire VM with all services for about 5 seconds.
Whenever I book a flight with Air France, I have to enter our birth dates off by one day. We’re in Canada, and the Air France servers are most likely in…France, or at least using a french time zone.
Anyways, I figure that someone decided to use some form of LocalTime midnight for the date. We’re 6 or 7 hours later, so when it translates it to French time, it’s a different day.
So, not a bug that I fixed, but one that I work around. It’s been this way for years, too.
The good news is that with JavaScript now having adopted Temporal in the standards all of that should be in the past as soon as apps switch to it. (And when Apple decides to implement the damn thing)
Was told to test a new database migration as they updated from mainframe to a current version of Oracle. First I was to test data integrity. Wrote a python script to pull from the old db and verify that record matched the new. Found sooooo many bugs, sent it to the dev. Half were real bugs, half were bugs with my script. Turns out they didn’t actually want a 1:1 on the new db they were going to change things cuz they could. Lots of back and forth and by the end my python script essentially mirrored the production code.
Then they wanted me to load test the new system. I wrote a quick script to query N rows from the db with X threads and blast the new service. Didn’t take long to write, presented my finished script and plan to pull in manual testers to also run my script as needed.
Dev was cocky because they’d setup auto scaling on AWS. Manager was concerned I’d taken time to write a script instead of just spamming the same message over and over with a tool like gatling.
First run I took down the system very quickly, like within a minute with just my laptop. Note, this was a system that was planned to handle 100s of millions of requests an hour. Dev was impressed, turned out to be a auto scaling misconfiguration. Second time it took a few mins but it died again. Config needed some more tweaks. Third time I took it down because of some queueing issue that raised it’s ugly head. Fourth time I took it down they found a bug in the code. Fifth, Sixth, and Seventh time I needed the manual tester’s help, but the devs were scratching their heads trying to figure out how I was taking them out in 5mins of sustained load. Took two teams of devs scratching their heads and several itterations through bringing the system down and even two separate devs walked through my code to see what on earth I was doing. To top it off it wouldn’t just take down the microservice, it would crash our entire AWS infrastructure for the company. Not just our project but all our aws projects in the building went down.
They finally found the problem. The silly little problem. Remember that data I pulled from the db and blasted them with? Well, we hit a poision pill situation. One of the records turns out was a manual tester’s favorite to use when running one of his tests. That one record had 30,000 attached sub-records. Everytime that tester ran that test case he just kept adding a sub-record to it. Dude didn’t know he was slowly crafting such a massive data bomb all those years that would take down all our AWS servers.
They had also fixed the problem with one service crashing the others. And that my friends is why you never skimp on good automated testing. When it went live there was no major issues or outages, though we did let it soak in parallel with prod for weeks to make sure we trusted the results.
GDB started printing emojis with MI mode: https://sourceware.org/bugzilla/show_bug.cgi?id=34501
“Yo, I hear you like emojis, so here are some emojis to debug, while you are debugging the debugger.”
(I’m exposing my name here for the sake of wtf, don’t pwn me.)
i was just a user, but i debugged a weird issue where one of those blue light reduction programs would start flickering and inverting day and night time during june.
turns out the dev had made the assumption that the sun always rises before it sets, and i was far enough north that the sun would start setting after midnight.
https://zainrizvi.io/blog/falsehoods-programmers-believe-about-time-zones/
https://youtu.be/-5wpm-gesOY
haha, knew what this was going to be. Already came close to posting it a couple of other times in this thread. :)
Such a classic.
Ooh, interesting one! 😄
This was on a Fortran project that had no encapsulation and way too many global variables.
Suddenly, one particular calculation was failing, and not in a floating point rounding error kind of way. After setting up several scenarios to generate various input values, I determined that Fortran’s absolute value function was returning the wrong results.
I couldn’t understand why, so I went to the Lead Architect. He poked at it for about four minutes and did some searches, and found where someone else had defined a global variable named ABS that was masking the built-in abs function. The function call argument [e.g. abs(3)] was being compiled as an array index on this ABS variable [which was not an array, so yay, undefined behaviour].
They did a 400 table join to return ‘true’ to test to see if they connected to the database; it took 48 minutes to run. Printed out, it took up 38 sheets of paper.
I replaced it with a dozen sequence checks and individual select statements for each table; it took 3 seconds to run. And it could accurately identify which tables and columns are missing. Printed out, it fit on a single page.
I found an issue in a software used by government employees.
A request would fail if the server and the client had different years in their system time which could happen if they were located in different time zones.
After a few calculations I asked: “Hey, what are the chances a government employee would do this important task between 6pm and midnight on December 31st?”
We decided not to prioritize the issue.
You realise that’s going to end up in the launch process for a nuke somehow.
I guess “this important task” could refer to firing a nuke, which to be fair is unlikely to happen at that moment in time.
Govt bureaucracy. It could be scheduling a janitor and still somehow end up in the checklist.
IT department really wanted to use their snake oil antivirus enterprise super secure hackproof™ software everywhere.
So it was installed on the CI servers.
Fastforward a few months and we were wondering why one build agent was no longer picking up jobs.
After a day of debugging we found out that a certain snake oil sofware tampered with a single machine internal TCP conncetion and somehow fucked it up in a way that all packets in one direction just vanished.
Disabling the snake oil software magically fixed the problem.
Should a CI server even have access to the Internet and need antivirus software? Maven access I guess.
It is good to have a build method that doesn’t require network access; at one time that used to be required for Debian builds. However, for a variety of reasons, most CI servers do have access to the network, even if there’s a firewall (or NAT) between them and the Internet.
I’ve been a dev for years and had my fair share of sillyness… can’t recall which was the most silly, but one very recent one was trying to debug a weird error. So I opened the VS solution did a search for the string that spat out the error. Found it in a catch clause and stabbed that function with breakpoints, and went off to test.
Hours later (too many to confess) after trying everything, I found out why I couldn’t get it to stop on the breakpoint. I had opened the wrong project, for the wrong app :O
It was an unlucky coincidence that it had the same literal error trace in code!
i was trying to figure out why ssh wasnt working only to realize I was using the wrong username. That took about 2 hours of time to realize as I was bashing my head at the wall while looking at the pubkey signature being the same on both machines.
Sign in to comment. Sign in