Technology

European so zero experience with Best Buy, but I feel we need to talk about how RAM incompatibility is no longer just a historic thing that used to happen back in the nineties and early 2000s, but one that quietly creeped back post 2020.

I manage business laptop fleets for a living. In the time of COVID-induced supply chain breakdowns, it was hard for our clients to get their hands on Custom-To-Order business laptops they'd usually get fully configured directly from the manufacturer (back then we were almost exclusively on Lenovo Thinkpads). Lead times for CTO laptops went from 2-4 weeks to 8-9 months.

Some clients couldn't wait that long and had us shop for off-the-shelf retail units we could upgrade to company standards ourselves. In 2022 we got a bunch of Thinkpad T14 Gen3 units with 16GB of soldered-on DDR4-3200 and one free SO-DIMM slot for a RAM upgrade. Quick and easy job - we got a bunch of Crucial 16GB DDR4-3200 cookies and went to work upgrading all the machines to 32GB. Machines upgraded fine, ran memory tests fine, and deployed a Windows image fine.

A few days later, on a Monday, we were supposed to start deploying them to users. One of the things each tech would do just before final handover was to run Lenovo System Update on each unit one more time for good measure. (This was a new model and they get quite a few BIOS updates in the first year after release as teething issues get ironed out.) And it turned out that this Monday Lenovo just made available a new BIOS version for the T14G3. Mind you, Thinkpad updates were rock-solid for decades at that point. And then they weren't.

When a machine failed to boot after a BIOS update the first thought was how lucky it was to catch a failing laptop even before it goes out to the user. "Once is an accident."

When a second machine failed to boot after the same BIOS update, I started getting nervous. "Twice is a coincidence."

When a third machine failed to boot, we pulled the plug on the deployment. "Three times is enemy action."

The machines were dead. Blank screen and fans running 100% indefinitely after power-up. Had Lenovo Premier Support dispatch engineers to site. They brought replacement motherboards. They replaced the motherboards (moved over our RAM and SSDs to the new ones) and booted successfully. Ran BIOS update. Did not boot. With three new "dead motherboards" they escalated the ticket.

It took three months and two more BIOS revisions for Lenovo to tell us that "our third party RAM is the culprit". By then we had already figured it out ourselves. Removing the Crucial sticks and letting the MB go through the automatic "memory reconditioning" process would actually bring these T14G3 laptops back to life with the latest BIOS and just their onboard RAM. Funnily enough, we had these same SODIMM sticks happily running in HP Elitebooks and Dell Latitudes without an issue. But there was some weird interaction between Lenovo's soldered-on RAM and Crucial's sticks.

Where Lenovo lost us as a customer was their official response when asked about the incompatibility and which modules would work: "The ones with a Lenovo sticker."

They had no idea what the issue was. They'd tested different SODIMMs and the models that happened to work they'd relabel as "Lenovo memory" and resell with 200% markup. They refused to publish a compatibility list or support anything without their sticker.

In the meantime we got a bunch of HP Elitebook 840 G8 laptops to cover for the unusable Thinkpad stock. These came with two SO-DIMM slots and no soldered memory, and we managed to get them with 2x16GB Samsung DDR4-3200 sticks. Each stick had an HP label, too. They also worked without a hitch when we swapped these out for the Crucials.

In the end we put the Samsung sticks into the Thinkpads and - funnily enough - they still work today with HP-labeled RAM. So do the Crucial sticks in HP laptops. Our own research concluded that it was single-rank / dual-rank difference between paired up modules that was causing issues (something you can't fully swap out if half of your RAM is soldered). We did get random freezes while testing HPs where we tried to pair a single-ranked Samsung module with a dual-ranked Crucial module. Lenovo refused to confirm that this was the real issue all along.

We even had an internal Spongebob comic about this:

After decades with Thinkpads, we went with Elitebooks. I remember it being heartbreaking for me at the time. But what was worse was the industry-wide return of the ghosts of the past: weird glitches, mysterious incompatibilities, and horrible degradation of once stellar enterprise quality control. And all of it has gone even more downhill since.