Fake USB Serial
During the final stages of creating the SHA2017 badge we hit a snag. It's an awesome story and it provides a bit of insight in fake ICs and how to deal with them.
This adventure begins right after our second 'sweatshop'. Sweatshops are a term we call our production sessions. The first round was for making the kits, so everything in the bag except for the badge (PCB). The second one was our main production one, we got 2000 boards (out of 4750) in, completely assembled and now we got to: connect the display, put software on them, let the hardware tester run, mount the displays on the working boards, fix the broken ones (mostly a solder joint on the touch-sensing IC), clean the PCB, add a gluedot for the display, mount the display, pack the assembled and programmed and tested board in an ESD bag and add it in the SHA2017 badge kit.
It was a fun session, we had food and drinks for everybody and the operation ran really smooth. At the end we gave all our volunteers their SHA2017 badge in advance, as a way of saying thanks (and getting more developers on board ;) ). We had a massive rate of succes, well over 97% of the boards worked without issue!
And then things got... weird.
A problem arose
Next day one of our volunteers got in touch that his SHA2017 badge wasn't working on battery. It wouldn't boot. It would boot on the USB cable. Also, an LED was lit, which shouldn't be lit on battery. Okay, weird fluke, whatever. Only then another person turned up with the same problem.
After some testing (we just received another box of boards) it turns out that over half the boards we tested (around 50) had the same issue.
So, the LED was connected to the USB-Serial chip, the CP2102 by Silicon Labs. SL gave us a donation of 2000 chips and we purchased another 2750 chips through our component-supplier. Using a magnifieing glass we did some sleuthing and found that the problematic boards all had a USB Serial chip with the same serial number:
And that's weird. Because that serial-number is not possible according to the datasheet from SiLabs. So...
ITS A FAKE
Yes, fake chips happen. We realized we need to fix this, else well over 2000 boards, randomly distributed among the working badges, would only work on USB and disappoint our visitors.
By now our chief hardware designer came up with a theory and a fix. The lit LED was a giveaway that the chip was back powered through the serial lines by the ESP32. The theory was that this back powering partially powered up the fake USB serial chip, which then toggled the reset line on the ESP32, preventing it to boot. This meant we couldn't fix this problem in software (if the problem turned up later, after boot we would just disable serial communication and only use the over the air update). In 15 minutes of the first reported problem a fix was devised. Cut a the TX trace and add in a 10k resistor. The problem was overcome, at least on one out of well over two thousand five hundred boards.
We then went a bit further with our analysis. Another member theorized that the chip might give a different result when it's registers would be queried. And in fact it did. Weirdly enough, the fake-chips gave the proper response according to the datasheet, while the official ones acted a bit out of spec.
With that information our software-dev wrote a new version of our Autoflasher. This version detects which USB-Serial is plugged in and flash a slightly different firmware to it. The only difference is the splash screen which people see when unpacking their badge. On the bottom right corner you can read 'F' or 'SL' for 'Fake' or 'Silicon Labs'. Or, as we told visitors 'Fast' and 'Slow', which was also true, the fake chips achieved much greater speed.
Again, as with our hardware tester, the permanent picture on the Eink greatly helped. We could now just push all boards through the software proces, then easily split them with the 'F' and 'SL' logos, package the 'SL' boards for camp and fix the 'F' boards.
So we did another sweatshop.
Not all heroes wear capes
By now we were days before the event. People were already building at the field and we were running out of time. We sent out a call to the Dutch hacker scene (and globally) and people responded in masses to come and help. Out of nowhere microscopes and soldering stations appeared, we overnighted a roll of resistors, scalpels and other bits. In fact when we were panicking about acquiring resistors, somebody offered to just jump in his car from Germany and drive a reel to the Hague. We felt our stress pushing in, but also the love and support from the community.
The final sweatshop at Revspace was immense. We took up the main space and the mess hall. Over thirty people attended to push through as many boards as we could to split them up so we at least had all the 'SL' boards done and packed for the event, and fixed as many 'F' ones as well. At the end, before loading up the truck, we had well over 2500 boards ready and packaged to be handed out at the entrance. At this point we also devised a voucher scheme with a unique token for if we failed, we would fix as many at camp or maybe even after camp.
All the badge-stuff moved to camp to the then brand new Badge Bar. The Bar was a last minute happy coincidence where we got the opportunity to have a bar tent as our badge hacking area. So we had a large tent, with a lot of tables and we had a budget to buy soldering stations and a few microscopes.
Then, again out of nowhere a bunch of hackers showed up at our tent and brought more tools and started fixing up the final boards. Come day one, we had enough badges to hand out to everybody and our tale had a happy ending.
In 2019 we decapped both chips at Revspace to get the final lowdown and prove that the chips were really different on the inside as well (as if we didn't know). We created some Fuming Nitric Acid and decapped a pair of our two USB Serial chips.
Our main lesson was: get the genuine article. Use the same components as in your prototype fase and verify you get the same. The second main lesson is that you need to trust the community. Badge.team is the most open team of all badge teams that we know of and it pays off. We could not have done this without all our enthusiastic volunteers that came to our rescue. We strongly believe that our openness (we regularly post on Twitter, chat on IRC, send out emails, we made an emulator and we made the schematic and code publicly available from the early concepts) helps the cause.