Taking over a Dead IoT Company
5 years after NYCTrainSign collapsed, I investigate why the company failed and end up writing an exploit to take over their fleet.
Back in 2017, NYCTrainSign was a company making replicas of the countdown timers that told you how long it would be until the next train came.
But instead of being hung up on the ceiling, you could put it on your desk as a tasteful part of your home.
The person responsible for marketing did a great job driving interest. I remember a lot of Facebook and Instagram posts showing off how the sign could be useful for cafes and pizzerias so their customers could see when they should leave for the train.
However, underneath the veneer of Instagram, the signs were full of subpar engineering and unsustainable costs.
In early 2018 the company stopped replying to social media posts and very few people had received their purchased signs. The company recommended that customers dispute the charge to try to get their money back.
Today, even new companies entering this space have had to deal with the fallout of NYCTrainSign.
Now, 5 years after the company collapsed, I acquired one of their signs to investigate why the company failed. Along the way I ended up taking over the company’s sign control domain and writing an exploit to get full control of any signs still in the field.
Getting a Sign
Sometime in 2021 I found someone on reddit that was selling a NYCTrainSign. They were one of the few who had received product from the company and were looking to get rid of it.
Amazingly, the original owner kept the original packaging.
The sign itself is housed in a wooden case that the company handmade. I believe the company had hired a woodworker out of college to make the cases.
As someone with little to no woodworking skills, the case looks pretty good. There are some bad corners but it wouldn’t look out of place in any home.
The sign internally was comprised of
- 2 LED Matrix Panels
- 1 Raspberry Pi 3
- 1 4GB MicroSD Card
- 1 LED Matrix HAT from Adafruit
- Wiring for the power supply connection
- Wiring for the LED Matrix Panels
- A small button wired to GPIO on the Pi
While the case is fine, internally it shouldn’t take much to tell that the sign is not that well made.
The Pi was only half screwed on by two loose screws. The button (intended for reset) was sort of just hanging around. There’s also a giant hole for some unknown reason. The power supply connection seems like it could break with too much movement and it was actually unplugged when I first got the sign.
The BOM is Too Damn High
A bill of materials or BOM is a list of the raw components that go into a product. Most electronics projects, especially serious ones, will have a detailed BOM that describes the item and price that it goes for.
Often the final cost of the BOM will be much lower than the retail price because of the cost of shipping, R&D, profit margin, etc. Small changes in BOM price can have a big impact on the final cost of an item. It’s not uncommon to switch vendors or parts to save just cents on the BOM.
One trick I use is that multiplying the BOM cost by 4 will often get you the retail price.
With access to a sign we can put together a hypothetical BOM:
- Raspberry Pi 3 - $35
- Adafruit LED Matrix Hat - $25
- LED Matrix * 2 - $60 ($30 each at least)
- One of their tweets specifically cite using a specific more expensive Adafruit component (https://www.adafruit.com/product/2278).
- 5V 2A Power Supply - $5 (Best Guess)
- Suspiciously this is not enough amperage to power everything at full power draw. Most guides recommend 4 to 10 amps.
- 4GB MicroSD Card - $7 (Best Guess)
- Wood Case - $15 (Best Guess)
- Miscellaneous Items
- Wiring, Buttons, Cabling, Screws, Packaging (the sign had some cardboard, a sheet of paper, and some bubble wrap and foam) - $3 (Best Guess)
So roughly the BOM cost was $150. If we were to apply our pricing trick, the suggested retail price should be at least $600.
They did not know the trick
Based on archived webpages the NYCTrainSign team was selling signs for $600 but there are articles mentioning prices like $300. Some people mention purchases for around $200 and even only paying $100.
It seems there were also plans to rent the signs at $30/month as well.
$300 is obviously too low of a price based on the BOM but yet $600 is probably too much. It’s just a sign after all.
Renting could have been an interesting idea but I think it would be very difficult to recoup the initial cash investment. Hardware businesses often need to have an initial cash injection to build inventory and then want to recoup that cash and gather profit as quickly as possible by selling their product.
Who Sold The Shovels
There is a saying that "during a gold rush, you should sell shovels".
In our tale, Adafruit sold the shovels. By using the Adafruit LED Matrix HAT, the BOM cost gets inflated by $25 or about 20%. The HAT is also entirely avoidable with a little engineering work.
This is because:
- The HAT is not absolutely necessary. The rpi-rgb-led-matrix library used by the sign includes direct wiring instructions and designs for a passive adapter board. These instructions & boards existed back in 2017.
- A cheaper HAT also existed back in 2017. At $2.10 it would have been significantly cheaper than the $25 Adafruit HAT. Also despite using the more expensive Adafruit HAT the sign for some reason flickers constantly.
- Most of the time Adafruit parts are used for prototyping but are replaced with something cheaper when going to production. This is likely factored in Adafruit's pricing scheme.
Now you might also think that The Raspberry Pi Foundation is also a shovel seller. The $35 cost is a lot to stomach and it’s the most expensive single item in the BOM.
Having thought about it, while it may seem like overkill and I wouldn't recommend it, I think using a Pi wasn’t a bad decision compared to using an Arduino or ESP32. It gave the company free marketing and an easy development environment.
In my opinion, if the NYCTrainSign team didn’t want to invest upfront engineering time in using a microcontroller like the ESP32, it would make sense to start with the Raspberry Pi 3, then switch to the Pi Zero W when it was released.
Eventually they should aim to switch to the ESP32 or similar and lower their BOM cost further but it doesn’t seem like starting with the Pi was overtly wrong as long as the long term plan would be to eventually replace it.
Reading the Code
Since the product is Raspberry Pi based, it’s simple to make a backup of the MicroSD card so that I can explore the filesystem and make changes as needed.
The sign’s codebase consists of some custom made Python & NodeJS code as well as a number of open source parts.
There are 2 primary custom components running on the Pi:
- The Python server (LED Server)
- The NodeJS server (Config Server)
The LED Server written in Python is responsible for drawing to the LED Matrix and getting train data from the company’s API. The LED server communicates with the Config Server to determine what settings the user has configured and then issues frequent HTTP calls to a remote server to get data like train arrivals and weather.
With the train data, the LED Server will generate an image or text locally and then render that on the LED Matrix.
The Config Server written in NodeJS is responsible for storing user configuration in a JSON file and receiving requests to retrieve and update that file. At boot time, the Config Server will pull the latest configuration from an HTTP server. In addition, the Config Server will connect to an AWS IoT Core endpoint to receive real time config updates from an MQTT server.
- On first startup, Wifi is configured with raspberry-wifi-conf which is an open source application that will have the Pi create a wireless network that the user is supposed to connect to and then provide actual WiFi connection details to.
- The Reset button is controlled by a small Python script that is run in the background. When the button is pressed the script deletes wifi settings, resets the hostname, deletes some remote monitoring functionality, and reboots the server.
- It seems that the company could remotely connect to a terminal on every sign. My sign appeared to have remote control software from https://www.dataplicity.com/ installed.
While looking through the code, I noticed various code quality issues:
- Their transit API didn't seem to consider that a station can have multiple train lines. Two lines at the same station would show as ending at the same stops.
- No discernable firmware update process. Perhaps the update process would have just been to have customers flash the MicroSD card with a new Pi image.
- A lot of the Python code merely uses system calls to make changes with the underlying system
- The Python LED server communicates with the NodeJS Config server to store/retrieve configuration. I suspect this was done because it was easier for the team to interact with AWS IoT from NodeJS but easier to interact with the display from Python.
- Frequent mixing of tabs and spaces due to misconfigured code editors
- Entire git history was saved on the MicroSD card
- Bash history saved on the MicroSD card
- Very little code reuse
Getting a Shell
It’s relatively simple to get a root shell on most Raspberry Pi’s as typically there’s no encryption on the MicroSD card. We can simply boot into single user mode and reset the password for the pi user.
However, while we have a shell now and we can play around, none of it really matters because the company’s API doesn’t exist anymore. The sign was programmed to use hard-coded local data when it has no internet access so any data that’s being shown is useless.
Recreating the Server
We could of course update the domain that the sign talks to but luckily the domain that the sign communicates to by default was available for purchase.
So I bought it.
With full control of the domain, we can create a new API based on what the sign is expecting and revive all of the signs that are out in the field. Potentially, we could also perform some kind of update to update signs to more modern software.
I reconstructed the endpoints for the train arrival times (at least for NYC) and weather data. For train times I decided to use the API behind https://wheresthefuckingtrain.com/. For weather data I used the OpenMeteo API.
Getting Sign Control
Like most IoT devices, the sign makes a lot of system calls. One call directly concatenates the sign ID into a shell command. With control of the server in theory we can directly get remote control of any train sign.
Since we control the domain, in theory, we can feed any sign a malicious sign ID and run any arbitrary command. We can then use this to register the sign to our new control server and give people control of their signs again.
After handwaving away some of the boring data wrangling details, our exploit looks like the following:
- The sign is turned on and it attempts to retrieve configuration. This will loop forever until the sign retrieves something.
- The sign will eventually send a request for an image logo. This request will contain two ID's unique to each sign. We store these IDs and create an exploit sign config.
- On the sign’s next config request we serve our exploit to the sign.
- We instruct the user to restart their sign and our exploit is run on restart
- The exploit updates any code that’s needed to pair it with our new server
Here is a video of the exploit running:
With the ability to remotely serve arbitrary sign data, we can officially say that our sign has been restored!
With full control of the domain, we are now the new NYCTrainSign captains.
So What Happened Back Then?
Too Many Discounts
I think the core issue is having a high BOM cost as well as selling a lot of signs at a discount.
Even during a beta I really don’t think you can make a product for $150 and then sell it for $117 without any venture capital backing.
As we discussed earlier, even at $300 the product is too cheap. The sign should have likely been selling at $600 from the very beginning.
The product had some ideas around serving ads but an LED sign isn’t really a “get big fast” kind of company. So selling lots of signs at a loss really just serves as marketing.
Too Many Ads
The NYCTrainSign company at one point had a Chief Marketing Officer as well as a Social Media Manager and a Social Media Assistant.
This seems excessive for a burgeoning startup.
At the time, the MTA was experiencing frequent delays which the media was heavily covering. It seems like the company got a lot of free interest very quickly. It likely would have been sufficient to coast off of free marketing.
Combined with the additional ads and marketing that the company was purchasing, there was a lot of demand for the product.
Not Enough Product
However despite the demand, the team could not produce the volumes needed. Not to mention that every sign was being manually built in Brooklyn which is obviously unsustainable with a low retail price. It also seems unlikely that the team was keeping enough inventory on hand so they were probably also affected by the lead time on sourcing parts.
Despite having no ability to fulfill orders, they continued to take them in; likely in a Ponzi-like attempt to get funds to fulfill the previous discounted orders.
At this point, the team told customers they were expecting a 6 month delay while they moved their production to China. Customers who didn’t want to wait could contact their credit card company to issue chargebacks.
Moving to China seems like a pipe dream but it never manifested. Shortly after their announcement the company was shut down and their office shuttered.
Too Many Cooks
Based on Linkedin information there were initially 4 founders. Over time it grew to 11 people and then at some point 15 people involved with the company altogether.
At even just 60k per founder, with the profit per sign sold being maybe $400, you would probably need to sell ~600 signs at full price per year to create enough revenue to run payroll.
One of the founders at some point released a screenshot of their sales activity and they had reached $250k in revenue in about two months.
However it’s unclear how much of that was profit since it seems like many signs were sold at a steep discount. The $600 price point seems excessive. Really the $200 or $300 price point makes more sense.
I imagine that in a city of about 8.5 million it shouldn’t be too hard to sell 600 signs in at least the first year. But next year you’d have to sell ~600 signs all over again. And this isn’t including the cost for any other employees, contractors, etc.
In my opinion, without a better strategy for having a lower BOM cost from the beginning or some kind of recurring revenue, the company would have quickly become unsustainable. Which it obviously did.
Good Idea, Good Timing, Bad Team, Bad Product
After the company was dead in the water, the founders & employees attempted to make some sort of consulting firm for some reason.
The founders have never come clean about what really happened and why so little product was shipped and where the money went. At least one founder says that they personally never received any money but the money had to have gone somewhere.
This lack of transparency would be what turned the mini darling company into something of a meme.
Today the company is completely gone and 3 out of 4 founders have publicly moved on. Many of the former employees still list the company on their LinkedIn. One founder (the CEO) keeps a low profile and can’t readily be found on the internet.
At the end of the day, the team had no discernible plan about what to do with their product and it really just seems like people who jumped into making a company without thinking carefully about what they’d do at every step. While you can maybe do that with a software company, it’s difficult to do that in the hardware space.
What’s really aggravating is that I truly believe that if the NYCTrainSign team spoke to someone with a background in electronics they would have been more successful. Instead it seems like their primary advisor was their college computer science professor.
It seems like NYCTrainSign just took a project that the CEO created in his spare time and then tried to sell it for $300 to $600 without productionizing it or thinking about what might happen afterwards.
In summary: good idea, good timing, bad team, bad product.
The founders & team for the most part do seem like honest people. I do not think they had any ill will.
They merely were some friends who did not have the experience needed to build a hardware business and were caught off-guard by the success of their marketing.
However, the CEO, Timothy Woo, should come clean about what happened with all the customer funds and what went wrong.
While I’m not sure if there’s a legal requirement to return funds, I do believe there’s a moral obligation to do right by your customer. Especially if you don’t end up giving them their purchase. At the very least I think an explanation is in order.
What Happens Now
When I originally embarked on this adventure I had dreams of building my own sign and then selling it as a product.
I prototyped a very useful sign using an ESP32 that I still use to this day.
However, the more I thought about it, the more I felt that I wasn’t the right person to bring this to market. My strengths are in software and maybe business, not in electrical engineering or woodworking.
Not to mention, LED signage is a crowded space with plenty of existing solutions. Not only are there plenty of LED signs on Amazon, but there are also more fancy signs like Tidbyt or this one I found on Etsy.
I’ve decided that instead of entering into a crowded space, I will continue working on my ESP32 sign mostly as a personal learning project.
For the community I will open source the underlying sign code for the NYCTrainSign as well as the reconstructed API server that includes the exploit code.
I will also maintain the new NYCTrainSign server so long as the hosting costs are fairly low. I don’t intend on adding any new features to the current NYCTrainSign but I do have some ideas for an improved sign firmware.
So if you have a sign from NYCTrainSign, try out the site that I created to manage the signs remotely.
If the instructions don’t work, file an issue. But if you don’t have an NYCTrainSign, definitely don’t buy one.Thanks to Sharan for putting up with me when I was overly interested in this sign, and thanks to Linda, Kai, and Soly for reviewing and editing this post.