machine vision

67 Articles

The map in action, along with a sample of the video feeds.

Hardware Store Marauder’s Map Is Clarkian Magic

December 20, 2025 by 23 Comments

The “Marauder’s Map” is a magical artifact from the Harry Potter franchise. That sort of magic isn’t real, but as Arthur C. Clarke famously pointed out, it doesn’t need to be — we have technology, and we can make our own magic now. Or, rather, [Dave] on the YouTube Channel Dave’s Armoury can make it.

[Dave]’s hardware store might be in a rough neighborhood, since it has 50 cameras’ worth of CCTV coverage. In this case, the stockman’s loss is the hacker’s gain, as [Dave] has talked his way into accessing all of those various camera feeds and is using machine vision to track every single human in the store.

Of course, locating individuals in a video feed is easy — to locate them in space from that feed, one first needs an accurate map. To do that, [Dave] first 3D scans the entire store with a rover. The scan is in full 3D, and it’s no small amount of data. On the rover, a Jetson AGX is required to handle it; on the bench, a beefy HP Z8 Fury workstation crunches the point cloud into a map. Luckily it came with 500 GB of RAM, since just opening the mesh file generated from that point cloud needs 126 GB. That is processed into a simple 2D floor plan. While the workflow is impressive, we can’t help but wonder if there was an easier way. (Maybe a tape measure?)

Once an accurate map has been generated, it turns out NVIDIA already has a turnkey solution for mapping video feeds to a 2D spatial map. When processing so much data — remember, there are 50 camera feeds in the store — it’s not ideal to be passing the image data from RAM to GPU and back again, but luckily NVIDIA’s “Deep Stream” pipeline will do object detection and tracking (including between different video streams) all on the GPU. There’s also pose estimation right in there for more accurate tracking of where a person is standing than just “inside this red box”. With 50 cameras, it’s all a bit much for one card, but luckily [Dave]’s workstation has two GPUs.

Once the coordinates are spat out of the neural networks, it’s relatively simple to put footprints on the map in true Harry Potter fashion. It really is magic, in the Clarkian sense, what you can do if you throw enough computing power at it.

Unfortunately for show-accuracy (or fortunately, if you prefer to avoid gross privacy violations), it doesn’t track every individual by name, but it does demonstrate the possibility with [Dave] and his robot. If you want a map of something… else… maybe check out this backyard project.

Continue reading “Hardware Store Marauder’s Map Is Clarkian Magic” →

Image

Self-Driving Cars And The Fight Over The Necessity Of Lidar

October 30, 2025 by 72 Comments

If you haven’t lived underneath a rock for the past decade or so, you will have seen a lot of arguing in the media by prominent figures and their respective fanbases about what the right sensor package is for autonomous vehicles, or ‘self-driving cars’ in popular parlance. As the task here is to effectively replicate what is achieved by the human Mark 1 eyeball and associated processing hardware in the evolutionary layers of patched-together wetware (‘human brain’), it might seem tempting to think that a bunch of modern RGB cameras and a zippy computer system could do the same vision task quite easily.

This is where reality throws a couple of curveballs. Although RGB cameras lack the evolutionary glitches like an inverted image sensor and a big dead spot where the optical nerve punches through said sensor layer, it turns out that the preprocessing performed in the retina, the processing in the visual cortex and analysis in the rest of the brain is really quite good at detecting objects, no doubt helped by millions of years of only those who managed to not get eaten by predators procreating in significant numbers.

Hence the solution of sticking something like a Lidar scanner on a car makes a lot of sense. Not only does this provide advanced details on one’s surroundings, but also isn’t bothered by rain and fog the way an RGB camera is. Having more and better quality information makes subsequent processing easier and more effective, or so it would seem.

Continue reading “Self-Driving Cars And The Fight Over The Necessity Of Lidar” →

Hackaday Links Column Banner

Hackaday Links: June 29, 2025

June 29, 2025 by 16 Comments

In today’s episode of “AI Is Why We Can’t Have Nice Things,” we feature the Hertz Corporation and its new AI-powered rental car damage scanners. Gone are the days when an overworked human in a snappy windbreaker would give your rental return a once-over with the old Mark Ones to make sure you hadn’t messed the car up too badly. Instead, Hertz is fielding up to 100 of these “MRI scanners for cars.” The “damage discovery tool” uses cameras to capture images of the car and compares them to a model that’s apparently been trained on nothing but showroom cars. Redditors who’ve had the displeasure of being subjected to this thing report being charged egregiously high damage fees for non-existent damage. To add insult to injury, if renters want to appeal those charges, they have to argue with a chatbot first, one that offers no path to speaking with a human. While this is likely to be quite a tidy profit center for Hertz, their customers still have a vote here, and backlash will likely lead the company to adjust the model to be a bit more lenient, if not outright scrapping the system.

Continue reading “Hackaday Links: June 29, 2025” →

A piano is pictured with two hands playing different notes, G outlined in orange and C outlined in blue.

AI Piano Teacher To Criticize Your Every Move

June 22, 2025 by 15 Comments

Learning new instruments is never a simple task on your own; nothing can beat the instant feedback of a teacher. In our new age of AI, why not have an AI companion complain when you’re off note? This is exactly what [Ada López] put together with their AI-Powered Piano Trainer.

The basics of the piano rely on rather simple boolean actions, either you press a key or not. Obviously, this sets up the piano for many fun projects, such as creative doorbells or helpful AI models. [Ada López] started their AI model with a custom dataset with images of playing specific notes on the piano. These images then get fed into Roboflow and trained using the YOLOv8 model.

Using the piano training has the model run on a laptop and only has a Raspberry Pi for video, and gives instant feedback to the pianist due to the demands of the model. Placing the Pi and an LCD screen for feedback into a simple enclosure allows the easy viewing of how good an AI model thinks you play piano. [Ada López] demos their device by playing Twinkle Twinkle Little Star but there is no reason why other songs couldn’t be added!

While there are simpler piano trainers out there relying on audio cues, this project presents a great opportunity for a fun project for anyone else wanting to take up the baton. If you want to get a little more from having to do less in the physical space, then this invisible piano is perfect for you!

Image

Cheap Endoscopic Camera Helps Automate Pressure Advance Calibration

March 21, 2025 by 27 Comments

The difference between 3D printing and good 3D printing comes down to attention to detail. There are so many settings and so many variables, each of which seems to impact the other to a degree that can make setting things up a maddening process. That makes anything that simplifies the process, such as this computer vision pressure advance attachment, a welcome addition to the printing toolchain.

If you haven’t run into the term “pressure advance” for FDM printing before, fear not; it’s pretty intuitive. It’s just a way to compensate for the elasticity of the molten plastic column in the extruder, which can cause variations in the amount of material deposited when the print head acceleration changes, such as at corners or when starting a new layer.

To automate his pressure advance calibration process, [Marius Wachtler] attached one of those dirt-cheap endoscope cameras to the print head of his modified Ender 3, pointing straight down and square with the bed. A test grid is printed in a corner of the bed, with each arm printed using a slightly different pressure advance setting. The camera takes a photo of the pattern, which is processed by computer vision to remove the background and measure the thickness of each line. The line with the least variation wins, and the pressure advance setting used to print that line is used for the rest of the print — no blubs, no blebs.

We’ve seen other pressure-advanced calibrators before, but we like this one because it seems so cheap and easy to put together. True, it does mean sending images off to the cloud for analysis, but that seems a small price to pay for the convenience. And [Marius] is hopeful that he’ll be able to run the model locally at some point; we’re looking forward to that.

Continue reading “Cheap Endoscopic Camera Helps Automate Pressure Advance Calibration” →

Hackaday Links Column Banner

Hackaday Links: November 24, 2024

November 24, 2024 by 6 Comments

We received belated word this week of the passage of Ward Christensen, who died unexpectedly back in October at the age of 78. If the name doesn’t ring a bell, that’s understandable, because the man behind the first computer BBS wasn’t much for the spotlight. Along with Randy Suess and in response to the Blizzard of ’78, which kept their Chicago computer club from meeting in person, Christensen created an electronic version of a community corkboard. Suess worked on the hardware while Christensen provided the software, leveraging his XMODEM file-sharing protocol. They dubbed their creation a “bulletin board system” and when the idea caught on, they happily shared their work so that other enthusiasts could build their own systems.

Continue reading “Hackaday Links: November 24, 2024” →

Image

Analyzing Feature Learning In Artificial Neural Networks And Neural Collapse

July 26, 2024 by 9 Comments

Artificial Neural Networks (ANNs) are commonly used for machine vision purposes, where they are tasked with object recognition. This is accomplished by taking a multi-layer network and using a training data set to configure the weights associated with each ‘neuron’. Due to the complexity of these ANNs for non-trivial data sets, it’s often hard to make head or tails of what the network is actually matching in a given (non-training data) input. In a March 2024 study (preprint) by [A. Radhakrishnan] and colleagues in Science an approach is provided to elucidate and diagnose this mystery somewhat, by using what they call the average gradient outer product (AGOP).

Defined as the uncentered covariance matrix of the ANN’s input-output gradients averaged over the training dataset, this property can provide information on the data set’s features used for predictions. This turns out to be strongly correlated with repetitive information, such as the presence of eyes in recognizing whether lipstick is being worn and star patterns in a car and truck data set rather than anything to do with the (highly variable) vehicles. None of this was perhaps too surprising, but a number of the same researchers used the same AGOP for elucidating the mechanism behind neural collapse (NC) in ANNs.

NC occurs when an ANN gets overtrained (overparametrized). In the preprint paper by [D. Beaglehole] et al. the AGOP is used to provide evidence for the mechanism behind NC during feature learning. Perhaps the biggest take-away from these papers is that while ANNs can be useful, they’re also incredibly complex and poorly understood. The more we learn about their properties, the more appropriately we can use them.