Skip to content
HN On Hacker News ↗

GitHub - aryamandarda/tamserve: Live, multi-camera object tracking with SAM 2 and EdgeTAM, on one GPU or many. One model per GPU serves every camera: faster than the original code, about 4x less GPU memory, identical masks.

▲ 123 points • 17 comments • by aryaman_darda • 6d ago • HN discussion ↗

Pangram verdict · v3.3

We believe this text is mainly AI, with some human-written content.

97 %

AI likelihood · overall

AI
3% human-written 97% AI-generated
SEGMENTS · HUMAN 1 of 16
SEGMENTS · AI 15 of 16
WORD COUNT 1,670
PEAK AI % 100% · §1
Analyzed
Nov 17
backend: pangram/v3.3
Segments scanned
16 windows
avg 104 words each
Distribution
3 / 97%
human / AI fraction
Verdict
AI
Pangram v3.3

Article text · 1,670 words · 16 segments analyzed

Human AI-generated
§1 AI · 100%

Live, multi-camera tracking with SAM 2, on one GPU or many.

§2 AI · 100%

SAM 2 and its smaller sibling EdgeTAM can track anything you point at, but their code was written for one recorded video at a time. tamserve is the serving engine they are missing: one copy of the model tracks every object on every camera, for as long as the cameras run.

§3 AI · 100%

Point at what to track, or name it ("dog", "the red mug") and tamserve finds it.

§4 Human · 0%

Why tamserve · New to video tracking? · Using it · Tracking by name · How it works · Results · Limits Six live cameras on one GPU, every one at 30 frames/s.

§5 AI · 100%

Watch the 32-second demo, or read how it was recorded. Why tamserve The models are excellent.

§6 AI · 100%

The code that comes with them was made for editing one recorded video, and the usual way to put it on several cameras is to run a separate copy of the model for each one. Memory piles up with every copy and with every minute of video. tamserve does what LLM servers such as vLLM do for chat: every camera shares one copy of the model, and whatever needs the GPU at a given moment goes through it together. That turns out to be faster as well as lighter, and because the model itself is not touched, the results are the same. the model's own code tamserve Several cameras one copy of the model per camera one copy for all of them GPU memory, 8 cameras 8.6 GB 2.2 GB Speed, 1 camera 177 frames/s 219 frames/s Speed, 8 cameras 280 frames/s 323 frames/s 64 cameras would need about 70 GB of GPU memory 321 frames/s in 5.7 GB GPU memory for an 800-frame video 9.8 GB 0.28 GB, however long it runs Starting to track something new mid-stream not possible on any frame Finding what to track you point at each object you point, or you name it and a Tracker finds and follows every match by itself Several GPUs you split the cameras between them by hand a Router places the cameras and gathers every result Accuracy on DAVIS 2017 0.8686 0.8685 These are for EdgeTAM on one RTX 4090, against EdgeTAM's own code at its fastest: compiled, one copy per camera, all sharing the GPU. Speeds are totals across the cameras, with one object each. With the larger SAM 2.1 models, tamserve runs at the same speed as Meta's current SAM 2 code and uses about a third of the GPU memory.

§7 AI · 100%

Results has the full tables. New to video tracking?

§8 AI · 100%

Say you have video, and you need to know where one particular thing is in every frame: this person, that forklift, the dog.

§9 AI · 100%

That is tracking.

§10 AI · 100%

SAM 2 and its smaller relatives, EdgeTAM and SAM 2.1, are models from Meta that do it very well. You point at the thing once, on a single frame, and the model draws its exact outline on every frame after that. tamserve sits between one of those models and your program. Your program hands it the latest frame from each camera and gets back the outline of every tracked object in each one. You can start tracking something new, or stop, at any moment, and cameras can come and go. It is meant for anyone who has to answer "where exactly is this thing, right now, in each of my video feeds": sports and broadcast graphics, warehouse and retail analytics, robotics, wildlife and lab monitoring, annotation tools. There are two ways to say what to track. You can point: a person clicking on the picture, or a detector, says where the thing is, and tamserve takes over from there. Or you can name it: a Tracker takes a word or a phrase, finds everything that matches on every camera, and follows it. Four words come up a lot below: word what it means here frame one image from a video mask the answer for one object in one frame: a grid of True and False the size of the frame, True on every pixel that belongs to the object prompt how you say what to track: a box around it, a click on it, a rough mask of it, or, with a Tracker, a word or a phrase stock the model's own code: the tracker in Meta's EdgeTAM repository, used exactly as it comes Using it Install pip install tamserve You need Python 3.10 or newer and an NVIDIA GPU. It does run on a CPU, which is handy for trying it out but much too slow for live video. The first time you load a model, tamserve downloads it into ~/.cache/tamserve, so that run needs an internet connection. EdgeTAM is 54 MB. The download is pinned to one version and verified before any of it is loaded. A live loop tamserve does not open cameras, and it does not run a loop of its own. Your program reads frames however it likes and calls engine.step() once per round, with the newest frame from each camera exactly as the decoder gives it. This example uses OpenCV to read two network cameras: import threading import cv2 # pip install opencv-python; any source of frames will do from tamserve import Box, Engine, build_video_predictor class Camera: """Reads one stream on a thread of its own and keeps its newest frame.""" def __init__(self, url): self.stream, self.newest, self.arrived = cv2.VideoCapture(url), None, threading.Event() threading.Thread(target=self.read, daemon=True).start() def read(self): while True: ok, picture = self.stream.read() if not ok: break # the stream has ended self.newest = cv2.cvtColor(picture, cv2.COLOR_BGR2RGB) # OpenCV gives BGR, the engine takes RGB self.arrived.set() def take(self): """Waits for a frame the loop has not seen yet, and returns it.""" self.arrived.wait() self.arrived.clear() return self.newest engine = Engine(build_video_predictor("cuda")) # loads EdgeTAM onto the GPU cameras = {"door": Camera("rtsp://door-camera/stream"), "yard": Camera("rtsp://yard-camera/stream")} # What to track and where it is in the first frame: a box as (left, top, right, bottom), in pixels. # In a real system this comes from a detector, or from someone drawing on the picture. start = {"door": {"courier": Box(410, 120, 560, 480)}} while True: # 1. Take the newest frame from each camera. frames = {name: camera.take() for name, camera in cameras.items()} # 2. One call tracks every object in every one of those frames. masks = engine.step(frames, start) start = None # the box was only needed once; from here on the engine tracks the courier by itself # 3.

§11 AI · 100%

Use the answer.

§12 AI · 100%

masks["door"]["courier"] is True on the courier's pixels and False everywhere else. for camera_name, objects in masks.items(): for object_name, mask in objects.items(): print(camera_name, object_name, "covers", int(mask.sum()), "pixels") Each call to step() takes one frame per camera and returns when the masks for those frames are ready. The engine remembers what it is tracking between calls, which is why the box is passed only once. Camera names and object names are yours to choose. The frames go in exactly as the decoder gives them. The engine uploads them and resizes them on the GPU, which takes about half a millisecond a frame where resizing on the CPU takes 16. The Camera class is there for speed too: with a thread per camera, the next round's frames are decoded while the GPU tracks this one. And because a camera keeps only its newest frame, a loop that falls behind skips frames and does not lag. The engine tracks from whatever frame it is given and does not need to see every one.

§13 AI · 100%

How fast can the loop go?

§14 AI · 100%

On an RTX 4090 with the compiled engine, eight 720p cameras and one object each, a loop like this one tracks 273 frames a second from video to masks, which is 34 for every camera. With 1080p cameras it is 238. Pointing at things To start tracking something, you give it a name and point at it in the current frame. There are three ways to point. All of them use the frame's own pixel coordinates, with x counted from the left edge and y from the top. prompt what it is Box(left, top, right, bottom) A rectangle around the thing. This is the most reliable, and it is what a detector gives you. Clicks([(x, y), ...]) One or more points on the thing, as from a mouse. A second list of 1s and 0s can mark each point as on the thing (1) or deliberately off it (0). a mask A True/False array the size of the frame, if you already have the outline from somewhere else. The four things you can do Start tracking something. Pass its name and where it is, together with the frame it is in.

§15 AI · 100%

You do this once.

§16 AI · 100%

Everything already being tracked carries on undisturbed. engine.step(frames, {"yard": {"dog": Box(300, 380, 420, 470)}}) Stop tracking something. engine.drop("door", "courier") # just the courier engine.drop("door") # everything on the door camera Fix a track that has gone wrong. Now and then an outline drifts. It slides onto the wrong person, or keeps only half of the object. Point at the object again under the same name, exactly as you did to start it. Tracking begins afresh from the new box, and the old, mistaken history is thrown away. engine.step(frames, {"door": {"courier": Box(455, 130, 600, 500)}}) # same name, new box Start something you noticed in the frame you just processed. step() moves every tracked object forward by one frame. Suppose a detector looked at the frame you just passed in and found a dog. Calling step() again with that same frame would move everything else forward twice. add() starts the dog on that frame and leaves the rest alone. engine.add(frames, {"yard": {"dog": Box(300, 380, 420, 470)}}) # frames: the ones you just gave step() Following what you name Everything above starts from someone pointing at the thing. A Tracker takes words instead. Give it a phrase, or several, and it finds every match on every camera, follows each one, picks up new ones as they appear and lets go of those that leave.