DORI

a smart fish doorbell

Spawning fish struggle to navigate past human-made barriers like dams, disrupting ecosystems. The ‘fish door- bell’ system in Utrecht, Netherlands, uses a live camera feed to allow tourists and viewers to open the dam when they see fish. However, this framework requires constant human monitoring. Our approach, DORI (Detection of River In-habitants), seeks to replace this with an automated system that detects and classifies fish, signaling the dam to open autonomously. We optimize the runtime and performance of the algorithm by integrating and testing the fish detection performance of the state-of-the-art pre-trained models such as Segment Anything Model (SAM), Fast Segment Anything (FastSAM), Recognize Anything Model (RAM), Tag2Text and Contrastive Language-Image Pre-training (CLIP) for image segmentation and classification in the DORI frame-work.

Pipeline overview for DORI. At each timestamp t, an image and corresponding timestamp are passed into the algorithm. Luminance of the image is calculated, and if the difference between the current luminance and the average luminance is above a threshold Lthresh, the image is segmented using FastSAM. The segment masks produced by Fast SAM are then classified by CLIP, and if the probability of the class ‘fish’ for any segment mask is greater than 0.5, the doorbell is indicated to be pushed.

Check out our Github for example Notebooks and code.

Stills from the fish doorbell, taken from TheFishDoorbell.