Counting the fish in the waters off Darwin has, until recently, been either deadeningly dull or potentially deadly, depending on the way you go about it.
The waters are home to salt water crocodiles (like this 4.27 metre specimen caught in June), sharks and jellyfish making diving particularly perilous.
So for the last few years, the Northern Territory Department of Primary Industry and Resources (DIPR) fisheries department has submerged frames fitted with a GoPro camera facing a bait bag, to film the fish that approach for a feed.
A few hours later, once the burley was gone, the frame would be hauled up and the footage taken back to shore, to be viewed and the fish identified and counted.
“That was hours of effort – usually by someone who has a PhD – sitting there watching video of fish swimming past for hours on end. And effectively saying ‘yes fish, not fish, yes fish’,” explains DIPR chief information officer, Rowan Dollar.
“As fascinating as that would be to spend hours doing, machines are getting better at it and are certainly a lot quicker,” he adds.
Earlier this year, DIPR partnered with Microsoft to develop a machine learning enabled alternative to the scientist sitting for tens of hours at a screen. The company has likened it to Facebook’s facial recognition feature, but for fish.
Snapper backer
In 2016, the NT government established a number of temporary reef fish protection areas, where commercial and recreational fishing are prohibited.
The aim of the protection areas is to prevent the over-fishing of vulnerable species such as the golden snapper and black jewfish.
“These are two commercially and recreationally important species in the Northern Territory, but research has proven that they had been overfished around the greater Darwin area,” says DIPR fisheries scientist, Dr Shane Penny.
It is these areas the camera monitoring work has taken place.
“Part of the research is around what the impacts are of closing those reefs to fishing – how quickly do the stocks come back? Do we have the same balance of stocks returning? Are there more of one species or less?” says Dollar.
The machine learning fish counting platform was built using Azure Cognitive Services, and the first iteration of the system was up and running within a month.
The challenge of being able to recognise what is and isn’t a fish and its species, was significant. The waters in the area are “green murk” with eight metre tides. The fish also present at different angles to the camera, meaning the system needed to be more sophisticated than recognising a face in a still photo.
Hundreds of thousands of hours of footage were fed into the engine.
“The solution was widely deployed within six months, and its identification powers have been progressively enhanced using machine learning ever since. The AI system is now able to identify a fish in a video with 95 to 99 per cent accuracy,” says Lee Hickin, national technology officer at Microsoft Australia.
The system is “getting pretty close” to being able to identify which species of fish is present, Dollar adds.
The open source solution has now been made available on GitHub.
“Freed from the mundane aspects of counting and identifying fish, scientists can instead take the insights from the AI solution and focus on making informed decisions that have significant environmental and economic impacts,” Hickin says.
Deep dive learning
The solution has potential uses elsewhere in the territory. Its use in monitoring feral fish in freshwater river systems is being explored.
“Again you don’t go swimming in those, we’ve got lots of crocodiles. I mean lots of crocodiles,” Dollar says.
There’s also an application to monitor commercial fishing trawlers.
“We could look into setting up a camera on a trawler that’s out at sea and doing on-the-fly identification of the catch, so we can start measuring by-catch. At the moment that’s being done by weight, but if we can do that by numbers we can get a better handle on how big the populations are,” says Dollar.
With a real-time population count “we can increase quotas and manage quotas for that fishery and that trawler at that point in time, when you’re out there fishing” Dollar adds.
Similar work – using machine learning to detect and identify fish species in footage captured aboard fishing vessels – is underway by scientists from CSIRO and Data61.
Their Wanda software is being trained on imagery supplied by the Australian Fisheries Management Authority (AFMA), and could replace the human observers that monitor major commercial fishing operations.
The Nature Conservancy’s FishFace concept won the popular vote in Google’s Australia Impact Challenge in 2016. The device is designed to “use facial recognition technology to automate the collation, at sea, of information on the species and numbers of fish caught, and use these data to inform management decisions” and is being trialled with fishermen in Indonesia.
Microsoft claims its solution comes at a time when global fish stocks are facing “greater pressure than ever before”.
“The fact that the solution is open source creates potential for similar platforms to be deployed in different settings around the world, to support important scientific endeavours that will benefit the earth’s environment and humanity,” the company says.