Neural model speeds and improves text file type detection by four times
Fast And Accurate Text Content File Type Identification
Machine Learning
Summary
It can be hard for computers to tell what kind of file they are looking at just by the content inside, especially when the usual clues like file extensions might be wrong. The authors created a new type of computer program that uses neural networks to recognize different kinds of text files, like source code, faster and more accurately than other tools. Their tests with open-source files show this new tool works better and is quicker than a popular existing method named Magika. It’s also smaller, which helps it run more easily on different computers.
What this means in practice
- •For cybersecurity teams: Identify text file types efficiently and accurately even when file extensions and magic numbers are unreliable, improving threat analysis processes.
- •For large software repositories: Automate tagging and organization of source code files with a faster, smaller, and more accurate detection tool to manage vast amounts of text files.
Authors
Manu Nandan, Michael Brautbar, Edward Raff
Abstract
A common requirement across organizations is to have a tool that can identify file types based on their contents, particularly in the cybersecurity domain where magic numbers and file extensions can not be trusted. While existing tools work well in practice, there is plenty of room for improvement either in terms of computational load and time for detection in the case of model based tools like Magika or in terms of accuracy of detection in the case of file parsing tools that use programming language constructs. In this study, we propose a neural network model for identification of types of text content files, especially source code, that is more accurate and faster than other available tools. Our experiments on open-source files indicate that it is not only more accurate on average for text-content file-type identification, but also approximately four times faster than Magika, while being 28% smaller in size.