My Harness Reported 0 Findings on 100 Files. All 100 Were Ignored.
A recent experience highlighted a critical issue where linting tools reported zero findings across many files, yet the actual number of issues was significantly higher. This discrepancy arose from three common silent failures. First, files can be ignored if they are outside the base path ESLint is configured to scan. Second, with flat configurations, certain file types like TypeScript might not be included by default, leading to them never being processed. Third, in monorepos, linting tools might analyze unreleased source code from linked node_modules instead of the published package. These failures are insidious because they do not produce errors or non-zero exit codes, leading to a false sense of security. The tools simply report successful completion with no findings. Such silent failures can lead to inaccurate reporting, as a low or zero finding count is often celebrated rather than investigated. To prevent this, it is crucial to verify that the linting tool actually processes the intended files. A simple test is to introduce code known to trigger specific rules and confirm the tool flags it. If this "canary" code is ignored, the entire linting result is unreliable. This underscores the importance of verifying the integrity of the tested corpus and the reproducibility of reported results.
node_modulesinstead of the published package. These failures are insidious because they do not produce errors or non-zero exit codes, leading to a false sense of security. The tools simply report successful completion with no findings. Such silent failures can lead to inaccurate reporting, as a low or zero finding count is often celebrated rather than investigated. To prevent this, it is crucial to verify that the linting tool actually processes the intended files. A simple test is to introduce code known to trigger specific rules and confirm the tool flags it. If this "canary" code is ignored, the entire linting result is unreliable. This underscores the importance of verifying the integrity of the tested corpus and the reproducibility of reported results.