I've marked my CLI project as open source but it's still counting towards my test limit

Unfortunately this will happen if you run snyk monitor with the --project-name flag, which we don't currently support in snyk test.

The --project-name flag is used to override the default project name, so if the default project name is goof but you specify a project-name of goofy we will look for a project called goofy and apply that project’s policy. As we don't support the flag for snyk test we won't find any policy for the default project goof and therefore count it as a private test.