Describe the bug

Lines in Cloudprober configuration files over 64KiB in length cannot be parsed by a bufio.Scanner are silently ignored

Cloudprober Version

0.14.2

To Reproduce

Take an existing configuration file for Cloudprober that is greater in size than 64KiB, and flatten it so it's all a single line.

Start Cloudprober using this file as configuration. No probes will be run, and querying the /config-parsed endpoint will show minimal/no config.

There is no reported error in the Cloudprober logs giving any indication that configuration was ignored or skipped.

Additional context

We machine-generate our cloudprober configuration files, and up until now have been writing these out via the .String() method on a ProberConfig. This outputs all the configuration on a single line.

Recentlly, when getting to about 55-60 probes in an individual configuration file, we have run into errors where Cloudprober has been executing zero probes on the workers where there are a larger number (55-60) of probes. Looking into this futher, it seems that handleIncludes makes use of a bufio.Scanner but doesn't explicitly check for error cases:

func handleIncludes(baseDir string, content []byte) (string, error) {
	// snip...
	
	scanner := bufio.NewScanner(bytes.NewReader(content))
	for scanner.Scan() {
		line := scanner.Text()
		// handle line
		// snip...
	}

	// scanner.Err() is not called here
	// snip...
}

In this case, we are running into the MaxScanTokenSize for a bufio.Scanner: https://pkg.go.dev/bufio#pkg-constants. The scanner stores a ErrTooLong internally, but this needs to be explicitly checked for by calling scanner.Err().

There are other error conditions that this code is susceptible to, not just our ErrTooLong case, and since these are never detected by calling scanner.Err(), they are not surfaced. Since scanner.Scan() also returns no output in the case where an error is generated, this results in no probes being parsed from the configuration file.