set track_avg_cnt: true by default

[mahf708]

          >@AaronDonahue should we maybe turn on avg count tracking automatically for this diag? Or, more in general, for any field that can have a certain extra data set in the header (something like "can_have_invalid_data" or something)?

Yes, I think we would want to do that rather than have users try to figure out why some fields are nan, or otherwise bad values.

Originally posted by @AaronDonahue in #2916 (comment)

[mahf708]

Also, note to fix this in cosp:

scream/components/eamxx/src/physics/cosp/eamxx_cosp.cpp

Lines 235 to 243 in a1b89eb

	// If not updating COSP statistics, set these to ZERO; this essentially weights
	// the ISCCP cloud properties by the sunlit mask. What will be output for time-averages
	// then is the time-average mask-weighted statistics; to get true averages, we need to
	// divide by the time-average of the mask. I.e., if M is the sunlit mask, and X is the ISCCP
	// statistic, then
	//
	// avg(X) = sum(M * X) / sum(M) = (sum(M * X)/N) / (sum(M)/N) = avg(M * X) / avg(M)
	//
	// TODO: mask this when/if the AD ever supports masked averages

And maybe another issue in cosp: E3SM-Project/E3SM#5495 (comment)

[mahf708]

@AaronDonahue + @bartgol: moving the discussion here.

Is there any downside of setting the default track_avg_cnt: true for everything? No need to special-case it, just turn it on everywhere?

[bartgol]

Having track_avg_cnt: true by default implies that we need to accumulate and write to file a few more fields. While the number of extra fields may not be remarkable, for use cases where we split output into many files (like prod runs often do), it may add up to a non negligible amount of extra output. As such, I am inclined to have IO detect whether or not track_avg_cnt should be on, and remove it from the yaml file options.

[mahf708]

Closing this as it is an item in #2938, so no need for a separate issue.