Provide utility for writing Properties to a file in a repeatable manner

[vpavic]

Instances of java.util.Properties are typically written to file using Properties#store. Since this method also writes a timestamp in a comment, its use is problematic in projects that need to have reproducible builds.

If Spring Framework could provide utility for writing Properties to a file in a repeatable manner, this would IMO be of great benefit to all Spring projects. There are already several issues/PRs that are blocked due to the shortcomings of Properties#store:

• Make it possible for the output from the BuildInfo task to be repeatable spring-boot#14494
• Ensure indexer output is deterministic and repeatable #22383

/cc @dsyer @wilkinsona

[sbrannen]

As @jhoeller mentioned elsewhere, Spring already has a PropertiesPersister interface and DefaultPropertiesPersister implementation.

So, we'll look into providing support to filter out comment lines from the output (likely enabled via a new boolean flag in DefaultPropertiesPersister), which should take care of the timestamp comment.

@vpavic, would that meet your needs?

[wilkinsona]

I'm not sure that filtering comments alone is sufficient. To be guaranteed to be repeatable I believe it's also necessary to order the keys. Without ordering the keys, you are reliant on repeatable hashing of the keys which may not hold true under all circumstances.

Gradle contains some code for writing properties in a repeatable manner which could serve as inspiration.

[vpavic]

Thanks for the feedback @sbrannen, I wasn't aware of PropertiesPersister - a custom implementation would probably do.

Thanks @wilkinsona, I forgot to mention there's also an ordering aspect to this.

[sbrannen]

I'm not sure that filtering comments alone is sufficient. To be guaranteed to be repeatable I believe it's also necessary to order the keys. Without ordering the keys, you are reliant on repeatable hashing of the keys which may not hold true under all circumstances.

OK. Thanks for bringing that up. We'll then have to take the order into account as well.

[sbrannen]

Design Notes

It will likely be easiest to support consistent ordering by extending Properties and overriding the implementation of keys().

To remove the comments, we will likely need to override store(OutputStream, String) and store(Writer, String) to filter out all comments.

Open Issues

@vpavic, @dsyer, @wilkinsona:

• Can you please confirm that you do not need this support for the storeToXML(...) variants?
• Is it acceptable to ignore (i.e., filter out) any user-supplied comments (provided via the String comment argument to store(OutputStream, String) or store(Writer, String))?

[vpavic]

I believe XML variants aren't needed.

Regarding PropertiesPersister itself, it seems a bit obscure as code search doesn't reveal any (production) uses even in Framework's codebase - Properties#store is actually used in all places that write Properties instances.

To me PropertiesPersister looks like an overhead for this, as simple property writing helper with two variants (one accepting OutputStream the other Writer) of static store method would be much simpler to consume across the Spring portfolio. Both load and XML related parts of the API don't seem particularly relevant to the core problem being discussed here.

[sbrannen]

Thanks for the feedback, @vpavic.

I am actually considering only a custom subclass of Properties. If I get that to work, I don't think there will be a need to implement a custom PropertiesPersister, modify DefaultPropertiesPersister, or introduce a new helper class.

[sbrannen]

OK, this feature is basically done, but I'd like to share it with you here first to see if it meets your needs.

New method in org.springframework.core.CollectionFactory (per recommendation by @jhoeller):

/**
 * Create a variant of {@code java.util.Properties} that sorts properties
 * alphanumerically based on their keys.
 *
 * <p>This can be useful when storing the {@link Properties} instance in a
 * properties file, since it allows such files to be generated in a repeatable
 * manner with consistent ordering of properties. Comments in generated
 * properties files can also be optionally omitted.
 *
 * @param omitComments {@code true} if comments should be omitted when
 * storing properties in a file
 * @return a new {@code Properties} instance
 * @since 5.2
 */
public static Properties createSortedProperties(boolean omitComments) {
	return new SortedProperties(omitComments);
}

Implementation of SortedProperties:

/**
 * Specialization of {@link Properties} that sorts properties alphanumerically
 * based on their keys.
 *
 * <p>This can be useful when storing the {@link Properties} instance in a
 * properties file, since it allows such files to be generated in a repeatable
 * manner with consistent ordering of properties.
 *
 * <p>Comments in generated properties files can also be optionally omitted
 * (see {@link SortedProperties#SortedProperties(boolean)}).
 *
 * @author Sam Brannen
 * @since 5.2
 * @see java.util.Properties
 */
class SortedProperties extends Properties {

	private static final String EOL = System.getProperty("line.separator");

	private static final Comparator<Entry<Object, Object>> entryComparator = //
		(entry1, entry2) -> String.valueOf(entry1.getKey()).compareTo(String.valueOf(entry2.getKey()));

	private final boolean omitComments;


	/**
	 * Construct a new {@code SortedProperties} instance that honors the supplied
	 * {@code omitComments} flag.
	 *
	 * @param omitComments {@code true} if comments should be omitted when
	 * storing properties in a file
	 */
	SortedProperties(boolean omitComments) {
		this.omitComments = omitComments;
	}

	@Override
	public void store(OutputStream out, String comments) throws IOException {
		ByteArrayOutputStream baos = new ByteArrayOutputStream();
		super.store(baos, (this.omitComments ? null : comments));
		String contents = new String(baos.toByteArray(), StandardCharsets.ISO_8859_1);
		for (String line : StringUtils.tokenizeToStringArray(contents, EOL)) {
			if (!omitComments || !line.startsWith("#")) {
				out.write((line + EOL).getBytes(StandardCharsets.ISO_8859_1));
			}
		}
	}

	@Override
	public void store(Writer writer, String comments) throws IOException {
		StringWriter stringWriter = new StringWriter();
		super.store(stringWriter, (this.omitComments ? null : comments));
		String contents = stringWriter.toString();
		for (String line : StringUtils.tokenizeToStringArray(contents, EOL)) {
			if (!omitComments || !line.startsWith("#")) {
				writer.write(line + EOL);
			}
		}
	}

	@Override
	public void storeToXML(OutputStream out, String comments) throws IOException {
		super.storeToXML(out, (this.omitComments ? null : comments));
	}

	@Override
	public void storeToXML(OutputStream out, String comments, String encoding) throws IOException {
		super.storeToXML(out, (this.omitComments ? null : comments), encoding);
	}

	/**
	 * Return a sorted enumeration of the keys in this {@link Properties} object.
	 * <p>The keys will be converted to strings if necessary using
	 * {@link String#valueOf(Object)} and sorted alphanumerically according to
	 * the natural order of strings.
	 */
	@Override
	public synchronized Enumeration<Object> keys() {
		Set<Object> sortedKeys = new TreeSet<>();
		for (Object key : super.keySet()) {
			sortedKeys.add(String.valueOf(key));
		}
		return Collections.enumeration(sortedKeys);
	}

	/**
	 * Return a sorted set of the entries in this {@link Properties} object.
	 * <p>The entries will be sorted based on their keys, and the keys will be
	 * converted to strings if necessary using {@link String#valueOf(Object)}
	 * and compared alphanumerically according to the natural order of strings.
	 */
	@Override
	public Set<Entry<Object, Object>> entrySet() {
		Set<Entry<Object, Object>> sortedEntries = new TreeSet<>(entryComparator);
		sortedEntries.addAll(super.entrySet());
		return Collections.synchronizedSet(sortedEntries);
	}

}

@vpavic, @dsyer, @wilkinsona, please let me know your thoughts.

Cheers,

Sam