The broken RSS feed mystery... when you are sure you didn't change anything, it just broke by itself

Kartris is a long-established open source ecommerce package. In order to keep users informed of updates and news, we have an RSS feed which is built into the back end admin. This connects to an RSS/XML document on our web site at a fixed URL.

It's not a critical feature, but because admin users see the feed everyday, they tend to notice when it's down and contact us. We had a couple of clients report it, and verified our end that it wasn't working on a local dev version of the software either.

The message isn't very helpful, it's just a default response we return when an error occurs and we can't show the feed.

The XML document is edited manually, so there is the possibility of breaking it with bad syntax, or invalid dates, for example. But since we normally check it is working after an update, and the last one was a couple of months before, it seemed unlikely this was the problem.

So it was time to run in debug to try to step through and see what was going wrong. This revealed that the XML was malformed. In particular, the opening and closing tags in many places did not match.

Each release version is represented by an "item" which looks like this in the XML:

 	<item>
	  <majorVersion>3</majorVersion>
	  <minorVersion>3.3000</minorVersion>
	  <beta>True</beta>
	  <releaseDate>Thu, 09 Jun 2022 14:25:00 GMT</releaseDate>
	 <type>Beta</type>
	  <bugFixes>
	* Fixed issue where new Worldpay DLL still reported itself as RBSWorldpay
	  </bugFixes>
	  <improvements>
	* New tblKartrisProductSearchIndex table used to store the min/max values of products. This makes for significantly better performance in searches or filtering by product price.
	* New triggers and sprocs that ensure this new index is maintained, with version, option or qty discount changes all triggering a recalculation.
	* Full clean up and rebuild of indexes of all tables, PKs using clustered indexes, consistent naming for all indexes, removed duplicate indexes, added some indexing to additional fields in places.
	  </improvements>
	</item>	

In particular, the error message suggested that the majorVersion tag had a lower case opening tag, but the closing one retained the camel case. We FTPed into the server and checked the file, and it looked good. Both tags were camel case, as they should be.

So we got back to thinking about why this stopped working outside of us changing the file. The XML parsing in Kartris is handled by .NET, so perhaps those components had been changed in a recent Windows Update, which was buggy? It seemed unlikely, and we know from experience that many people are quite tardy in running updates. But we soon eliminated this as a possibility by viewing the incoming stream and seeing the lowercase opening tags in there.

We then did what we probably should have done earlier, which is to view the feed URL in a browser, rather than view the document via FTP. Viewing the source of this confirmed the opening tags were indeed lowercase. So that was how our site was serving the document, even though the document was not really like that.

Then the penny dropped. The site is served through CloudFlare, and has CSS, Javascript and HTML minification turned on. That removes whitespace and comments, but since that involves rewriting the HTML to some degree, it seemed likely that something was going wrong there.

We turned off HTML minification for the site on the CloudFlare panel, and cleared caches. Sure enough, the problem disappeared!

So this appears to be a bug in CloudFlare that has been introduced at some point in the last few days or possibly weeks. The HTML minification changes the case of tags. Curiously though, only opening tags, the closing tags were left unchanged. Changing the case in this way has no effect on the document size, so there does not appear to be any reason or benefit in doing this. We can only assume it is an unintended consequence of an "improvement" they have made.

We have posted the issue to CloudFlare's public forum.