New files[dd].xml filesize reduction suggestion.

xforx · Post by **xforx** » 2004-04-15 12:35

Hi,
The new filelist format (xml) has made filelists quite big thanks to TTH (which is a good thing) and the not very size conscious XML format.

For instance taking a demo files1.xml weighting ~1.8mb and replacing the longish tag/attribute names (Directory -> D, File -> F, Size -> S) reduces it to ~1.3mb, bz2 compressed the difference is smaller (280kb vs 204kb --> 76kb) but still hefty and I'll bet it scales very nicely as filesize grows.

So my suggestion is simple, change the tag/attribute names. For every file list a client downloads reducing the size by ~70k on average (might total to MB of data) isn't a bad tradeoff to consider (taking into account that the only down side I can think of is slightly weirder attribute names, a thing which can be clarified in comments).

Post by **GargoyleMT** » 2004-04-16 00:54

Interesting bit of size micro-management.

Nothing else varied - your list was fully hashed, so no new hashes or files were added to it? You did the replacement inside DC++ as well, not using an external archiver, etc.?

Sedulus · Post by **Sedulus** » 2004-04-16 06:02

I do not get the same results

Code: Select all

[0:wza:utf8test]$ cat files50.xml |sed -e 's/<\(\/\?\)Directory/<\1D/g' -e 's/<File /<F /' -e 's/ Size="/ S="/' -e 's/ Name="/ N="/' > files50.xml.smaller
[0:wza:utf8test]$ ll
total 13436
-rw-r-----    1 walter   users     7233367 Apr 14 21:41 files50.xml
-rw-r--r--    1 walter   users     6499629 Apr 16 07:59 files50.xml.smaller
[0:wza:utf8test]$ bzip2 *
[0:wza:utf8test]$ ll
total 4096
-rw-r-----    1 walter   users     2089908 Apr 14 21:41 files50.xml.bz2
-rw-r--r--    1 walter   users     2090208 Apr 16 07:59 files50.xml.smaller.bz2

ivulfusbar · Post by **ivulfusbar** » 2004-04-16 07:42

Code: Select all

ll *
-rw-r-----    1 fusbar   users     7347564 2004-04-16 09:53 files1.xml.bz2
-rw-r--r--    1 fusbar   users     7331777 2004-04-16 09:55 files1.xml.smaller.bz2

Repetition of Seds example... and i concure with the results.

xforx · Post by **xforx** » 2004-04-16 09:51

GargoyleMT wrote:Nothing else varied - your list was fully hashed, so no new hashes or files were added to it? You did the replacement inside DC++ as well, not using an external archiver, etc.?

Well, actually, filelist wasn't fully hashed (didn't have the stamina to do that for a test). And yes, I used an external bz2 archiver, however I did compare the original bz2 and the one created by the other program (they were on par).

About the simmingly non fitting results, my filelist was 1.8mb without hashes, hash data introduces both bulk and entropy to the file so I have no idea how your lists compare to the one I tested (sizewise), can you (ivulfusbar or Sedulus) try repeating the experiment with a filelist of around 2mb (not including the hash) and post complete data (size before compression and after).

Sedulus · Post by **Sedulus** » 2004-04-16 15:33

without the tth, it's better, but still no real difference

and as everyones filelist will have no tth-less entries in the future, hopefully, it's even less convincing

Code: Select all

[0:wza:utf8test]$ ll *notth*
-rw-r--r--    1 walter   users     4763581 Apr 16 17:31 files50.xml.notth
-rw-r--r--    1 walter   users      745259 Apr 16 17:31 files50.xml.notth.bz2
-rw-r--r--    1 walter   users     4029843 Apr 16 17:29 files50.xml.smaller.notth
-rw-r--r--    1 walter   users      724792 Apr 16 17:30 files50.xml.smaller.notth.bz2

ivulfusbar · Post by **ivulfusbar** » 2004-04-16 16:35

From a mathematical viewpoint it quite obvious that it doesn't matter much.. if you consider the algorithm used in the bz2-format. I guess we could write down a mathematical proof if you want... ,))

xforx · Post by **xforx** » 2004-04-17 17:14

Yes, actually I would like to see you give a mathematical proof given no compression algorithm I know of is perfect (bz2 including); for instance take a several GB's file containing only one letter say 'a', theoretically the only information you need to reproduce it is it's size and the letter. Try compressing it and you'll get a much bigger file which incidentally can also be compressed with the same algorithm to a smaller size, so, it does matter. In this case I think more tests are needed with different list files and besides given there are no drawbacks I can't see a reason not to implement it.

Post by **GargoyleMT** » 2004-04-17 20:37

xforx wrote:given there are no drawbacks I can't see a reason not to implement it.

Drawback one: XML file lists after thic change will be incompatible with old versions
Drawback two: it doesn't save enough space to be worth it

New files[dd].xml filesize reduction suggestion.

New files[dd].xml filesize reduction suggestion.

Who is online