Jump to content
  • Announcements

    • AndalayBay

      Orphan Attachments   07/31/2018

      I have been doing some housekeeping lately and I've noticed that I had a lot of orphaned attachments. Attachments get orphaned when the PM or post is deleted without removing the attachment first. Deleting a PM or post does not delete the attachment and the file or image remain on the server. I'd like to ask all members to go through their attachments and delete any attachments you don't need anymore or those that have been orphaned. Where can I get a list of my attachments? Click on your display name in the upper right corner of the forums and pick "My Attachments" from the drop-down list. How can I tell an attachment is orphaned? If the PM has been deleted, you'll see a message like this in your attachment list: Unfortunately there is no message if the post has been deleted, so please check your old posts. We do purge old birthday threads every once in a while. Also some hosted projects have been shut down, so you may have orphaned attachments on one of those locations. Thanks!
Sign in to follow this  
Visceral Moonlight

Basic Comparison of Archive Formats

Recommended Posts

This will be a general overview of various archive formats.

 

For this thread, I'll be focusing on some of the more common ones.

 

Fist a list:

 

We'll be covering tar, RAR, 7z, and ZIP.

 

tar is a file format for archiving only. It keeps file permissions, making it ideal for backups. It is an open format, meaning a license isn't needed to make a utility to support tar archives. tar is also the name of the Unix utility used to work with them. tar doesn't use compression, so you need to compress it separately after making a tar archive. gzip and bzip2 are commonly used for this, giving you archive names that end with .tar.gz and .tar.bz2

 

RAR is a proprietary format. Archives can be created using WinRAR. Being proprietary, a license is needed to support the format and the company can always say that one can no longer support the file format. For unpacking, 7-Zip has support but does not support creation.

 

7z is 7-Zip's main archive format, which is licensed under the LGPL. It doesn't retain file permissions, as I recall. It supports the following compression algorithms: LZMA, LZMA2, Bzip2, DEFLATE, and PPMd. LZMA and LZMA2 have improved compression ratios over the other three with LZMA2 having improved multi-threading support over LZMA.

 

ZIP is supported by just about every OS but doesn't compress very well. It is the only archive format that Windows supports without installing another program.

 

 

Out of the archives that do compression, 7z (using either LZMA or LZMA2) and RAR tend to compress files to the smallest sizes. They're actually about equal in the amount of compression. However, due to RAR being proprietary and having little support outside of the proprietary programs, I recommend that one use 7z. When it comes to RAR, there's always the chance that you'll have to pay to extract the contents in the future and certain operating systems are harder to work with to extract RAR files. Since the two archives compress roughly the same, the fact that 7z isn't proprietary is a huge plus and, in my opinion, is the superior format.

 

As for compression options for 7z, there are quite a few. The main ones are the compression level, threads, and solid compression. With more threads, file size may vary but it'll be faster to compress. As for compression level, this is a trade-off. At the highest setting, you save the most space. However, it also takes the most time to pack and unpack. Solid compression tells it to consider all of the data as a single unit, so you get a smaller file size. Again, it also takes longer to pack and unpack.

 

Personally, I use it archive files so I set the compression to level 9 using solid compression.

 

 

For transferring files between systems (like a file system structure) that you wish to keep the permissions, tar is probably going to be your go-to format to use. If you want to ensure that everyone can use an archive, ZIP is a good fit for that since almost every OS has built-in support for ZIP archives.

 

 

 

And that's a general guide on archive formats. Hopefully, it comes in handy. :)

Share this post


Link to post
Share on other sites

Hey, thanks for this. Just been investigating LZMA2 and it'll mean 7zip'll use all 8 cores (hyper-threading) rather than the 2 which LZMA does. I've been using LZMA for years without knowing this! That'll help when compressing some larger archives.

 

Also. Wow, ARJ: there's a blast from the past. Remember messing around with that on the DOS command line. Kids these days, don't even know they're born with their shiny GUIs...

 

Not sure if it's relevant or needed but back in the day (before 7zip really took off) a number of mods for Morrowind were distributed using the (proprietary) ACE format. ACE seemed to deliver better compression for media files like textures than either ZIP or RAR. Those were the main MW formats: mainly ZIP, but RAR and ACE too. Someone snuck onto Bethsoft and said: "Hey, I don't understand why you guys don't use 7zip." People looked at the compression (still a lot of people on dial up even then) and were like "Oh. My. God." Heh.

Edited by Dragon32

Share this post


Link to post
Share on other sites

Still doing research but it is looking like Peazip is the tool of choice for unpacking ACE archives.

I've been hearing a lot of good things about Peazip recently. I may have to have a look.

Share this post


Link to post
Share on other sites

Haven't gotten to PeaZip yet but I did do some more testing with 7-Zip.

 

I updated to 9.38 yesterday and compressed about ten archives with my usual "Ultra" settings using a single thread and all of the archives wound up at least 150 kB smaller. These archives were a mix of RAR, Zip, and 7z.

 

For my next test, I'll try compressing files using the same settings under both 9.20 and 9.38 to see how things compare.

 

9.38 does have one killer feature: the ability to grab a file's CRC by simply right-clicking on it. This feature is incredibly useful.

 

This is getting to be kind of fun. :D

Edited by deaths_soul

Share this post


Link to post
Share on other sites

I found a good reference page for the command-line parameters for 7-Zip: http://sevenzip.sourceforge.jp/chm/cmdline/switches/method.htm

 

To further explore the compression ratios of archive types and algorithms, I'll be experimenting with those switches to see how speed and size compare. My primary focus is size as I always seem to not have enough storage space but speed is also important for systems that are mainly for storage, like file servers.

 

Edit:

 

Some of those settings can set in the GUI like, dictionary size.

Edited by deaths_soul

Share this post


Link to post
Share on other sites

LZMA2 was created for XZ format and it includes changes that are good for that stream compression format. Also LZMA2 is better than LZMA, if you compress already compressed data. LZMA2 is faster for 4-threads, if you compress big file (more than 256 MB), so 7-Zip will be able to split it to blocks. More about...7zip

 

Brian

Share this post


Link to post
Share on other sites

The latest WinRAR version has both the new RAR5 format and the RAR format available for users to select which format an archive will be created.  Note that if an archive has the RAR5 format it cannot be opened with older WinRAR versions.

 

Another thing I've notice is that the archives and larger archives created in WinRAR is smaller than the previous versions created.  In comparison to 7z the RAR archives is smaller, even the larger archives is smaller, than to 7z.

Share this post


Link to post
Share on other sites

With 7z archives, you need to tweak the settings to get the optimum ratios as described. In some cases, RAR still gives better ratios but, overall, 7z with LZMA2 gives better ratios for the majority of the archives and is constantly improving.

 

Generally, the development versions of 7-Zip are safe to use and contain the latest modifications so those are the versions that I recommend using.

 

I've done a lot of testing using a variety of archives and in the cases RAR wins, it's usually by a few MBs, or even KBs, with the settings I use. For reference, I test with archives at least a few gigabytes in size.

 

The cases aren't as often as 7z winning out so I recommend 7z with LZMA2 to get consistent results.

Edited by Visceral Moonlight

Share this post


Link to post
Share on other sites

LZMA2 is faster for 4-threads, if you compress big file (more than 256 MB), so 7-Zip will be able to split it to blocks.  If the zip file exceeds that size, 7-zip will split it into multiple files automatically, such as integration_serviceLog.zip.001, integration_serviceLog.zip.002, etc. (Way back when, PK Zip used this to span zip files across multiple floppy disks.) You'll need all the files to be present to unzip them. LZMA2 was created for XZ format and it includes changes that are good for that stream compression format. Also LZMA2 is better than LZMA, if you compress already compressed data. LZMA decoder is simple. 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×