Aug 16, 2011
tom
No Comments

How can I visualize hard disk space with millions of files?

Question

We have a hard disk that is 600 Gigs and nearly full. It’s been filled up with 18,501,765 million files (mostly small 19k images) and 7,142,132 folders. It’s very difficult to find out where exactly all the space has gone too. Our regular cleanup procedures are not clearing up enough space which means we need to look at this drive as a whole and determine what is out there and what can be moved or removed. We’ve tried several applications and so far they have either blown up or simply ran for an amazing amount of time to complete.

Server Information

  • Operating System: Windows Server 2003
  • File System: NTFS

Solution

Space ObServer was able to read through 18,501,765 million files and 7,142,132 folders without taking up hardly any memory. I’m sure this is mostly due to the fact that it uses a SQL backend to store all of the data. It unfortunately the most expensive of all the products at $259.95 per server.

Attempted Solutions

During my research I tried several different solutions both pay and free. I kept a list of the products I tried below for everyone’s information.

Free Software

Pay Software

Updates

Update #1: The server I am attempting to analyze has 2 GB of RAM and most products that I try seem to try and keep the file/folder information in memory. This tends to run out much too quickly with 18,501,765 million files and 7,142,132 folders.

Update #2: Looks like the developers of WinDirStat got involved enough to tell us that it can compile under 64-bit. That gives it more memory to work with but I’m not sure if it’s going to be enough unless they can persist to disk.

Answer

Assuming your OS is Windows…

Either way you slice it, tabulating millions of files is always going to take a long time and will be restricted by the I/O of the disk itself. I recommend TreeSize Professional. Or maybe SpaceObServer. You could give the freeware version of TreeSize a try as well.

Check more discussion of this question.

Related posts:

  1. Does Cloned VM use the same Hard Disk space of Guest?
  2. Hard Disk Un-used space
  3. How to delete data and free up disk space in Oracle DBMS 9i
  4. Clean windows and ubuntu to free up disk space
  5. rm on a directory with millions of files

Leave a comment