RAMDrive Users [Brainstorm]

[Legolash2o]

Many users are starting to use RAMDrive's, we need to get together and brain storm for ideas of how integration can be faster or any ideas relating to speed.

 

At the moment i'm added the '/ScratchDir' switch to DISM so that in the next release it will use your RAMDrive, if WinToolkit is told to used that anyway.

[crashfly]

I like the idea of using a RAM disk (just started using one due to the discussion about it), however I think we need to figure out a "baseline" for what the bare minimum needed for integrating.  Some have pointed out that one needs a substantial drive to make all of the changes.  Likely we also need to have minimum RAM amount specs to even be able to work with a large enough RAM disk.

[Stimpy]

Do we have any clue to a rough size estimate for the RAMDrive?  How is the ScratchDir used?  Is it only for decompressing CABs or is the whole Windows image compiled there?

 

EDIT...

 

OK, using the new v1.4.1.9,  I have started a test integration, and have noticed a folder called WinToolkit_Temp in the root of my C: drive.  I will work on the assumption that this is the scratch folder, if so, then it's peak size got to 10.6GB, this is based on a Windows 8 Ent x86 integration with only tweaks and 61 hotfixes applied... No addons.

 

Judging by how large the temp folder got, it would be insane to use a RAMDisk, as you would need a system with more than 16GB of RAM, at least 24GB would be my recommendation, as the RAMDisk size would have to be at least 16GB, and maybe larger for some people that integrate schit-loads of large programs and addons.

 

 

I still think that DiskIO (which on my SSD powered system, is running at between 0 and 21% disk utilisation during hotfix integration) is not the reason a high-end PC is slow at integrating.  DSIM or maybe WinToolkit itself is the limiting factor, in my opinion.  DiskIO peaks during the final stage of integration, while the image is actually being made, this part takes around 1 to 2 minutes on my system.  The next operation that maxes out DiskIO is when you make an ISO image, this takes around 1 Minute, maybe 2 if it's a large image, over 6GB.

 

Maybe a RAMDisk could help a lower end system (with a crazy amount of memory) that is still using a mechanical HDD, in this case, UPGRADE to an SSD, as it's stupid to run a high-end system with a mechanical HDD these days.

 

 

BTW, this test integration took 23 Minutes on my system.  No RAMDisk used.  I would predict that I could have saved maybe 1 to 3 minutes of my life, if I was using a RAM disk. Maybe!

 

 

System specifications:

Intel i7 2600K @ 4.6GHz HT Enabled

16GB 2156MHz Memory, @ CAS 9

240GB Intel 520 SSD @6Gbps

Edited January 24, 2013 by Stimpy

[Thiersee]

Another factor you should not forget on low-end computer:

 

CPU must work a lot faster as in normal case, temp goes high and cooling may getting a problem.

 

On my computer, for example, I got the CPU to 40-60 % on all 4 cores and temp of 42-44° by using HDD; using SSD the CPU goes t0 100% on all 4 cores and temp to 53-54°.

 

OK, the temps are not so dangerous, but sometimes it beeps, that temps are going to high.

 

MfG, Thiersee

[Stimpy]

Aaaaahhhh, I have found the REAL scratch directory, and can see how it works...  Interesting that it only serves as a place where the hotfixes are decompressed to.

 

Looking at the scratch directory only, a fairly small RAMDrive could be created, maybe 2GB minimum, 4GB recommended as a safety net to catch any larger hotfixes.  One hotfix (KB2769165) got the directory up to just over 1.66GB.

 

But for those wanting to cover all bases, the entire ToolKit directory in the Temp folder should be cached, as the cabs are decompressing outside of the scratch directory (C:\Users\Username\AppData\Local\Temp\WinToolkit).

 

UPDATE...  So WinToolkit actually has 2 folders (found so far) that are being used as a cache. The first one being C:\WinToolkit_Temp and the other contained in C:\Users\Username\AppData\Local\Temp\WinToolkit.  A RAMDisk would need to contain both of these directories for performance sake, and would require a 16GB minimum RAMDisk partition, with 24GB being recommended.  That's too rich for my taste, as the 2 or 3 minute gain is just not worth the cost in RAM.  This is based on the fact that an advanced, or high-end PC uses a good SSD, so YMMV.

 

Again, my observations on performance from my previous post apply.  This is not the holy grail, I'm pretty sure of that.

Edited January 24, 2013 by Stimpy

[Legolash2o]

The C:\WinToolkit_Temp will most likely just be for the mounting.

[Stimpy]

The C:\WinToolkit_Temp will most likely just be for the mounting.

 

Iv'e noticed that it's size changes all the time while the Toolkit is integrating the hotfixes.  I guess it's only updated after each hotfix has been processed by DSIM, but on a mechanical HD, this would cause head thrashing, and would slow down performance on such a machine.

 

But why anyone who has a system with 16GB or more RAM would be using a mechanical HD is beyond me.

[Stimpy]

I'm just throwing this out there for debate...

 

Does anyone think that the Scratchdir command of DSIM could supposed to be used for debugging purposes, rather than speeding it up?  As if it's run without the scratchdir command, wouldn't it just be doing it's work in RAM anyway?

Edited January 24, 2013 by Stimpy

[Legolash2o]

If you don't provide a scratchdir command it uses windows default temp directory. Apparently you can use also the scratchdir switch when mounting too. I wonder if that is also moved to RAMDrive speed will be increased?

[crashfly]

If you don't provide a scratchdir command it uses windows default temp directory. Apparently you can use also the scratchdir switch when mounting too. I wonder if that is also moved to RAMDrive speed will be increased?

And that might be an issue if one uses the RAM disk as the temp directory for windows (and it is too small).

 

Since the bulk of DISM space usage will be on the mount point (until changes are written back to the WIM file on unmount), it would seem to be a good suggestion to leave the mount point on the fastest drive (preferrably an SSD with enough space).  As previously pointed out, using 16gb of ram for a RAM disk is not possible for the majority of people.  So at least the scratch directory to a RAM disk would work with a minimal size of 4gb.  (Luckily, I am already there are that one.)

[caphp]

I'm sorry Lego, but the speed with /ScratchDir ( v1.4.1.9 ) is a little bit slower than before.

 

I tested this way:

 

Ramdrive and SSD and WinToolkit 1.4.1.9
 

Win Toolkit Options – Main (my changes)
 

Error Logging OFF
Mount Logging OFF
Registry Logging OFF

 
Win Toolkit Options – Misc (my changes)

 
Win Toolkit Temp Folder  (RAM  1GB   T:\Temp)
Win Toolkit Mount Folder (RAM 20GB   M:\Mount)
‘Update Catalog‘ Download Folder (RAM   M:\Updates\McRip Windows 7 x64)
 

Using Windows 7 Professional x64 SP1 (X17-59885.iso unpacked on SSD   C:\w7iso)

Downloading McRip Windows 7 x64 Updates (2013/01), 384 *.msu files

Using Win Toolkit v1.4.1.9 and All-In-One Integrator
 

Result: 46m31s

 

Refering to my last tests, the result is 1 minute slower than with v1.4.1.8

 

 

There are two processes ( dism.exe and dismhost.exe ) which permanently writes to "C:\Windows\Logs\DISM\dism.log" and "C:\$LogFile (NTFS-Volumeprotokoll)".

Maybe there will by a very little speedup, when these processes will write to ramdrive ?

 

But the main difference between using a ramdrive last summer and today with WinToolkit is the speed of update integration.

The speed now is about 9s-10s per update integration.

Last summer ( sorry, but I can't refer to the exact toolkit version ), it was about only 2s per update.

So it was possible to integrate all updates plus programs and drivers in about 14 minutes.

[Stimpy]

If you don't provide a scratchdir command it uses windows default temp directory. Apparently you can use also the scratchdir switch when mounting too. I wonder if that is also moved to RAMDrive speed will be increased?

 

Yeah, I agree, it sounds like that's what it's used for.  I just wonder how much use a 20GB minimum sized RAM disk will be used.

 

I will say it again, if you have 24GB of system RAM that your willing to make a 20GB RAMDisk with, and your still using a mechanical hard drive, you need you head examining!  Upgrade your system!

 

I cannot help test this, as I don't have 24+GB of RAM going spare.  Otherwise it could be interesting to play with.  I would still put money on not getting more than a 5% speed increase on a good system with an SSD.  On a system with 24GB of RAM and a mechanical HD, the benefits would be huge.

[Legolash2o]

I manage to use 6GB RAM Drive ok, but for some reason it won't create a RAM Drive in Windows 8 x64 SSD would be a great upgrade for a PC, it's like having a new computer.

 

I could add an option for the user to be able to select:

 

Win Toolkit Temp

Mount Temp

ScratchDir Temp

 

People also forget their anti-virus will slow down the update process a lot. The best and only way to make it way faster is if I removed DISM and made Win Toolkit handle everything manually.

[Stimpy]

Sounds like some interesting ideas Lego,  Give people the power to do stuff the way they want it done.  As long as you don't give yourself a brain fart making it happen  :gleam:

 

I do think DISM is the limiting factor here, at least on a high-end PC.  But can you make the Toolkit handle all this stuff directly?  Hats off to you if you can pull that one off!  :prop:

[Legolash2o]

I do think DISM is the limiting factor here, at least on a high-end PC.  But can you make the Toolkit handle all this stuff directly?  Hats off to you if you can pull that one off!  :prop:

 

DISM probably had a whole team of people to create it, i suppose I could monitor an integration and see how it works fully.

[crashfly]

DISM probably had a whole team of people to create it, i suppose I could monitor an integration and see how it works fully.

I am of the mind of "why re-invent the wheel?"  While "doing it" yourself could give speed benefits, I think it would take away from the work on your program done so far.  But hey, you *are* the developer.

[caphp]

I am of the mind of "why re-invent the wheel?"  While "doing it" yourself could give speed benefits, I think it would take away from the work on your program done so far.  But hey, you *are* the developer.

Normally I would commit your statement, but "this" wheel has a lot of rough edges and when it is possible to make the wheel round again, this would be more than nice !

[Legolash2o]

I can't do anything at the minute as I'm currently updating my Win7Disk with the January updates. I've noticed it can sometimes take a while when it cleaning temp files during integration. Maybe I can fix that

[amaltom61]

Is there really any speed increase while using a ram drive compared to a SSD and HDD

[Legolash2o]

SSD vs HDD... Hell yes there is a difference. Haven't really tested with RAMDrive.

[crashfly]

Is there really any speed increase while using a ram drive compared to a SSD and HDD

 

As Legolash2o mentioned, hell yes.  For more direct comparisons, I found a performance test someone did compariing SSD and HDD.  I do realize that the hardware they are using are macs, but the same performance increase should hold over on windows hardware. link: http://www.tomshardware.com/reviews/ssd-notebook-portable,1913-5.html

[Legolash2o]

Those results in that link are definitely old. my SSD (average) opens word up in less than a second.

[deadrop]

If you are using SSD, even though the additional speedup from using a RAM drive may only provide a slight boost, the RAM drive also has the advantage of slightly reducing the wear and tear on your SSD (since SSD's have a limited, but large, number of write cycles).  If the RAM is available,and you don't anticipate needing the RAM for other tasks while integration is processing, I think it makes sense to use the RAM drive.

 

It might be helpful if there were some recommendations on which locations (disk properties - write and read speeds (for large and small files) and capacity) are preferred for various groups of files.  For example, if you have only one traditional HDD, your integration times will probably be long, since the disk will be thrashing.  Putting some of the files (perhaps updates) on a USB 3.0 flash drive might speed up the overall integration, even if the USB drive is slower.  If two HDD spindles are available, which files should be located on different HDD's to minimize thrashing?  If a 2GB RAM drive is available, which part of integration would most benefit from having its files on the RAM drive?

 

I am of the mind of "why re-invent the wheel?"  While "doing it" yourself could give speed benefits, I think it would take away from the work on your program done so far.  But hey, you *are* the developer.

I agree.  Staying with DISM also has the advantage that resulting image is (as far as I know) "supportable", since it was built with MicroSoft provided tools (as long as only a subset of the WinTookit features are used).

[caphp]

Is there really any speed increase while using a ram drive compared to a SSD and HDD

 

No personal attack, but why are people asking this question ?

 

A average HDD read/writes about 100 MB/s.

A good SSD read/writes about 500 MB/s.

 

...and no one asks, if a SSD is faster than a HDD, but...

 

A average DDR3 (1600MHz) with 9-9-9-27 timings read/writes about 15000 MB/s

 

So why are there so many users asking, if a RAMDRIVE is faster than a SSD, or if it can make any sense using it ? :g:

[crashfly]

So why are there so many users asking, if a RAMDRIVE is faster than a SSD, or if it can make any sense using it ? :g:

I think the last part is what is being considered.  Many of us have a SSD.  Some of us have a large amount of RAM > 12GB.  We are just discussing the balance between how much of a RAM drive is necessary over traditional HDD/SSD space.  Note that balance is the key word here.