Jump to content
  • Announcements

    • AndalayBay

      Leaving IP.Board   11/13/2017

      See full announcement here.
    • AndalayBay

      New Theme Set Up   11/14/2017

      We have a new theme created by Vincent that's now available to everyone. It has been set as the default theme, so everyone should see it when you refresh the page. If you still aren't seeing it, you can select it from the Theme drop-down at the bottom of the page. It's called Assimilation. Thanks Vincent. Awesome looking theme!
alt3rn1ty

[Guide] Manual Cleaning Skyrim / Skyrim SE Master Files

Recommended Posts


Cleaning the Official Master ESMs


This guide assumes using TES5Edit v3.2 on Skyrim Nexus,
Or SSEEdit v3.2 on Skyrim SE Nexus

Due to this guide being dual purpose ( For Skyrim and Skyrim SE ) for the rest of this guide I will refer to both tools as xEdit.
Screenshots of tools used may be one or the other, or older versions, which does not matter, the images are only to illustrate the method / options used.

 

Why Clean the Master Files ?

Firstly because the masters have entries that are identical to the same records in Skyrim.esm or other DLC esms'. They exist because Bethesda may have looked at something in the CK and an unneeded entry was auto included in the plugin even though the item was not altered in any way. The Official Creation Kits are notoriously buggy and randomly create dirty / wild edits, often when the author of the plugin is completely unaware. Wherever that plugin is placed in your load order its records overwrite all the conflicting records from plugins loaded before it ( the rule of one ) resetting the settings back to the values contained in the Official Bethesda DLC. It won't cause crashes, it just changes the values of plugins loaded before it. Which can alter mods that you have for Weapon Damage, Armor, Lighting, Food Effects and so on. The masters are very early in your load order but there is potential for a mod to be made as a fake.esm, and placed among them, and so ITMs in a later loading master may cause problems for that mods esm. Chance is remote that a master will affect another master, and this procedure is best used on all of your mods plugins, but cleaning everything of ITMs ( Identical to Master records ) causes no harm, is more optimal giving the game less to process in your load order, and so it is best to get rid of these completely unnecessary dirty edits.

 

The Second reason is that Bethesda chose to delete some things that are in the Official DLC. Any mods loaded with references to deleted records from the Official Bethesda DLC will cause your game to crash. This problem particularly affects older mods ( especially mods that were made before newer official patches were released, with more deleted references the old mod did not anticipate - It will also become problematic for the Skyrim Special Edition community where old Original Skyrim mods are being converted to SSE, and Bethesda have deleted even more records from the plugins before they released the newer plugins for that version of the game ). xEdit can restore and properly assign values to these records that will disable them and still allow mods to access them. This is done using the "Undelete and Disable References" option.

 

For further explanations of why it is still recommended to clean the games masters .. Read on from this post,

Zilav and Arthmoor, most valued technical and vastly experienced modding authors, weigh in on the subject.

 

 

The following mostly apply to mod authors, but worth knowing about for mod users too :

 

xEdit will also report when a mod has Deleted NavMeshes as part of the report from Automatic cleaning. Like deleted references, any mod that references a deleted NavMesh will cause Skyrim to Crash. Properly optimizing your mods NavMeshes and checking your mod for Deleted Vanilla NavMeshes ( which can also be caused by a CK wild edit even if you did not do it yourself ) is important. Mods altering the same cell and the same NavMeshes when your mod is not optimized will cause Skyrim to Crash. Poorly optimized NavMeshes with errors reported by the CK will make Skyrim unstable. Instabilities like fast travelling to a location and Skyrim crashes. Note the ones found to be deleted in the games masters, cannot be undeleted. To fix deleted Navmeshes in your mods, Arthmoor has provided a walkthrough in Skyrim - Fixing Navmesh Deletion in TES5Edit

 

Manually cleaning your mods is also important to remove wild edits. This is mostly down to the experience of Mod Authors to solve such problems, but there are a few noted later in this guide which are in the DLCs which everyone can easily Manually clean. Some mods can have accidental Wild Edits in them caused by the author looking at how Bethesda did something they wish their mod to do as well. These Wild Edits often prevent Skyrim from doing things like advancing quests, spawning NPCs, assigning dialogue to NPCs, preventing NPCs from patrol areas they are assigned to. They can also alter Vanilla Lighting and Triggers that the author wished to use. All of these things affect any plugin with conflicting records loaded before a mod with Wild Edits.

 

Mod authors - Learn to use xEdit, and ensure the only records in your mod plugins are what you would expect to be in there, its the most important tool the community can make use of when used properly.

 

Mod Users - Follow this guide...

 

 

Before moving on to the Manual cleaning, something everyone should do prior to Manual Cleaning :


Automatic cleaning of Bethesda's ESMs with xEdit

With the games Original esm's installed ( You can use Steam to Verify Integrity of Game Cache of Skyrims files to ensure you have good error free copies of the original master files ), and in accordance with the following wiki article http://www.creationkit.com/index.php?title=Skyrim_Dirty_Plugins_List :


Load up xEdit.

1. Right click the plugin selection screen and select "none"


large.xEdit1.png.4e327d2849d996c61b870d6


2. Tick the relevant esm to edit, and click okay ( If you have not cleaned any of your Master files yet, the first one to tick will be Skyrims "Update.esm" ), then click Okay


large.xEdit2.png.a538b216ea95a79a5e15ef5


After each of the following actions, wait for a message in the message window that the previous operation has finished / Done :

3. Right click the plugin after you get the "Background Loader : Finished" message,and choose "Apply Filter for Cleaning"



large.xEdit3.png.8392a40f0139e2199b57b4e


Wait until Filtering is finished then ..

4. Right click the plugin and choose Remove Identical to Master Records



large.xEdit4.png.43f1c1457ae7c6d949de243


Wait until it finishes then ..

5. Right click the plugin and choose Undelete and Disable references



large.xEdit5.png.8c600acaeb8e116f182f03b


Wait until it finishes then ..

6. Close xEdit, and it should check with you that you wish to save the plugin ( this only happens if you have made any changes to the plugin to save, if it just closes .. Then you have not cleaned anything )


Rinse and repeat the Automatic cleaning ( steps 1 - 6 above ) for each of the master files.

Working from first to load, to last, not including Skyrim.esm or any unofficial patches
( No point doing Skyrim.esm, and the unofficial patches are already done and should not be cleaned )

So clean in this order

Update.esm

Dawnguard.esm

Dawnguard.esm ( Yes it needs to be done twice )

Hearthfire.esm

Dragonborn.esm


Dawnguard.esm needs to be cleaned twice ( as of xEdit 3.1 onwards - After doing the automatic cleaning routine once on Dawnguard.esm, and saving it, load it up again in xEdit and you will be able to clean a further 6 ITMs ) :




Skyrim Original

Update.esm
Apply filter for cleaning
[Removing "Identical to Master" records done]  Processed Records: 1290, Removed Records: 96
[undeleting and Disabling References done]  Processed Records: 1194, Undeleted Records: 3

Dawnguard.esm ( First time around )
Apply filter for cleaning
[Removing "Identical to Master" records done]  Processed Records: 98027, Removed Records: 630
[undeleting and Disabling References done]  Processed Records: 97397, Undeleted Records: 82
<Warning: Plugin contains 57 deleted NavMeshes which can not be undeleted>
Dawnguard.esm ( Second time around )
Apply filter for cleaning
[Removing "Identical to Master" records done]  Processed Records: 97396, Removed Records: 6
[undeleting and Disabling References done]  Processed Records: 97390, Undeleted Records: 0
<Warning: Plugin contains 57 deleted NavMeshes which can not be undeleted>

Hearthfires.esm
Apply filter for cleaning
[Removing "Identical to Master" records done]  Processed Records: 17797, Removed Records: 184
[undeleting and Disabling References done]  Processed Records: 17613, Undeleted Records: 11
<Warning: Plugin contains 5 deleted NavMeshes which can not be undeleted>

Dragonborn.esm
Apply filter for cleaning
[Removing "Identical to Master" records done]  Processed Records: 214386, Removed Records: 68
[undeleting and Disabling References done]  Processed Records: 214318, Undeleted Records: 8
<Warning: Plugin contains 1 deleted NavMeshes which can not be undeleted>




Skyrim Special Edition

Update.esm
Apply filter for cleaning
[Removing "Identical to Master" records done]  Processed Records: 17216, Removed Records: 233
[undeleting and Disabling References done]  Processed Records: 16983, Undeleted Records: 91

Dawnguard.esm ( First time around )
Apply filter for cleaning
[Removing "Identical to Master" records done]  Processed Records: 98676, Removed Records: 628
[undeleting and Disabling References done]  Processed Records: 98048, Undeleted Records: 82
<Warning: Plugin contains 57 deleted NavMeshes which can not be undeleted>
Dawnguard.esm ( Second time around )
Apply filter for cleaning
[Removing "Identical to Master" records done]  Processed Records: 98047, Removed Records: 6
[undeleting and Disabling References done]  Processed Records: 98041, Undeleted Records: 0
<Warning: Plugin contains 57 deleted NavMeshes which can not be undeleted>

Hearthfires.esm
Apply filter for cleaning
[Removing "Identical to Master" records done]  Processed Records: 17830, Removed Records: 178
[undeleting and Disabling References done]  Processed Records: 17652, Undeleted Records: 11
<Warning: Plugin contains 5 deleted NavMeshes which can not be undeleted>

Dragonborn.esm
Apply filter for cleaning
[Removing "Identical to Master" records done]  Processed Records: 214498, Removed Records: 69
[undeleting and Disabling References done]  Processed Records: 214429, Undeleted Records: 8
<Warning: Plugin contains 1 deleted NavMeshes which can not be undeleted>


------------------



Dawnguard.esm needs manual cleaning aswell as automatic cleaning


After the automated cleaning is done, you can also now manually clean a few more Wild edits xEdit will not have touched ...

Recently Arthmoor has brought to the attention of the community additional information regarding manual cleaning of Dawnguard.esm, which everyone needs to do for their own setup same as automatic cleaning ( because nobody can legally upload official master files anywhere, everyone needs to do their own )



First load up xEdit

When the plugin selection comes up, right click and select None
Then put a tick in the box just for Dawnguard.esm, click Okay
After its finished loading, right click Dawnguard.esm and choose "Filter for Cleaning"


1. For "CELL 00016BCF: Remove XEZN subrecord referring to RiftenRatwayZone [ECZN:0009FBB9]. Otherwise it blocks the official fix in Update.esm." .... Expand the records as in the following screenshot, and right click the indicated sub-record, and choose Remove



large.58af75b0a7fef_Cleaning1.png.4d9fd9


2. For "CELL 0001FA4C: Wild edit. Remove this record. It's a testing cell." .... Expand the records as in the following screenshot, and right click the indicated record, and choose Remove


large.58af75e78d857_Cleaning2.png.cf5c54


3. For "CELL 0006C3B6: Wild edit. Remove this record. It's a testing cell." .... Expand the records as in the following screenshot, and right click the indicated record, and choose Remove


large.58af761722959_Cleaning3.png.e4213b


NOTE : This guide used to include cleaning instructions for "CELL 00039F67: Wild edit. Remove this record. It's a testing cell" ( The WICourier edit ) - But since the new version of TES5Edit 3.1+ now cleans that as part of the automated cleaning ( which you should have done prior to manual cleaning ), you no longer need to clean it manually afterwards.


-----------------------------------------



CRCs of the various stages for comparison
( The following cleaned with xEdit 3.2 )



Skyrim Original


large.58af240ec3efa_TES5EditMasterCRCs.p


Update.esm Original - CRC = E5B67BDA MD5 = 2476E42699D4D9236DDD2EA8B31F612D
Update.esm Auto Clean - CRC = 18401373 MD5 = 27CF2C3E379BC853F4AC574BCEC7A0B9
Dawnguard.esm Original - CRC = BD72CCF3 MD5 = 5BD2F339925A5FB96F9F0DA7F96FD8CE
Dawnguard.esm Auto Clean First time - CRC = 6391C6DC MD5 = 0602D8450CC51F8C9EF8AB531B6EDAEB
Dawnguard.esm Auto Clean Second time - CRC = 94A47F31 MD5 = D3D38017BCDA07C1ABB1C343A9266634
Dawnguard.esm Auto plus Manual Clean - CRC = 50FB6A39 MD5 = 74561A2CF8644ACC8BE857E0705996D5
Hearthfire.esm Original - CRC = AF82CE6A MD5 = 71C3D20EA9F8510E29C36F2A426247C2
Hearthfire.esm Auto Clean - CRC = FBB4B8FA MD5 = E91E103C7F541C34F7CDD53AD68EDA3A
Dragonborn.esm Original - CRC = A9F83BFF MD5 = 37D8CBFFCB460011CE2BF3AABD007A2D
Dragonborn.esm Auto Clean - CRC = E0715431 MD5 = 7D78A3CA1E4D3700DB5D7F5DDAF5F7D2



Skyrim Special Edition


large.58af2428e577f_SSEEditMasterCRCs.pn


Update.esm Original - CRC = 8a234131 MD5 = A6E4D668C8E297960D3CBB78104CA8E3
Update.esm Auto Clean - CRC = fde85196 MD5 = E1D557E74F585FDA253DA613DD53FD08
Dawnguard.esm Original - CRC = 051fcdc8 MD5 = E124FE26F161A722D52CCD2BC4CD1FA5
Dawnguard.esm Auto Clean First time - CRC = 09ff5624 MD5 = 6119377530847F5FBBC758FEDDA8512A
Dawnguard.esm Auto Clean Second time - CRC = debdefee MD5 = 887FABC6396E285694A83F1DB407C209
Dawnguard.esm Auto plus Manual Clean - CRC = fa97d737 MD5 = C522A66C97596D3170EDB231A9B732E8
Hearthfire.esm Original - CRC = fbd3fb4c MD5 = C8D10DC5D674D63D4C8FAD04ACF3AFDC
Hearthfire.esm Auto Clean - CRC = 6b0a76a3 MD5 = EB8E4BD7EAE01A3A87E2343B95BD2782
Dragonborn.esm Original - CRC = c7e30215 MD5 = AFCEC0A936DBF4A3C0CDA2ACBD4813ED
Dragonborn.esm Auto Clean - CRC = ffe01cc5 MD5 = 5FB86435A02093A279D34DC305882519




( Note : If there is only one file inside a folder within a zip, 7zip shows that files CRC alongside the folder - I checked each plugins within the folders individually to be sure the Screenshot CRCs above are all correct, and of course they also match the online CRC checks )

 

Now that the Master files are cleaned, you can go through the rest of your Load Order using Automatic cleaning of ITMs and UDRs on all your mods plugins. The sequence of cleaning mods plugins should be after you have your Load Order correct, clean them with the last to load being the last to clean. Mod authors should have done them already, so most will probably not need cleaning. Also look out for any mod specific cleaning instructions in the mods description. For example the Unofficial Patches will not need any cleaning, they are already done, and the remaining ITMs in those plugins should be left alone because they do have a purpose ..
( its a very rare occasion when this is true ).



To check CRCs Drag and Drop resulting files here https://emn178.github.io/online-tools/crc32_checksum.html

To Check MD5 checksums Drag and Drop resulting files here http://onlinemd5.com/

The xEdit Work In Progress Development topic is at the following link 
http://afkmods.iguanadons.net/index.php?/topic/3750-wipz-tes5edit/page-20#entry152079

Edited by alt3rn1ty

Share this post


Link to post
Share on other sites

Sticky ?

Edit : Damn, can this be moved to the Skyrim Guides section of the forum ( noticed it after posting ) - EDIT : TY .. O_o That was quick

Edited by alt3rn1ty

Share this post


Link to post
Share on other sites

Thanks but while you were swiftly doing that .. I was creating another one in this - the correct part of the forum - So the other duplicate ( now edited to read REMOVE ME ) needs deleted -->> http://www.theassimilationlab.com/forums/topic/14508-guide-manual-cleaning-skyrim-master-files/

 

:blush: ? Is "Juggle the forum" an allowed game ?

Edited by alt3rn1ty

Share this post


Link to post
Share on other sites

;) NP. Will update it when TES5Edit 3.0.33 goes live one day .. No.3 on the manual cleaning will no longer need to be in the guide because 3.0.33 cleans that with automatic cleaning  :beerchug:

 

Edit : Bah!, I was enjoying being a young Amoeba, this post makes me whale fodder

 

Anyway back on topic ... Cleaning stuff - Doooo eeeeet ! :)

Edited by alt3rn1ty

Share this post


Link to post
Share on other sites

Now that TES5Edit 3.1.0 is released - Removed the WICourier edit ( which is cleaned by normal auto cleaning using the newer version of TES5Edit - so not necessary in this manual cleaning guide anymore ) - Previous screenshot 4 and instructions have become number 3 in the OP in place of the removed instructions.

Edited by alt3rn1ty

Share this post


Link to post
Share on other sites

Slight update to the description wording, and redid the screenshots hopefully for more clarity and prevent any misunderstanding of exactly what should be edited.

Share this post


Link to post
Share on other sites

Updated the OP - To reflect Release of TES5Edit 3.1.2 on Nexus

 

Also there was a discrepancy of how many records were processed after the last edit noting 68 ITMs instead of 64 on Dragonborn.esm, the processed records of UDRs now reads 214318 processed records instead of 214322 ( minus the 4 additional records removed prior to doing UDRs )

Share this post


Link to post
Share on other sites

I miss the sounds from Morrowind more than anything, though Dragonborn brought a few back which was very nice.

 

I'm well into Fallout 4 right now, but just updated the OP to include MD5 hash for each file cleaning stage in case anyone else needs them.

Share this post


Link to post
Share on other sites

All ITM and UDR results are from xEdit 3.1.2

 

( more accurately - I was using the WIPz xEdit before 3.1.2 was released, which became 3.1.2 - But have checked since 3.1.2 was released and the results have been corroborated by other folk on AFK Mods topic )

I have not used any of the more recent WIPz which were purely for starting Fallout 4 support ( in fact the latest WIPz is now only available from the FO4Edit page on Nexus as noted in the OP of the WIP topic, which you would rename to TES5Edit.exe ), and do not bring any refinements for Skyrim

Share this post


Link to post
Share on other sites

To anyone who still needs to follow this guide :


 


I am currently updating it ( some old direct links to wiki paragraphs have been lost due to Bethesda re-organising their wiki's )


 


So to become less dependant upon other links and information, I am fleshing it out a bit with more images


 


It also needs an update to file CRCs ( at the end of the first post ) due to Zilavs recent changes to xEdit


 


So you now need to be a bit more up to date and use the Work In Progress version of xEdit available from Fallout 4 Nexus ( link for this in the first post has also been updated )


Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×