Jump to content
DirectorGunner

How to Guide for Extracting and Modding Star Citizen assets

Recommended Posts

Howdy Base forum,

This is from my thread on spectrum. I hope you find it thorough, and above all, helpful!

(P.S. if it helps, will add pics later)

 
How to Guide for Extracting and Modding Star Citizen assets
From the older days of 2014 to now
 
Welcome!
This is a detailed tutorial for extracting and to be able to start modifying/playing with existing assets in any 3D program.
Before this we had a very long legacy thread which was hard to sift through for directions on how to do anything.
With the new spectrum,
the following is a thorough How-To guide for new modders in the modding community.
Tips and updates are usually shared on Markemp's converter thread so I recommend bookmarking that main thread.
 
This tutorial still works using Markemp's new 3.x converter in Step 4.
 
Step 1:
-Copy or download all the Star Citizen build pak files you want to work with (see 2nd post for how to download any build) to a new folder on a drive where you have plenty of space and towards the root of the drive.
If you copy these into a deeply nested folder, you'll run into NTFS folder tree/filename length limits in windows IIRC.
 
ON OLD VERSION PRE 3.0 SC BUILDS
-unpack everything using 7zip via 7za.exe and the following bat file.
7za.exe needs to be copied to same directory you're doing this in
You can run the bat file via Windows Command Prompt.
If you're in windows explorer, hold down shift and right click on a nearby folder,
you'll be given the option to open that folder in windows command prompt which may save you some typing.
 
ON VERSION 3.0 AND AFTER SC BUILDS:
Use SCLeaksP4KBuster to unpack the massive single archive Data.p4k file
 

Step 2:

-Run Eppi's texture converter in your unpacked root folder
legacy forum thread is here:
And his github here
It completely automates the texture conversion process with many output options as well as getting many previously missed nuances like emission maps, and gloss maps separate from the normals. Keep in mind some 3D programs like C4D don't recognize the alpha channel in decal maps. You can use Photoshop to disable all channels but the alpha channel on the decal map, select all copy, enable RGB channels and disable alpha channel, select RGB channel, paste into place, delete previous layer or flatten, save as new tiff with suffix _ALPHA so you can find it easily. You should have a black and white image where the white is all the decals and black is transparency. If you're making your own build with his github code, be sure to use windows 10 SDK, thanks to Barnes102 for sharing that tip!
 

 

Step 3:
-Use Alluran's HoloXPLOR.DataForge tool on source material XMLs you want to convert to proper MTLs manually.
This part after unserializing can get a little complicated so I'll write about that in more detail in Step 5 below. You can drag and drop entire folders on the HoloXPLOR.DataForge 2.6.0-ptu build to convert all serialized files recursively. It does take a while on a full build.
 

 

Step 4:
-Run Markemp's converter to start getting file file types you want. I highly recommend choosing the DAE file option.
- - This is the line of code I use in command prompt for OBJs (not recommended) https://pastebin.com/kpMBxV8a
- - This is the powershell script I use... { WARNING: study windows powershell basics before trying this, using powershell can be dangerous to your windows installation if you type something you shouldn't } ...for DAE files which are more robust and include the helper nodes. Replace the directory information I used with what you have. I had my converter is a different folder location (one directory up) which is why I needed to run this powershell script as opposed to the one Markemp gave in his tutorial.
foreach ($file in (get-childitem -Recurse *.cga,*.cgf,*.skin,*.chr)) { a:\created\sc263\cgf-converter.exe $file -tif -objectdir "a:\created\sc263\data" }
 
Step 5:
SKIP TO BOTTOM OF THIS STEP, this old info just keep for record if you're working on PRE Alpha 3.0 builds
-Convert your unserialized XMLs material directions to standard Wavefront MTLs so your OBJs import with textures 70%+ correctly
This will save you a lot of time and even more time if automated via a script.
For some of your assets, you'll have a non-standard serialized XML which breaks down each material referenced in your new object files.
Un-serialize them with step 3 to begin this step.
The best way to explain this is to just show you.
We'll use the Aurora as an example, lines beginning with hashtags are comments.
 
You need to go from this
To this
Star Citizen - How to convert XML material blocks to MTL
Here's what to convert to in the MTL as you write it
  • Ka for indirect color values,
  • Kd for diffuse values, (the 0 to 1 decimal values can be times by 255 per value to get RGB standard colors if you want to adjust them easily)
  • Ks for specular values,
  • Ns for specular exponent value I believe,
  • Ke is for emission via mtl on lots of engines from doing some online research/reading.
  • d for transparency,
  • map_bump for ddna file,
  • map_Kd for diffuse map,
  • map_d for opacity texture map,
  • map_Ks for specular map,
  • map_Ns for specular map something.. usually is the exact same file as specular map with Markemp's cgf converter's outputs
  • disp for displacement map,
  • decal for decal map,
Currently, what's missing from this MTL conversion process is: blend map directions, gloss map directions, environmental map directions, and a few other details related to custom shaders
 
Here's an update for the new RGBA blend map for LayerBlend shader that CIG now used for assets like armor and helmets
See thread by alex on this
 
Step 6:
-Use whatever 3D program you're most comfortable with.Blender is imo the best free option. But C4D is also user friendly, and Max and Maya are more professional solutions and accessible with free access for students.
 
Step 7:

 

EXTRA NOTES:
I use the Holoview models CTM (use OpenCTM to convert them to OBJ) as references when rebuilding the ships.You can find the holoview CTM download links quickly by using a URL sniffer plugin for your browser and just observe when you load the holoviewer for the ship model you want.The lists on reddit are missing many of the ships and CIG may eventually change the holoview models to fix the ones that aren't correct. So imo it's best just to look and get them per ship you're working on at the time. The couple things missing from this tutorial is how to convert cryengine prefabs to rebuild large structures automatically via the prefab files.Like hangars and bases.Ship prefabs (such as weapon and thruster placement) might be done programmatically because I have not found prefab directions yet in the game files for that. I asked CIG in the Ask a Dev forum concerning this but not sure if I'll ever get an answer in this forum. https://robertsspaceindustries.com/spectrum/community/SC/forum/50259/thread/for-the-love-of-the-modding-community-teach-us-how

 

Dymek Made a new conversion kit to use Star Citizen 3.0 assets IN CRYENGINE, check out his thread
https://robertsspaceindustries.com/spectrum/community/SC/forum/50172/thread/star-citizen-in-cryengine-tools-converters-shaders Good luck, pop into the art thread linked at the top of this thread whenever you need help.
 

Some clarifications and additions to above.

How to download previous builds
As a backer (afaik) you can download any of the previous public builds down to version 1.1.5 v2 , provided CIG keeps them hosted and allows access to them.
AFAIK, as long as you're logged into your client, you should be able to download both the JSON and build files for the Public build you want to convert assets from.
The following are all public builds
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-1.3.0/273590.json   -this is SC 1.3
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-115/169905.json   -this is SC 1.1.5 Final
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-116/173766.json   -this is SC 1.1.6
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-116/186168.json   -this is SC 1.1.6a
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-120/248856j.json   -this is SC 1.2
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.0.0/305217.json   -this is SC 2.0
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.1.0/311865.json   -this is SC 2.1.0
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.1.0/313699.json   -this is SC 2.1.1
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.1.0/315233.json   -this is SC 2.1.2
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.2.0/327398.json   -this is SC 2.2.0
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.2.0/328656.json   -this is SC 2.2.1
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.2.0/330834.json   -this is SC 2.2.2
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.2.0/333246.json   -this is SC 2.2.3
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.3.0/336649.json   -this is SC 2.3.0
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.3.0/341778.json   -this is SC 2.3.1
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.4.0/363258-xml.json   -this is SC 2.4.0
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.4.0/368250-code.json   -this is SC 2.4.1
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.5.0/393743.json   -this is SC 2.5 final release
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.6.0/490789-c.json   -this is SC 2.6.0
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.6.0/506099.json   -this is SC 2.6.1
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.6.0/521494-c.json   -this is SC 2.6.2
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.6.0/523988-c.json   -this is SC 2.6.3
You can open any of these json files in Microsoft Wordpad (for readability) or note pad or a json viewer of your choice.
To assemble the links, you would use the web root URL + build prefix + the directory tree + the pak file name.
So for Build 2.6.3
#1 you take the web root URL
http://1.webseed.robertsspaceindustries.com/
#2 Then add the Build prefix after the web root URL, which is located toward the bottom of the json
eg
http://1.webseed.robertsspaceindustries.com/GameBuilds/sc-alpha-2.6.0/523988-c/StarCitizen
#3 Then add the directory tree and pak file contained in the filelist tree in the json
eg
http://1.webseed.robertsspaceindustries.com/GameBuilds/sc-alpha-2.6.0/523988-c/StarCitizen/Data/Objects-part0.pak

Provided CIG continues to allow backers to download previous builds for modding/fan art, the above directions should work until a newer system is implemented.
 *at one point I was downloading a bunch of old builds for testing conversions and I got blocked from downloading more pak files from that build, I logged into my launcher and then all of a sudden I was able to download the rest of the build pak files manually. So that lead me to believe that IP access for builds is authorized by the launcher, but it could be just a fluke. However this happened twice for me and each time, re-logging into the launcher worked in allowing me to continue to download the rest of the pak files from old builds. So.. seems this is the case but it may not be. 

Now for another tidbit of info.
Decal Textures and Alpha maps
So if you noticed, after converting textures with EPPI's tool.
Alpha channels are still in the diffusion and specular map files for many textures.
These are typically not use-able as is in most 3D programs.
You'll need to extract the alpha channel with Photoshop or another image editing software to be usable.

Here's what I do in Photoshop
Open diffusion or specular tif file
Check if alpha is all white or has any black
if white close file, everything okay check next file
if anything other than all white present proceed.
Select alpha channel and disable RGB channels on background (the texture)
Copy
Create new layer
Make sure new layer is selected in layer viewer
RGB channel for this new layer should be visible by default after creation, but confirm
select > Edit > Paste Special > Paste into place
You should end up with a black and white alpha map in the new layer
Save as a copy > Tif > no layers and no alpha and as a copy check marked > save tiff with _ALPHA at end of file name before .tif extension
Use this file in the opacity map in Substance painter or in any 3D program for the decal material.
Now in addition to alpha maps, sometimes it's not an alpha map but an emmisive map or some other kind of odd map (occasionally you'll get something that's not an emmisive or an alpha.. use your best judgement).
For the emmisive maps, you'll need to maintain your color from the RGB channels.
What I do is copy the alpha channel out as a layer mask to get the color in the emmisive (would be white areas) and the rest of the non-emmisive areas are black.

Decals:
Method on getting decals to display properly when they are exploded/lifted off the mesh on their own polys
https://robertsspaceindustries.com/spectrum/community/SC/forum/50172/thread/unstoppable-decal-issues-solved-addendum-to-asset-

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • By Weehamster
      LAST EDIT: 18/Aug/2018
      Hi everyone /
      I mainly have ships/vehicles that are either from lore, concept, development or being re-worked, and I constantly add more to the albums.
      NOTE: I've started to keep/restore the albums for ships that are now flyable as I've had a few requests for them. 
      Aopoa (Xi'An)
      Aegis
      Anvil
      Argo Astronautics
      BIRC (Banu)
      Consolidated Outland
      Crusader
      Drake
      Esperia
      Kruger
      MISC
      Origin
      RSI
      RSI/Aegis
      Tumbril
      Vanduul
      --------------------------------------------------------------------------------------------------
      Ship Components
      -------------------------------------------- BONUS --------------------------------------------
      Characters
      First Person Components
      Places 
      Places: Terra
      Places: Odin
      Places: Stanton
      Gameplay
      Other
       
    • By Gallitin
      The Star Citizen Base Marketplace is a separate subforum within the main Star Citizen Base forums. It is intended for people to use this subform to trade their Star Citizen items or other items.
      All Forum Rules apply to the Marketplace forums as well. In addition to the forum rules there is a separate set of rules specifically for the Marketplace. These are listed below.
       
      Marketplace Rules
      1: In the instance that we are contacted about any member here being a “bad trader” or a “troll,” we reserve the right to turn any and all IP and personal information we have on that member over to any law enforcement authorities and/or any other group or individuals we desire. If you do not agree with this rule, do not trade on our boards.
      2: Use the correct tag for your thread.
      WTS = Want to Sell, WTT = Want to Trade, WTB = Want to Buy 3: No misleading titles or item descriptions.
      4: No Price or “interest” checks
      5: You must post a price unless the item is purely For Trade. (trading for cash still means For Sale) Posts advertising external sites must include ships and prices.
      6: No pirated software sales or trades. No sale of NFR or Academic software. OEM software must be sold/traded in accordance with the terms of that software, usually with a piece of hardware.
      7: Selling or Trading of CD-KEY(s) without the original disks or digital software is strictly prohibited unless the software was originally sold by the publisher without physical media
      8: No auction style posts.
      9: No multi-level marketing schemes. (Pyramid Schemes)
      10: No sales of coupons or any other freely distributed items such as free beta keys, discount codes, or anything else obtained free from a public source. etc.
      11: No panhandling (Requests for money, paypal, or goods and services for free)
      12: No trading money for money. (Borrowing money with higher payback)
      13: No thread crapping or derailing.
      Any communications on the threads should be about expressing interest in either selling or purchasing ONLY. Posts debating the price, offering “options”, debating the value, asking about the origins, offering payment information, discussions of purchase, etc… are derailing of threads, will not be tolerated, and will be subject to aggressive moderation if necessary. Questions must be sent to sellers/buyers via PM. 14: No alcohol, tobacco, firearms, weapons of any kind or pornography (anything that can be classified “Adult Material”) are to be sold here….. nothing that is governed by the Federal Government as interstate trade. Likewise, no pharmaceuticals, legal or otherwise, can be sold here.
      15: The buying, selling, trading or solicitation of services is prohibited.
      16: Selling merchandise using Imperium or SCB's symbols or imagery without the express permission of the site owners who have copyright is prohibited.
      17: Requesting a paypal transaction to be sent using “to friends and family” (which gives the buyer no protection) is not allowed. It violates PayPal's TOS. (formerly called “gift” payment in paypal)
      18: No requests for personally identifying information that is unreasonable (i.e. requesting driver's license information, real address information, etc…)
      19: No attempts to sell referral code uses.
      20: Due to high incidences of fraud and abuse, no sales or attempts to swap virtual items from other games are allowed. (e.g. swaps or sales of in-game WoW gold, CS:GO weapons/skins, DOTA gear, etc…)
      21: Once a transaction is complete make sure you leave feedback for the user.
      22: Members who wish to post on our marketplace may be required to provide verification information to the site admins And compliance with these requests is mandatory.
      The above rules are made for the protection of all. Failure to comply with the rules will result in suspension of your privileges to post in this forum, or forfeiture of your forum membership according to the severity of the infraction. The moderators and administrators of this board reserve the right to modify, edit, move or delete any part or all of a post or thread for any reason stated above. Forum staff also reserves the right to ban any user, at any time, and for any reason.
      If you see something suspicious or have a concern, click the “Report” button on the post and explain why. Forum moderators will check it fairly quickly.
    • By Dillweed68
      I’m new to the group. 
       
      I like long walks at sunset, sand between my toes and giving away starships. 
       
      Yeah, that's right, I said it.  Wanna fight about it...LOL
       
      Hoping to meet many who want to interact and play the game.  

    • By Ostia
      Hello Once More @Imperium Members!
      This month we’ll be continuing the format from last time! In case you missed it, here is the explanation of the changes and some new information related to how Game Days will operate in the near future:
      We wanted to improve the experience of Game Day overall and felt that with Star Citizen in its current state just doesn’t warrant the amount of effort that we had once been able to put into it. IE. Custom Scenarios, Hype Trailers, Streaming, Competitions. Eventually that will change again and we can get back to the game that we are so anxiously awaiting. Until that time we would like to keep Game Day’s focus as a team building event that gives us a regular time each month that we can set aside to play games as an org. 
      You’ll notice at the top of this post there is a poll with several games as options. If you plan on taking part in this Game Day, please vote for your favorite choice from this list, and those that play in the chosen game will be added into a raffle! The poll will close on this next Friday, the 25th of May.
      You’ll still be free to join in with anyone else that would be playing Star Citizen, so don’t worry if you don’t have any of the games in the list.
      If you would like to suggest game to play together as an org on future Game Day’s please leave us suggestions in this thread and we will add them to the poll for the next event.
      We’ll also be shortening the duration of Game Day for this month. Instead of 1600 UTC through 0400 UTC, we’ll be starting at 1800 UTC and run to 0400 UTC.
      Joining Game Day
      For those of you that want to make a return visit, or show up for the first time, you only need to enter the Landing Channel on Teamspeak (ts.starcitizenbase.com), and one of our Hosts will direct you to the action!
      Raffles
      As for most other Game Day’s, we will be providing raffles for participants that play games in Star Citizen. However, this month we will be only conducting a participation raffle, so if you join any Star Citizen games created by other Imperium members, you’ll be entered into the raffle!
      As mentioned above, we will also be conducting a participation raffle for the winning game from the poll. Said game will have several TS channels created for players to meet up and play together, and doing so will enter you into this second raffle as well!
      Since we are hosting two games at once, be sure to check in with an officer or specialist that your name was added to the raffle of the game you played in! (As much as we may pretend otherwise, our Game Day hosts are still human, and you wouldn’t want to miss out on your chance to win free stuff)
      The prizes for both raffles will be $10 Star Citizen gift cards, and the winners of the raffles will be PM’d in the following week to confirm their prizes.
      New to Star Citizen?
      Read this post for information on the game and our Game Days:
    • By Narayan
      Greetings citizens!
      Recently I have already made a video about 600i, but the CIG launched a contest for the creation of the 600i commercial video, and I decided to making one another video.
      You can upvote my submission in the contest post here
      Enjoy Watching!
       
       
       
×