Jump to content
DirectorGunner

How to Guide for Extracting and Modding Star Citizen assets

Recommended Posts

Howdy Base forum,

This is from my thread on spectrum. I hope you find it thorough, and above all, helpful!

(P.S. if it helps, will add pics later)

 
How to Guide for Extracting and Modding Star Citizen assets
From the older days of 2014 to now
 
Welcome!
This is a detailed tutorial for extracting and to be able to start modifying/playing with existing assets in any 3D program.
Before this we had a very long legacy thread which was hard to sift through for directions on how to do anything.
With the new spectrum,
the following is a thorough How-To guide for new modders in the modding community.
Tips and updates are usually shared on Markemp's converter thread so I recommend bookmarking that main thread.
 
This tutorial still works using Markemp's new 3.x converter in Step 4.
 
Step 1:
-Copy or download all the Star Citizen build pak files you want to work with (see 2nd post for how to download any build) to a new folder on a drive where you have plenty of space and towards the root of the drive.
If you copy these into a deeply nested folder, you'll run into NTFS folder tree/filename length limits in windows IIRC.
 
ON OLD VERSION PRE 3.0 SC BUILDS
-unpack everything using 7zip via 7za.exe and the following bat file.
7za.exe needs to be copied to same directory you're doing this in
You can run the bat file via Windows Command Prompt.
If you're in windows explorer, hold down shift and right click on a nearby folder,
you'll be given the option to open that folder in windows command prompt which may save you some typing.
 
ON VERSION 3.0 AND AFTER SC BUILDS:
Use SCLeaksP4KBuster to unpack the massive single archive Data.p4k file
 

Step 2:

-Run Eppi's texture converter in your unpacked root folder
legacy forum thread is here:
And his github here
It completely automates the texture conversion process with many output options as well as getting many previously missed nuances like emission maps, and gloss maps separate from the normals. Keep in mind some 3D programs like C4D don't recognize the alpha channel in decal maps. You can use Photoshop to disable all channels but the alpha channel on the decal map, select all copy, enable RGB channels and disable alpha channel, select RGB channel, paste into place, delete previous layer or flatten, save as new tiff with suffix _ALPHA so you can find it easily. You should have a black and white image where the white is all the decals and black is transparency. If you're making your own build with his github code, be sure to use windows 10 SDK, thanks to Barnes102 for sharing that tip!
 

 

Step 3:
-Use Alluran's HoloXPLOR.DataForge tool on source material XMLs you want to convert to proper MTLs manually.
This part after unserializing can get a little complicated so I'll write about that in more detail in Step 5 below. You can drag and drop entire folders on the HoloXPLOR.DataForge 2.6.0-ptu build to convert all serialized files recursively. It does take a while on a full build.
 

 

Step 4:
-Run Markemp's converter to start getting file file types you want. I highly recommend choosing the DAE file option.
- - This is the line of code I use in command prompt for OBJs (not recommended) https://pastebin.com/kpMBxV8a
- - This is the powershell script I use... { WARNING: study windows powershell basics before trying this, using powershell can be dangerous to your windows installation if you type something you shouldn't } ...for DAE files which are more robust and include the helper nodes. Replace the directory information I used with what you have. I had my converter is a different folder location (one directory up) which is why I needed to run this powershell script as opposed to the one Markemp gave in his tutorial.
foreach ($file in (get-childitem -Recurse *.cga,*.cgf,*.skin,*.chr)) { a:\created\sc263\cgf-converter.exe $file -tif -objectdir "a:\created\sc263\data" }
 
Step 5:
SKIP TO BOTTOM OF THIS STEP, this old info just keep for record if you're working on PRE Alpha 3.0 builds
-Convert your unserialized XMLs material directions to standard Wavefront MTLs so your OBJs import with textures 70%+ correctly
This will save you a lot of time and even more time if automated via a script.
For some of your assets, you'll have a non-standard serialized XML which breaks down each material referenced in your new object files.
Un-serialize them with step 3 to begin this step.
The best way to explain this is to just show you.
We'll use the Aurora as an example, lines beginning with hashtags are comments.
 
You need to go from this
To this
Star Citizen - How to convert XML material blocks to MTL
Here's what to convert to in the MTL as you write it
  • Ka for indirect color values,
  • Kd for diffuse values, (the 0 to 1 decimal values can be times by 255 per value to get RGB standard colors if you want to adjust them easily)
  • Ks for specular values,
  • Ns for specular exponent value I believe,
  • Ke is for emission via mtl on lots of engines from doing some online research/reading.
  • d for transparency,
  • map_bump for ddna file,
  • map_Kd for diffuse map,
  • map_d for opacity texture map,
  • map_Ks for specular map,
  • map_Ns for specular map something.. usually is the exact same file as specular map with Markemp's cgf converter's outputs
  • disp for displacement map,
  • decal for decal map,
Currently, what's missing from this MTL conversion process is: blend map directions, gloss map directions, environmental map directions, and a few other details related to custom shaders
 
Here's an update for the new RGBA blend map for LayerBlend shader that CIG now used for assets like armor and helmets
See thread by alex on this
 
Step 6:
-Use whatever 3D program you're most comfortable with.Blender is imo the best free option. But C4D is also user friendly, and Max and Maya are more professional solutions and accessible with free access for students.
 
Step 7:

 

EXTRA NOTES:
I use the Holoview models CTM (use OpenCTM to convert them to OBJ) as references when rebuilding the ships.You can find the holoview CTM download links quickly by using a URL sniffer plugin for your browser and just observe when you load the holoviewer for the ship model you want.The lists on reddit are missing many of the ships and CIG may eventually change the holoview models to fix the ones that aren't correct. So imo it's best just to look and get them per ship you're working on at the time. The couple things missing from this tutorial is how to convert cryengine prefabs to rebuild large structures automatically via the prefab files.Like hangars and bases.Ship prefabs (such as weapon and thruster placement) might be done programmatically because I have not found prefab directions yet in the game files for that. I asked CIG in the Ask a Dev forum concerning this but not sure if I'll ever get an answer in this forum. https://robertsspaceindustries.com/spectrum/community/SC/forum/50259/thread/for-the-love-of-the-modding-community-teach-us-how

 

Dymek Made a new conversion kit to use Star Citizen 3.0 assets IN CRYENGINE, check out his thread
https://robertsspaceindustries.com/spectrum/community/SC/forum/50172/thread/star-citizen-in-cryengine-tools-converters-shaders Good luck, pop into the art thread linked at the top of this thread whenever you need help.
 

Some clarifications and additions to above.

How to download previous builds
As a backer (afaik) you can download any of the previous public builds down to version 1.1.5 v2 , provided CIG keeps them hosted and allows access to them.
AFAIK, as long as you're logged into your client, you should be able to download both the JSON and build files for the Public build you want to convert assets from.
The following are all public builds
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-1.3.0/273590.json   -this is SC 1.3
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-115/169905.json   -this is SC 1.1.5 Final
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-116/173766.json   -this is SC 1.1.6
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-116/186168.json   -this is SC 1.1.6a
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-120/248856j.json   -this is SC 1.2
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.0.0/305217.json   -this is SC 2.0
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.1.0/311865.json   -this is SC 2.1.0
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.1.0/313699.json   -this is SC 2.1.1
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.1.0/315233.json   -this is SC 2.1.2
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.2.0/327398.json   -this is SC 2.2.0
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.2.0/328656.json   -this is SC 2.2.1
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.2.0/330834.json   -this is SC 2.2.2
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.2.0/333246.json   -this is SC 2.2.3
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.3.0/336649.json   -this is SC 2.3.0
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.3.0/341778.json   -this is SC 2.3.1
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.4.0/363258-xml.json   -this is SC 2.4.0
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.4.0/368250-code.json   -this is SC 2.4.1
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.5.0/393743.json   -this is SC 2.5 final release
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.6.0/490789-c.json   -this is SC 2.6.0
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.6.0/506099.json   -this is SC 2.6.1
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.6.0/521494-c.json   -this is SC 2.6.2
http://1.webseed.robertsspaceindustries.com/FileIndex/sc-alpha-2.6.0/523988-c.json   -this is SC 2.6.3
You can open any of these json files in Microsoft Wordpad (for readability) or note pad or a json viewer of your choice.
To assemble the links, you would use the web root URL + build prefix + the directory tree + the pak file name.
So for Build 2.6.3
#1 you take the web root URL
http://1.webseed.robertsspaceindustries.com/
#2 Then add the Build prefix after the web root URL, which is located toward the bottom of the json
eg
http://1.webseed.robertsspaceindustries.com/GameBuilds/sc-alpha-2.6.0/523988-c/StarCitizen
#3 Then add the directory tree and pak file contained in the filelist tree in the json
eg
http://1.webseed.robertsspaceindustries.com/GameBuilds/sc-alpha-2.6.0/523988-c/StarCitizen/Data/Objects-part0.pak

Provided CIG continues to allow backers to download previous builds for modding/fan art, the above directions should work until a newer system is implemented.
 *at one point I was downloading a bunch of old builds for testing conversions and I got blocked from downloading more pak files from that build, I logged into my launcher and then all of a sudden I was able to download the rest of the build pak files manually. So that lead me to believe that IP access for builds is authorized by the launcher, but it could be just a fluke. However this happened twice for me and each time, re-logging into the launcher worked in allowing me to continue to download the rest of the pak files from old builds. So.. seems this is the case but it may not be. 

Now for another tidbit of info.
Decal Textures and Alpha maps
So if you noticed, after converting textures with EPPI's tool.
Alpha channels are still in the diffusion and specular map files for many textures.
These are typically not use-able as is in most 3D programs.
You'll need to extract the alpha channel with Photoshop or another image editing software to be usable.

Here's what I do in Photoshop
Open diffusion or specular tif file
Check if alpha is all white or has any black
if white close file, everything okay check next file
if anything other than all white present proceed.
Select alpha channel and disable RGB channels on background (the texture)
Copy
Create new layer
Make sure new layer is selected in layer viewer
RGB channel for this new layer should be visible by default after creation, but confirm
select > Edit > Paste Special > Paste into place
You should end up with a black and white alpha map in the new layer
Save as a copy > Tif > no layers and no alpha and as a copy check marked > save tiff with _ALPHA at end of file name before .tif extension
Use this file in the opacity map in Substance painter or in any 3D program for the decal material.
Now in addition to alpha maps, sometimes it's not an alpha map but an emmisive map or some other kind of odd map (occasionally you'll get something that's not an emmisive or an alpha.. use your best judgement).
For the emmisive maps, you'll need to maintain your color from the RGB channels.
What I do is copy the alpha channel out as a layer mask to get the color in the emmisive (would be white areas) and the rest of the non-emmisive areas are black.

Decals:
Method on getting decals to display properly when they are exploded/lifted off the mesh on their own polys
https://robertsspaceindustries.com/spectrum/community/SC/forum/50172/thread/unstoppable-decal-issues-solved-addendum-to-asset-

 

Share this post


Link to post
Share on other sites

Is there any way we can get an update to this kind of thing? Alot of the links seem to be broken or out of date. Been poking around in the files mainly to play around with 3d software and learn more about it while also being a SC fanboy. I was want my own 3d model of my carrack XD

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • By WarWulf
      Is immersion, and are haptics, still relevant?

      Sony has recently revealed their new controller... and it looks like they've replaced the usual rumble pad with actual haptics and force feedback. In their blog post, Sony detailed the "variety of powerful sensations" haptic feedback can bring to gameplay, like "the slow grittiness of driving a car through mud." Haptic feedback is the same technology behind the Nintendo Switch's HD rumble and we know the Xbox Series X controller will make use of it as well.
      Probably the most famous haptic controller in this community will be the Microsoft Sidewinder Force Feedback 2, which hasn't been available commercially since 2003. This controller is now much sought after by a relatively small group of people, and although haptic feedback is not widely supported, it is still incorporated into some of the more detailed flight sims even after 20 years.
      So what were some of the features that this provided? It's been a long time since I let mine go, but I do remember the demos which were part of the joysticks toolkit. This allowed you to feel simulated experiences such as a something similar to dragging a pencil over corrugated cardboard (intermittent resistance), feeling a difficult spot within the joysticks range of motion, resistance in one direction, etc. It might be nostalgia, but I definitely recall being impressed by these demonstrations. I was even more impressed when I fired it up in my first flight simulation haptic experience. I had never had the experience of fighting the winds before and it was obviously something that Microsoft had spend time refining.
      So why, if this controller was as legendary as its fans would tell you, did it die off? Patents. The company Haptic Technology had developed the technology and owned the patents to it, and Microsoft and Sony were not paying to license the patents. It would be easy to see this is a patent troll manoeuvre, however the truth is that without these licenses, innovations would probably be fewer and far between. There were a number of court cases, Microsoft settled early paying a dividend and purchasing a portion of the company. Sony were more resistant. The court cases went on for a number of years, during which they were appealing a court decision that would have ceased production of the PlayStation line until the patented technology was removed. Eventually it all closed down and while it never really made any big headlines, even those outlets following the situation lost interest. No console had haptic feedback, favouring the rumble technology instead, and immersion continued to produce technology aimed towards industrial production and military products.
      Now we're caught up in the timeline, what is the latest development? The patents for a number of applications for haptic technology, including those used in the Sidewinder FF, have finally expired after being extended by the holding company. Those die hard fans have been theorising about a resurgence of this niche technology. However there has largely been radio silence on the subject. Theory-crafting was simply that. Until the consoles announced that they were going to be providing their own haptics.
      This changes the lay of the land. As any good PC gamer knows (even if they may not like it) a large portion of the industry is motivated by console technologies above those on PC. There are always areas which are PC-centric, and the PC is always at the forefront of the technology curve, but frankly consoles and mobiles are where the big money is these days. And you can't argue with profits.
      So with this possibly becoming something that will be developed for and supported again, the big question is; Is it still relevant? And how would we want it to be implemented?
      This isn't an attempt at an opinion piece, although my opinion will bleed into it. I'm genuinely interested in whether or not this has piqued enough curiosity for people to vote with their wallets. How will people want it to apply to Star Citizen, if it is implemented? Feeling the buffeting of atmosphere and the kiss of the tarmac when flying and landing? Simulating faults in the system making controls less responsive? Or perhaps something more in-depth?
      Is this even relevant enough for people to notice?
      I look forward to your comments, if any.
       
    • By Weehamster
      LAST EDIT: 5/Apr/2020
      Hi everyone /
      I have images/gifs of ships/vehicles that are either flyable or in concept, development, re-worked or from Star Citizen lore.
      There are also many other categories, including anything from Star Systems to Characters to Ship/FPS Equipment.
      I constantly add more to the albums so don't be afraid to come back here anytime for the latest images.
      Please Enjoy.
      VEHICLES:
      SHIPS
      LAND VEHICLES
      ------------------------------------------------------------------------------------------------
      COMPONENTS:
      SHIP WEAPONS - GUNS
      SHIP WEAPONS - MISSILES / TORPEDOES / MINES
      OTHER SHIP COMPONENTS
      FIRST PERSON WEAPONS
      FIRST PERSON WEAPONS - ATTACHMENTS
      OTHER FIRST PERSON COMPONENTS
      -------------------------------------------- BONUS --------------------------------------------
      CHARACTERS:
      ------------------------------------------------------------------------------------------------
      PLACES: 
      ------------------------------------------------------------------------------------------------
      GAMEPLAY:
      ------------------------------------------------------------------------------------------------
      SQUADRON 42:
      ------------------------------------------------------------------------------------------------
      OTHER:
       
    • By Zelix-Xzavior
      (FFK)Fleet of The Faithful Knights
      The holy Knight has long since been forgotten for 1,500 years, For centuries the Holy Order of Knights had dedicated themselves to the safe passage of pilgrims to the Holy Land. They protected the travelers from thieves, robbers, and entire enemy armies. They were elite holy warriors. They dedicated their lives, fortunes, and sacred honor to this cause. Through force, they brought peace and security to the land.
      Now, in the year 2949, the Holy Order of Knights has once again risen to combat the ever rising threat of the Vanduul, and of Pirates, that plague this universe. The Fleet of the Faithful Knights is a beacon of light in a dark verse - and we are growing in numbers daily. Do you have the strength and bravery to be a Faithfull Knight? A Faithful Knight is truly a fearless warrior that will NEVER shy away from battle. If you are Honorable, you will welcome us. If you are also Brave, then you will wish to join us. And, if you are an enemy of the pure, the righteous, and the kind, then YOU WILL FEAR US.

      We are looking for:
      Fleet gunners, fighter squadron pilots, bombers, traders/merchants to help generate capital to operate our fleet, explorers, military scouts, industrial (miners, salvage, science, agriculture), and support roles to help with medical/triage, refueling, and repair.
      What we offer:
      Whatever your gameplay style is, we have it and need your help. You will have the protection of the Knights and will be a member of a truly powerful organization that has one mission: To be a beacon of light in a dark verse, and crush the darkness wherever it hides.

      For all new members. Please join us in our Discord and for some game play as we consider your application.
      Join our Discord here: https://discord.gg/GDpwYws
      Our website: Faithfulknights.com Rsi Org. page: https://robertsspaceindustries.com/orgs/FFK
       
       

       
       
       
×
×
  • Create New...