power crystals' XML patcher and data exporter (now under old management)

User avatar
The W:AB Noob
Lieutenant General
Posts: 4575
Joined: Fri 12 Jul 2013 22:29
Location: United States, Central Time Zone
Contact:

Re: power crystals' XML patcher and data exporter (now under old management)

Postby The W:AB Noob » Mon 12 Sep 2016 06:05

power crystals wrote:
The W:AB Noob wrote:Powercrystals, just a small request: can you upload the source code for the Data Exporter? Thanks.


I have no problem with this (though keep in mind this source isn't meant to be read by others so it's a mess of confusing lamdas and crap). By this you mean the table exporter thing, right?

Sorry, yes, the table exporter.

Btw, the index command still listed outdated files, but don't stress too much. It looks like it was harder than I expected to implement it without hardcoding. It'll just be something I do manually. Thanks for working so hard on it though.

power crystals
Corporal
Posts: 44
Joined: Tue 8 Apr 2014 00:18
Contact:

Re: power crystals' XML patcher and data exporter (now under old management)

Postby power crystals » Wed 14 Sep 2016 04:52

Sorry for the delays. Been busier IRL than average lately.

peer wrote:Oh. Are you just scraping the enohkas tools or something?

Yes (see below). I didn't see a need to reinvent the binary parser when I just wanted to automate stuff.

The W:AB Noob wrote:Sorry, yes, the table exporter.

Behold, the table exporter in all of its messy glory, spoiler tags because it's large relative to most posts here:
Spoiler : C# :
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Xml.Linq;

using moddingSuite.BL;
using moddingSuite.BL.Ndf;
using moddingSuite.BL.TGV;
using moddingSuite.Model.Edata;
using moddingSuite.Model.Ndfbin;
using moddingSuite.Model.Ndfbin.Types.AllTypes;
using moddingSuite.Model.Textures;
using moddingSuite.Model.Trad;

namespace WGTableExporter
{
public class Program
{
private static string _singleFile;
private static string _singleTable;

public static void Main(string[] args)
{
if(args.Length == 0)
{
// * *
Console.WriteLine("Usage: WGTableExporter.exe <file.dat> [<file.ndfbin|file.dic>] [<tablename>]");
Console.WriteLine("For data (.ndfbin) entries:");
Console.WriteLine(" An xml file will be created with the same name as the table");
Console.WriteLine(" If no third argument is specified, all tables will be exported");
Console.WriteLine("For localization (.dic) entries:");
Console.WriteLine(" A csv file will be created with the same name as the table");
Console.WriteLine(" The third argument has no meaning (the entire entry is always exported)");
Console.WriteLine("For texture (.tgv) entries:");
Console.WriteLine(" A dds file will be created with the same name as the texture");
Console.WriteLine("All files are created in a folder named after the .dat file they came from");
Console.WriteLine(" and will maintain the same folder structure as they originally had in the .dat");
Console.WriteLine("If the second argument is not provided, the entire .dat file will be exported");
Console.WriteLine("ZZ_Win.dat appears to have a .shc file that confuses the moddingSuite library,");
Console.WriteLine(" so any errors about that file may be safely ignored.");
Console.WriteLine();
Console.WriteLine("Note that names are case-sensitive - TAmmunition is not tammunition,");
Console.WriteLine(" and that for .ndfbin/.dic/etc files you need the entire path to the file");
Console.WriteLine(" e.g. pc\\ndf\\patchable\\gfx\\everything.ndfbin");
}
else if(!File.Exists(args[0]))
{
Console.WriteLine("The specified file {0} does not exist", args[0]);
}
else
{
if(args.Length > 1)
{
_singleFile = args[1];
}
if(args.Length > 2)
{
_singleTable = args[2];
}

EdataManager em = new EdataManager(args[0]);
em.ParseEdataFile();

ExportDataFile(em, Path.GetFileNameWithoutExtension(args[0]));
}
}

private static void ExportDataFile(EdataManager file, string rootDir)
{
foreach(EdataContentFile ef in file.Files)
{
if(_singleFile != null && _singleFile != ef.Path)
{
continue;
}

string exportDir = Path.Combine(rootDir, Path.GetDirectoryName(ef.Path) ?? "");

if(!Directory.Exists(exportDir))
{
Directory.CreateDirectory(exportDir);
}

try
{
if(ef.FileType == EdataFileType.Ndfbin)
{
NdfBinary b = (new NdfbinReader()).Read(file.GetRawData(ef));

ExportNdf(b, Path.Combine(exportDir, Path.GetFileNameWithoutExtension(ef.Name)));
}
else if(ef.FileType == EdataFileType.Dictionary)
{
TradManager tm = new TradManager(file.GetRawData(ef));

ExportLocalizationTable(tm, Path.Combine(exportDir, Path.GetFileNameWithoutExtension(ef.Name) + ".csv"));
}
else if(ef.FileType == EdataFileType.Image)
{
TgvFile tgv = new TgvReader().Read(file.GetRawData(ef));
byte[] dds = (new TgvDDSWriter()).CreateDDSFile(tgv);
using(FileStream fs = new FileStream(Path.Combine(exportDir, Path.GetFileNameWithoutExtension(ef.Path) + ".dds"), FileMode.OpenOrCreate))
{
fs.Write(dds, 0, dds.Length);
fs.Flush();
}
}
}
catch(InvalidDataException)
{
Console.WriteLine("An internal error was encountered while trying to read {0}!", ef.Path);
}
}
}

private static void ExportNdf(NdfBinary entry, string outputDir)
{
foreach(NdfClass table in entry.Classes)
{
if(_singleTable != null && _singleTable != table.Name)
{
continue;
}

if(!Directory.Exists(outputDir))
{
Directory.CreateDirectory(outputDir);
}

ExportNdfTable(table, Path.Combine(outputDir, table.Name + ".xml"));
}
}

private static void ExportNdfTable(NdfClass table, string outputFilename)
{
XDocument d = new XDocument();
XElement root = new XElement(table.Name + "s");
d.Add(root);

foreach(NdfObject obj in table.Instances)
{
XElement oe = new XElement(table.Name);
oe.Add(new XAttribute("id", obj.Id));
foreach(NdfPropertyValue pv in obj.PropertyValues)
{
if(pv.Value is NdfCollection)
{
NdfCollection c = (NdfCollection)(pv.Value);
XElement ce = new XElement(pv.Property.Name);
foreach(var m in c)
{
ce.Add(new XElement("collectionelement", m.Value));
}
oe.Add(ce);
}
else
{
oe.Add(new XElement(pv.Property.Name, pv.Value));
}
}
root.Add(oe);
}

d.Save(outputFilename);
}

private static void ExportLocalizationTable(TradManager table, string outputFilename)
{
StreamWriter sw = new StreamWriter(outputFilename);
foreach(TradEntry te in table.Entries)
{
sw.WriteLine("{0}, {1}", te.HashView, te.Content);
}
sw.Flush();
sw.Dispose();
}
}
}
Requires a reference to the moddingSuite exe but otherwise you should be able to just drop that thing into VS (or monodevelop or what have you) and run.

The W:AB Noob wrote:Btw, the index command still listed outdated files, but don't stress too much. It looks like it was harder than I expected to implement it without hardcoding. It'll just be something I do manually. Thanks for working so hard on it though.

Really? I'm curious as to what I'm missing because I'm up to like 3 different instances of "oh, this time I get it". Can you post:
  • Your file layout that's causing the issue
  • The csv it produces
  • A csv modified to what you'd expect to see
For #2 and #3 feel free to reduce it to just a few rows on both so long as the difference that's causing the problem is present.

peer
Private First-Class
Posts: 6
Joined: Wed 7 Sep 2016 13:52
Contact:

Re: power crystals' XML patcher and data exporter (now under old management)

Postby peer » Wed 14 Sep 2016 04:57

power crystals wrote:Sorry for the delays. Been busier IRL than average lately.

peer wrote:Oh. Are you just scraping the enohkas tools or something?

Yes (see below). I didn't see a need to reinvent the binary parser when I just wanted to automate stuff.


Ah. I'm rewriting it in java, getting a mono env is kind of a pain. I'm just about to finish a file exporter similar to this. Thanks for posting!

power crystals
Corporal
Posts: 44
Joined: Tue 8 Apr 2014 00:18
Contact:

Re: power crystals' XML patcher and data exporter (now under old management)

Postby power crystals » Wed 14 Sep 2016 23:40

peer wrote:Ah. I'm rewriting it in java, getting a mono env is kind of a pain. I'm just about to finish a file exporter similar to this. Thanks for posting!

There's things about it I've thought about forking it to change (like every value being of type Object or the overuse of lamdas making retrieving instances absurdly expensive, but I can't bring myself to care enough to do it. Good luck! If my years doing Minecraft mods taught me anything it's that I hate java, but if you hate mono just as much as I hate java that seems reasonable.

Honest question, how are you going to do this with java's lack of unsigned types? Pretty sure there's uints all over that thing.

peer
Private First-Class
Posts: 6
Joined: Wed 7 Sep 2016 13:52
Contact:

Re: power crystals' XML patcher and data exporter (now under old management)

Postby peer » Thu 15 Sep 2016 00:20

power crystals wrote:
peer wrote:Honest question, how are you going to do this with java's lack of unsigned types? Pretty sure there's uints all over that thing.


I've only just gotten to the point where I've exported the file structure/data to disk, I havent actually started parsing the files individually. Are they long u_int? Because i could just store them unsigned in a long (lossily).

power crystals
Corporal
Posts: 44
Joined: Tue 8 Apr 2014 00:18
Contact:

Re: power crystals' XML patcher and data exporter (now under old management)

Postby power crystals » Thu 15 Sep 2016 02:33

Mostly uint32, but not always. From NdfBinReader.cs:

Code: Select all

            ms.Read(buffer, 0, buffer.Length);
            header.FooterOffset = BitConverter.ToUInt64(buffer, 0);

            ms.Read(buffer, 0, buffer.Length);
            header.HeaderSize = BitConverter.ToUInt64(buffer, 0);

            ms.Read(buffer, 0, buffer.Length);
            header.FullFileSizeUncomp = BitConverter.ToUInt64(buffer, 0);


e: not saying that's the only place, but it's the first I spotted
e2: to be fair, you'd probably be fine operating them as a signed int64 because the odds of that 64th bit mattering are pretty low. I just like to hate on java ;)

peer
Private First-Class
Posts: 6
Joined: Wed 7 Sep 2016 13:52
Contact:

Re: power crystals' XML patcher and data exporter (now under old management)

Postby peer » Thu 15 Sep 2016 04:07

power crystals wrote:Mostly uint32, but not always. From NdfBinReader.cs:

Code: Select all

            ms.Read(buffer, 0, buffer.Length);
            header.FooterOffset = BitConverter.ToUInt64(buffer, 0);

            ms.Read(buffer, 0, buffer.Length);
            header.HeaderSize = BitConverter.ToUInt64(buffer, 0);

            ms.Read(buffer, 0, buffer.Length);
            header.FullFileSizeUncomp = BitConverter.ToUInt64(buffer, 0);


e: not saying that's the only place, but it's the first I spotted
e2: to be fair, you'd probably be fine operating them as a signed int64 because the odds of that 64th bit mattering are pretty low. I just like to hate on java ;)


Worst case scenario I can use Big Numbers.

AJE
Sergeant Major of the Army
Posts: 384
Joined: Sat 7 Sep 2013 07:22
Contact:

Re: power crystals' XML patcher and data exporter (now under old management)

Postby AJE » Sat 24 Sep 2016 11:14

Do I need to know how to code well in order to use the patcher?

I'm finally beginning to plan a mod that's too big for me to be able to redo it every time Eugen releases an update, but I have no experience coding. While I can mostly understand the sample patches provided, there are still some inconsistencies that I can't understand, and if that's because this tool requires coding experience to understand, I'll have to find another way of making a large mod.

User avatar
The W:AB Noob
Lieutenant General
Posts: 4575
Joined: Fri 12 Jul 2013 22:29
Location: United States, Central Time Zone
Contact:

Re: power crystals' XML patcher and data exporter (now under old management)

Postby The W:AB Noob » Sat 24 Sep 2016 16:21

AJE wrote:Do I need to know how to code well in order to use the patcher?

I'm finally beginning to plan a mod that's too big for me to be able to redo it every time Eugen releases an update, but I have no experience coding. While I can mostly understand the sample patches provided, there are still some inconsistencies that I can't understand, and if that's because this tool requires coding experience to understand, I'll have to find another way of making a large mod.

Friend me on Steam @ The W:RD Noob if you haven't already and I can help you use and understand the patcher if you want.

power crystals
Corporal
Posts: 44
Joined: Tue 8 Apr 2014 00:18
Contact:

Re: power crystals' XML patcher and data exporter (now under old management)

Postby power crystals » Sat 24 Sep 2016 18:21

AJE wrote:Do I need to know how to code well in order to use the patcher?

I'm finally beginning to plan a mod that's too big for me to be able to redo it every time Eugen releases an update, but I have no experience coding. While I can mostly understand the sample patches provided, there are still some inconsistencies that I can't understand, and if that's because this tool requires coding experience to understand, I'll have to find another way of making a large mod.


You shouldn't have to, but that isn't necessarily the case, if nothing else due to the fact that I've never gotten around to writing actual documentation. If you have any questions (and W:AB noob can't answer them I guess :P) feel free to ask here.

Return to “Modding”

Who is online

Users browsing this forum: No registered users and 4 guests