Spiral Knights - SpiralSpy XML to OBJ

SpiralSpy XML to OBJ

80 replies [Last post]

Sat, 01/10/2015 - 18:39

Xan-The-Dragon

I want to hear what you guys think of the capabilities of my program.

Here's what I did:
While looking through a program called "SpiralSpy" (A SK model viewer), I noted that it used Three Rings's libraries to read and write client-side game models, as well as export to the XML format (I think the libraries are called "Clyde", "Narya", and "Nenya").

When I saw potential for using the models outside of Spiral Knights (further, SpiralSpy), I designed a code in RBX_LUA (ROBLOX's Sandboxed version of Lua 5.1) that converts the XMLs to something that ROBLOX can read. After the community had told me "I don't want to have to get ROBLOX just to convert my models!" (Pretty much), I designed a program in Java that is capable of converting these XMLs. I have MANY updates to do.

Java Program (Updated 5:27 PM (Central Daylight Time) on March 29, 2015):
Status: No bugs found (Stable!)

GET IT HERE: http://www.mediafire.com/download/4x9nzn9b3snyz08

Updates/Fixes:
* Fixed bug that caused program to not run. Details: I had, while experimenting with textures, forgot to add a condition that checked if no texture existed. In this case, it would assume the texture existed but then complain (and break) because it doesn't actually exist.

Currently working on:
Textures

EXTRA!
I have asked the creator(s) of Spiral Spy to incorporate this conversion system into Spiral Spy - if things work out as intended, you may be able to export as .OBJ directly from Spiral Spy!

Sat, 01/10/2015 - 18:50

Traevelliath

Well considering that Spiral Spy has been advertised on this very forum and you can find several Spiral Spy comics in the Treasure Vault, I don't see anything wrong with what you're doing.

Just be careful because Spiral Spy will sometimes find models that aren't yet implemented, and those are generally considered off limits.

Sat, 01/10/2015 - 18:55

Xan-The-Dragon

The Code

Alright.

Also, I've decided to release both the Lua code and a tutorial on how to use it until otherwise told to remove it by a staff member. Here's the video (which has the required resources)

https://www.youtube.com/watch?v=FLfI5QQFAic

Sat, 01/10/2015 - 20:39

Dahall

Nice! Hey, can you make an importer/exporter for Blender?

Sat, 01/10/2015 - 22:01

Gkku

Wooooow

I don't know about the ToS stuff, but as someone who loves SK and is pretty tight with the folks over at ROBLOX, this is extremely cool.

But yeah, texturing would be impossible at the moment. Unless you feel like tediously turning the texture into several hundred decals that you'd then apply to every individual face.

Sun, 01/11/2015 - 00:55

Glacies

Obligatory "Oh, man. Oh, god, oh man, oh god, oh man."

This is absolutely impressive. After I get some sleep and prep up my computer, I plan on exporting and seeing how well it fairs. Is it possible to also make an import tool or is that out of the question at the moment?

Sun, 01/11/2015 - 03:19

Krakob

Using their open source libraries isn't agains the ToS, but using their models probably is. Questions regarding the Terms of Service should be sent to tos@threerings.com, anyway. Try that in the future.

Sun, 01/11/2015 - 07:22

Xan-The-Dragon

Replies

@Dahall - Indeed I could! IF YOU WANT AN OBJ TO IMPORT: To do this, I need to first set up the part of the code that reads bone data and will be able to separate the body parts (as opposed to what it currently does where it's all one shape with ___ triangles). Plugin? I don't know Python, but I am sure I could take the time to learn it just for you guys :)

@Krakob - Alright! I'll be sure to do that in the future.

@Glacies - Well, I'm releasing an update JUST after this message that stops an error that will sometimes happen on larger models (you will note this on the model in C:\Program Files\Steam\SteamApps\common\Spiral Knights\rsrc\world\prop\clockworks\gears\1unit_teeth\cw-copper_12teeth.dat), which if you like the technical stuff was caused by a missing table entry due to the index being all weird. It was easy to fix, all I had to do was add a few conditions.

@Gkku - Yeah, that's true with texturing. Though last night I did get an interesting idea -- What if I exported the model as an OBJ file (because on ROBLOX studio you can do that. Right click "Triangles" in Workspace, and click "Export Selection...") and set up my own MTL that would read the PNG file, export it using Blender (because sometimes if you delete the PNG after importing the OBJ that uses it, it will create its own MTL that tells the colors and the faces and such) and read that MTL. That /might/ work.

@All - I am unsure if this is possible, but I believe you don't have to have a ROBLOX account to get the model. You can give it a shot.
At the bottom of the studio window, there's a bar that says "Run a code" -- type in this:

game:GetService("InsertService"):LoadAsset(202121392).Parent = workspace

Sun, 01/11/2015 - 12:46

Xan-The-Dragon

Going further...

I am considering something that can read the other types of models (i.e skyboxes) and render them properly. Right now, I'm stumped on how the XML applies bone data to certain triangles, and am actively looking for how to do it. Otherwise, update 4 will have skybox and effect support if I can get it working.

Sun, 01/11/2015 - 13:24

Gbot-Vtwo

!!!

Sun, 01/11/2015 - 16:04

#10

Xan-The-Dragon

Github and progress

I'm new to Github (as in JUST started), but I am going to be posting my progress there.

https://github.com/XanthicDragon/SpiralSpy-XML-Reader

Wed, 01/14/2015 - 05:29

#11

Krakob

If you wanna learn Python, I recommend this page. It has a lot of good links and you can probably get started with Python in half an hour if your prior knowledge is sufficient, which I'm quite certain it is given that you're able to pull this off.

As for continuing this project, I suggest you go for a standalone converter or a Blender/3ds Max plugin (I vote Max, but that's just because it's the tool I use :P ). Most people who use this would probably prefer going Spiral Spy -> your program -> generic 3d model file -> game engine of choice. Having Roblox be a requirement feels a bit backwards. A huge plus with this is that you wouldn't be restricted by Roblox' engine, which, if I understand you correctly, does not support textures. For those who would like to use this for non-Roblox purposes, being required to manually re-do the texture mapping and all would be a real pain. All that said, I don't know too much about technical 3d stuff and whatnot so although I wish I could, I don't think I can help you much. I do know Python moderately well though, so if you'll start toying with that I could try to help you out if needed.

As for the whole against the ToS or not thing, I'd guess that the GMs are cool with this given that it's been a few days since you published this thread and they've yet to reply or close it down.

Wed, 01/14/2015 - 06:06

#12

Dahall

Blender can convert to .fbx so there's that.

Wed, 01/14/2015 - 10:35

#13

Xan-The-Dragon

Alright, Krakob. I'll be sure to do that. The reason I did it in ROBLOX was because that was the first coding language I learned, thus the most familiar. Like I had said, I should have a plugin for Blender relatively soon. I want to finish the Lua code first so I at least have something to build off of. I could start developing a "beta" version of the importer which will create one single shape for the whole mesh, but I don't expect bone data (You know, the stuff that says "Triangles A, B, C, D, E, and F are for the _____ part!") to be ready any time soon, since the way SpiralSpy handles them is still somewhat foreign to me. There seems to be different methods for characters, skyboxes, and just plain models in getting bones. I'm learning rather quickly, so it's still debatable for what I could do with my Python code on the view if it should be a single item in 3D space or a set of them for the bones. Also, thanks for your suggestion on where I should learn Python. It should drastically speed up the process if it's the proper resource (which I am sure it is). I'll be sure to sufficiently learn in the free time I have, so I can get this out to the community ASAP.

Wed, 01/14/2015 - 19:24

#14

Xan-The-Dragon

I'm curious, community:

Would you rather have me develop a plugin for blender?
~OR~
Design a program in Java that you can execute to convert the XML to a very universal OBJ?

The reason I ask this is because of the way string manipulation works in Python - It's being VERY difficult, and hindering my ability to design this plugin. Also, I don't know python too well (I checked out the tutorial, if you're worried I didn't try!), and know java a bit better.

Wed, 01/14/2015 - 23:00

#15

Glacies

I personally would prefer the latter.

I myself use Autodesk 3ds Max when modeling as I prefer it over Blender. (for whatever reason I have never gotten it down to this day.)

Universal OBJ gives a lot more openings along with various forms of what can and cannot be completed in a preferred model program. It also sounds like the latter would be much easier for yourself to work with and personally seems easier to work with overall.

Thu, 01/15/2015 - 01:11

#16

Dahall

Sure, Blender can import from .obj format as well.

Thu, 01/15/2015 - 03:33

#17

Krakob

The latter sounds like a better idea, but I think it might in reality be harder, as Blender (probably) provides lots of tools to modify and create 3d data for plugin writers. As for string manipulation, why is it relevant? I'm guessing you're thinking about parsing the XML, in which case, there's a module for that. There are modules for most things in Python, really. If there isn't in the standard python library, there probably is in the Python Package Index.

Thu, 01/15/2015 - 07:48

#18

Xan-The-Dragon

Yes, String Manipulation is for the XML handling. I was simply doing it in Python in the same method that I had done in Lua. If there is a module for it, that helps a TON.

With 3DS max? I don't own that software, and don't know how plugins are designed. Yes, I can watch a tutorial, but how do I test it? I don't have any friends who own this software, and feel it'd be a pain in the butt to have to do this over the internet. Unfortunately, 3DS max just may be out of the window.

If I were to look at the application scope of this, a Java program would be optimal. I'm sure I can find a class on Java that is designed for exporting other model formats, including one specifically designed for exporting 3DS Max's filetypes. Other than that, I could continue working on the Python importer for Blender.

My point with this is that I would prefer to stick with what I know, but I'll let you decide again and that's what I will go with.

Thu, 01/15/2015 - 07:35

#19

Krakob

You can get a 3-year student license on 3ds Max if you simply are a student. If not, it does indeed get quite expensive. Either way, I think it'd be best if you were to go with what you know and although there may be a big amount of 3ds Max, it'd be easier for them to just grab Belnder whereas acquiring 3ds Max isn't quite as easy for Blender users.

I suggest you go with your Java idea, anyway. If it's what you know best, it's what would yield the best results.

Thu, 01/15/2015 - 08:10

#20

Dahall

"You can get a 3-year student

"You can get a 3-year student license on 3ds Max if you simply are a student. If not, it does indeed get quite expensive"

Being a pirate has it's perks

Thu, 01/15/2015 - 08:41

#21

Krakob

Shhhhh, the cyber police might be listening!

Thu, 01/15/2015 - 10:51

#22

Eternity-Terminal

Shhhhh, the cyber police might be listening!

You rang?

(Note: I am not really the cyber police. This is for humor. Now pretend that this disclaimer didn't exist and react in a way which will tingle my humor receptors.)

Thu, 01/15/2015 - 13:03

#23

Dahall

Good Luck

Sun, 01/18/2015 - 13:46

#24

Xan-The-Dragon

Party Time!

Alright guys! The Java version of this program is out!

Notes: I haven't compiled it to run if you double-click the jar file. LUCKY FOR YOU, I have designed run scripts in both Windows (.bat) and Mac (.app) format.

I have to have the two versions (Java 7 and 8) because executing it on Mac seemed to get all mad that the classes were out of date.

Java 7 version:
http://www.mediafire.com/download/ewd7pivd8cwwdgm

Java 8 Version:
http://www.mediafire.com/download/plv9zqvtlukvc7m

If it doesn't work, just tell me.

Sun, 01/18/2015 - 13:53

#25

Xan-The-Dragon

Problems

I noted some problems with the program's ability to convert models. Sorry about the false sense of happiness! I will have them updated ASAP

Mon, 01/19/2015 - 04:10

#26

Krakob

This is very exciting! Can't wait to test it out later today.

Mon, 01/19/2015 - 09:34

#27

Dahall

@OP

Thanks for doing this! btw, the plural of vertex is vertices.

EDIT: Yep, you know why I'm editing this.

From the looks of it, the triangles are connected in random and have rougher vertex placement compared to SpiralSpy's one. That particular model is of the T1 sabre, or at least that's what it was supposed to be.

EDIT2: I have deleted all the faces and edges from the model and it seems the vertices are placed accurately. Also, it seems that I may not have completely copied the data which could explain why the blade is too short. Here's the corrected version

EDIT3: the incomplete import seems to be from your application. Can you tell what causes the mesh to be partially imported?

EDIT4!!!!!: So I managed to swap the game files in SK. Here's an acheron that looks like the saucepan

Great progress Nonetheless!! Please do keep it up.

Mon, 01/19/2015 - 23:29

#28

Petater

More selective ToS enforcement... why hasn't this thread been removed? I thought spiralspy edits were a no-go.

Mon, 01/19/2015 - 20:21

#29

Fehzor

^LOL, story of my life

Mon, 01/19/2015 - 20:23

#30

Inalaman

What do you have against spiral-spy? :/

Mon, 01/19/2015 - 20:27

#31

Petater

Nothing, just the weird instance of some things being qualified as violating ToS while spiral-spy posts are abound. Feeling like we'll get warnings and bans for posts that we feel don't qualify for such actions because of the precedent spiral spy posts set kinda make us salty.

Mon, 01/19/2015 - 20:39

#32

Skepticraven

↓

Tater loves to derail topics to make them about him. It's a privilege of being salty that he gets.

By post #6, I'm fairly sure this thread has gotten a look-over by staff.

Mon, 01/19/2015 - 20:43

#33

Fehzor

Like lets pretend for a minute that the rules applied to me = the rules applied to Spy.

http://forums.spiralknights.com/en/node/105126

If I post a spoiler, I get a warning. This is true because I accidentally did that and got a warning. Specifically, anything from the game's files constitutes a spoiler. Similarly, if I deliberately lie to people for no reason, I get a warning. This is because for me, that is considered spam-- I asked after receiving a warning for deliberately deceiving people. Ergo, Spiral-Spy MUST be breaking the rules that have been applied to me-- if Spy is making up the information then it is "spam" that constitutes some sort of warning, and if Spy is getting information from elsewhere it constitutes a "spoiler". The only acceptable alternative, researching the rates, would take a huge sample of data that would have been impossible for him. But it really isn't just Spy, this kind of thing happens all of the time, like in the frying pan picture above-- the terms of service specifically prohibit modifying files like that.

It's not that big of a deal all things considered, it just drives me nuts because I'm a sack of garbage that nit picks about that kind of thing on the internet, and am mostly just butthurt because no one appreciates or validates anything I say or do. I know this to be true because I have been told so.

Mon, 01/19/2015 - 23:31

#34

Petater

@skeptic

Except it utilizes the installed copy of spiral knights or the fact that to display the models and integrate with the sk install, reverse engineering had to be done. Skeptic you know as well as I do the ToS can be used to nuke all spiral spy posts. Selective line iteming of ToS as the "allow" reason can be nullified with other various statements bent to the degree that steam thread was nuked by.

Mon, 01/19/2015 - 20:48

#35

Thinslayer

@Petater

The GMs themselves said they are cool with it. They even provided reasons and linked to the appropriate section of the ToS.

See for yourself.

Mon, 01/19/2015 - 20:52

#36

Fehzor

@Thinslayer

No, they said the opposite in the link you mentioned-

Any attempt to modify or disassemble Spiral Knights in any shape or form is against the Terms of Service.

The post in question showed a swapped out model file in an installation of the game.

Mon, 01/19/2015 - 20:53

#37

Petater

". Any
attempt to modify or disassemble Spiral Knights in any shape or form is
against the Terms of Service. "

Yeah, in order for spiral spy to exist that rule had to be broken, as spiral spy can only exist by reverse engineering of spiral knights.

Mon, 01/19/2015 - 20:55

#38

Thinslayer

@Fehzor

That's the rub. SpiralSpy doesn't disassemble Spiral Knights; it peeks into its files. That's a very different animal. Like other posters have said, that's like banning Microsoft Word because it can look at their text files. There is nothing inherently illegal about it, as the GMs said.

Mon, 01/19/2015 - 20:56

#39

Midnight-Dj

Moral of the story, just draw Spiral knights on paper , no one can sue you for that.

Mon, 01/19/2015 - 20:58

#40

Thinslayer

Legality

My guess is that the devs and GMs are wary of SpiralSpy, but they have no legal recourse against it (seeing as it just peeks into the files without disassembling anything) so long as the players behave themselves in how it's used.

Mon, 01/19/2015 - 23:40

#41

Petater

Yes there is, Spiral Spy uses information about the rendering procedure that can only be gleaned from reverse engineering the format and runtime heavily. Or is the reason that steam login without steam thread was removed invalid?

Mon, 01/19/2015 - 21:02

#42

Fehzor

Spiral Spy didn't modify it, Dahall did.

http://forums.spiralknights.com/en/node/107593#comment-976531

Or no, Petater means Spiral Spy in general. That's my mistake. Either way, I'll be leaving this argument now, before I get in trouble for it. I also don't want to derail it any more than we already have.

Mon, 01/19/2015 - 21:02

#43

Petater

Arguments on the application of ToS is something that needs to be finalized so we don't get more salty players.

Mon, 01/19/2015 - 21:03

#44

Thinslayer

@Petater

"[it] can only be gleaned from reverse engineering the format and runtime heavily."
How do you know this?

"Or is the reason that steam login without steam thread invalid?"
What thread? I haven't seen it. Please link to it.

"FYI it modified a text file with standard non-proprietary entries that were not even a part of spiral knights, nor a part of their intellectual property, yet got removed because spiral knights might read it and alter what it does."
The confusing grammar prevents me from knowing what "it" refers to. Clarify please.

Mon, 01/19/2015 - 21:08

#45

Skepticraven

↓

A better thread for this derailment.

@Thinslayer
About 4 threads in the support subforum that I described over here, since the threads themselves have been hidden.

Mon, 01/19/2015 - 23:42

#46

Petater

> "What thread? I haven't seen it. Please link to it."

It's been removed for breaking ToS for the reason of modifying the runtime of spiral knights.

> ""[it] can only be gleaned from reverse engineering the format and runtime heavily."
How do you know this?"

Because OpenGL renderers that magically decode the custom file format for 3D models don't just poof out of thin air, even when utilizing the same base code libraries.

Mon, 01/19/2015 - 21:14

#47

Thinslayer

@Petater & Skepticraven

Yeah, I think that other thread is better for this discussion. This post sums it up well, I think.

Mon, 01/19/2015 - 23:44

#48

Petater

And what that thread summed up was what I was stating earlier. Curation by selective enforcement of ToS by being on their good or bad side. I'm just wondering if you see why I'm salty and why I bring up spiral spy into this mess.

Mon, 01/19/2015 - 21:32

#49

Avihr

Void

So you rather have them censor SpiralSpy related threads so we can be salty about them not letting us have nice things? :P

Mon, 01/19/2015 - 21:39

#50

Petater

How about allow images generated from it, but linking the download to spiral spy (other thread linked to the spiralspy dl site) or sharing 3d model code no, since those have precedents for being removed when it's other various material.

Search

SpiralSpy XML to OBJ