MIPINDEX

a miptexture indexer for DOS

Copyright (c) 1996 by Edward Kiser

  The beaten path
  will lead you to your doom.
  That is its revenge
  for being beaten.

    -- Zen proverb

[0] Table of contents

  1. System requirements
  2. Installation of the executable
  3. What mipindex can do, and how to make it do it
    1. Getting the command syntaxes
    2. "Dir" for miptextures
    3. Building WAD2 files
    4. Converting BSP <-> DST
    5. Miscellaneous quirks
    6. Examples
  4. History and Bugs; Updates and Bugfixes
  5. License & legalese
  6. About the source code
  7. Submitting changes and new features
  8. Acknowledgments
  9. About the author
I might have written things in this file which you already know. I do not intend to insult your intelligence, but I cannot read your mind. Please forgive me if I bore you -- and don't read the boring parts more than once!

[1] System requirements

This program was (and can be again) compiled with DJGPP 2.01 and ought to run on any DOS system which can run Quake. You will need CWSDPMI.EXE if it says "Load Error: No DPMI." You can probably use the CWSDPMI that came with Quake, or you can download a new version from the same places that offer DJGPP. To find some places that have it, go to the DJGPP homepage at http://www.delorie.com/djgpp/.

Under Windows 95, Mipindex does not need CWSDPMI.

[2] Installation of the executable

Put the EXE in whatever directory you prefer. Put CWSDPMI.EXE with it if you need to. Then you're done.

If you put Mipindex on your PATH, or in the directory from which it is used, you won't have to type as much when you invoke it.

If you have PGP, you can get my public key from the Mipindex homepage (the URL is at the bottom of this file) and check the signatures on everything. Just so you know they're for real, and not modified versions. This file's signature is attached, MIPINDEX.EXE has the signature MIPINDEX.SIG, and SOURCE.ZIP has the signature SOURCE.SIG.

PGP is not required to actually use MIPINDEX -- only to verify its official-ness. If you want PGP, though, the public key on my homepage has links to the U.S. and International distribution sites. Be sure you choose the right one.

[3] What Mipindex can do, and how to make it do it

Mipindex 1.2 can:

Now I'll describe these things.

Getting the command syntaxes

Just type Mipindex by itself to get help and see all the syntaxes for the various options. If Mipindex ever doesn't understand your options, it'll tell you its syntax again.

Dir for miptextures

Mipindex can act like a dir command for miptextures. It only lists each miptexture the first time it appears. It can process BSP (version 29 only, which corresponds to Quake 1.00 - 1.06, I think), WAD2, IDX (a Mipindex-specific format you'll read about shortly), and PAK files. Mipindex also understands a certain 768-byte file called palette.lmp. palette.lmp is the only file Mipindex treats as having a single miptexture in it.

Mipindex will complain if it doesn't understand a file type, and then it will move on to the next file without pausing.

The output can be piped or redirected, just like any other DOS command.

Mipindex doesn't care whether you use backward or forward slashes. Parameters are processed in the order given; wildcards are expanded in alphabetical order.

You can't do a -list on a DST with this version of Mipindex.

Building WAD2 files

Mipindex can also build WAD2 files for use with map editors or QBSP or both. No two miptextures are "identical" in the WAD2 that results. This keeps it small.

Mipindex considers miptextures identical if their names and sizes are identical. If the names of two textures are the same but the sizes are different, like metal5_8 in registered Quake, Mipindex calls them different and includes them both. QBSP only looks at names, though -- that fact should convince you, if you create your own textures, to give them unique names. metal5_8 is an oddity.

Mipindex does not allow empty space to exist in its output WAD2. You can build a WAD2 from another WAD2 to get rid of free space created by other utilities. This rebuilds the entire WAD2. (It also gets rid of everything but miptextures and the first palette. You've been warned.)

It does not matter whether you have unpacked your PAK files or not -- except, if you have, you have to specify more files to build a WAD2 from.

It is possible that Quake contains miptextures with identical names and sizes but different image data. Mipindex will think they are the same. A future version may keep CRCs of miptextures to respect the distinctions.

A palette is considered to be a special kind of miptexture. It is indexed and written to WAD2s just like a miptexture, except its name is always palette and its size is always 256 x 3, which is illegal for an actual miptexture. (Quake never finds out Mipindex is thinking this way.) Because all palettes have the same name and size, Mipindex thinks they are all alike and only allows one palette in an index or a WAD2. Right now, though, Quake only has one palette anyway, and this behavior shouldn't cause any problems.

Converting BSP <-> DST

The DST is a new file format introduced with this program. DST means "distributable." Right now only BSPs can be converted to DSTs, but conversion of DST <-> other file formats may be supported in future versions. Depending on how many miptextures a BSP file uses, a DST can be 25% to 33% smaller.

A DST file is not compressed. It can be compressed with an archiver. It is smaller because it doesn't have miptextures in it.

When Mipindex converts a BSP to a DST, it uses a miptexture index. All the data in the DST is copied from the BSP, except for the miptextures in the index (i.e. with "identical" copies in the index) -- only their names are copied. If the index covers all of id Software's textures, the DST will not contain any of them.

For the reverse conversion, Mipindex uses the index to recover the original textures that were removed. In a future version, maybe commonly requested textures will be cached, but for now this seeks all over your hard drive. The reverse conversion will not result in a byte-for-byte identical BSP. Empty space between the miptextures will be removed, and if there were multiple references to the same miptexture in the original BSP (a possibility which QBSP should eliminate anyway), each reference will have its own copy in the resulting BSP. Also, the "lumps" in the BSP will be put in "order." However, the files play just the same in Quake. I've tested it many times with many BSPs. You can too.

A DST may have interesting legal properties. The creator of a BSP typically cannot distribute the BSP for profit, because it typically contains id Software's textures, and is therefore a sort of collaborative work with id. id Software doesn't like people making money off of the redistribution of their textures. However, with Mipindex, you can make a DST which does not contain id Software's textures. In theory, this DST is the sole and entire property of the map creator, and can then be sold if the map creator wishes.

Sometimes (only in specific instances), Mipindex can leave some of id Software's miptextures inside the DST. This can happen if an id miptexture is not in your index, or if you've created a "derivative work" miptexture and given it a new name.

Also, if a BSP contains a new texture with an old name and size, Mipindex will remove it without noticing that the image data is different, and the BSP that other people get will contain the wrong image data.

It is your responsibility to check for these situations and avoid them.

It is still illegal to sell BSPs or DSTs of a level without the level creator's permission. If you are the level creator, this is not a problem. Otherwise you'd better ask.

Also, you may have to sign the COMEXP license to use the Quake trademark for your own commercial gain. (Or you can just say you're selling Mipindex DSTs and hope your customers draw the intended conclusion.)

An IDX (index) file format is supported, so that Mipindex can index several files and use textures from all of them for BSP <-> DST conversion. This IDX file contains the full paths of its sources, and byte-offsets directly into the files, so if you move or alter the files you've indexed, you'll have to reindex them.

You can use wildcards with the conversion commands. Mipindex will not overwrite or alter existing files.

At the end of conversion, Mipindex prints a summary of its successes and failures. Failure does not mean that no output file was created. Sometimes a bad output file is created. If Mipindex reports an error, delete the corresponding output file.

Miscellaneous quirks

When running under Windows 95, Mipindex supports long filenames. This can be particularly insidious at times, such as when you accidentally enter

      mipindex -wad2 my.wad2 d:*.pak
and it creates my.wad2. You can rename it, but... look out for this.

Mipindex's support of long file names may not be complete. I don't recommend you use it. There's really no need; Quake doesn't support LFNs anyway.

If you have any comments, requests, or bug reports, send them to me at edkiser@jaxnet.com and I'll read them. And, if time permits, I'll do something about them.

Or, if you prefer, fix the bug yourself, and tell me how you did it.

Examples

To get a directory of miptextures, use mipindex -list *.pak or mipindex -list *.bsp. The list is quite long so you'll want to pipe it through a pager like more or Vernon D. Buerg's list (not included).

To build a WAD2, use something like mipindex -wad2 output.wad *.pak.

To convert BSPs and DSTs, first you have to make an index. The command is similar to building a WAD2:

      mipindex -idx my.idx *.pak
or

      mipindex -idx my.idx maps/*.bsp
. Then, you use the index for the conversion:

      mipindex -undst my.idx ..\addons\maps\schneier.dst
or

      mipindex -dst my.idx ..\mylevels\maps\*.bsp
. The latter will convert all your BSPs to DSTs.

Mipindex never deletes, modifies, or replaces files. It only creates new files. The best way to learn to use it is to play with it and read the manual -- though not necessarily in that order.

[4] History and Bugs; Updates and Bugfixes

Version 1.0 of this program was only a WAD2 builder. It had a "theoretical" bug where miptextures were not deliberately aligned at multiples of 16. (They were accidentally aligned at multiples of 16. The bug is theoretical because I never saw it actually do any damage.) It also compared only names of miptextures.

Version 1.1 compared names and sizes and introduced BSP <-> DST conversion. But it had the same theoretical bug, and a second bug in BSP <-> DST conversion, where certain files would seem to have invalid textures and crash Quake as re-expanded BSPs.

Version 1.1a fixed both of those bugs. I've tested BSP <-> DST conversion on several Quake levels downloaded from the Internet, and it works every time. So far.

Version 1.1b fixed an annoying bug in the treatment of the palette during WAD2 building.

Version 1.1c corrected some minor errors in the documentation, added a reference to the new Mipindex homepage, and reduced the size of the executable by 100K by removing debug information. It was never uploaded to ftp.cdrom.com, and I never advertised it on the usenet.

This version, 1.2, comes with full source code and is distrubuted under the terms of the Free Software Foundation's General Public License. The old TERMS of USE have been replaced by the GPL. The -bsp option was changed to -undst, but -bsp will still work if you were using Mipindex in batch files.

Now and then I may release new "official" versions of this program. To see if you have the latest version, check the USENET group rec.games.computer.quake.editing to see if I've posted anything, or check ftp.cdrom.com to see if they carry a later version than the one you have. You can also send me an e-mail and I'll tell you what the latest version is, and at least one place where you can download it.

[5] License & Legalese

Everything in this distribution is Copyright (c) 1996 by Edward Kiser. The rules for distribution of the program are covered in the file COPYING.TXT, which was included in this archive.

This program does not contain any encryption algorithms and so does not fall under any export restrictions. :-)

The TERMS of USE that used to be here are now gone. They were redundant anyway; the things they prohibited were already illegal.

Quake is a trademark of id Software, and this program was written to work with Quake. However, id Software has nothing else whatever to do with this little project -- in fact, they may not even know it exists, even after all this time -- so they and I will appreciate it if you don't call them up with technical support questions about this program. Best to leave them alone so they will come up with more great games and game engines. Thanks.

This program uses the Standard Template Library (STL), which is Copyright (c) 1994 by Hewlett-Packard Company. It bears the following notice with regard to STL:

    Copyright (c) 1994
    Hewlett-Packard Company

    Permission to use, copy, modify, distribute and sell this software
    and its documentation for any purpose is hereby granted without fee,
    provided that the above copyright notice appear in all copies and
    that both that copyright notice and this permission notice appear
    in supporting documentation.  Hewlett-Packard Company makes no
    representations about the suitability of this software for any
    purpose.  It is provided "as is" without express or implied warranty.
Hewlett-Packard can't offer technical support for Mipindex, either.

I'm holding to the position that neither the Mipindex executable nor the source is a "copy" of STL. That's why neither bears the copyright notice for STL.

Provided you abide by the GPL, you can distribute Mipindex, or modified versions thereof, with your Total Conversion Projects.

The release of a new version does not invalidate your right to distribute an older version.

[6] About the source

This is the first version of Mipindex to come with full source code. The source is sloppy enough to be embarrassing, but I hope it's clear enough to work with.

STL still has some problems under DJGPP, but it has problems under all compilers so far, and it may be a while before the compiler gods get it all figured out. The makefile actually compiles a module called small.cpp that ameliorates those problems somewhat. (It #includes all the cpp files -- an ugly solution, but it works.) The compile time is not appreciably lengthened because the STL header files are bigger than the whole body of Mipindex code, and the executable ends up smaller, because with separate object modules, the linker lets each one have its own copies of the instantiated templates.

If you want to do it the old way, type make bigold.exe.

On my Pentium-60 the program takes almost one minute to compile. It takes longer with separate modules.

The module breakdown is as follows:

MIPINDEX
main module of the program
MINDEX
the class of miptexture indeces, and related functions (usually only one instance of this class is created at a time)
BSP-DST
conversion functions
QSTRUCTS
classes for quake objects, such as miptextures (somewhat sloppy)
UTILITY
miscellaneous functions
The code should be more or less self-explanatory. I've done what I can.

It compiles with only two warnings, both out of qstructs.cpp:

    initializer-string for array of chars is too long
You can safely ignore them, or fix them; your choice.

Anyone who would like to write a guided tour document is welcome to do so; I may even add it to the distribution if I'm offered the opportunity.

[7] Submitting changes and new features

Your modifications, by themselves, are your property. You can distribute them all you want. The distribution of modified versions of Mipindex, either source or executable, is covered by the GPL in COPYING.TXT. So you can distribute those all you want too.

But there's a certain sense of pride in knowing you've contributed... [Drum roll please!] to the... OFFICIAL VERSION! [Cymbols!]

If you'd like a shot at making your modifications a part of the official Mipindex, you can submit them to me for inclusion in the next official version of Mipindex.

If I decide to incorporate your modifications, it will become part of the official Mipindex or part of an official variant. All submissions will be treated as offers to me of the perpetual, non-exclusive, royalty-free right to incorporate those modifications into Mipindex, and distribute that new version of Mipindex under the GPL (which means your modifications, too, will be subject to modification). Should I choose to accept your offer, your name will be permanently placed in the Acknowledgments (or somesuch) in the documentation, and the history of the program (history.txt). Comments in the code will surround your changes as long as your changes are present.

I will not pay money for contributions! Don't even ask!

Submissions will only be accepted by E-mail. The entire submission should be in one E-mail. If you send me several E-mails, I should be able to pick and choose which ones to incorporate; don't send me a patch #2 which requires me to accept patch #1 (unless I already accepted #1.) Some hints on getting accepted:

I reserve the right to change my mind, but I'll tell you if I do.

I reserve the right to modify your modifications, either immediately after acceptance or later on; I'll treat it as if you modified the code and I modified it afterward, and the history will reflect it that way.

I reserve the right to take as long as I wish to consider your offer. If you think I'm taking too long, you can send an E-mail and withdraw the offer. When I have accepted or rejected your offer, I'll tell you -- though I won't necessarily tell you why.

All types of modifications (bugfix, new feature, design change) will be considered. I can't consider ports, because I probably don't have access to the platforms you are porting to, so I can't be sure the thing works properly. I hope you'll understand if I don't want to put my name on things whose performance I can't verify. But it's GPL, you can port it on your own all you want.

I'd appreciate bytewise file compatibility with DSTs. If you can't do that, change the file extension to avoid user confusion. Thanks.

If you alter or extend the DST format, your alterations should continue to recognize the current DST format, even if they no longer generate it.

No such preference applies to the IDX format. (It's easy to reindex stuff.)

[8] Acknowledgments

I'd like to thank: Thanks to Remy Horton who told me why some editors were choking on Mipindex's WAD2 files. (One of those editors, I discovered, was Quest.)

I'd also like to thank YOU for giving this a try. Thanks. If nobody ever tried this program, it would be pointless to have written it.

[9] About the Author

My name is Edward Kiser. "Kiser" rhymes with "Wiser".

You can e-mail me at edkiser@jaxnet.com. You can also visit the Mipindex homepage.

I do all sorts of stuff besides program and play Quake, but I've never quite been able to confine myself to any one thing. I am a dabbler; I have a lot of hobbies, of which Quake and Programming are but two.

As to my career, I'm a student majoring in Electrical Engineering. According to my university, I know absolutely nothing about programming. Mipindex should be proof to the contrary, but they won't accept it. Ah, bureaucracy. My senior design project is all-software though, which ought to raise some eyebrows...

Visit the Libertarian party at http://www.lp.org/