Resource files management IV

Previous posts:
Resource files management. Part I
Resource files management. Part II
Resource files management. Part III

Okay, so far we've been writing helper classes and now we're getting to the main stuff: ResourceFile class.
ResourceFile is pretty simple, let's take a look over its interface:

class ResourceFile
{
protected:
    ResFileHeader mFileHeader;       // Resource file header structure
    std::vector<Lump*> mLumpList;    // List holding every lump
                                     // of the resource file

    std::string mFileName;           // Local storage of the
                                     // resource file's name
    OpenMode mFileMode;              // Current file access mode
    std::fstream mFileStream;        // Primary stream for reading data
    std::fstream mSaveStream;        // Helper stream for saving data

    bool LoadLumpList();             // Loads each lump (info) from file
    bool SaveLumpList();             // Saves each lump (info) to file
    void UnloadLumpList();           // Clears mLumpList
    bool SaveLumps();                // Saves each lump (data) to file

public:
    ResourceFile() {}
    ResourceFile(const std::string& FileName, OpenMode FileMode)
        { this->Open(FileName, FileMode); }
    ~ResourceFile();

    // Open existing resource file and read data or create new file
    bool Open (const std::string& FileName, OpenMode FileMode);

    // Save data to resource file
    bool Save (const std::string& AltFileName = std::string());

    // Close resource file and clear data from memory
    void Close();

    // Add new lump to resource file from external file
    bool CreateLumpFromFile (const std::string& Name,
                             const std::string& FileName);

    // Add new lump to resource file using data from memory
    bool CreateLump (const std::string& Name, void* Data = NULL);

    // Delete selected lump from resource file
    bool DeleteLump (const std::string& Name);

    // Load data for selected lump
    bool LoadLump   (const std::string& Name);

    // Unload data for selected lump
    bool UnloadLump (const std::string& Name);

    // Accessor functions
    std::vector<Lump*>* GetLumpList();
    Lump* GetLump(const std::string& Name);
};

That's it - simple resource file class. Now you can take it and use in your projects, though you may want to tweak it more - add compression, encryption, internal directory hierarchies and so on, there is much to do more.
But now I'll just stop on this simpliest implementation of resource file. There is several things I would like to mention.
First of all as you may notice resource file reads or writes lumps in two separate passes. Let's look at opening resource file logic:
At the beginning ResourceFile checks if we want to open existing file or create a new file. If we are creating new file, then it just opens a new file for writing, but if we want to open an existing file, then the ResourceFile class opens that file and starts reading useful information from it.
First it reads mFileHeader, from which it gets information about the file, more specifically how much lumps are stored in file and where is the lump information list. Having this information the ResourceFile class starts creating N lumps according to information stored in ResFileHeader and reads information of each lump from lump information list in file.
After performing these operations we are having something like this, assuming there are two lumps in a file:

ResourceFile -
    mLumpList[0] -
        mLumpInfo -
            Size     = 215
            Position = 25
            Name     = "Grass Texture"
        mData = NULL
    mLumpList[1] -
        mLumpInfo -
            Size     = 198
            Position = 240
            Name     = "Sand Texture"
        mData = NULL

As you can see, after having this information we can dynamically load the data for any lump we want in our lump list, we just specify the name of the texture, ResourceFile finds it in a file according to the stored information and loads its data to the memory. This is how LoadLump works.

Now the second thing is about saving files. The saving procedure is implemented this way:

1. Write mFileHeader
2. Write all lumps data
3. Get current position in file and update mFileHeader.LumpListOffset
4. Write all lumps information
5. Return to the beginning of the file and rewrite mFileHeader

Here you can see that we need to write resource file header twice. Well that is because I'm pretty lazy now for writing better methods. The reason we need to write file header twice is because ResourceFile doesn't knows where is the lump info list is located in file before it actually writes it. Therefore when we get to the position where the lump info list will be written, we update the mFileHeader with this information and proceed. After all information is written we get back to the beginning of the file and rewrite the header. Of course this can be battled if you write updating of LumpListOffset each time you are adding or removing lump to mLumpList, i.e. add lump's size to the LumpListOffset if you're adding a lump or subtract the size if you're removing it. Also, if you're creating a new file you'll need to set the initial size of LumpListOffset to be equal to size of the ResFileHeader, because the LumpListOffset is equal to ResFileHeader size + sum of each lump size. This is not needed when you're reading existing file as initial value of LumpListOffset is loaded from file.

Now I must admit that the previous saving sequence is not real. The full sequence looks like:

1. Create temporary file
2. Write mFileHeader
3. Write all lumps data:

    for each lump
    if lump's data loaded in memory
    3.1 Write data from memory to file
    else
    3.2 Copy data from existing file to temporary file

4. Get current position in file and update mFileHeader.LumpListOffset
5. Write all lumps information
6. Return to the beginning of the file and rewrite mFileHeader
7. Close both files
8. Remove original file if we're overwriting it or remove alternative-name file (Save As)
9. Rename the temporary file to original or alternate name

As you can see, there is much going at the background. Though, this is not the best solution (imagine doubling 8 GB resource file), but it is easier to implement.

Phew, that was long one.
What to do next? You can go further and create GUI editor of resource files. Well, my friend made one already, it is Qt based.

Source included:
ResourceFile.h
ResourceFile.cpp

Please, feel free to contact me if you're having problems compiling this source.

Resource files management III

Resource files management. Part I
Resource files management. Part II

In previous post I wrote about implementation of data structures of the resource file.
Today's topic will be serialization of resource files.

File types

Ok, to retrieve our resources first of all we'll need to define basic methods of reading and writing with files.
In C++ it could be done simply using file streams. Refresh your memory on std::fstream methods if necessary.
Next, the format of file must be chosen. There are two choices to go from here: text/binary format.
When using textual format the data in file is represented as human-readable text.

myresource.dat:
[LumpCount] = 3
[LumpListOffset] = 215
...
(Note: .dat filename extension is not a requirement, you can use anything you like, however you should choose unique extension that is not used by other programs)

Whereas, binary files contain non-readable characters and from human point of view its just a mess.
However, data in binary files is always packed and so it takes less storage than text files. On the other hand, binary files are platform-dependent, i.e. int variable on x64 platform will take twice more than the same int on x86 platform. Text files don't suffer from this issue - text is always the same on any other platform.
And lastly you should know that files you will store in resource file probably would be of binary format, therefore if you are up to using text files you should think about how to write binary data to text file.
That's said I'm sticking up to binary format and below are the simple writing/reading methods of data to/from binary file.

Serialization

According to wikipedia:

Serialization is the process of converting a data structure or object into a sequence of bits so that it can be stored in a file or memory buffer, or transmitted across a network connection link to be "resurrected" later in the same or another computer environment. When the resulting series of bits is reread according to the serialization format, it can be used to create a semantically identical clone of the original object.

What it says is, serialization is a tecnhique of writing data of objects in program to another source (in our case it is a file), so that it is possible anytime to read back the data stored in file and load it into the objects.
Unfortunately, C++ do not provide native support for serialization. Therefore, we'll need to write own serialization methods to write our data to file.

I've done simple functions of writing/reading basic types and std::strings with a file, you may write additional functions that suite yourself (std::vector etc.)
Here is the source: Serialization.h
The code is pretty self-explanatory, though if you have questions, just ask. Next, I'll write about how to use this serialization stuff.

Okay, last time we discussed the data structures of resource file and I've dropped functionality of Lump and ResourceFile classes on purpose for the sake of clarity. Now, I'll define methods of Lump class (ResourceFile is left for the next post).

Lump interface is pretty simple:

class Lump
{
protected:
    LumpInfo  mLumpInfo;
    void*     mData;

public:
    Lump();
    ~Lump();

    bool LoadData (std::fstream&);        // Read data from file
    bool SaveData (std::fstream&);        // Write data to file
    void UnloadData();                    // Clear data

    void LoadInfo (std::fstream&);        // Read information from file
    void SaveInfo (std::fstream&);        // Write information to file

    // accessors and mutators
    ...
};

That's all. Lump class implements information reading and writing methods that simply call serialization write() and read() methods for each data member of mLumpInfo:
void Lump::SaveInfo (std::fstream& fs)
{
    Serialization::Binary::Write(fs, mLumpInfo.Size);
    Serialization::Binary::Write(fs, mLumpInfo.Position);
    Serialization::Binary::Write(fs, mLumpInfo.Name);
}

void Lump::LoadInfo (std::fstream& fs)
{
    Serialization::Binary::Read(fs, mLumpInfo.Size);
    Serialization::Binary::Read(fs, mLumpInfo.Position);
    Serialization::Binary::Read(fs, mLumpInfo.Name);
}

Also Lump can dynamically load and unload lump content from file to memory. Here is the full source:
Lump.h
Lump.cpp

Feel free to ask any questions you have. Next time I'll write about ResourceFile class.

Resource files management. Part IV

Resource files management II

Resource files management. Part I

Last time I've described resource files and their data structure. Let's continue with management of resource files.

Lump and ResourceFile objects
Resource files are implemented as objects and may be included in other programs to utilize resource files. I'm using C++, though the provided concept may be written in any other programming language.

In the previous post we have learned that resource file consists of header, lump's data and lump's information. Therefore, we'll need two classes: Lump and ResourceFile and two additional data structures: ResFileHeader, LumpInfo:

// Structure defining the header of our resource files.
struct ResFileHeader
{
    unsigned long Version;                  // Resource file version
    unsigned long LumpCount;                // Number of lumps in res.file
    unsigned long LumpListOffset;           // Position of the first
                                            // Lump info list element
};

// Structure of each element in the lump info list.
// Stores information which can be used to retrieve the contents of a lump.
struct LumpInfo
{
    unsigned long Size;                     // Size of the lump, in bytes
    unsigned long Position;                 // Position in the res.file
    std::string   Name;                     // Name of the lump
};

// Class representing single resource stored in a resource file.
// Stores lump information and lump contents
class Lump
{
protected:
    LumpInfo mLumpInfo;                     // Our lump information
    void*    mData;                         // Lump's contents
...
};

// Class representing single resource file.
// Stores resource file header and list of lumps
class ResourceFile
{
protected:
    ResFileHeader mFileHeader;
    std::vector<Lump*> mLumpList;
...
};

That's it. We'll discuss functionality of two classes later, right now we know the structure of our resource file. Once again, from abstract view the structure of resource file looks like:
[Resource File]
-   [ResFileHeader]
-   [Lump]
    -   [LumpInfo]
    -   [Data]
-   [Lump]
    -   [LumpInfo]
    -   [Data]
    ...

whereas in practice LumpInfo is separated from Lump:
[Resource File]
-   [ResFileHeader]
-   [Lump]
    -   [Data]
-   [Lump]
    -   [Data]
    ...    
-   [LumpInfo]
-   [LumpInfo]
    ...

Next time I'll write about basic functionality that Lump and ResourceFile implement.

Resource file management. Part III
Resource file management. Part IV