Jump to content
saebel

OBSE Plugin / Array Processing C++ Expert Assistance Needed

Recommended Posts

Hoping someone with knowledge sees this.

So I'm working on revamping my SDR.dll via OBSE's plugin system.  I'm trying to improve the overall processing time, get rid of memory leaks, etc., to the best of my ability.

Before I get into my problem, I should explain the structure.   There are two "Master" arrays used.

The first one is the Master COV Map, which uses nested maps to store custom object values.  It's structure is as follows:
map ["Parent Mod" (string), "Child Refs" (map["BaseRefID" (UInt32), "COV Groups" (map["GroupID" (UInt32), "COV Data Set" (map["Data Key" (string), "Data Value" (num, string, or ref)])])])]

The second one is the ModGroup ID Array, which is a simple array to assign an ID (the key) to a Mod Name / Group Name pairing.  The Group ID is then used for setting COV values (above) when functions are called to set/get/mod the COVs.  It will look like this:

[0] "SDR.esp|Actor Values"

[1] "SDR.esp|Weapon Values"

[2] "SomeOtherMod.esp|Name of the Group"

I have already completed the core programming needed to set/get/mod values and to add/create maps on the fly as needed.  However, I am concerned about bloat.

I have created some clean up processes that removes data from invalid references and mods that are no longer in the active load order.  However, they are all done through scripts through the SDR.esm.  I would much rather port those features into the SDR.dll to boost performance.

I need to be able to loop through all the Master data and remove any Group ID maps, reference maps, and mod maps that are no longer valid or active - removing entries as I go.  The Group ID array does not need entries removed, because I don't want the key numbers to shift (that can impact data stored in the active save game).  So if a group gets deleted, the entry will be set to "". 

In the SDR.dll, I'm using the OBSE Array Interface system (PluginAPI.h), which has all the functions I need - except one: there is no method to *remove* a key/value pair!!!

If the arrays/maps were just a standard C++ implementation, it would be no problem.  But the OBSE Array Interface has a very whacky construct.  I've tried tapping into ArrayVar.h/.cpp files to see if I can bypass the OBSE Array Interface and use some of the functions deeper under the hood, but I haven't had any luck.  C++ is not my strong suit, and I've come to a dead stop on how to handle this.

Suggestions of any kind would be extremely welcome.

Share this post


Link to post
Share on other sites

This looks like it’s not just C++, but an intimate knowledge of the OBSE code as well. Have you tried contacting shadeMe on GitHub or Nexus? I just checked GitHub and he was active two days ago, but it doesn’t look like there’s any messaging system. Here’s his Nexus profile.

I will poke a couple of folks for you, but as I said, I don’t think it’s just a C++ issue.

Share this post


Link to post
Share on other sites

This may be crude, saebel, as I have no knowledge of plugins, but can't you use native C++ code to delete keys/values?  Then use the OBSE api to talk between the plugin and esp?  Build your array and operate on it within the C++ plugin?  Like I said, I don't know shit about the OBSE api or plugins in general.

Share this post


Link to post
Share on other sites

Tried using Maps or Stringmaps (only difference is data type of the key; map=numeric keys)?

http://obse.silverlock.org/obse_command_doc.html#Array_Variables
Arrays according to the OBSE docs do not allow gaps. If you used ar_Erase on the element of an array, you would necessarily cause an index shift.
"An Array behaves like arrays in most programming languages: the key is an unsigned integer starting at zero, and there are no gaps between elements. (In other words, if an element exists at indexes 1 and 3 then an element necessarily exists at 0 and 2)"

The map type of array allows gaps, floating point keys, etc. I'm sure all the goodies you want are there without diving into the guts, so try constructing it like

(array_var) ar_Construct arrayType:map

Share this post


Link to post
Share on other sites

This is in a DLL, not game code. It’s written in C++ using the OBSE libraries. @Malonn, that’s a good idea. @saebel, you could do as Malonn suggested. I also saw something in the OBSE array library that suggested they do their own garbage keeping. We got our server fixed up and I got git installed, but I haven’t had a chance to upload the code to the server. Instead I’ve been tearing my hair out all day as I try to get Fail2ban fixed. The server upgrade broke it. :mad:

Share this post


Link to post
Share on other sites
16 hours ago, Malonn said:

This may be crude, saebel, as I have no knowledge of plugins, but can't you use native C++ code to delete keys/values?  Then use the OBSE api to talk between the plugin and esp?  Build your array and operate on it within the C++ plugin?  Like I said, I don't know shit about the OBSE api or plugins in general.

In order to do what you suggested, I would have to do the following:

  1. Copy the OBSE array into a C++ array.
  2. Use native C++ to find and remove the key/value pair.
  3. Copy the native C++ array into a new OBSE array.
  4. Replace the old OBSE array with the new one.

So here's the deal with how they set up the OBSE Array API.  They provided an example of how to do step 3.  But they did not provide an example of how to do step 1.  Considering how many tiers and subarrays we are talking about, I can't believe that coming up with a set of functions to constantly transpose each of the OBSE arrays into C++ and back again would be of any performance benefit.

I suppose it may be possible to keep all the arrays in C++ locally during the game session, and only do the conversion upon game load/save.  But the OBSE API has everything else already covered in terms of storing integers, floats, and strings.

I can already drill down to an OBSE key/value pair using their API.  I just want to be able to remove it.

I guess one other possible method would be to take the existing OBSE array that needs the key/value removed, and generate a duplicate of the array by rolling through and adding the key/values, then skip the key/value I want to remove.  But again, that just seems like extra-processing intensive.

Share this post


Link to post
Share on other sites
15 hours ago, darkrumbleking said:

Tried using Maps or Stringmaps (only difference is data type of the key; map=numeric keys)?

http://obse.silverlock.org/obse_command_doc.html#Array_Variables
Arrays according to the OBSE docs do not allow gaps. If you used ar_Erase on the element of an array, you would necessarily cause an index shift.
"An Array behaves like arrays in most programming languages: the key is an unsigned integer starting at zero, and there are no gaps between elements. (In other words, if an element exists at indexes 1 and 3 then an element necessarily exists at 0 and 2)"

The map type of array allows gaps, floating point keys, etc. I'm sure all the goodies you want are there without diving into the guts, so try constructing it like


(array_var) ar_Construct arrayType:map

I already have solved my problem using scripting and OBSE array functions.  But I would prefer to move all of that into the .dll plugin to improve performance when it comes to loading/save times.  With a small load order and a short adventure time, performance hit is negligible.  I'm concerned about heavily modded games with many hours of play time.  The arrays for those will be really large, and processing very time consuming (I imagine).

Share this post


Link to post
Share on other sites

To copy an mixed array like that into a C++ usable form might want an std::Variant. This is new stuff, and would take a bit of doing to set up. Is it possible to split the mixed array into smaller ones of different types? Posting some code might help as well. :)

Share this post


Link to post
Share on other sites

There is a lot of code to sort through.  But essentially it comes down to the following:
There is code in OBSE that allows you to erase elements.  But it requires knowing the array ID number.

UInt32 ArrayVarMap::EraseElements(ArrayID id, const ArrayKey& lo, const ArrayKey& hi)
{
	ArrayVar* var = Get(id);
	if (!var || lo.KeyType() != hi.KeyType() || lo.KeyType() != var->KeyType())
		return -1;

	// find first elem to erase
	std::map<ArrayKey, ArrayElement>::iterator iter = var->m_elements.begin();
	while (iter != var->m_elements.end() && iter->first < lo)
		++iter;

	UInt32 numErased = 0;

	// erase. if element is an arrayID, clean up that array
	while (iter != var->m_elements.end() && iter->first <= hi)
	{
		iter->second.Unset();
		iter = var->m_elements.erase(iter);
		numErased++;
	}

	// if array is packed we must shift elements down
	if (var->IsPacked())
		var->Pack();

	return numErased;
}


There is code that allows you to get the array ID number, but it requires getting the array as a passed argument via a script. 

// From Comands_Array.cpp

static bool Cmd_ar_Erase_Execute(COMMAND_ARGS)
{
	// returns num elems erased or -1 on error
	UInt32 numErased = -1;
	ExpressionEvaluator eval(PASS_COMMAND_ARGS);
	if (eval.ExtractArgs() && eval.Arg(0))
	{
		if (eval.Arg(0)->CanConvertTo(kTokenType_Array))
		{
			ArrayID arrID = eval.Arg(0)->GetArray();
			
			if (eval.Arg(1)) {
				// are we erasing a range or a single element?
				const Slice* slice = eval.Arg(1)->GetSlice();
				if (slice)
				{
					ArrayKey lo, hi;
					slice->GetArrayBounds(lo, hi);
					numErased = g_ArrayMap.EraseElements(arrID, lo, hi);
				}
				else
				{
					if (eval.Arg(1)->CanConvertTo(kTokenType_String))
					{
						ArrayKey toErase(eval.Arg(1)->GetString());
						numErased = g_ArrayMap.EraseElements(arrID, toErase, toErase);
					}
					else if (eval.Arg(1)->CanConvertTo(kTokenType_Number))
					{
						ArrayKey toErase(eval.Arg(1)->GetNumber());
						numErased = g_ArrayMap.EraseElements(arrID, toErase, toErase);
					}
				}
			}
			else {
				// second arg omitted - erase all elements of array
				numErased = g_ArrayMap.EraseAllElements(arrID);
			}
		}
	}

	if (numErased == -1) {
		*result = -1.0;
	}
	else {
		*result = numErased;
	}

	return true;
}

The "GetArray()" function that passes the Array ID is buried in the Script Token section of OBSE.  For example:

// From ScriptTokens.cpp

#if OBLIVION
ArrayID ScriptToken::GetArray() const
{
	if (type == kTokenType_Array)
		return value.arrID;
	else if (type == kTokenType_ArrayVar)
		return value.var->data;
	else
		return 0;
}

// snip

ArrayID ArrayElementToken::GetArray() const
{
	ArrayID out = 0;
	g_ArrayMap.GetElementArray(GetOwningArrayID(), key, &out);
	return out;
}

Within the OBSE plugin API framework, I haven't found any internal function for getting the array ID outside of the scripted method of passing an argument.  However, I *can* get the array ID via a script, using the following code:

scn sdrGetArrayID

array_var aArray
string_var sArrayID
short iArrayID

Begin Function {aArray}
	; calculates and saves the Array ID
	Let sArrayID := sv_Construct "%g" aArray
	Let iArrayID := ToNumber sArrayID
	sv_Destruct sArrayID
	SetFunctionValue iArrayID
End

What my mod does is:
- On game load, either retrieve or create the master arrays that store all the data.
- Using my scripted function, it then determines the array ID of the master IDs and stores them in custom game setting variables
- Those game setting variables can then be accessed by the sdr.dll to grab the master arrays and process them.

*However*, when the subarrays are created and stored inside each parent array, all of that is handled from within the sdr.dll functions.  And because there are no script arguments involved in the lower tiers, I can't figure out a way to extract the arrID.

I have tried code like the following:

ArrayID GetArrayID(OBSEArray* arr) {
	ArrayID out = 0;
	g_ArrayMap.GetElementArray(GetOwningArrayID(), arr, &out);
	return out;
}

But none of my experiments have worked.

:(

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×