Introduction
Using a native C++ library from C# is never straightforward but ones
involving Quantative Finance present their own unique set of challenges
in part due to the mathematical focus of those that work in this field - aka
the "Quants".
This article uses some advanced techniques
(in the C# and C++ languages, in the Windows O/S and even in Visual Studio
itself) to provide a high performance, concurrent, typesafe C# API:
- Presents a Variant-centric
design for a typical native, non-threadsafe, C-like Quant
library including analytic and numeric pricing of basic ("vanilla") options
(via Financial Numerical Recipes);
- A single C++ Translator DLL to wrap this library in a generic way and to
translate exceptions;
- And a very typesafe, Quant-centric, C# API allowing for
concurrency.
Some of the techniques and technologies demonstrated include:
- Generated API via XML markup using inbuilt Visual Studio support for
XSDs and T4 templates;
- Support for skipping or defaulting arguments, mapping "magic" strings to
enumerations and handling C++ unions in C#;
- Use of generic disposable structs to eliminate all boxing
and minimise heap (GC) use;
- Direct run time generation of P-Invoke functors including manual DLL loading and
decoding;
- Native Variant memory handling in C#
including typesafe reuse between native calls;
- High performant C# Variant wrapper using unsafe C# code
for direct BSTR and SAFEARRAY en-/de-coding;
- Use of C++ macros, variadic templates and runtime walking of the Export Address Table
to create the single native Translator DLL;
- Advanced use of Windows Side by Side
manifests to add concurrency to a non-threadsafe native
library.
Motivation
This section shows
a simple example followed by real Quant library-style calls:
The First Example
Given a C++ signature for Function4 within the NativeAPI.DLL like this:
extern "C" VARIANT __declspec(dllexport) __stdcall
Function4(VARIANT* _index, VARIANT* _arg1, VARIANT* _arg2, VARIANT* _arg3)
This is often deliberately done to provide an intuitive VBA declaration like
this:
Declare Function Function4 Lib "NativeAPI" Alias "_Function4@16" _
(Index, Arg1, Optional Arg2, Optional Arg3)
Unfortunately the obvious P-Invoke signature is not supported (as of .NET
v4.5):
[DllImport("NativeAPI.DLL")]
extern static object Function4(
ref object index, ref object first,
ref object second, ref object third);
So the ugly mangled name must be used with all arguments shifted to expose
the hidden return argument that is part of the stdcall calling convention being
used:
[DllImport("NativeAPI.DLL", EntryPoint = "_Function4@16")]
extern static void Function4(
out object result,
ref object index, ref object first,
ref object second, ref object third);
The use of objects by reference makes everything non-typesafe, inefficient
and painful because .NET will marshal structs, interfaces and classes into their
COM equivalents causing runtime errors that are hard to troubleshoot:
try
{
object index = 1;
object first = "first";
object second = null;
object third = "third";
object res;
Function4(out res, ref index, ref first, ref second, ref third);
The bigger problem is that any C++ exceptions are swallowed by .NET and
replaced with the same generic message, losing the essential error information
provided by the Quant library:
index = null;
Function4_Works(out res, ref index, ref first, ref second, ref third);
}
catch (Exception ex)
{
var msg = ex.Message;
}
The First Solution
The function is marked up using Visual Studio's inbuilt support for XML
editing constrained to an XSD:
<function id="Function4" type="?Any">
<arg id="Indexer" type="Integer"/>
<arg id="Choice1" type="?Any"/>
<arg id="Choice2" type="?Any"/>
<arg id="Choice3" type="?Any"/>
</function>
Then the function can be efficiently called and the result retrieved with the
native C++ exception being optionally rethrown as a C# one with full fidelity:
using (var fn = NativeFunctions.Function4.New())
{
fn.Indexer.Set(1);
fn.Choice1.Set(1.23);
fn.Choice2.Set("String");
NativeFunctions.Result choice = fn.Invoke();
The debugger view at this point is instructive, accurately showing the
native Variant type and value as interpreted by .NET itself:
The code continues, showing efficient re-calling and exception translation:
if (choice.Native.IsDouble)
{
double dbl = choice.GetDouble();
}
fn.Indexer.Set(3);
choice = fn.Invoke();
if (choice.Native.IsEmpty)
{
}
fn.Indexer.Set(-1);
try
{
choice = fn.Invoke();
}
catch (NativeFunctionException ex)
{
var msg = ex.Message;
}
}
The Complex Example
Given C++ signatures for functions within the NativeAPI.DLL as follows:
#define API(FUNCTION) VARIANT __declspec(dllexport) __stdcall FUNCTION
API(CreateList)(VARIANT* _name, VARIANT* _list);
API(FunctionComplex)(VARIANT* _payOff, VARIANT* _frq, VARIANT* _overrides,
VARIANT* _endDates, VARIANT* _roll, VARIANT* _ccy, VARIANT* _curve,
VARIANT* _options, VARIANT* _schedule, VARIANT* _barrier);
Then the ugly mangled name must be used with all arguments shifted as before:
[DllImport("NativeAPI.DLL", EntryPoint = "?CreateList@Native@@YG?AUtagVARIANT@@PAU2@0@Z")]
extern static void CreateList(out object result, ref object name, ref object values);
[DllImport("NativeAPI.DLL", EntryPoint = "?FunctionComplex@Native@@YG?AUtagVARIANT@@PAU2@000000000@Z")]
extern static void FunctionComplex(out object result, ref object payoff, ref object frq, ref object overrides,
ref object endDates, ref object roll, ref object ccy, ref object curve, ref object options,
ref object schedule, ref object barrier);
Firstly the overrides must be created as a named list within the Quant
library:
object dummy;
object name = "Overrides";
object values = new object[,]
{
{"Amount", "PayDelay"},
{0.9, 0.5},
{2, 2}
};
CreateList(out dummy, ref name, ref values);
Assume the schedule is stored in a typesafe structure somewhere:
var schedule = new KeyStruct<DateTime, DateTime, decimal>[]
{
KeyStruct.Create(DateTime.Now, DateTime.Now.AddMonths(1), 1.0m),
KeyStruct.Create(DateTime.Now.AddMonths(2), DateTime.Now.AddMonths(3), 1.1m)
};
The call setup is now quite tricky and requires intimate knowledge of the
array shapes and magic strings as defined by different Quants working on the
library often at different times:
object result;
object payoff = "D";
object frq = "FourWeekly";
object overrides = "!" + name;
object endDates = new object[,] { { DateTime.Now.Date, DateTime.Now.AddMonths(1).Date } };
object roll = 7;
object ccy = "AUD";
object curve = 0.1;
object options = new object[,] { { "Extrapolate", true } };
object schedule2 = new object[,] { { DateTime.Now.Date, DateTime.Now.AddMonths(1).Date } };
var _barrier = new object[schedule.Length, 3 + 1];
for (int i=0; i<schedule.Length; i++)
{
_barrier[i, 0] = schedule[i].Key.Date;
_barrier[i, 1] = schedule[i].Value1.Date;
_barrier[i, 2] = (double)schedule[i].Value2;
}
object barrier = _barrier;
FunctionComplex(out result, ref payoff, ref frq, ref overrides, ref endDates,
ref roll, ref ccy, ref curve, ref options, ref schedule2, ref barrier);
Processing the result is equally painful especially if the type is not fixed:
double res2 = (double)result;
The Complex Solution
The functions are again marked up but with a richer syntax:
<create id="CreateList" type="List">
<name/>
<arg id="Data" type="Any" isArray="2d"/>
</create>
<function id="FunctionComplex" type="Double">
<enum id="Payoff" type="QuantBarrierType"/>
<argT id="PaymentFrequency" type="?EnumOrString" T="QuantFrequency"/>
<argT id="Overrides" type="?NamedList" T="QuantOverrides"/>
<arg id="EndDates" type="?Date" isArray="1d"/>
<arg id="Roll" type="?StringOrNumber"/>
<arg id="SettlementCcy" type="?String"/>
<arg id="SettlementCurve" type="?Curve"/>
<argT id="Options" type="?Map" T="QuantCurveOptions"/>
<argT id="Schedule" type="Matrix" T="QuantAsianAveragingSchedule"/>
<argT id="LowBarrier" type="?Matrix" T="QuantContinuousBarrierSchedule,
QuantDiscreteBarrierSchedule"/>
</function>
Again the overrides are created first but this time using enumerations and
without any boxing:
var overrides = QuantNamedList<QuantOverrides>.New("Overrides");
using (var bldr = overrides.NewBuilder(2, 2))
{
bldr.Add(QuantOverrides.Amount).SetDoubleColumn(new[] { 0.9, 0.5 });
bldr.Add(QuantOverrides.PaymentLag).SetIntegerColumn(new[] { 2, 2 });
bldr.Invoke();
}
Assuming the same typesafe structure for the schedule exists, then the setup
is far easier and more efficient than before. Observe how an array of structures
can be set directly without any boxing or (unbounded) heap usage:
using (var fn = NativeFunctions.FunctionComplex.New())
{
fn.EndDates.Set(new[] { DateTime.Now, DateTime.Now.AddMonths(1) });
var bldBarrier = fn.LowBarrier.SetWithBuilder1(2);
bldBarrier.StartDates.Set(schedule, t => t.Key);
bldBarrier.EndDates.Set(schedule, t => t.Value1);
bldBarrier.Levels.Set(schedule, t => (double)t.Value2);
Support for typed matrices via IDisposable structs and custom mapping of enumerations to strings:
using (var bldOptions = fn.Options.SetWithBuilder(1))
{
bldOptions.Add(QuantCurveOptions.Extrapolate, true);
}
fn.Payoff.Set(QuantBarrierType.Discrete);
Full typesafety for the string names used including in unions with numbers:
fn.Overrides.Set(overrides);
fn.SettlementCurve.Set(0.1);
More unions including those mixing enumerations, strings and numbers:
fn.PaymentFrequency.Set(QuantFrequency.FourWeekly);
fn.Roll.Set(7);
Choice between simpler or more efficient but complex approaches:
var bldAsian = fn.Schedule.SetWithBuilder(2);
bldAsian.Dates.SetAt(0, DateTime.Now);
bldAsian.Dates.SetAt(1, DateTime.Now.AddMonths(1));
Choice of exception re-throwing or manual error handling and full typesafety
of the result without any unnecessary heap usage (no use of 'object' even for
nested arrays):
fn.SettlementCcy.Set("AUD");
double res = fn.Invoke();
}
The debugger view at this point is again instructive, accurately showing the
native Variant type and value as interpreted by .NET itself. Most importantly,
you see the actual values passed to the function, such as "D" for the Payoff
even though an enumeration was used to set it:
Classic Option Pricing with simple Volatility
Start with a local scope to show how to reuse a VARIANT value across function
calls: in this case, the name of the fixings/observations for the underlying:
using (var scope = NativeFunctions.Scope())
{
var fixings = VariantFixings.New(scope);
Now load the fixings/observations into the Quant libary, assign an arbitrary
name, and enable reuse of the BSTR. Note how the library is
expecting a typed matrix: the first column containing the dates and the last
column the corresponding values:
using (var fn = NativeFunctions.CreateFixings.New())
{
fn.AsOf.Set(DateTime.Now);
fn.QuantName.Set("USDFixings");
var builder = fn.DatesAndValues.SetWithBuilder(3);
builder.Dates.Set(new[] { DateTime.Now, DateTime.Now.AddMonths(1), DateTime.Now.AddMonths(3) });
builder.Numbers.Set(new[] { 1.1, 1.5, 1.9 });
fn.Invoke();
fn.QuantName.Save(fixings);
}
First price the option three months before expiry with flat risk free rate
and volatility. Also calculate one risk measure (delta):
double pvAnalytic;
using (var fn = NativeFunctions.ModelOptionBlackScholes.New(QuantMeasure.PV, QuantMeasure.Delta))
{
fn.AsOf.Set(DateTime.Now);
fn.ExpiryDate.Set(DateTime.Now.AddMonths(3));
fn.Payoff.Set(QuantCallOrPut.Call);
fn.Rate.Set(0.15);
fn.Spot.Set(1.1);
fn.Strike.Set(1.0);
fn.Vol.Set(0.20);
var res = fn.Invoke();
The result is a vector of VARIANTs, one for each measure. By knowing the
measure types this can be efficiently decoded without any boxing:
pvAnalytic = res.GetResult().GetDouble();
var delta = res.GetResult().GetDouble();
It is now straightforward to re-price the option at expiry, at which
point the fixings/observations must be supplied. This is done below in a fully
type safe manner that efficiently reuses the previously allocated BSTR for the
name:
fn.AsOf.Set(DateTime.Now.AddMonths(3));
fn.Fixings.SetFromShared(fixings);
res = fn.Invoke();
Results are again easy and efficient to access including handling of the empty case:
var pvExpired = res.GetResult().GetDouble();
var bIsEmpty = res.IsEmpty;
delta = res.GetResult().GetDouble();
}
}
Numerical Option Pricing with complex Volatility
A complex volatility process is represented here by an unnamed (or named) map
containing a named one, something
like this:
"VolProcess" -> {
{ "VolatilityProcess", 0.20 },
};
{
{"Vol", "!VolProcess" },
{ "VolType", "Flat"}
}
This tricky layout (including the named map) can be created as follows:
var vp = QuantVolProcess<QuantVolOptions>.New("VolProcess", 0.20);
vp.SetNoOptions();
The pricing then proceeds as before:
double pvNumerical;
using (var fn = NativeFunctions.ModelOptionFD.New(QuantMeasure.PV, QuantMeasure.Delta))
{
fn.AsOf.Set(DateTime.Now);
fn.ExpiryDate.Set(DateTime.Now.AddMonths(3));
fn.Payoff.Set(QuantCallOrPut.Call);
fn.Rate.Set(0.15);
fn.Spot.Set(1.1);
fn.Strike.Set(1.0);
fn.VolProcess.Set(vp);
var res = fn.Invoke();
pvNumerical = res.GetResult().GetDouble();
var bIsEmpty = res.IsEmpty;
var delta = res.GetResult().GetDouble();
Options to control the numeric algorithm can now be set before recalculating:
var options = fn.Options.SetWithBuilder(2);
options.Add(QuantFDModelOptions.SpotSteps, 100);
options.Add(QuantFDModelOptions.TimeSteps, 100);
options.ThrowIfIncomplete();
res = fn.Invoke();
pvNumerical = res.GetResult().GetDouble();
}
var pvDiff = pvNumerical - pvAnalytic;
Solution Architecture
The example C++ Quant library design is divided up into a main NativeAPI.DLL
and a dependent Utils.DLL. It is deliberately not thread safe in any way:
specifically, static variables are used so only one instance of these DLLs is
expected to be loaded into a single process and all
calls to the library must occur from a single thread.
This kind of setup is common in Quant libraries that were primarily written
for Excel 2003 and earlier; i.e. before Microsoft introduced concurrency support
for XLLs. It is also still common if the library was primarily for use with VBA
which does not support multiple threads.
By introducing a single C++ Translator.DLL, it is possible to automatically
enumerate the public exports of the NativeAPI.DLL and decode the number of
arguments at runtime. This then allows for the right wrapper function to be
chained around the original to support API argument count changes between
versions and exception translation.
Concurrency can then be achieved by introducing a Windows SxS manifest file
and duplicating this along with the binaries, once for each thread desired.
Managed Topics
Discussion of the pure C# oriented design and implementation (the advanced
interop is in the following section).
XSD Constrained XML Editing
Visual Studio has a good XML editor that provides suggestions, constraints
and error checking in the presence of an XSD. It's XSD editor, however, really
works best in pure XML mode. The first sections of the NativeFunctions.XSD
follow:
="1.0"="utf-8"
<xs:schema id="NativeFunctions"
targetNamespace="http://www.tovica.com/schemas/cp3/nativefunctions/1"
elementFormDefault="qualified"
xmlns="http://www.tovica.com/schemas/cp3/nativefunctions/1"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
>
<xs:simpleType name="PascalCaseString">
<xs:restriction base="xs:string">
<xs:pattern value="[A-Z][a-zA-Z0-9]*"/>
</xs:restriction>
</xs:simpleType>
...
The supported data types are:
<xs:simpleType name="Types">
<xs:restriction base="xs:string">
<xs:enumeration value="Double"/>
<xs:enumeration value="Date"/>
<xs:enumeration value="DateTime"/>
<xs:enumeration value="String"/>
<xs:enumeration value="Integer"/>
<xs:enumeration value="Boolean"/>
<xs:enumeration value="Any"/>
<a name="900"></a> <xs:enumeration value="Curve"/>
<xs:enumeration value="Fixings"/>
<xs:enumeration value="Surface"/>
<xs:enumeration value="StringOrNumber"/>
<xs:enumeration value="?Double"/>
<xs:enumeration value="?Date"/>
<xs:enumeration value="?DateTime"/>
<xs:enumeration value="?String"/>
<xs:enumeration value="?Integer"/>
<xs:enumeration value="?Boolean"/>
<xs:enumeration value="?Any"/>
<xs:enumeration value="?Curve"/>
<xs:enumeration value="?Fixings"/>
<xs:enumeration value="?StringOrNumber"/>
</xs:restriction>
</xs:simpleType>
Generics are also supported, normally for enumerations or sharing/reusing VARIANTs:
<xs:simpleType name="Generics">
<xs:restriction base="xs:string">
<xs:enumeration value="NamedMap"/>
<xs:enumeration value="NamedList"/>
<xs:enumeration value="Map"/>
<xs:enumeration value="List"/>
<xs:enumeration value="Matrix"/>
<xs:enumeration value="EnumOrString"/>
<xs:enumeration value="EnumOrNumber"/>
<xs:enumeration value="EnumBool"/>
<xs:enumeration value="?NamedMap"/>
<xs:enumeration value="?NamedList"/>
<xs:enumeration value="?Map"/>
<xs:enumeration value="?List"/>
<xs:enumeration value="?Matrix"/>
<xs:enumeration value="?EnumOrString"/>
<xs:enumeration value="?EnumOrNumber"/>
</xs:restriction>
</xs:simpleType>
The root element shows the overall design of the XML:
<xs:element name="root">
<xs:complexType>
<xs:choice maxOccurs="unbounded">
<xs:element name="templated" type="Templated"/>
<xs:element name="function" type="Function"/>
<xs:element name="create">
<xs:complexType>
<xs:sequence>
<xs:element name="measures" type="Measures" minOccurs="0"/>
<xs:element name="name"/>
<xs:group ref="ArgumentsGroup" maxOccurs="unbounded"/>
</xs:sequence>
<xs:attribute name="id" type="PascalCaseString" use="required"/>
<xs:attribute name="type" type="NameTypes" use="required"/>
</xs:complexType>
</xs:element>
</xs:choice>
</xs:complexType>
</xs:element>
The corresponding NativeFunctions.XML just needs to have the right header in
order for Visual Studio to pick up the XSD:
="1.0"="utf-8"
<root xmlns="http://www.tovica.com/schemas/cp3/nativefunctions/1"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.tovica.com/schemas/cp3/nativefunctions/1 NativeFunctions.xsd">
<function id="Function3" type="String">
<arg id="FirstArg" type="String"/>
<arg id="SecondArg" type="?String"/>
<arg id="ThirdArg" type="?String"/>
</function>
...
The following example shows how arguments can be skipped or default values assigned:
<function id="FunctionWithSkip" type="Date">
<arg id="AsOf" type="Date"/>
<arg id="Days" type="Integer">0</arg>
<arg id="Months" type="Integer">0</arg>
<arg id="Years" type="Integer">0</arg>
<skip id="BusDayConv"/>
<skip id="Calendars"/>
</function>
And these final ones show the more advanced abilities to add the type
safety to the Quant library, including how Measures are handled for the model
calls:
<create id="CreateMap" type="Map">
<name/>
<arg id="Data" type="Any" isArray="2d"/>
</create>
<create id="CreateFixings" type="Fixings">
<name/>
<arg id="AsOf" type="Date"/>
<argT id="DatesAndValues" type="Matrix" T="QuantDatesAndNumbers"/>
</create>
<templated id="ModelOptionBlackScholes">
<measures useLegacyMethod="yes"/>
<arg id="AsOf" type="Date"/>
<bool id="Payoff" type="QuantCallOrPut"/>
<arg id="Spot" type="Double"/>
<arg id="Strike" type="Double"/>
<arg id="ExpiryDate" type="Date"/>
<arg id="Rate" type="Curve"/>
<arg id="Vol" type="Surface"/>
<arg id="Fixings" type="?Fixings"/>
</templated>
<templated id="ModelOptionFD">
<measures/>
<arg id="AsOf" type="Date"/>
<bool id="Payoff" type="QuantCallOrPut"/>
<arg id="Spot" type="Double"/>
<arg id="Strike" type="Double"/>
<arg id="ExpiryDate" type="Date"/>
<arg id="Rate" type="Curve"/>
<process type="QuantVolOptions"/>
<arg id="Fixings" type="?Fixings"/>
<argT id="Options" type="?Map" T="QuantFDModelOptions"/>
</templated>
Code Generation using T4
Next to the NativeFunctions.XML and XSD files the project contains a TT file.
This is an ASP.NET-like language that Visual Studio uses to allow code
generation. Editing of the TT file (or choosing "Run Custom Tool" off the
right-click menu on it) causes Visual Studio to compile and run the code. The
top of the file is pretty standard:
<#@ template debug="true" hostspecific="true" language="C#" #>
<#@ output extension=".cs" #>
<#@ assembly name="System.Xml" #>
<#@ assembly name="System.Core" #>
<#@ assembly name="System.Xml.Linq" #>
<#@ import namespace="System.Collections.Generic" #>
<#@ import namespace="System.Linq" #>
<#@ import namespace="System.Xml.Linq" #>
<#@ import namespace="System.IO" #>
<#@ import namespace="System" #>
<#
Dictionary<string, Wrapper> wrappers;
var count = Initialise(out wrappers);
this.WriteLine("//Generated {0:N0} function wrappers", count);
#>
using ManagedAPI.Functions.Arguments;
...
The output file will be NativeFunctions.CS. The Initialise() call is
responsible for parsing the NativeFunctions.XML file into a reasonable data
structure, "wrappers". This is then repeatedly looped over to create the code:
namespace ManagedAPI.Functions
{
partial class NativeFunctions
{
public enum Function
{
None = 0,
<#
this.PushIndent(new string(' ', 4 * 3));
foreach (var p in wrappers)
{
this.WriteLine("{0},", p.Key);
}
this.PopIndent();
#>
}
See the NativeFunctions.TT file for all the gory details.
Function Arguments
Generation is also used here (Arguments.TT) to avoid the large amount of repetitive tedious
code required to support all the argument types in their basic scalar, vector
and shared modes. Some code reduction was achieved by reusing these structs for
the function result as well. For example, the generated bit of a basic required
string argument is:
namespace Arguments
{
[DebuggerDisplay("{Native.Type}: {Native.Boxed}")]
public partial struct RequiredStringArg
{
readonly NativeArguments Buffer;
public Variant Native { get { return Buffer[Index]; } }
public readonly string Name;
public readonly int Index;
static void ThrowIfEmpty(Variant v, string name)
{
if (v.IsEmpty) throw new NativeMissingValueException(name);
}
internal void ThrowIfEmpty()
{
ThrowIfEmpty(Native, Name);
}
internal static String Get(string name, IntPtr r)
{
var v = new Variant(r);
ThrowIfEmpty(v, name);
return v.GetString(name);
}
internal void ThrowIfBadType()
{
ThrowIfEmpty();
if (!Native.IsString) throw new NativeTypeMismatchException(Name);
}
internal RequiredStringArg(string name, NativeArguments buffer, int idx)
{
Name = name;
Buffer = buffer;
Index = idx;
}
public void Set(String value)
{
Native.SetString(value);
}
public void SetFromShared<SHARED>(SHARED sharedVariant)
where SHARED : Interfaces.IVariantString
{
Buffer.UseVariantAgain(sharedVariant.Native, Index);
}
public void Save(Variants.VariantString sharedVariant)
{
Buffer.StoreArgumentForReuse(sharedVariant.Native, Index);
}
public void SetFromResult<R>(R v)
where R : NativeFunctions.IResult
{
Buffer.UseVariantAgain(v.Native, Index);
}
}
}
Leaving only the following to be typed in manually:
partial struct RequiredStringArg
{
public void Set(StringBuilder value)
{
Native.SetString(value);
}
}
Additional functionality around buffer handling including scoped buffers and
shared native arguments or results of different scopes can be found in:
- NativeBuffer - the native VARIANT array for arguments and the result;
- NativeArguments - passing the NativeBuffer to the function and dealing
with argument scope;
- TypedVariantCache - global storage of native arguments for reuse;
- Builders and NamedBuilders - helper structs for efficiently setting
typed arrays or matrices including Maps and Lists;
- QuantVolProcess, QuantFixings, QuantEnumOrNumber, etc - many helper
structs for setting arguments and holding typed results and unions.
Functions as Disposable structs
The general pattern for the functions themselves is to generate them totally
from the NativeFunctions.XML file markup as structures that implement
IDisposable - i.e. are normally used exclusively within 'using' statements.
The first bits generated are the constants for convenience and the essential field
for NativeArguments which itself has the link back to the NativeInstance via
Thread Local Storage:
public partial struct FunctionWithSkip : IDisposable
{
public const NativeFunctions.Function Type = NativeFunctions.Function.FunctionWithSkip;
public readonly static string Name = "FunctionWithSkip";
public readonly static string Result = "FunctionWithSkip.Result";
public NativeArguments Native;
Each argument follows as a field. Note that any field can be skipped or
defaulted, not just the ones at the end of the call:
const int HighestArgCount = 4;
public RequiredDateArg AsOf;
public RequiredIntegerArg Days;
public RequiredIntegerArg Months;
public RequiredIntegerArg Years;
This approach relies on 'New()' to create the structure and in there the
argument slot number is explicitly passed to each argument, allowing for any of
them to be skipped or even the order changed without altering any calling code.
Note also that a different buffer can be provided for advanced usage such as one
buffer per model, or nested calls:
public static FunctionWithSkip New(NativeArguments buffer = null)
{
if (buffer == null) buffer = NativeInstance.Current.Functions.DefaultBuffer;
var res = new FunctionWithSkip
{
Native = buffer,
AsOf = new RequiredDateArg("FunctionWithSkip.AsOf", buffer, 0),
Days = new RequiredIntegerArg("FunctionWithSkip.Days", buffer, 1),
Months = new RequiredIntegerArg("FunctionWithSkip.Months", buffer, 2),
Years = new RequiredIntegerArg("FunctionWithSkip.Years", buffer, 3),
};
res.SetDefaults();
return res;
}
Basic argument type checks and scalar default settings follow with extension
points via partial functions:
[Conditional("DEBUG")]
void VerifyArguments()
{
AsOf.ThrowIfBadType();
Days.ThrowIfBadType();
Months.ThrowIfBadType();
Years.ThrowIfBadType();
CustomVerifyArguments();
}
partial void CustomVerifyArguments();
void SetDefaults()
{
Days.Set(0);
Months.Set(0);
Years.Set(0);
CustomSetDefaults();
}
partial void CustomSetDefaults();
Since DLL exports are being used via a generic Translator the Quant library
native binaries can be changed without recompiling (or re-releasing) the managed
application that uses them. This means it is possible that not only may function
argument counts change but the availability of functions may also. The following
allows for programmatical checking so that the managed application can degrade
functionality cleanly if it is not available:
public bool IsSupported { get { return NativeInstance.Current.Functions.IsSupported(Type); } }
public void ThrowIfNotSupported()
{
NativeInstance.Current.Functions.ThrowIfNotSupported(Type);
}
Once all the arguments have been supplied the function can be invoked, any
exceptions propagated and the result type-checked all in one go. Note how the
argument structure is re-used to perform this latter check:
public Date Invoke()
{
VerifyArguments();
var raw = Native.Call(Type);
return RequiredDateArg.Get(Result, raw);
}
Alternatively, any exception can be avoided by splitting the call up. Between
or after the second call, methods are available off NativeArguments to access
the raw result directly, the raw error code or its decoded value:
public bool InvokeBegin()
{
VerifyArguments();
var code = Native.CallRaw(Type);
return code == 0;
}
public Date InvokeEnd()
{
var raw = Native.RawResult;
return RequiredDateArg.Get(Result, raw);
}
The lifetime of the native result can be transferred to a different scope -
i.e. this stops the 'using' block from freeing the result along with the
arguments:
public void SaveResults(ref VariantResult shared)
{
Native.StoreResultForReuse(ref shared, Result);
}
Finally the end of the function including the freeing of the native values
(but not the NativeBuffer itself) of the arguments and result is generated:
public void ResetToDefaults()
{
Free();
SetDefaults();
}
internal void Free()
{
Native.FreeForReuse(HighestArgCount);
}
void IDisposable.Dispose()
{
Free();
}
}
Additional options for accessing the results are required for when doing so
can be expensive. For example, the following is provided for a vector of
strings:
public String[] Invoke()
{
var a = new Adaptors.ArrayOutput<String>();
Invoke(ref a);
return a;
}
public int Invoke<T>(ref T output) where T : Variant.IOutput<String>
{
VerifyArguments();
var raw = Native.Call(Type);
return RequiredStringVectorArg.Get(Result, raw, ref output);
}
public Results InvokeAsVariant()
{
VerifyArguments();
var raw = Native.Call(Type);
return RequiredStringVectorArg.Get(Result, raw);
}
Finally functions that accept Measure lists also have additional support such as
multiple ways to set the Measures themselves. This trades efficiency for
convenience:
public static ModelOptionFD New(ICollection<QuantMeasure> measures)
{
var buffer = NativeInstance.Current.Functions.DefaultBuffer;
buffer.Functions.SetMeasures(buffer[0], measures, measures.Count);
return New(buffer);
}
public void SetMeasures(IEnumerable<QuantMeasure> measures, int count)
{
Native.Functions.SetMeasures(Measures.Native, measures, count);
}
Typed Matrices
The reliance on untyped arrays is particularly painful especially when the
required shape and content change depending on some other argument. For example,
FunctionComplex() has an argument that is marked up as:
<argT id="LowBarrier" type="?Matrix" T="QuantContinuousBarrierSchedule,
QuantDiscreteBarrierSchedule"/>
This generates the following union of those two types:
public OptionalMatrixArg<QuantContinuousBarrierSchedule,QuantDiscreteBarrierSchedule> LowBarrier;
Taken from earlier, the following shows how a QuantContinousBarrierSchedule can be set
using the first builder:
var bldBarrier = fn.LowBarrier.SetWithBuilder1(2);
bldBarrier.StartDates.Set(schedule, t => t.Key);
The argument is manually coded as a generic accepting the list of types for
the union, in this case, two:
public struct OptionalMatrixArg<T, U>
where T : struct, IMatrixBuilder<T>
where U : struct, IMatrixBuilder<U>
{
readonly NativeArguments Buffer;
public Variant Native { get { return Buffer[Index]; } }
public readonly string Name;
public readonly int Index;
....
Each builder can then be exposed by deferring to the appropriate generic type's implemented interface:
public T SetWithBuilder1(int rows, int cols = -1)
{
var m = new T();
return m.New(Native, rows, cols);
}
Exposing something like StartDates above involves making a series of
argument-like structures that operate instead of pre-allocated mixed matrices,
column-by-column as follows:
public partial struct RequiredDateColumn
{
readonly Variant.Vector Native;
public void Set(IList<Date> value)
{
Native.SetDateColumn(value);
}
public void SetAt(int idx, Date value)
{
Native[idx].SetDate(value);
}
....
}
Finally the QuantContinuousBarrierSchedule can be manually coded using the
above. Note how in this case the Quant library accepts zero rebates but not
empty pay dates, making things more complex:
public struct QuantContinuousBarrierSchedule : IMatrixBuilder<QuantContinuousBarrierSchedule>
{
public static readonly string Name = "ContinuousSchedule";
public const int MaxColumns = 5;
public readonly RequiredDateColumn StartDates, EndDates;
public readonly RequiredDoubleColumn Levels;
public readonly OptionalDoubleColumn Rebates;
public readonly OptionalDateColumn PayDates;
public bool HasPayDates { get { return Columns == MaxColumns; } }
public readonly int Columns;
internal QuantContinuousBarrierSchedule(Variant v, int rows, int cols = -1)
: this()
{
Columns = cols <= 4 ? 4 : MaxColumns;
var vector = v.SetNewMatrix(Columns, rows);
StartDates = new RequiredDateColumn(vector); vector++;
EndDates = new RequiredDateColumn(vector); vector++;
Levels = new RequiredDoubleColumn(vector); vector++;
Rebates = new OptionalDoubleColumn(vector); vector++;
if (Columns == MaxColumns) PayDates = new OptionalDateColumn(vector); vector++;
}
The QuantDiscreteBarrierSchedule has a different number of columns and types.
For more schedules, see QuantAsianAveragingSchedule and
QuantFaderBarrierSchedule.
Mapping with Enumerations
Enumerations with attributes is a one way to declaratively support the
replacement of "magic" strings by clearer enumerations. For example, the
following would ensure that "C" was passed to the Quant library if the user
specified Continuous for the argument:
public enum QuantBarrierType
{
[EnumName("C")]
Continuous,
[EnumName("D")]
Discrete,
}
Similarly, the replacement of annoying booleans with clear two-value
enumerations makes for a much better API. For example, the following would pass
FALSE to the Quant library if the user specified Put for the argument or TRUE if
Call:
public enum QuantCallOrPut
{
Put = 0,
Call,
}
The Quant library may support many alternatives and those values may need to
be normalised to a single one when writing XML compliant with an XSD for
example. In the following all the alternatives map to the primary name (the
first one):
public enum QuantFrequency
{
[EnumName("Daily", "1D")]
Daily,
[EnumName("Weekly", "1W")]
Weekly,
[EnumName("TwoWeekly", "BiWeekly", "2W")]
TwoWeekly,
[EnumName("FourWeekly", "28D", "4W")]
FourWeekly,
[EnumName("Monthly", "1M")]
Monthly,
[EnumName("Quarterly", "3M")]
Quarterly,
[EnumName("SemiAnnual", "Semi", "6M")]
SemiAnnual,
[EnumName("Annual", "1Y", "12M")]
Annual,
Term,
}
Finally, some of the names chosen by the Quant library developers may be just
too obscure or ambiguous:
public enum QuantMeasure
{
AccrualEndDates = 0,
AccrualStartDates,
[EnumName("FullDiscountRho")]
BucketedDiscountRho,
Cashflows,
Coupons,
Dates,
[EnumName("DCFs")]
DayCountFractions,
Delta,
[EnumName("DFs")]
DiscountFactors,
....
Additionally the well known problem with converting enumerations to strings
comes into play here - i.e. repeated calls to ToString() on an enum are not only
slow but result in a duplicate (non-interned) string and allow for numbers to be
specified.
The following shows the first part of the implementation of a class that
resolves these issues reasonably efficiently:
[DebuggerStepThrough]
public static class Enum<T> where T : struct, IConvertible
{
#region Accessors
public readonly static string Name;
public readonly static string[] SortedNames;
public readonly static T[] SortedValues;
public static int Count { get { return SortedNames.Length; } }
public readonly static SortedList<string, string> Alternatives;
public static T GetValueByIndex(int idx)
{
return SortedValues[_name2value[idx]];
}
public static string GetNameByIndex(int idx)
{
return SortedNames[_value2name[idx]];
}
#endregion
There are many methods available including extension methods - see
EnumConverter.cs for details. The initalisation is done via the static
constructor as shown:
#region One time initialisation
readonly static int[] _value2name, _name2value;
static Enum()
{
var type = typeof(T);
if (!type.IsEnum) throw new InvalidOperationException("Type is not an enum: " + type.FullName);
Name = type.Name;
SortedValues = (T[])Enum.GetValues(type);
var fields = typeof(T).GetFields(BindingFlags.Public | BindingFlags.Static);
SortedNames = new string[fields.Length];
for (int i = 0; i < fields.Length; i++)
{
var field = fields[i];
var name = field.Name;
var attributes = (EnumNameAttribute[])field.GetCustomAttributes(typeof(EnumNameAttribute), false);
if (attributes.Length == 1)
{
name = attributes[0].Name;
var alternatives = attributes[0].Alternatives;
if (alternatives != null)
{
Alternatives = Alternatives ?? new SortedList<string, string>(fields.Length, StringComparer.Ordinal);
foreach (var a in alternatives) Alternatives.Add(a, name);
}
}
SortedNames[i] = name;
}
Array.Sort(SortedNames, SortedValues, StringComparer.Ordinal);
_value2name = new int[SortedNames.Length];
for (int i = 0; i < _value2name.Length; i++) _value2name[i] = i;
Array.Sort(SortedValues, _value2name, Comparer);
_name2value = new int[SortedNames.Length];
for (int i = 0; i < _name2value.Length; i++) _name2value[_value2name[i]] = i;
}
#endregion
Native Topics
Discussion of the advanced C# interop topics and the Windows C++
implementation.
The Translator
The C# code needs a P-Invoke friendly API to call and that is what the
Translator.DLL provides, starting with the walking of the Export Address Table
of the Quant library (NativeAPI.DLL) to report on all exported functions at run
time:
extern "C" HRESULT __declspec(dllexport) __stdcall
GetExportedFunctions(HMODULE module, LPSAFEARRAY* pDemangled, LPSAFEARRAY* pAddresses)
{
DWORD cNames = 0;
*pDemangled = nullptr;
*pAddresses = nullptr;
auto pHeaders = ImageNtHeader(module);
if (pHeaders)
{
auto pTable = (PIMAGE_EXPORT_DIRECTORY)::ImageDirectoryEntryToData(module, TRUE,
IMAGE_DIRECTORY_ENTRY_EXPORT, &cNames);
cNames = 0;
if (pTable)
{
cNames = pTable->NumberOfNames;
auto pNames = (DWORD*)(pTable->AddressOfNames + (DWORD)module);
auto pFunctions = (DWORD*)(pTable->AddressOfFunctions + (DWORD)module);
if (cNames > 0 && pNames && pFunctions)
{
*pDemangled = ::SafeArrayCreateVector(VT_BSTR, 0, cNames);
*pAddresses = ::SafeArrayCreateVector(VT_I4, 0, cNames);
auto pCurrD = (BSTR*)(*pDemangled)->pvData;
auto pCurrA = (LPVOID*)(*pAddresses)->pvData;
auto buffer = new char[MaxUndecoratedNameLength + 1];
auto i = cNames;
do
{
auto pName = (LPCH)(*pNames + (DWORD)module);
auto pFunction = (PROC)(*pFunctions + (DWORD)module);
if (pName && pName[0] && pFunction)
{
auto cbSymbol = ::UnDecorateSymbolName(pName, buffer,
MaxUndecoratedNameLength + 1, UNDNAME_COMPLETE);
if (cbSymbol > 0 && cbSymbol < MaxUndecoratedNameLength
- 4 )
{
*pCurrA = pFunction;
*pCurrD = (BSTR)::SysAllocStringLen(nullptr, cbSymbol);
::MultiByteToWideChar(CP_UTF7, 0, buffer, cbSymbol + 1, *pCurrD, cbSymbol);
}
}
++pCurrA;
++pCurrD;
++pFunctions;
++pNames;
--i;
} while (i > 0);
delete buffer;
}
}
}
return cNames > 0 ? S_OK : E_FAIL;
}
The corresponding P-Invoke signature in NativeFactory is then:
delegate int _GetExportedFunctions(IntPtr module,
[MarshalAs(UnmanagedType.SafeArray, SafeArraySubType = VarEnum.VT_BSTR)]
out string[] pDemangled,
[MarshalAs(UnmanagedType.SafeArray, SafeArraySubType = VarEnum.VT_I4)]
out IntPtr[] pAddresses
);
The Translator itself has to have thread-affinity so that the same instance
can be easily shared across multiple copies of the Quant library while still
providing a place to store the native C++ exception message, one per copy:
static __declspec(thread) char TLS_Error[255];
extern "C" LPSTR __declspec(dllexport) __stdcall
GetThreadLastError(void)
{
return TLS_Error;
}
Macros and a variadic template are used to more easily create identical
wrapper functions differing only by the number of arguments expected:
#define F_ARGS(NUMBER, ...) &args[NUMBER]
#define DECLARE_FUNCTION(COUNT) \
template<typename... Params> \
static HRESULT __stdcall \
MACRO_CAT(Invoke, COUNT)(LPVOID fn, VARIANT* args) \
{ \
try \
{ \
args[0] = ((VARIANT(__stdcall *)(Params*...))fn)(MACRO_ARGS(COUNT,F_ARGS)); \
return S_OK; \
} \
catch (std::exception& ex) \
{ \
::strncpy_s(TLS_Error, ex.what(), _TRUNCATE); \
return E_FAIL; \
} \
}
Taking the case when COUNT is 3 this expands out into something roughly
equivalent to this:
HRESULT Invoke3(FUNCTION2 fn, VARIANT* args)
{
try
{
args[0] = FUNCTION2(&args[1], &args[2]);
return S_OK;
}
catch (std::exception& ex)
{
TLS_Error = ex.what();
return E_FAIL;
}
}
That is, given the address of any Quant library function that takes two
arguments, call it with those arguments, storing the result and recording any
exception message in the TLS buffer.
Crucially this implies the same P-Invoke
signature can be used for all Quant library functions and so it does not matter
if the number of arguments changes between Quant library versions. This allows for
the Quant library to be changed without recompiling the
Translator or, indeed, the C# application using it:
delegate int _FunctionWrapper(IntPtr function, IntPtr buffer);
internal const int MaxFunctionArguments = 12;
readonly static _FunctionWrapper[] _FunctionWrappers =
new _FunctionWrapper[MaxFunctionArguments];
As referenced in the above code, the number of arguments supported has to be
fixed at compile time. For simplicity, this is done in the C# as well, rather than just
exporting another function from the Translator. The macros are then used to "stamp
out" the code as follows:
DECLARE_FUNCTION(0)
DECLARE_FUNCTION(1)
...
DECLARE_FUNCTION(12)
static const PROC Invokers[] =
{
ADDRESS_FUNCTION(0),
ADDRESS_FUNCTION(1),
...
ADDRESS_FUNCTION(12),
};
extern "C" LPVOID __declspec(dllexport) __stdcall
GetFunctionWrapper(int argc)
{
if (argc >= 0 && argc < _countof(Invokers)) return Invokers[argc];
return nullptr;
}
Finally, the NativeFactory in the Managed.DLL is responsible for locating the
Translator.DLL, loading it and fixing up all entry points as shown below:
public static bool SearchForNativeBinariesAndSetupTranslator()
{
if (!SearchForNativeBinaries()) return false;
var check = Path.Combine(Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location), TRANSLATOR);
if (!File.Exists(check)) return false;
var hTranslator = NativeHelpers.NativeLoadLibrary(TRANSLATOR);
_ExportedFunctions = NativeHelpers.GetDelegate<_GetExportedFunctions>(hTranslator, "_GetExportedFunctions@12");
_GetLastError = NativeHelpers.GetDelegate<GetLastErrorFunctionType>(hTranslator, "_GetThreadLastError@0");
Variant.NativeFree = NativeHelpers.GetDelegate<Variant.VariantHelperType>(hTranslator, "_VariantsFree@8");
Variant.NativeInit = NativeHelpers.GetDelegate<Variant.VariantHelperType>(hTranslator, "_VariantsInit@8");
var getWrapper = NativeHelpers.GetDelegate<_GetFunctionWrapper>(hTranslator, "_GetFunctionWrapper@4");
for (int i = 0; i < MaxFunctionArguments; i++)
{
var wrapper = getWrapper(i);
Debug.Assert(wrapper != null, "MaxArgument count needs to be increased in Translator");
_FunctionWrappers[i] = NativeHelpers.GetDelegate<_FunctionWrapper>(hTranslator, wrapper);
}
return true;
}
The Translator exposes both the addresses of the function wrappers and the full
mangled names of all the functions exported by the NativeAPI. For performance
reasons, it is better to "bake in" the addresses directly into the IL - i.e.
given a function in NativeAPI and it's corresponding wrapper in the Translator,
create a single C# delegate as follows:
internal delegate int NativeFunctionType(IntPtr buffer);
internal static NativeFunctionType CreateFunction(IntPtr address, int argc)
{
if (_SxSPathsByVersion == null) throw new InvalidOperationException("Call SearchForNativeBinariesAndSetupTranslator first");
if (argc >= _FunctionWrappers.Length) throw new IndexOutOfRangeException("GetFunction: max function arguments exceeded");
var wrapper = _FunctionWrappers[argc];
var arg1 = Expression.Parameter(typeof(IntPtr), "buffer");
var exp = Expression.Invoke(Expression.Constant(wrapper), Expression.Constant(address), arg1);
var lambda = Expression.Lambda<NativeFunctionType>(exp, arg1);
return lambda.Compile();
}
Concurrency and Manifests
The Translator.DLL is loaded in the traditional manner: via the Windows
LoadLibrary function as follows:
internal static IntPtr NativeLoadLibrary(string fullFileName)
{
var res = LoadLibraryW(fullFileName);
if (res != IntPtr.Zero) return res;
var err = Marshal.GetLastWin32Error();
throw new FileLoadException(string.Format("Unable to load library (0x{0:X8}) '{1}'", err, fullFileName));
}
However in order to load more than one instance of the same physical group of
interdependent DLLs, the following is required:
- Each physical group must have a unique path - e.g. be in a different
subfolder like "Build\Debug\1\*" and "Build\Debug\2\*";
- Every DLL must have any embedded manifests removed - e.g. this will be
present if linking dynamically with the CRT etc;
- A single manifest must be created that lists all the DLLs in the
physical group (and the CRT etc if necessary);
- LoadLibrary is used but within an Activation Context - i.e. with that
manifest active.
So in each subfolder "1" and "2" there is a copy of the NativeAPI and it's
dependent Utils - and for simplicity both were linked statically with the CRT so
they have no embedded manifest. Now an XML file such as "NativeAPI.Manifest"
can be created in both locations with identical content:
='1.0'='UTF-8'='yes'
<assembly xmlns='urn:schemas-microsoft-com:asm.v1' manifestVersion='1.0'>
<assemblyIdentity type='win32' name='NativeAPI' version='1.0.0.0' processorArchitecture='x86'/>
<description>Native API</description>
<file name='NativeAPI.DLL'/>
<file name='Utils.DLL'/>
</assembly>
The manifest is then used to create an Activation Context only for the
duration of the LoadLibrary call. This happens concurrently on different
threads, one per physical group or instance, and C#'s version of TLS is used to
"assign" the NativeInstance class to the current thread:
public class NativeInstance
{
public readonly int Id;
public readonly Version Version;
public readonly string EntryPoint;
public readonly string ManifestName;
public readonly NativeFunctions Functions;
public static NativeInstance Current { [DebuggerStepThrough] get { return _currentInstance; } }
#region Native loading
[ThreadStatic]
static NativeInstance _currentInstance;
internal NativeInstance(int instanceId, Version version, string targetDLL, string manifest = null)
{
if (_currentInstance != null)
throw new InvalidOperationException("An NativeInstance is already assigned to this thread: "
+ _currentInstance.Id);
Id = instanceId;
Version = version;
EntryPoint = targetDLL;
uint? cookie = null;
if (manifest != null)
{
ManifestName = Path.GetFileName(manifest);
cookie = NativeHelpers.NativePushActivationContext(Path.GetDirectoryName(manifest), manifest);
}
try
{
var hModule = NativeHelpers.NativeLoadLibrary(targetDLL);
Functions = new NativeFunctions(this, hModule, version);
}
finally
{
if (cookie != null) NativeHelpers.NativePopActivationContext(cookie.Value);
}
_currentInstance = this;
Functions.MapAllUsedMeasures();
}
#endregion
Decoding Variants
The NativeAPI is based solely on VARIANTs which, given .NET's originally
incarnation as "Com+ v2", unsurprisingly can be mapped almost perfectly to an
Object, however:
- The commonly used VT_ERROR type is indistinguishable from an integer
(like VT_I4);
- All the Marshal APIs relating to this were designed before Generics in
.NET v2.0 so everything has to be boxed both ways;
- The array mapping requires each element having to be
boxed as well.
This latter problem is particularly bad for Quant libraries since pricing of
Financial Instruments (e.g. Options) often involves asking also for quite a few
risk and informational "Measures" at the same time - i.e. the result is an array
of arrays of different shapes.
On the positive side, the possible input and output types and array shapes
are normally quite small. For example, passing a vector (a 1D array) is
converted across to a 2D array - so only this latter type needs to be handled.
Manual decoding is used including unsafe C# code to avoid the issues with
.NET's inbuilt support, all based around the address of a single native block of
memory (16 bytes) holding the VARIANT as follows:
public struct Variant
{
readonly IntPtr _variant;
#region Helpers
#region Supported variant types
readonly static VarEnum[] _supported = new[]
{
VarEnum.VT_EMPTY,
VarEnum.VT_I4,
VarEnum.VT_R8,
VarEnum.VT_DATE,
VarEnum.VT_BSTR,
VarEnum.VT_BOOL,
VarEnum.VT_VARIANT|VarEnum.VT_ARRAY,
};
static bool VarTypeIsSupported(VarEnum type)
{
return Array.BinarySearch(_supported, type, Enum<VarEnum>.Comparer) >= 0;
}
static Variant()
{
Array.Sort(_supported);
}
#endregion
Since the API exposed by the Translator involves a contiguous block of
VARIANTs, the decoder can attach to any one of these as follows:
internal Variant(IntPtr ptr, int index)
{
_variant = ptr + index * 16;
}
A VARIANT is a classic C union with a 2-byte discriminator which can be
safely cleared to zero (VT_EMPTY) provided that it doesn't contain a BSTR or a
SAFEARRAY:
#region Destruction
[SuppressUnmanagedCodeSecurity]
internal void Free()
{
if (IsReferenceType) NativeHelpers.NativeNativeFree(_variant);
else Reset();
}
[SuppressUnmanagedCodeSecurity]
internal void Reset()
{
Marshal.WriteInt16(_variant, (short)VarEnum.VT_EMPTY);
}
#endregion
To facilitate debugging, especially in the presence of inconsistent Quant
library use of VARIANT types, it is useful to expose both the raw type without
any flags (like VT_ARRAY) and the .NET's interpretation of the raw data:
public VarEnum Type { get { return (VarEnum)((byte)VarType); } }
internal VarEnum VarType { get { return (VarEnum)Marshal.ReadInt16(_variant); } }
public object Boxed { get { return !IsEmpty ? Marshal.GetObjectForNativeVariant(_variant) : null; } }
A nested type is used to handle vectors - i.e. it represents the data payload
of a SAFEARRAY with element type VARIANT:
public struct Vector
{
readonly IntPtr _variant;
readonly int _count;
public static readonly Vector None;
#region Construction
internal Vector(IntPtr ptr, int count)
{
_variant = ptr;
_count = count;
}
private Vector(IntPtr ptr, int index, int count)
{
_variant = ptr + index * 16;
_count = count;
}
#endregion
Again to facilitate .NET debugging, the raw type and .NET's interpretation of
the raw data are exposed:
public object Boxed { get { return Marshal.GetObjectsForNativeVariants(_variant, _count); } }
For style reasons all unsafe (static) C# usage is scoped. Direct decoding
relies on looking at the native memory layout of a VARIANT:
[SuppressUnmanagedCodeSecurity]
internal unsafe class Unsafe
{
static VarEnum GetType(byte* pVariant)
{
return (VarEnum)(*(ushort*)pVariant);
}
delegate T VariantGet<T>(byte* pVariant);
static DateTime UncheckedDate(byte* pVariant)
{
return DateTime.FromOADate(*(double*)(pVariant + 8));
}
static double UncheckedDouble(byte* pVariant)
{
return *(double*)(pVariant + 8);
}
static int UncheckedInteger(byte* pVariant)
{
return *(int*)(pVariant + 8);
}
static bool UncheckedBoolean(byte* pVariant)
{
return *(int*)(pVariant + 8) != 0;
}
static string UncheckedString(byte* pVariant)
{
var ptr = *(int*)(pVariant + 8);
return ptr != 0 ? Marshal.PtrToStringBSTR(new IntPtr(ptr)) : null;
}
Note the way the boolean mapping is defined as "not 0" to avoid subtle issues
where VBA strictly compares only against "-1". Similarly, zero length BSTRs are
not expected to be employed by the Quant library, something carefully checked
for in the reverse case.
The following routine is used to manually decode a BSTR directly into a
StringBuilder to avoid the use of immutable .NET strings entirely where
possible. It relies on the native memory layout having the length before the
start of the (UTF16) string data itself:
internal static int GetBSTR(IntPtr pBStr, StringBuilder sb)
{
if (pBStr != IntPtr.Zero)
{
var len = Marshal.ReadInt32(pBStr - 4) / 2;
if (len > 0)
{
sb.EnsureCapacity(len + sb.Length);
for (int i = 0; i < len; i++)
{
sb.Append((char)Marshal.ReadInt16(pBStr, i * 2));
}
}
}
return sb.Length;
}
The general approach taken is then to call a series of routines for each type
as shown for the "double" case below which uses NaN to represent the VT_EMPTY
case:
static T GenericGetElement<T>(string field, VarEnum check, byte* pv,
VariantGet<T> uncheckedFunction, T @default = default(T))
{
VarEnum type;
if ((type = GetType(pv)) == check)
{
return uncheckedFunction(pv);
}
if (type == VarEnum.VT_EMPTY || type == VarEnum.VT_ERROR) return @default;
throw ErrorCOM(field, "Type mismatch - expected {0} but got {1}", check.ToString(), type.ToString());
}
internal static double GetDouble(string field, IntPtr v)
{
return GenericGetElement<double>(field, VarEnum.VT_R8, (byte*)v.ToPointer(), UncheckedDouble, double.NaN);
}
internal double GetDouble(string field)
{
return Unsafe.GetDouble(field, _variant);
}
For array decoding, a common anti-boxing pattern is employed: using a generic
constrained to an interface directly as an argument. Additionally, 'ref' is used
to ensure only the address of the struct implementing the interface is passed
around and also for any state to be updated. Examination of the IL byte code for this approach shows that
"constrained calls" are used and no boxing occurs:
public interface IOutput<T>
{
void OnMatrix(int row, int col, T data);
void OnColumn(int row, T data);
}
The actual decoding of the SAFEARRAY involves some tedious unsafe C# code so
is not shown here but the routines around it are shown to illustrate the
anti-boxing pattern:
static void UncheckedGetColumn<T, O>(string field, VarEnum check, int rows, ref byte* pvData,
VariantGet<T> f, ref O output, T @default = default(T))
where O : IOutput<T>
where T : IEquatable<T>
{
if (pvData == null || rows <= 0) return;
output.OnColumn(~rows, @default);
for (int i = 0; i < rows; i++)
{
var raw = GenericGetElement(field, check, pvData, f, @default);
if (!EqualityComparer<T>.Default.Equals(raw, @default)) output.OnColumn(i, raw);
pvData += 16;
}
}
static int UncheckedGetVector<T, O>(string field, VarEnum check, byte* pVariant,
VariantGet<T> f, ref O output, T @default = default(T))
where O : IOutput<T>
where T : IEquatable<T>
{
int cDim, len, dummy;
var pvData = UncheckedArray(field, pVariant, out cDim, out len, out dummy);
if (pvData == null) return 0;
if (cDim == 2) throw ErrorCOM(field, "Internal vector problem - vector expected but got matrix");
UncheckedGetColumn(field, check, len, ref pvData, f, ref output, @default);
return len;
}
static int GenericGetVector<T, O>(string field, VarEnum check, byte* pv,
VariantGet<T> uncheckedFunction, ref O output, T @default = default(T))
where O : IOutput<T>
where T : IEquatable<T>
{
VarEnum type;
if ((type = GetType(pv)) == (VarEnum.VT_ARRAY | VarEnum.VT_VARIANT))
{
return UncheckedGetVector(field, check, pv, uncheckedFunction, ref output, @default);
}
if (type == VarEnum.VT_EMPTY || type == VarEnum.VT_ERROR) return 0;
throw ErrorCOM(field, "Type mismatch - expected {0} but got {1}", check.ToString(), type.ToString());
}
There are many routines provided covering all the types including specific
support for matrices of the same type and also the more common mixed type - i.e.
where each column is a different type.
The corresponding setting routines are slightly more tricky as they must
normalise the inputs. For example, the following shows how P-Invoke's special
handling of StringBuilders can be used to avoid duplication when creating a
BSTR:
static void UncheckedNewString(byte* pVariant, StringBuilder value)
{
if (value.Length > 0)
{
*(int*)(pVariant + 8) = NativeHelpers.NativeAllocBSTR(value).ToInt32();
return;
}
*(int*)(pVariant + 8) = 0;
}
[DllImport(OLEAUT32_DLL, ...)]
private static extern IntPtr SysAllocStringLen([In] StringBuilder sb, int cch);
The anti-boxing pattern is used along with a complex delegate-based signature
for the key function for streaming data out of a collection of structures into a single column
of the one type without any boxing:
static class Singleton<T>
where T : IEquatable<T>
{
public static readonly Func<T, T> Identity = x => x;
}
static byte* UncheckedSetColumn<TCollection, TElement, TEnumerable>(VarEnum type, int skip, int take, byte* pvData,
VariantSet<TElement> setter, TEnumerable input, Func<TCollection, TElement> selector, TElement @default)
where TElement : IEquatable<TElement>
where TEnumerable : IEnumerable<TCollection>
...
Running the Examples
There are three mutually exclusive code branches - i.e. two of them
commented out:
- The default one loads the two copies of the Quant binaries in
side-by-side mode and runs them in two separate threads:
var t1 = new Thread(x => Run((bool)x));
t1.Start(false);
var t2 = new Thread(x => Run((bool)x));
t2.Start(false);
t1.Join();
t2.Join();
- Uncomment the following to run the standard P-Invoke single-instance
approach:
PInvoke();
return;
- And uncomment the following to run the singleton approach for easier
debugging:
Run(bSingleInstance: true);
return;
Note that this latter mode is best when making changes since the side-by-side
mode requires any changes to the NativeAPI.DLL or Utils.DLL to be manually
duplicated into the 1 and 2 subfolders of Debug.
References
Financial Numerical Recipes
SAFEARRAY native struct
VARIANT native struct
BSTR native type
History
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
