Tag Archives: C#

Statically Compiled LINQ Queries Are Broken In .NET 4.0

Written by William Roush on January 19, 2014 at 5:31 pm

Diving into how a minor change in error handling in .NET 4.0 has broken using compiled LINQ queries as per the MSDN documentation.

Query was compiled for a different mapping source than the one associated with the specified DataContext.

When working on high performing LINQ code this error can cause a massive amount of headaches. This StackOverflow post blames the problem on using multiple LINQ mappings (which the same mappings from different DataContexts will count as "different mappings"). In the example below, we’re going to use the same mapping, but different instances which is extremely common for short-lived DataContexts (and reusing DataContexts come with a long list of problematic side-effects).

namespace ConsoleApplication1
{
    using System;
    using System.Data.Linq;
    using System.Linq;

    class Program
    {
        protected static Func<MyContext, Guid, IQueryable<Post>> Query =
            CompiledQuery.Compile<MyContext, Guid, IQueryable<Post>>(
                (dc, id) =>
                    dc.Posts
                        .Where(p => p.AuthorID == id)
            );

        static void Main(string[] args)
        {
            Guid id = new Guid("340d5914-9d5c-485b-bb8b-9fb97d42be95");
            Guid id2 = new Guid("2453b616-739f-458f-b2e5-54ec7d028785");

            using (var dc = new MyContext("Database.sdf"))
            {
                Console.WriteLine("{0} = {1}", id, Query(dc, id).Count());
            }

            using (var dc = new MyContext("Database.sdf"))
            {
                Console.WriteLine("{0} = {1}", id2, Query(dc, id2).Count());
            }

            Console.WriteLine("Done");
            Console.ReadKey();
        }
    }
}

This example follows MSDN’s examples, yet I’ve seen people recommending you do this to resolve the changes in .NET 4.0:

protected static Func<MyContext, string, IQueryable<Post>> Query
{
    get
    {
        return
            CompiledQuery.Compile<MyContext, string, IQueryable<Post>>(
                 (dc, id) =>
                    dc.Posts
                        .Where(p => p.AuthorID == id)
            );
    }
}

Wait a second! I’m recompiling on every get, right? I’ve seen claims it doesn’t. However peeking at the IL code doesn’t hint at that, the process is as follows:

  • Check if the query is assignable from ITable, if so let the Lambda function compile it.
  • Create a new CompiledQuery object (just stores the Lambda function as a local variable called “query”).
  • Compile the query using the provider specified by the DataContext (always arg0).

At no point is there a cache check, the only place a cache could be placed is in the provider (which SqlProvider doesn’t have one), and that would be a complete maintenance mess if it was done that way.

Using a test application (code is available at https://bitbucket.org/StrangeWill/blog-csharp-static-compiled-linq-errors/, use the db.sql file to generate the database, please use a local installation of MSSQL server to give the best speed possible so that we can evaluate query compilation times), we’re going to force invoking the CompiledQuery.Compile method on every iteration (10,000 by default) by passing in delegates as opposed to passing in the resulting compiled query.

QueryCompiled Average: 0.5639ms
QueryCompiledGet Average: 1.709ms
Individual Queries Average: 2.1312ms
QueryCompiled Different Context (.NET 3.5 only) Average: 0.6051ms
QueryCompiledGet Different Context Average: 1.7518ms
Individual Queries Different Context Average: 2.0723ms

We’re no longer seeing the 1/4 the runtime you get with the compiled query. The primary problem lies in this block of code found in CompiledQuery:

if (context.Mapping.MappingSource != this.mappingSource)
{
	throw Error.QueryWasCompiledForDifferentMappingSource();
}

This is where the CompiledQuery will check and enforce that you’re using the same mapper, the problem is that System.Data.Linq.Mapping.AttributeMappingSource doesn’t provide an Equals override! So it’s just comparing whether or not they’re the same instance of an object, as opposed to them being equal.

There are a few fixes for this:

  • Use the getter method, and understand that performance benefits will mainly be seen where the result from the property is cached and reused in the same context.
  • Implement your own version of the CompiledQuery class.
  • Reuse DataContexts (typically not recommended! You really shouldn’t…).
  • Stick with .NET 3.5 (ick).
  • Update: RyanF below details sharing a MappingSource below in the comments. This is by far the best solution.

Tamper-proof Licensing Using Cryptographic Signatures

Written by William Roush on November 4, 2013 at 9:15 pm

I’m going to dive right into writing a basic licensing system using strong established cryptographical signing processes to prevent tampering or the creation of a key generator. I’m going to digitally sign the plain-text license using OpenPGP’s private key, this will generate a human readable license file, but modifying the payload will cause an exception to be thrown and tampering detected.

Solutions

I’m going to go over the basic options available. There are many others and there are a lot of hybrids of these various types, but for a quick introduction to licensing options, this should get you up to speed.

No Licensing

Of course the option for no licensing is always available. This provides no protection against a user copying your application and giving it to another user. Sometimes the overhead of writing and maintaining licensing code may not be worth the effort.

Plaintext Keys

Plaintext keys store user information, dates, times and keys in various locations, such as files on the hard disk, registry keys or resource files. These are fairly trivial to change and circumvention information can be passed around by word of mouth. You can attempt to store keys in hidden places like some other software, but these can and likely will be tracked down, and depending on if your method is ethically unsound, can land you unfavorable opinion with your would-be customers.

These are very easy, and can be a preferred choice if you’re not concerned with someone attempting to crack your licensing code, but may want some features granted by having licensing support in your software.

Symmetrical Cryptography / Hash Verification

Both of these methods fall under the same pros/cons for the most part. Symmetrical keys will encrypt the entire license file and should use some sort of secure algorithm (ex: AES256) with a secret key. Hash verification may store the data encrypted or unencrypted, but will hash the key information either with a secret method or a secret salt and store that with the license file, where you can validate that it hasn’t been tampered with, tampered keys fail validation.

Both of these suffer from the same issue: the code to decrypt/hash and verify your license files provide all the information needed to encrypt/hash new license files, allowing someone to create a license generator. In other languages these is a higher barrier of entry to be able to reverse engineer a native library, but in .NET your code can be fairly easily decompiled and methods like this reverse engineered (generally even when obfuscated).

Asymmetrical Cryptography

Asymmetrical cryptography, also known as Public-key cryptography is a method of encrypting data using the public key, and only being able to decrypt it with the private key (this can also be used for verification purposes by reversing the usage of the keys). In our example, we’ll use the example of digital signatures: where we use the private key for signing the license, and the public key to verify it hasn’t been tampered with.

Without the private key, a user cannot issue new licenses, a key generator will be impossible to make without one. The best bet resides in the malicious person to take one of two paths: to modify the software to remove the license requirement entirely, or modify the software to use a different public key, so they can use their own private key to generate licenses.

We’re on a better path now: a malicious user that has to patch our binaries to circumvent licensing will incur the overhead of having to patch every binary we release, while we can’t ever stop circumvention, we can make it annoying!

Online Verification

Online verification is one of the hardest methods, if not the hardest to circumvent. However a lot has to be assumed about your customers. You need to assume your users will be online at regular intervals, additional the use of software they’ve purchased will be entirely dependent on you keeping your licensing servers up and running.

Circumvention requires you to rewrite binaries to no longer check online, or spoof authentication servers.

PGP and You

I’m going to useĀ OpenPGP, an open encryption standard that includes public-key cryptography. This library is used a lot for signing/encrypting e-mails and code among various programming groups, and I’ll continue to use this later for various examples such as signing code in a later blog.

Generating Your Key Pair

I recommend using GPG4Win, an easy to use application for creating and managing GPG and X509 certificates.

Select "File" > "New Certificate"

Select “File” > “New Certificate”

We want to create an OpenPGP key pair.

We want to create an OpenPGP key pair.

Enter the certificate information.

Enter the certificate information.

Your private key is the key to your kingdom, use a strong passphrase.

You want to save the private key (*.gpg) somewhere where your application can access it.

You want to save the private key (*.gpg) somewhere where your application can access it.

You'll want to export the public key too.

You’ll want to export the public key too.

Dealing with PGP On .NET

I’m going to use a very commonly used cryptography library for .NET BouncyCastle, mainly because it’s extremely powerful and licensed under MIT a very open license friendly to both open source projects and closed source.

An Example License File

We’re going to go ahead with the most basic setup: we’re not going to encrypt the payload so we can understand what is going on, for further obscurity you can encrypt the payload or even the entire license, but it wont add much additional security (just make it harder to understand what is going on till they decompile your .NET code).

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

William Roush
1/1/2018 12:00:00 AM
1/1/2014 12:00:00 AM
-----BEGIN PGP SIGNATURE-----
Version: BCPG C# v1.7.4114.6375

iQFYBAABAgBCBQJSdv5OOxxSb3VzaFRlY2ggRXhhbXBsZSBDb2RlIFNpZ25pbmcg
S2V5IDxleGFtcGxlQHJvdXNodGVjaC5uZXQ+AAoJEGyAwS0mpw1Ki5MH/jQAKiur
FF06Q7D0BeaUE6eBk40I2LocZwSE2Osx94Tdux08L8vpaOUG4xBx6UsNKusvrbaV
hSxeU1e1LkO3QMu8+//FAFsJizTYjYZRoRe167WFRlQbteZdgeB1S+HMdDhsBs+Z
3+FGAulhs7EBmRkgSoYEX0NdJc1uEfW5kRr4KfL4v+m9UU6Z796KWFzOi8xqTcRG
eO9cNV7QtAzBOpI2CVuq6KuGj5VjNtQsxUrUNl6U/giV61Y5TYOfRnL47kX4U6sf
Uzp/iRZqemi+n7iBjMsO37+baANySaBRSKUG0EXtQM9sVxHZ7aGoXpjsU+oeWtBY
iWPAPX1ePwQ6LcM=
=57ao
-----END PGP SIGNATURE-----

Writing Your License File Management Code

First I’m going to outline some basics, we’re going to store a name and two dates, a software expiration date and a support expiration date. Additionally I’ll override ToString() for testing purposes, and write a method that returns the license data as a byte array that’ll go into our signed license file.

using System;
using System.IO;
using System.Linq;
using System.Text;

using Org.BouncyCastle.Bcpg.OpenPgp;
using Org.BouncyCastle.Bcpg;
using Org.BouncyCastle.Asn1.Ocsp;
using Org.BouncyCastle.Security;

public class License
{
    public string Name { get; set; }
    public DateTime SoftwareExpires { get; set; }
    public DateTime SupportExpires { get; set; }

    protected PgpPrivateKey PrivateKey { get; set; }
    public PgpPublicKey PublicKey { get; set; }

    public override string ToString()
    {
        return string.Format("Name: {1}{0}Software Expires: {2}{0}Support Expires: {3}",
            Environment.NewLine,
            this.Name,
            this.SoftwareExpires,
            this.SupportExpires
        );
    }

    private byte[] PrepareLicenseForStorage()
    {
        var licenseData = string.Format(
            "{0}\n{1}\n{2}",
            this.Name,
            this.SoftwareExpires.ToString(),
            this.SupportExpires.ToString()
        );

        return Encoding.UTF8.GetBytes(licenseData);
    }

Next we’re going to define how to read a public key out of a file.

public void ReadPublicKey(string path)
{
    using (var keyFileStream = File.OpenRead(path))
    using (var pgpDecoderStream = PgpUtilities.GetDecoderStream(keyFileStream))
    {
        var keyRingBundle = new PgpPublicKeyRingBundle(pgpDecoderStream);
        var keyRings = keyRingBundle.GetKeyRings();
        foreach (PgpPublicKeyRing keyRing in keyRings)
        {
            this.PublicKey = keyRing.GetPublicKey();
            break;
        }
    }
}

Then we’re going to define how to read a private key out of a file.

public void ReadPrivateKey(string path)
{
    using (var keyFileStream = File.OpenRead(path))
    using (var pgpDecoderStream = PgpUtilities.GetDecoderStream(keyFileStream))
    {
        // Get the key ring from the private key file.
        var keyRingBundle = new PgpSecretKeyRingBundle(pgpDecoderStream);
        var keyRings = keyRingBundle.GetKeyRings();

        // Get the first key from the key ring (it's an enumerable, so it's annoying).
        PgpSecretKey secretKey = null;
        foreach(PgpSecretKeyRing keyRing in keyRings)
        {
            secretKey = (PgpSecretKey)keyRing.GetSecretKey();
            break;
        }

        // Get the private key, we don't have a pass phrase on this (you should though!). 
        // Do NOT hard code this into the library, take it as a parameter from your key generation
        // software.
        this.PrivateKey = secretKey.ExtractPrivateKey("".ToCharArray());
    }
}

Next we’re going to define the code for creating a license file.

public void CreateLicenseFile(string fileName)
{
    var hashAlgorithm = HashAlgorithmTag.Sha1;
    using (var fileWriter = new FileStream(fileName, FileMode.Create))
    using (var outputStream = new ArmoredOutputStream(fileWriter))
    {
        // Continue our signature.
        var signatureGenerator = new PgpSignatureGenerator(this.PublicKey.Algorithm, hashAlgorithm);
        signatureGenerator.InitSign(PgpSignature.BinaryDocument, this.PrivateKey);
        foreach (string userId in this.PublicKey.GetUserIds())
        {
            var pgpSigatureSubpacketGenerator = new PgpSignatureSubpacketGenerator();
            pgpSigatureSubpacketGenerator.SetSignerUserId(false, userId);
            signatureGenerator.SetHashedSubpackets(pgpSigatureSubpacketGenerator.Generate());
            break;
        }

        var licenseDataBuffer = this.PrepareLicenseForStorage();

        // We're going to write out the cleartext portion.
        outputStream.BeginClearText(hashAlgorithm);
        outputStream.Write(licenseDataBuffer, 0, licenseDataBuffer.Length);

        // This updates our signature with the data in our license.
        signatureGenerator.Update(licenseDataBuffer, 0, licenseDataBuffer.Length);
        outputStream.EndClearText();

        // Add newline after text so we start the PGP signature header on the next line.
        byte[] newLineBytes = Encoding.UTF8.GetBytes(Environment.NewLine);
        fileWriter.Write(newLineBytes, 0, newLineBytes.Length);

        // Outputs the signature into the file.
        using (var bcpgStream = new BcpgOutputStream(outputStream))
        {
            signatureGenerator.Generate().Encode(bcpgStream);
        }
    }
}

Finally, we need to be able to create a license object from a signed license file.

public static License LoadLicenseFromFile(string fileName, PgpPublicKey publicKey)
{
    using (var inputStream = new FileStream(fileName, FileMode.Open))
    using (var armoredStream = new ArmoredInputStream(inputStream))
    using (var memoryStream = new MemoryStream())
    {
        // We're going to read the payload of the signed text in.
        while (armoredStream.IsClearText())
        {
            memoryStream.WriteByte((byte)armoredStream.ReadByte());
        }
        
        // Load the signature data from the armored file and initalize it with the public key.
        PgpObjectFactory pgpFact = new PgpObjectFactory(armoredStream);
        PgpSignatureList p3 = (PgpSignatureList)pgpFact.NextPgpObject();
        PgpSignature sig = p3[0];
        sig.InitVerify(publicKey);

        // We're going to seek to the beginning, and strip off the final "/r/n-" from the message.
        memoryStream.Seek(0, SeekOrigin.Begin);
        for (int i = 0; i < memoryStream.Length - 3; i++)
        {
            sig.Update((byte)memoryStream.ReadByte());
        }
        
        if (!sig.Verify())
        {
            throw new Exception("Verification Failed")
        }

        // Parse our stored license data.
        string[] licenseData = Encoding.UTF8.GetString(memoryStream.ToArray()).Split('\n');
        return new License
        {
            Name = licenseData[0],
            SoftwareExpires = DateTime.Parse(licenseData[1]),
            SupportExpires = DateTime.Parse(licenseData[2])
        };   
    }
}

From here we have the underlying infrastructure needed to write a license file that cannot be tampered with and that our application can verify is correct.

Changes To Production Code

First in production you’ll never want to ship your licensing software with your private key or with your secret phrase, that stays with you for generating new keys. Secondly we’ll want to read the private key from some internal resource so a malicious hacker can’t just replace your public key file without modifying binaries.