TestFlask CLI

TestFlask CLI is a command line interface/tool for various TestFlask operations.

You can find it inside the TestFlask repo as a console application project TestFlask.CLI. You can register testflask.exe to your %PATH% environment variable to use it directly inside the command prompt.

Currently, it supports a single verb unit that generates unit tests for a given TestFlask project inside a specified TestFlask API url.

For options for specific verbs, you can generate help menu with

testflask –help

Explanation of options for unit verb is as follows

Option (shortcut)	Option (long)	Description
-t	–testfw	(Default: mstest) Unit test framework (Only mstest is supported currently)
-a	–api	Required. TestFlask api host url that holds scenario for which unit test will be generated.
-p	–project	Required. TestFlask project key that owns the scenarios
-c	–classname	(Default: TestFlaskTests_Auto.cs) Output class name. Output file name will be [classname]_Auto.cs
-n	–namespace	(Default: TestFlaskAutoTests) Namespace for your generated test class
-m	–mode	(Default: aot) Test generation mode. Use aot to embed test data into test files, and use jit to fetch test data on execution time.
-l	–labels	Comma seperated list of labels of scenarios for which to generate unit tests
	–help	Display this help screen.
	–version	Display version information.

testflask unit

Here is an example that generates unit tests for QuickPay.Demo

testflask unit -a http://localhost:12079 -p QuickPayDemo -c QuickTests -n Demo.QuickPay.Tests -m jit

This command outputs a file named QuickTests_Auto.cs inside the same folder with testflask.exe.

Have a look at the generated file. As it is generated with option -m jit, a test method will load scenario data on the fly from TestFlask.API configured inside test project’s app.config. It means that if you alter response objects inside TestFlask Manager, you can instantly see changes inside the test execution.

You can also debug your services using these tests, or even create code coverage.

/****************************************************************************
*																		 	*
*	This class is auto generated by TestFlask CLI on 10/15/2018 10:14:11 PM	    *
*	https://github.com/FatihSahin/test-flask                                *
*	Implement provider methods and step assertions inside another file.		*
*																		 	*
****************************************************************************/
using System;
using System.Collections;
using System.Collections.Generic;
using System.Collections.Specialized;
using System.IO;
using System.Linq;
using System.Reflection;
using System.Runtime.Serialization.Formatters;
using System.Text;
using System.Web;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using TestFlask.Aspects.ApiClient;
using TestFlask.Aspects.Enums;
using TestFlask.Models.Context;
using TestFlask.Models.Entity;
using TestFlask.Aspects.Context;
using Demo.QuickPay.Data.Context;
using Demo.QuickPay.Biz.Models;

namespace Demo.QuickPay.Tests
{
    [TestClass]
    public partial class QuickTests
    {
        private bool isEmbedded = false;

        #region Conventional

        private static IEnumerable<Scenario> embeddedScenarios { get; set; }

        private JsonSerializerSettings jsonSerializerSettings = new JsonSerializerSettings
        {
            TypeNameHandling = TypeNameHandling.All,
            TypeNameAssemblyFormat = FormatterAssemblyStyle.Simple
        };

        [ClassInitialize]
        public static void ClassSetUp(TestContext context)
        {
            embeddedScenarios = ReadEmbeddedScenarios();
            DoClassSetUp(context);
        }

        [ClassCleanup]
        public static void ClassTearDown()
        {
            embeddedScenarios = null;
            DoClassTearDown();
        }

        private static IEnumerable<Scenario> ReadEmbeddedScenarios()
        {
            string fileName = "QuickTests_Auto_Embed.txt";

            if (!File.Exists(fileName))
            {
                return null;
            }

            List<Scenario> embeddedScenarios = new List<Scenario>();

            string line;
            using (System.IO.StreamReader fileReader = new System.IO.StreamReader(fileName))
            {
                while ((line = fileReader.ReadLine()) != null)
                {
                    var json = TestFlask.Models.Utils.CompressUtil.DecompressString(line);
                    var scenario = JsonConvert.DeserializeObject<Scenario>(json);
                    embeddedScenarios.Add(scenario);
                }
            }

            return embeddedScenarios;
        }

        private void ProvideTestFlaskHttpContext(Step step, TestModes testMode)
        {
            HttpContext.Current = new HttpContext(
                new HttpRequest("", "http://tempuri.org", ""),
                new HttpResponse(new StringWriter())
                );

            var invocation = step.GetRootInvocation();

            // In order to by pass Platform not supported exception
            // http://bigjimindc.blogspot.com.tr/2007/07/ms-kb928365-aspnet-requestheadersadd.html
            AddHeaderToRequest(HttpContext.Current.Request, ContextKeys.ProjectKey, invocation.ProjectKey);
            AddHeaderToRequest(HttpContext.Current.Request, ContextKeys.ScenarioNo, invocation.ScenarioNo.ToString());
            AddHeaderToRequest(HttpContext.Current.Request, ContextKeys.StepNo, invocation.StepNo.ToString());
            AddHeaderToRequest(HttpContext.Current.Request, ContextKeys.TestMode, testMode.ToString());

            TestFlaskContext.LoadedStep = step;
        }

        private void AddHeaderToRequest(HttpRequest request, string key, string value)
        {
            NameValueCollection headers = request.Headers;

            Type t = headers.GetType();
            ArrayList item = new ArrayList();

            // Remove read-only limitation on headers
            t.InvokeMember("MakeReadWrite", BindingFlags.InvokeMethod | BindingFlags.NonPublic | BindingFlags.Instance, null, headers, null);
            t.InvokeMember("InvalidateCachedArrays", BindingFlags.InvokeMethod | BindingFlags.NonPublic | BindingFlags.Instance, null, headers, null);
            item.Add(value);
            t.InvokeMember("BaseAdd", BindingFlags.InvokeMethod | BindingFlags.NonPublic | BindingFlags.Instance, null, headers, new object[] { key, item });
            t.InvokeMember("MakeReadOnly", BindingFlags.InvokeMethod | BindingFlags.NonPublic | BindingFlags.Instance, null, headers, null);
        }

        private Step GetLoadedStep(long stepNo, bool isEmbedded)
        {
            Step step = null;

            if (isEmbedded)
            {
                step = embeddedScenarios?.SelectMany(sc => sc.Steps).SingleOrDefault(st => st.StepNo == stepNo);
            }

            if (step == null)
            {
                TestFlaskApi api = new TestFlaskApi();
                step = api.LoadStep(stepNo);
            }

            return step;
        }

        private void HandleAssertion(Invocation rootInvocation, object responseObject, Exception exception, Action stepAssertion)
        {
            if ((!rootInvocation.IsFaulted && exception == null) || (rootInvocation.IsFaulted && exception != null))
            {
                stepAssertion();
            }
            else if (exception != null)
            {
                string exceptionStr = JToken.Parse(JsonConvert.SerializeObject(exception, jsonSerializerSettings)).ToString(Formatting.Indented);
                Assert.Fail($"Expected proper response of type {rootInvocation.ResponseType} but got exception =>{Environment.NewLine}{exceptionStr}{Environment.NewLine}{GetExceptionStackOutput()}");
            }
            else
            {
                string responseStr = JToken.Parse(JsonConvert.SerializeObject(responseObject, jsonSerializerSettings)).ToString(Formatting.Indented);
                Assert.Fail($"Expected exception of type {rootInvocation.ExceptionType} but got response =>{Environment.NewLine}{responseStr}");
            }
        }

        private string GetExceptionStackOutput()
        {
            StringBuilder strBuilder = new StringBuilder();
            IEnumerable<Invocation> exceptionalInvocations = TestFlaskContext.InvocationStack.ExceptionStack;

            strBuilder.AppendLine("**** TestFlask Exception Stack Snapshot ****");
            foreach (var excInv in exceptionalInvocations)
            {
                strBuilder.AppendLine("\t**** Faulty Invocation ****");
                strBuilder.AppendLine($"\t\tMethod => {excInv.InvocationSignature}");
                strBuilder.AppendLine($"\t\tInvocation Mode => {excInv.InvocationMode}");
                if (!string.IsNullOrWhiteSpace(excInv.RequestDisplayInfo))
                {
                    strBuilder.AppendLine($"\t\tRequest Info => {excInv.RequestDisplayInfo}");
                }
                strBuilder.AppendLine($"\t\tRequest => ");
                strBuilder.AppendLine(JToken.Parse(excInv.Request).ToString(Formatting.Indented));
                strBuilder.AppendLine($"\t\tExceptionType => {excInv.ExceptionType}");
                strBuilder.AppendLine($"\t\tException => ");
                strBuilder.AppendLine(JToken.Parse(excInv.Exception).ToString(Formatting.Indented));
            }

            return strBuilder.ToString();
        }

        private Invocation PrepareStep(long stepNo, TestModes testMode, bool isEmbedded)
        {
            Step loadedStep = GetLoadedStep(stepNo, isEmbedded);
            var rootInvocation = loadedStep.GetRootInvocation();
            ProvideTestFlaskHttpContext(loadedStep, testMode);
            ProvideOperationContext(rootInvocation);
            return rootInvocation;
        }

        #endregion

        #region Scenario33_MoneyTransferSuccess

        [TestMethod]
        [TestCategory("TestFlask")]
        public void Scenario33_MoneyTransferSuccess()
        {
            Scenario33_MoneyTransferSuccess_Step49_transferMoneySuccessWithAmount5();
        }

        private void Scenario33_MoneyTransferSuccess_Step49_transferMoneySuccessWithAmount5()
        {
            var rootInvocation = PrepareStep(49, TestFlask.Aspects.Enums.TestModes.Assert, isEmbedded);
            var requestObject = JsonConvert.DeserializeObject<object[]>(rootInvocation.Request, jsonSerializerSettings).First() as Demo.QuickPay.Data.Context.Payment;

            //Set up additional behaviour for method args
            SetUp_Scenario33_MoneyTransferSuccess_Step49_transferMoneySuccessWithAmount5(requestObject);

            Demo.QuickPay.Biz.Models.PaymentResult responseObject = null;
            Exception exception = null;

            try { responseObject = subjectPaymentController.TransferMoney(requestObject); }
            catch (Exception ex) { exception = ex; }

            HandleAssertion(rootInvocation, responseObject, exception,
                () => Assert_Scenario33_MoneyTransferSuccess_Step49_transferMoneySuccessWithAmount5(responseObject, exception));
        }

        #endregion

        #region Scenario34_MoneyTransferFailure

        [TestMethod]
        [TestCategory("TestFlask")]
        public void Scenario34_MoneyTransferFailure()
        {
            Scenario34_MoneyTransferFailure_Step50_transferMoneyFailureWithClosedDebitAccount();
        }

        private void Scenario34_MoneyTransferFailure_Step50_transferMoneyFailureWithClosedDebitAccount()
        {
            var rootInvocation = PrepareStep(50, TestFlask.Aspects.Enums.TestModes.Assert, isEmbedded);
            var requestObject = JsonConvert.DeserializeObject<object[]>(rootInvocation.Request, jsonSerializerSettings).First() as Demo.QuickPay.Data.Context.Payment;

            //Set up additional behaviour for method args
            SetUp_Scenario34_MoneyTransferFailure_Step50_transferMoneyFailureWithClosedDebitAccount(requestObject);

            Demo.QuickPay.Biz.Models.PaymentResult responseObject = null;
            Exception exception = null;

            try { responseObject = subjectPaymentController.TransferMoney(requestObject); }
            catch (Exception ex) { exception = ex; }

            HandleAssertion(rootInvocation, responseObject, exception,
                () => Assert_Scenario34_MoneyTransferFailure_Step50_transferMoneyFailureWithClosedDebitAccount(responseObject, exception));
        }

        #endregion

        private Demo.QuickPay.WebApi.Controllers.PaymentController subjectPaymentController;
    }
}

It is expected from the developer to write a partial class file that complements test assertions and initialization logic in another file. The motivation here is to keep the auto generated file untouched and generic by implementing custom logic in your own file.

This is a custom partial class file QuickTests.cs that complements tests in auto generated file QuickTests_Auto.cs

using System;
using Demo.QuickPay.Biz.Models;
using Demo.QuickPay.Data.Context;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using TestFlask.Models.Entity;

namespace Demo.QuickPay.Tests
{
    public partial class QuickTests
    {
        private static void DoClassSetUp(TestContext context)
        {
        }

        private static void DoClassTearDown()
        {
        }

        private void ProvideOperationContext(Invocation rootInvocation)
        {
        }

        [TestInitialize]
        // will be empty on template 
        public void ProvideSubjects() {

            subjectPaymentController = new WebApi.Controllers.PaymentController();
        }

        private void SetUp_Scenario33_MoneyTransferSuccess_Step49_transferMoneySuccessWithAmount5(Payment payment)
        {
            //no additional setup
        }

        private void Assert_Scenario33_MoneyTransferSuccess_Step49_transferMoneySuccessWithAmount5(PaymentResult paymentResult, Exception exception)
        {
            // a successful scenario assertion
            Assert.IsTrue(paymentResult.IsSuccessful);
        }

        private void SetUp_Scenario34_MoneyTransferFailure_Step50_transferMoneyFailureWithClosedDebitAccount(Payment requestObject)
        {
            //no additional setup
        }

        private void Assert_Scenario34_MoneyTransferFailure_Step50_transferMoneyFailureWithClosedDebitAccount(PaymentResult paymentResult, Exception exception)
        {
            //this is a failure scenario, so result must be unsuccessful.
            Assert.IsFalse(paymentResult.IsSuccessful);
            //payment failed with closed account so we expect an error code here.
            Assert.AreEqual("DEBIT_ACC_CLOSED", paymentResult.ErrorCode);
        }
    }
}

Embedding scenario data into a file for unit tests

If you trigger testflask unit for option with -m aot like here

testflask unit -a http://localhost:12079 -p QuickPayDemo -c QuickTests -n Demo.QuickPay.Tests -m aot

CLI will generate one additional file QuickTests_Auto_Embed.txt that contains all scenario data inside it. TestFlask tests than will load the file during test class initialization and replay the invocations from that file. Therefore, you will be able to run test fully isolated inside an auto TFS build or a remote build orchestration tool.

TestFlask Mocking and testing made simple

TestFlask CLI

testflask unit

Embedding scenario data into a file for unit tests