Simulate a Fault
Last Updated: November 15th, 2024
Customers use AWS Fault Injection Service (FIS) and Cutover to align data and processes. Using a Cutover runbook, you can trigger one or more FIS experiments in AWS and design Cutover workflows around that.
There are various FIS experiments designed to simulate disruptions, such as issues with Availability Zones, RDS, or CPU stress scenarios. In this document, we will focus on creating an experiment that disrupts EC2 instances. Using our guide, you can create an experiment template with a defined capacity limit for EC2 instances. Once the template is created, the experiment will trigger a success code upon completion, initiating any dependent tasks that follow.
We will now go through the steps required to configure Cutover with FIS, covering the integration points between the Cutover platform and the FIS solution.
Note: The configuration shared in this guide is intended to describe how to set up Cutover to trigger an experiment with FIS. Use this guide as a reference to understand the changes and match to your current configuration where necessary.
Set up your integrations in a few easy steps, read on to find out how in just 9 minutes.
Prerequisites
Prior to integration set up, you must have the following:
- An understanding of how our Custom Integration works. You can read more here.
- An understanding of FIS and how experiments work.
- In Cutover, you will need to assign both the Integration Admin and Custom Fields Admin role types to the user who is configuring the integration.
- Configure AWS to have your own required mandatory fields for example:some text
- Build Name Id
Changes to be made in FIS
Prior to creating your Custom Integration, you will need to make changes in FIS. Please read the following sections to find out what the changes are.
Create Experiment Templates in FIS
For our example, you will need to create an experiment template that enforces a capacity instance limit for EC2. Experiment templates allow you to define a set of actions that can be applied on a target or environment. Once an experiment template has been created, you can use Cutover to trigger it.
Open the FIS console and click Create experiment template. FIS has a template already pre-defined that you can use:
EC2. aws:ec2:api-insufficient-instance-capacity-error.
You should set an insufficient instance capacity and then you can also set a target on what you want to enforce this action on, once the experiment gets started.
As displayed in the image below, you can set the target to be applied to a particular role in the AWS environment. We will want this to be restricted and have an insufficient instance capacity limit once we start the experiment.
You need to ensure that the role that you are using for the target is also created via the Identity and Access Management (IAM) console and that the role has permissions with FIS. In the below image you can see all the FIS roles needed.
The full list of roles (as shown above) are the following:
- AmazonEC2FullAccess
- AWSFaultInjectionSimulatorEC2Access
- AWSFaultInjectionSimulatorECSAccess
- AWSFaultInjectionSimulatorEKSAccess
- AWSFaultInjectionSimulatorNetworkAccess
- AWSFaultInjectionSimulatorRDSAccess
- AWSFaultInjectionSimulatorSSMAccess
Additional documentation on these policies can be found here: https://docs.aws.amazon.com/aws-managed-policy/latest/reference/policy-list.html
In addition we’ve added a custom inline policy to start the FIS experiment as shown on the next page, where 871003340xxxx is the name of the AWS account this is created in:
Using the role created above for Cutover to call the FIS integration, you now need to include a trust relationship that allows for the Cutover AWS environment to assume this role in the FIS environment. See the image below for how this would look in the AWS Trust relationships tab. You can see the Cutover AWS environment(992112038979) has been set up to assume this role.
Once that's done, you can now configure the FIS integration via the Cutover Custom Integration portal.
Steps for creating a Custom integration
In order to create a Custom integration, you will need to follow the steps listed below - each step will be explained in more detail further in the guide:
1. Set up your integration connection
2. Add in an integration action
3. Add in your integration settings in the General tab
4. Create your Custom Fields
5. Include the authorization type to be used by the integration requests in the Authorization tab
6. Add in your request properties in the Request tab
7. Fill in any polling settings in the Polling tab - (if you are using polling)
Create a custom integration connection
Once the prerequisites are met, you can create your custom integration in Cutover by following these steps:
1. Click Settings (the cog icon at the bottom of the sidebar), then click Integrations.
2. Any previously configured custom integrations will be displayed on the Integrations Connections page.
3. To find out how to set up a custom integration or see step by step guides for some of our most requested integrations, click on Learn more at the top of the Integrations Connections page.
4. If you’re ready to build your integration, click Create Integration.
5. The New Integration Connection modal is shown. You can choose from two types of integration:
Custom integration: This type of integration gives you the flexibility to create integrations based on your requirements via any authorization.
Predefined integration: This type of integration has been set up with predefined fields and default authorization.
To find out more about the types of integration, click the appropriate radio button and then click the Learn more button inside the New Integration Connection modal window.
6. If you select the Predefined Integration radio option, select an integration from the Integration dropdown list. The information required will change depending on the integration chosen. Enter any mandatory fields and click Create.
The newly created integration will be listed at the bottom of the Integrations Connections page.
Note: As this guide is based on our Custom Integration, please see our Help Center for further information on setting up any predefined integrations.
7. If you select the Custom Integration radio option, you will need to enter a name and (optionally) an image URL for your integration (if this is left blank, a default image URL will be displayed). Click CREATE.
The newly created integration will be listed at the bottom of the Integrations Connections page.
Add an integration action
This section will cover the following example where you can use a Cutover task to trigger an experiment in FIS.
To do this, add an integration action to your newly created custom integration connection by following these steps:
1. In Settings > Integrations, click on your new custom integration from the list in the Integration Connections screen.
2. In the Edit panel, click + New in the Integration Actions section.
3. In the new action modal, select Build a custom integration from the Action dropdown.
General tab
In the General tab, enter the following details:
Name: Enter a name for your integration action. In our configuration, we have used the integration action name FIS-TriggerTemplate.
Image URL: This is the image of the icon that is associated and displayed with the integration action. We would use a URL for our FIS image, you are able to choose an image url of your choice.
Trigger: Select On Task Start.
Visibility: Select which workspace you would like your integration action to appear or select global for your integration to appear in all workspaces.
Note: You cannot change the visibility of an integration action once it has been created. Also, you will need to set up the integration action for every workspace it is required for.
Create Custom Fields
In order to create your custom fields, you will need to make sure you have the Custom Fields Admin role.
Note: You can read more about creating custom fields in our Help Center.
Below is a list of the custom fields, fields types and additional settings which should be entered when creating the custom fields in our example. You may want to add additional custom fields, depending on your requirements.
Once the above steps are completed, you can navigate back to your integration action - Settings > Integration connection > Integration Action settings.
Authorization tab
The Authorization settings tab allows you to define the type of authorization used by integration requests (in both request and polling phases). To get the best outcome from our Example integrations, we’d like you to use your own authorization set up to fully benefit and see your integration fully working. You can see our Authorization pages on the Developer Portal to find out the options available to you and how to configure these.
Fill out your Authorization tab, in this example we are using AWS STS SigV4 authorization. To find out the service code for the next field click here and you can find the specific AWS service code that you need to use. Using our example, as FIS is running in London it will be fis.eu-west-2.amazonaws.com and the service code will be FIS. Add the ID of the AWS environment that the FIS integration is in into the External ID field and for the Role ARN field use the role you provisioned earlier in the Create Experiment Templates section. For the Region field, enter the region where the FIS experiment is deployed.
Request tab
For the Request tab, see the FIS docs to understand what the request syntax should be. At a minimum, we need clientToken and experimentTemplate Id for the Outbound payload. The client token needs to be a unique value that you send on each request to AWS, so you can use task.id since it will be a unique value that gets sent in the Outbound payload. Using the task id though will mean that the task cannot be refired, because the client token will no longer be unique. Also, you need to include the experimentTemplateId which will be the ID of the experiment you already defined in AWS. You can create this as a custom field in Cutover so that you can dynamically input and pass the template ID from a Cutover task. For further information, see our Custom Fields help center article.
1. Select your chosen request type, in this example we are using a HTTP POST request.
2. Fill in your URL: Enter the API endpoint URL of your AWS instance (for example: https://fis.eu-west-2.amazonaws.com/experiments
3. In the request header, use the information as displayed below:
4. In ‘Execution mode’, set this to Fire and forget.
If required, you can choose to map the outbound payload to additional custom fields but in our example we will just trigger the experiment.
Polling tab
There are options for Polling but we will not cover these as the experiment does not need to be polled once it has started.
EC2 Launch Set up
Once you have set up FIS, you need to simulate it in a runbook. This will show you what an FIS experiment would look like. As our example is based on the creation of an experiment template with an EC2 capacity limit, you will need to create another Custom integration to launch EC2 instances. This helps to enforce a EC2 capacity limit restriction on the experiment template you created.
To set up the EC2 launch integration, there is no prior setup needed in AWS as you will be launching the instances from Cutover. Following the same process as before; name your Custom integration and then set the Authorization for the instance.
Next steps
In this guide you have learned how to successfully build and set up the AWS Fault Injection Service (FIS) integration. You should thoroughly test your integration and address any issues that may arise during the testing phase.
We encourage you to maintain detailed documentation of your integration configuration and any changes you make in the future. This will be invaluable for troubleshooting and maintenance.
If you are interested in integrations and would like to create further integrations in Cutover, please get in touch with your Customer Success Manager (CSM).
If you would like to know more about Cutover please contact info@cutover.com.
Thank you for using this guide, and we wish you every success with your integration project. If you have any feedback or suggestions for improving this documentation, please feel free to send it to docs@cutover.com.
