Machine Migration
This document details how to migrate machines, their state, profiles, jobs and/or workorders from one DRP endpoint to another.
Prerequisites¶
This feature was introduced in tip, prior to v4.11. You will need tip, v4.11 or greater versions of digital-rebar and drpcli.
The source and target endpoints must be configured with Multi Site Manager. Please see .
Operation¶
There are two endpoints with the following machines.
[admin@drp1 ~]$ drpcli machines list | jq -r '.[].Name'
birdy
drp1.rackn
drp2.rackn
[admin@drp2 ~]$ drpcli machines list | jq -r '.[].Name'
drp2.rackn
Here is an example machine to migrate from drp1 to drp2.
[admin@drp1 ~]$ drpcli machines show Name:birdy
{
"Address": "",
"Arch": "amd64",
"Available": true,
"BootEnv": "sledgehammer",
"Bundle": "",
"Context": "",
"CurrentJob": "1ed352bc-1303-63f2-94d2-19d9b7aba263",
"CurrentTask": 2,
"Description": "",
"Endpoint": "",
"Errors": [],
"Fingerprint": {
"CSNHash": "",
"CloudInstanceID": "",
"MemoryIds": [],
"SSNHash": "",
"SystemUUID": ""
},
"HardwareAddrs": [],
"Locked": false,
"Meta": {
"feature-flags": "change-stage-v2",
"machine-role": "machine"
},
"Name": "birdy",
"OS": "",
"Params": {},
"Partial": false,
"PendingWorkOrders": 0,
"Pool": "default",
"PoolAllocated": false,
"PoolStatus": "Free",
"Profiles": [
"example-profile",
"conflicting-profile"
],
"ReadOnly": false,
"RetryTaskAttempt": 0,
"Runnable": true,
"RunningWorkOrders": 0,
"Secret": "h5kYqfCu6ehiAVz0",
"Stage": "hello-world",
"TaskErrorStacks": [],
"Tasks": [
"stage:hello-world",
"hello-world"
],
"Uuid": "01d8828f-2f7d-4a1c-a1fa-f668274c7c44",
"Validated": true,
"WorkOrderMode": false,
"Workflow": "hello-world",
"WorkflowComplete": true
}
Some jobs have been created that are associated with the machine.
[admin@drp1 ~]$ drpcli jobs list Machine=01d8828f-2f7d-4a1c-a1fa-f668274c7c44
[
{
"Archived": false,
"Available": true,
"BootEnv": "sledgehammer",
"Bundle": "",
"Context": "",
"Current": false,
"CurrentIndex": 0,
"EndTime": "2022-09-15T15:22:35.937596048-04:00",
"Endpoint": "",
"Errors": [],
"ExitState": "complete",
"Machine": "01d8828f-2f7d-4a1c-a1fa-f668274c7c44",
"Meta": {
"feature-flags": "change-stage-v2",
"machine-role": "machine"
},
"NextIndex": 1,
"Previous": "00000000-0000-0000-0000-000000000000",
"ReadOnly": false,
"ResultErrors": null,
"Stage": "hello-world",
"StartTime": "2022-09-15T15:22:35.937594733-04:00",
"State": "finished",
"Task": "stage:hello-world",
"Uuid": "1ed352bc-1301-695a-94d2-8dc4daeea809",
"Validated": true,
"Workflow": "hello-world"
},
{
"Archived": false,
"Available": true,
"BootEnv": "sledgehammer",
"Bundle": "",
"Context": "",
"Current": true,
"CurrentIndex": 1,
"EndTime": "2022-09-15T15:23:04.49280118-04:00",
"Endpoint": "",
"Errors": [],
"ExitState": "",
"Machine": "01d8828f-2f7d-4a1c-a1fa-f668274c7c44",
"Meta": {
"feature-flags": "change-stage-v2",
"machine-role": "machine"
},
"NextIndex": 2,
"Previous": "1ed352bc-1301-695a-94d2-8dc4daeea809",
"ReadOnly": false,
"ResultErrors": null,
"Stage": "hello-world",
"StartTime": "2022-09-15T15:22:35.938191239-04:00",
"State": "finished",
"Task": "hello-world",
"Uuid": "1ed352bc-1303-63f2-94d2-19d9b7aba263",
"Validated": true,
"Workflow": "hello-world"
}
]
The machine must not have WorkOrderMode set to true in order to migrate. Migrate the machine.
[admin@drp1 ~]$ drpcli machines migrate Name:birdy to "$(drpcli -E https://drp2:8092 machines migrateTicket)"
Machine Name:birdy migrated to https://drp2:8092
The machine has migrated. Notice the drp1 machine has been locked.
[admin@drp2 ~]$ drpcli machines list | jq -r '.[].Name'
birdy
drp2.rackn
$ drpcli machines show Name:birdy
{
"Address": "",
"Arch": "amd64",
"Available": false,
"BootEnv": "sledgehammer",
"Bundle": "",
"Context": "",
"CurrentJob": "1ed352bc-1303-63f2-94d2-19d9b7aba263",
"CurrentTask": 2,
"Description": "",
"Endpoint": "",
"Errors": [],
"Fingerprint": {
"CSNHash": "",
"CloudInstanceID": "",
"MemoryIds": [],
"SSNHash": "",
"SystemUUID": ""
},
"HardwareAddrs": [],
"Locked": false,
"Meta": {
"feature-flags": "change-stage-v2",
"machine-role": "machine"
},
"Name": "birdy",
"OS": "",
"Params": {},
"Partial": false,
"PendingWorkOrders": 0,
"Pool": "default",
"PoolAllocated": false,
"PoolStatus": "Free",
"Profiles": [
"example-profile",
"conflicting-profile"
],
"ReadOnly": false,
"RetryTaskAttempt": 0,
"Runnable": false,
"RunningWorkOrders": 0,
"Secret": "h5kYqfCu6ehiAVz0",
"Stage": "hello-world",
"TaskErrorStacks": [],
"Tasks": [
"stage:hello-world",
"hello-world"
],
"Uuid": "01d8828f-2f7d-4a1c-a1fa-f668274c7c44",
"Validated": true,
"WorkOrderMode": false,
"Workflow": "hello-world",
"WorkflowComplete": true
}
The machine object on drp1 no longer has tasks, profiles, params, etc. Under Meta, we now have a redirect-endpoint
that tells the old endpoint where it should redirect to.
[admin@drp1 ~]$ drpcli machines show Name:birdy
{
"Address": "",
"Arch": "amd64",
"Available": true,
"BootEnv": "local",
"Bundle": "",
"Context": "",
"CurrentJob": "",
"CurrentTask": 0,
"Description": "Migrated to https://drp2:8092",
"Endpoint": "",
"Errors": [],
"Fingerprint": {
"CSNHash": "",
"CloudInstanceID": "",
"MemoryIds": [],
"SSNHash": "",
"SystemUUID": ""
},
"HardwareAddrs": [],
"Locked": true,
"Meta": {
"feature-flags": "change-stage-v2",
"redirect-endpoint": "https://drp2:8092"
},
"Name": "birdy",
"OS": "",
"Params": {},
"Partial": false,
"PendingWorkOrders": 0,
"Pool": "",
"PoolAllocated": false,
"PoolStatus": "",
"Profiles": [],
"ReadOnly": false,
"RetryTaskAttempt": 0,
"Runnable": false,
"RunningWorkOrders": 0,
"Secret": "h5kYqfCu6ehiAVz0",
"Stage": "none",
"TaskErrorStacks": [],
"Tasks": [],
"Uuid": "01d8828f-2f7d-4a1c-a1fa-f668274c7c44",
"Validated": true,
"WorkOrderMode": false,
"Workflow": "",
"WorkflowComplete": false
}
The jobs have migrated as well.
[admin@drp2 ~]$ drpcli jobs list Machine=01d8828f-2f7d-4a1c-a1fa-f668274c7c44
[
{
"Archived": false,
"Available": true,
"BootEnv": "sledgehammer",
"Bundle": "",
"Context": "",
"Current": false,
"CurrentIndex": 0,
"EndTime": "2022-09-15T15:22:35.937596048-04:00",
"Endpoint": "",
"Errors": [],
"ExitState": "complete",
"Machine": "01d8828f-2f7d-4a1c-a1fa-f668274c7c44",
"Meta": {
"feature-flags": "change-stage-v2",
"machine-role": "machine"
},
"NextIndex": 1,
"Previous": "00000000-0000-0000-0000-000000000000",
"ReadOnly": false,
"ResultErrors": null,
"Stage": "hello-world",
"StartTime": "2022-09-15T15:22:35.937594733-04:00",
"State": "finished",
"Task": "stage:hello-world",
"Uuid": "1ed352bc-1301-695a-94d2-8dc4daeea809",
"Validated": true,
"Workflow": "hello-world"
},
{
"Archived": false,
"Available": true,
"BootEnv": "sledgehammer",
"Bundle": "",
"Context": "",
"Current": true,
"CurrentIndex": 1,
"EndTime": "2022-09-15T15:23:04.49280118-04:00",
"Endpoint": "",
"Errors": [],
"ExitState": "",
"Machine": "01d8828f-2f7d-4a1c-a1fa-f668274c7c44",
"Meta": {
"feature-flags": "change-stage-v2",
"machine-role": "machine",
"redirect-endpoint": "https://drp2:8092"
},
"NextIndex": 2,
"Previous": "1ed352bc-1301-695a-94d2-8dc4daeea809",
"ReadOnly": false,
"ResultErrors": null,
"Stage": "hello-world",
"StartTime": "2022-09-15T15:22:35.938191239-04:00",
"State": "finished",
"Task": "hello-world",
"Uuid": "1ed352bc-1303-63f2-94d2-19d9b7aba263",
"Validated": true,
"Workflow": "hello-world"
}
]
The original jobs on drp1 will still exist, however they now have the Meta.redirect-endpoint field set similar to the
machine on drp1.
$ drpcli jobs list Machine=01d8828f-2f7d-4a1c-a1fa-f668274c7c44
[
{
"Archived": true,
"Available": true,
"BootEnv": "sledgehammer",
"Bundle": "",
"Context": "",
"Current": false,
"CurrentIndex": 0,
"EndTime": "2022-09-15T15:22:35.937596048-04:00",
"Endpoint": "",
"Errors": [],
"ExitState": "complete",
"Machine": "01d8828f-2f7d-4a1c-a1fa-f668274c7c44",
"Meta": {
"feature-flags": "change-stage-v2",
"machine-role": "machine",
"redirect-endpoint": "https://drp2:8092"
},
"NextIndex": 1,
"Previous": "00000000-0000-0000-0000-000000000000",
"ReadOnly": false,
"ResultErrors": null,
"Stage": "hello-world",
"StartTime": "2022-09-15T15:22:35.937594733-04:00",
"State": "finished",
"Task": "stage:hello-world",
"Uuid": "1ed352bc-1301-695a-94d2-8dc4daeea809",
"Validated": true,
"Workflow": "hello-world"
},
{
"Archived": false,
"Available": true,
"BootEnv": "sledgehammer",
"Bundle": "",
"Context": "",
"Current": true,
"CurrentIndex": 1,
"EndTime": "2022-09-15T15:23:04.49280118-04:00",
"Endpoint": "",
"Errors": [],
"ExitState": "",
"Machine": "01d8828f-2f7d-4a1c-a1fa-f668274c7c44",
"Meta": {
"feature-flags": "change-stage-v2",
"machine-role": "machine",
"redirect-endpoint": "https://drp2:8092"
},
"NextIndex": 2,
"Previous": "1ed352bc-1301-695a-94d2-8dc4daeea809",
"ReadOnly": false,
"ResultErrors": null,
"Stage": "hello-world",
"StartTime": "2022-09-15T15:22:35.938191239-04:00",
"State": "finished",
"Task": "hello-world",
"Uuid": "1ed352bc-1303-63f2-94d2-19d9b7aba263",
"Validated": true,
"Workflow": "hello-world"
}
]
If a profile has the same name between two endpoints, the destination endpoint profile wins during migration.
[admin@drp1 ~]$ drpcli profiles show conflicting-profile
{
"Available": true,
"Bundle": "",
"Description": "",
"Documentation": "",
"Endpoint": "",
"Errors": [],
"Meta": {},
"Name": "conflicting-profile",
"Params": {
"drp1-only-param": true
},
"Partial": false,
"Profiles": [],
"ReadOnly": false,
"Validated": true
}
[admin@drp2 ~]$ drpcli profiles show conflicting-profile
{
"Available": true,
"Bundle": "",
"Description": "",
"Documentation": "",
"Endpoint": "",
"Errors": [],
"Meta": {},
"Name": "conflicting-profile",
"Params": {
"drp2-only-param": true
},
"Partial": false,
"Profiles": [],
"ReadOnly": false,
"Validated": true
}
Automated Machine Migration¶
The workflow migrate-machine will migrate a machine from one endpoint to another.
While the task attempts to verify both endpoints are correctly setup before migration, it is important that contexts are setup and configured. Specifically the drpcli-runner and ansible-runner contexts should be available and work on both endpoints.
The following parameters are required:
migrate-machine/new-drp-server-api-url: URL to the endpoint (https://ip:8092 as an example.)migrate-machine/new-drp-server-token: A token for the new endpoint that can create a machine migrateTicket. This token must be created by using thedrpcli users tokencommand against the new endpoint.
The following parameters are optional:
migrate-machine/skip-machine-migration: If set totrue, the task to move the machine object to the new endpoint will be skipped. This is primarily for machines that might have been manually migrated, but still need the agent updated to the new endpoint.migrate-machine/skip-machine-destroy: If set totrue, the stub machine object will be left on the old endpoint after migration. The default behavior is to leave the object on the old endpoint.migrate-machine/old-drp-server-token: A token for the old endpoint that is needed to destroy the machine object left on the old endpoint.
The following parameters should not be touched and are for internal processing:
migrate-machine/esxi-ssh-priv-key:migrate-machine/esxi-ssh-pub-key:
Because ESXi does not have the same drpcli-based agent, the workflow cannot leverage all of the features available for other operating systems. It is required that sshd is enabled and keys are configured to access root on the machine being migrated. If the agent is running on the machine, the workflow will attempt to start sshd, copy over the existing configuration if it exists, then place a temporary key. Once migrated, the ssh service would be cleaned up, if already enabled and configured, the backed up keys will be restored and the temporary key removed. If the ssh service wasn't started originally, it will be stopped and disabled.