Category: DevOps

Structuring YAML Pipelines and nested variables in Azure DevOps

When managing pipelines for large and complex repositories with multiple ‘Platforms’, each containing multiple apps and services, then the folder structure and variable strategy can be complicated. However, if this is done right, then the payoff for template reuse is dramatic.

Here, I outline my approach on the pipeline folder and YAML structure only. The variable structure allows for a full set of naming conventions to easily default across all your projects apps and delegate standards to organisation and platform levels. This, ideally, leaves only app specific configurations for your dev teams to manage and worry about.

This strategy rests on top of my general approach to structuring a mono-repository. For more details on that see Mono-repository folder structures.

Continue reading “Structuring YAML Pipelines and nested variables in Azure DevOps”

Mono-repository folder structures

Every developer has their own way of structuring their code base. There is no right or wrong way, but some strategies have at least had some logical thought ๐Ÿ˜‰

This is a sample of how I generally structure my mono-repos when they need to scale to many organisational platforms, apps, and projects.

Continue reading “Mono-repository folder structures”

How to publish a multi-page Azure DevOps Wiki to PDF (and pipeline it)

Although you can print a single page of your wiki to a PDF using the browser, it’s problematic when you have a more complex structured multi-page wiki and you need to distribute or archive it as a single file.

Fortunately thanks to the great initiative by Max Melcher and his AzureDevOps.WikiPDFExport tool, combined with Richard Fennell’s WIKI PDF Export Tasks, we can not only produce pretty good multi-page PDF’s of our Wiki’s, but to also create a Pipeline to automate their production.

The documentation for both these tools is good, but I have included here some additional tips and more complete steps to quickly get your pipelines setup.

Pre-requirements

To follow the steps outlined below you will need to:

  1. Download the latest azuredevops-export-wiki.exe from GitHub (or the source code and build it yourself)
    • I create a local folder like C:\MyApps\AzureDevOps-Export-Wiki\ and drop the EXE there. Then I can execute all my command lines and see the outputs there too.
  2. Add the WIKI PDF Export Tasks extension in the Visual Studio Marketplace to your Azure DevOps Organisation. Click here WIKI PDF Export Tasks – Visual Studio Marketplace

The Setup

Assume we have an Azure DevOps (Azdo) project called ‘MyAzdoProject‘. This has a default code repository with the same name and once created, a wiki repository called ‘MyAzdoProject.wiki‘.

You can clone this Wiki repo by selecting the ‘Clone wiki’ from the Wiki menu.

In the code repository, I have created a folder called /resources/wiki-pdf-styles/ to hold the

  • Header HTML template file
  • Footer HTML template file
  • CSS Style Sheet

In the Wiki, we may have documentation for several Apps and each may have several sections such as Architecture, UX design, Requirements notes etc..

For this illustration I am only wanting to output the Architecture pages and subpages for App1. So everything below /App1/Architecture/** in the wiki.

The Resource Files

My resource files are as follows (name of the files include the apps ‘Short Code’ ‘app1’ so each app can have independent files):

header-app1.html

<div style='padding-left: 10px; margin: 0; -webkit-print-color-adjust: exact; border-bottom:1px solid grey; color: grey; width: 100%; text-align: left; font-size: 6px;'>
Nicholas Rogoff - My Cool App 1 - Architecture
</div>

footer-app1.html

<div style='padding-right: 10px; padding-top:2px; margin: 0; -webkit-print-color-adjust: exact; border-top:1px solid grey; color: grey; width: 100%; text-align: right; font-size: 6px;'>Copyright Nicholas Rogoff 2023 |
 Printed: <span class='date'></span> | Page: </span><span class='pageNumber'></span> of <span class='totalPages'></span>
</div>

styles.css

body {
  font-family: "Segoe UI Emoji", Tahoma, Geneva, Verdana, sans-serif;
  font-size: 10pt;
}

h1 {
  font-size: 25pt;
  color: #521e59;
}

h2 {
  font-size: 20pt;
  color: #3b868d;
}

h3 {
  font-size: 15pt;
  color: #f39000;
}

h4 {
  font-size: 12pt;
  color: #ec644a;
}

img {
  max-width: 100%;
  max-height: 800px;
}

/* Workaround to add a cover page */
.toc {
  page-break-after: always;
}

/* target a span with class title inside an h1 */
h1 span.title {
  page-break-before: avoid !important;
  page-break-after: always;
  padding-top: 25%;
  text-align: center;
  font-size: 48px;
}

/* make tables have a grey border */
table {
  border-collapse: collapse;
  border: 1px solid #ccc;
}

/* make table cells have a grey border */
td,
th {
  border: 1px solid #ccc;
  padding: 5px;
}

The Command Line

You can manually run the azure-export-wiki.exe (download the latest from here Releases ยท MaxMelcher/AzureDevOps.WikiPDFExport (github.com)) locally on a clone of your wiki repository. This is useful not just to output the PDF, but also to quickly refine your customizations, such as, parameters, templates and CSS.

I have used the following parameters:

  • -p / –path
    • Path to the wiki folder. If not provided, the current folder of the executable is used.
  • -o / –output
    • The path to the export file including the filename, e.g. c:\output.pdf
  • –footer-template-path, –header-template-path
    • local path to the header and footer html files
  • –css
    • local path to the CSS file
  • –breakPage
    • Adds a page break for each wiki page/file
  • –pathToHeading
    • Adds a path to the heading of each page. This can be formatted in the CSS
  • –highight-code
    • Highlight code blocks using highligh.js
  • –globaltoc
    • This sets the title for a global Table of Contents. As you will see, I have used this, in combination with the CSS to add in a main header Title.

…so my final command line looks like this:

.\azuredevops-export-wiki.exe 
  -p "C:\GitRepos\MyAzdoProject.wiki\MyAzdoProject\App1\Architecture" 
  -o "output.pdf"  
  --footer-template-path "C:\GitRepos\MyAzdoProject\MyAzdoProject\resources\wiki-pdf-styles\footer-cdhui.html"  
  --header-template-path "C:\GitRepos\MyAzdoProject\MyAzdoProject\resources\wiki-pdf-styles\header-cdhui.html" 
  --css "C:\GitRepos\MyAzdoProject\MyAzdoProject\resources\wiki-pdf-styles\styles.css" 
  --breakPage 
  --pathToHeading 
  --highlight-code 
  --globaltoc "<span class='title'>Nicholas Rogoff Cool App 1 Architecture Wiki</span>" -v

You can run and refine this command line locally and generate the output.

You can also do a lot more styling with the CSS than I have done and refine it to your requirements. Just use the –debug flag in the command line above and the intermediate HTML file is produced. You can then see all the classes that you can play with.

The Pipeline

I decided to create a YAML Pipeline Template, as I often have many apps and extensive wiki documentation. Printing the whole Wiki to a PDF is not feasible, and hits limitations, so I have a several pipelines to output distinct parts of the wiki structure.

The YAML Task Template (publish-wiki-to-pdf-cd-task-template.yml)

Everything in this template is parameterized to allow flexible usage. You can also set the defaults and simplify the consuming pipelines.

# Task template for generating the PDF from the Wiki

parameters:
  - name: LocalWikiCloneFolder
    displayName: The local path to clone the wiki repo to
    type: string
    default: '$(System.DefaultWorkingDirectory)\wikirepo'
  - name: WikiRootExportPath
    displayName: The path in the Wiki to export to PDF
    type: string
  - name: ProjectShortCode
    displayName: The short code for the project. Used to pick up the custom headers and footers files
    type: string
  - name: CustomFilesPath
    displayName: The local path to the custom files on the build agent in the main repo
    type: string
    default: '\resources\wiki-pdf-styles\**'
  - name: PdfOutputFileName
    displayName: The filename for the output pdf. Do not include the extension
    type: string
    default: '$(ProjectShortCode)-Wiki.pdf'
  - name: WikiRepoUrl
    displayName: The URL of the Wiki repo
    type: string
    default: 'https://myorg@dev.azure.com/myorg/MyAzdoProject/_git/MyAzdoProject.wiki'
  - name: PdfTitleHeading
    displayName: The title heading for the PDF
    type: string
    default: 'Nicholas Rogoff - $(ProjectShortCode) - Wiki'
    
steps:
- task: CopyFiles@2
  displayName: Copy-Headers-Footers-Styles
  inputs:
    Contents: '$(CustomFilesPath)'
    TargetFolder: '$(System.DefaultWorkingDirectory)\styles\'
    OverWrite: true
  enabled: false
- task: WikiPdfExportTask@3
  displayName: Create-PDF
  inputs:
    cloneRepo: true
    repo: '$(WikiRepoUrl)'
    useAgentToken: true
    localpath: '$(LocalWikiCloneFolder)'
    rootExportPath: '$(WikiRootExportPath)'
    outputFile: '$(System.DefaultWorkingDirectory)\$(PdfOutputFileName).pdf'
    ExtraParameters: '--footer-template-path "$(System.DefaultWorkingDirectory)\resources\wiki-pdf-styles\footer-$(ProjectShortCode).html" --header-template-path "$(System.DefaultWorkingDirectory)\resources\wiki-pdf-styles\header-$(ProjectShortCode).html" --css "$(System.DefaultWorkingDirectory)\resources\wiki-pdf-styles\styles.css" --breakPage --pathToHeading --highlight-code --globaltoc "<span class=''title''>$(PdfTitleHeading)</span>" -v'
- task: PublishBuildArtifacts@1
  displayName: Publish-Artifact
  inputs:
    PathtoPublish: '$(System.DefaultWorkingDirectory)\$(PdfOutputFileName).pdf'
    ArtifactName: 'drop'
    publishLocation: 'Container'

The main pipeline

# Publishes the wiki to PDFs

trigger:
- none
pr: none

# schedules:
# - cron: "0 0 * * *"
#   displayName: Daily midnight wiki publish
#   branches:
#     include:
#     - main

#   always: true

pool:
  vmImage: windows-latest

# Setting the build number to the date as work-around to include in the Title as $(Build.BuildNumber)
name: $(Date:yyyy-MM-dd)

variables:
- name: projectShortCode
  value: 'app1'
- name: localWikiCloneFolder
  value: '$(System.DefaultWorkingDirectory)\wikirepo'
- name: wikiRootExportPath
  value: '$(localWikiCloneFolder)\MyAzdoProject\Projects\CDH-UI\Architecture'
- name: customFilesPath
  value: '\resources\wiki-pdf-styles\**'
- name: wikiRepoUrl
  value: 'https://myorg@dev.azure.com/m/MyorgyAzdoProject/_git/MyAzdoProject.wiki'
- name: pdfOutputFilename
  value: '$(ProjectShortCode)-Architecture-Wiki.pdf'
- name: pdfTitleHeading
  value: 'Nicholas Rogoff - $(ProjectShortCode) - Architecture Wiki $(Build.BuildNumber)'

steps:
- template: './templates/publish-wiki-to-pdf-cd-task-template.yml'
  parameters:
    LocalWikiCloneFolder: $(localWikiCloneFolder)
    WikiRootExportPath: '$(wikiRootExportPath)'
    ProjectShortCode: '$(projectShortCode)'
    CustomFilesPath: '$(customFilesPath)'
    PdfOutputFileName: '$(pdfOutputFilename)'
    WikiRepoUrl: '$(wikiRepoUrl)'
    PdfTitleHeading: '$(pdfTitleHeading)'

I have left in some running options here. The default is completely manual, but I have added for reference, commented out, the format for a scheduled operation, as well as on every change (not recommended).

I have also used the ‘name:’ (which is referenced as $(Build.BuildNumber)), to create a date in a format I wanted for the Header page.

When this pipeline runs the PDF artifact can be downloaded. You can obviously add a new step to copy the file to any destination that suits your requirements.

Azure Key Vault – List Secrets that have not been set

When working across multiple environments, with restricted access, it can by difficult tracking which Key Vault secrets have been configured and which still need values set.

On my projects I like to give the dev teams autonomy on the dev environments setting their own Key Vault Keys and Secrets (following an agreed convention).

Periodically we then copy these keys to the other environment Key Vaults (see my other post on https://home-5012866824.webspace-host.com/wordpress/2021/08/09/azure-key-vault-script-for-copying-secrets-from-one-to-another/ )

The following script can be run to output all the Key Vault secrets, or only show the ones that need configuration. It can also be set to output pure markdown like…

# hms-sapp-kv-qa-ne (2021-07-03 14:31:04.590)

| Secret Name                                           | Needs Configuration                    |
|-------------------------------------------------------|----------------------------------------|
| apim-api-subscription-primary-key                     | ******                                 |
| apim-tests-subscription-primary-key                   | ******                                 |
| b2c-client-id-extensions-app                          | ******                                 |
| b2c-client-id-postman                                 | ******                                 |
| saleforce-client-id                                   | Needs Configuration                    |
| salesforce-client-secret                              | Needs Configuration                    |
| sql-database-ro-password                              | ******                                 |
| sql-database-rw-password                              | ******                                 |

that can be copy and pasted into the Azure or Github Wiki too and displays like the table below when rendered.

hms-sapp-kv-qa-ne (2021-07-03 14:31:04.590)

Secret Name Needs Configuration
apim-api-subscription-primary-key ******
apim-tests-subscription-primary-key ******
b2c-client-id-extensions-app ******
b2c-client-id-postman ******
saleforce-client-id Needs Configuration
salesforce-client-secret Needs Configuration
sql-database-ro-password ******
sql-database-rw-password ******

The script will output ‘Needs Configuration’ for any Secret values that are either blank, contain ‘Needs Configuration’ or ‘TBC’.

Script below, aslo contains examples:

<#PSScriptInfo
.VERSION 1.1
.GUName 48b4b27a-b77e-41e6-8a37-b3767da5caee
.AUTHOR Nicholas Rogoff

.RELEASENOTES
Initial version.
#>
<# 
.SYNOPSIS 
Copy Key Vault Secrets from one Vault to another. If the secret name exists it will NOT overwrite the value. 
 
.DESCRIPTION 
Loops through all secrets and copies them or fills them with a 'Needs Configuration'.
Will not copy secrets of ContentType application/x-pkcs12

PRE-REQUIREMENT
---------------
Run 'Import-Module Az.Accounts'
Run 'Import-Module Az.KeyVault'
You need to be logged into Azure and have the access necessary rights to both Key Vaults.

.INPUTS
None. You cannot pipe objects to this!

.OUTPUTS
None.

.PARAMETER SrcSubscriptionName
This is the Source Subscription Name

.PARAMETER SrcKvName
The name of the Source Key Vault

.PARAMETER MarkdownOnly
Output Markdown Only

.PARAMETER NameOnly
Set to only copy across the secret name and NOT the actual secret. The secret will be populated with 'Needs Configuration'

.NOTES
  Version:        1.1
  Author:         Nicholas Rogoff
  Creation Date:  2021-08-09
  Purpose/Change: Refined for publication
   
.EXAMPLE 
PS> .\List-UnSet-KeyVault-Secrets.ps1 -SrcSubscriptionName $srcSubscriptionName -SrcKvName $srcKvName 
This will list the secrets keys in the specified key vault and diplay 'Needs Configuration' for those that are blank, contain 'Needs Configuration' or 'TBC'

.EXAMPLE 
PS> .\List-UnSet-KeyVault-Secrets.ps1 -SrcSubscriptionName $srcSubscriptionName -SrcKvName $srcKvName -MarkdownOnly
Use for updating the Wiki. This will list the secrets keys, as MarkDown ONLY, in the specified key vault and diplay 'Needs Configuration' for those that are blank, contain 'Needs Configuration' or 'TBC'


.EXAMPLE 
PS> .\List-UnSet-KeyVault-Secrets.ps1 -SrcSubscriptionName $srcSubscriptionName -SrcKvName $srcKvName -ShowSecrets
This will list the secrets keys in the specified key vault and show the secrets


#>
#---------------------------------------------------------[Script Parameters]------------------------------------------------------
[CmdletBinding()]
Param(
  [Parameter(Mandatory = $true, HelpMessage = "This is the Source Subscription Name")]
  [string] $SrcSubscriptionName,
  [Parameter(Mandatory = $true, HelpMessage = "The name of the Source Key Vault")]
  [string] $SrcKvName,
  [Parameter(Mandatory = $false, HelpMessage = "Output Markdown Only")]
  [switch] $MarkdownOnly,
  [Parameter(Mandatory = $false, HelpMessage = "Output the secrets set")]
  [switch] $ShowSecrets
)

#---------------------------------------------------------[Initialisations]--------------------------------------------------------

# Set Error Action to Silently Continue
$ErrorActionPreference = 'Continue'

#----------------------------------------------------------[Declarations]----------------------------------------------------------
# Any Global Declarations go here

$SecretsFound = @()

#----------------------------------------------------------[Functions]----------------------------------------------------------

# Inline If Function
Function IIf($If, $Right, $Wrong) { If ($If) { $Right } Else { $Wrong } }

#-----------------------------------------------------------[Execution]------------------------------------------------------------

$checked = 0
$failed = 0

if (!$MarkdownOnly) {
  Write-Host "======================================================"
  Write-Host ($(Get-Date -Format 'yyyy-MM-dd HH:mm:ss.fff') + " Starting to check secrets in " + $SrcKvName + "... ") -ForegroundColor Blue
  Write-Host "======================================================"
}
else {
  Write-Host ("# " + $SrcKvName + " (" + $(Get-Date -Format 'yyyy-MM-dd HH:mm:ss.fff') + ")") -ForegroundColor Blue
}
Write-Host ""



$sourceSecrets = Get-AzKeyVaultSecret -VaultName $SrcKvName | Where-Object { $_.ContentType -notmatch "application/x-pkcs12" }

if (!$MarkdownOnly) {
  # Headers
  Write-Host -NoNewline "|".PadRight(60, "-")
  Write-Host -NoNewline "|".PadRight(60, "-")
  Write-Host "|"
}

Write-Host -NoNewline "| Secret Name".PadRight(60)
if (!$ShowSecrets) {
  Write-Host -NoNewline "| Needs Configuration".PadRight(60)  
}
else {
  Write-Host -NoNewline "| Secret Value".PadRight(60)      
}
Write-Host "|"

Write-Host -NoNewline "|".PadRight(60, "-")
Write-Host -NoNewline "|".PadRight(60, "-")
Write-Host "|"

ForEach ($sourceSecret in $sourceSecrets) {
  $Error.clear()

  $name = $sourceSecret.Name
  $plainTxtSecret = Get-AzKeyVaultSecret -VaultName $srckvName -Name $name -AsPlainText

  if ($plainTxtSecret -eq "Needs Configuration" -or $plainTxtSecret -eq "TBC" -or !$plainTxtSecret) {
    $secretToShow = $plainTxtSecret
  }
  elseif ($ShowSecrets) {
    $secretToShow = $plainTxtSecret
  }
  else {
    $secretToShow = "******"
  }

  Write-Host -NoNewline "| $name".PadRight(60)
  if ($secretToShow -eq "Needs Configuration" -or $plainTxtSecret -eq "TBC" -or !$plainTxtSecret) {
    Write-Host -NoNewline "| $secretToShow".PadRight(60) -ForegroundColor Magenta  
  }
  else {
    Write-Host -NoNewline "| $secretToShow".PadRight(60) -ForegroundColor DarkGray      
  }
  Write-Host "|"

  if (!$Error[0]) {
    $checked += 1
  }
  else {
    $failed += 1
    Write-Error "!! Failed to get secret $name"
  }
}

if (!$MarkdownOnly) {
  Write-Host -NoNewline "|".PadRight(60, "-")
  Write-Host -NoNewline "|".PadRight(60, "-")    
  Write-Host "|"

  Write-Host ""
  Write-Host ""
  Write-Host "================================="
  Write-Host "Completed Key Vault Secrets Copy"
  Write-Host "Checked: $checked"
  Write-Host "Failed: $failed"
  Write-Host "================================="
}
else {
  Write-Host ""
  Write-Host ("**Total Secrets Listed: " + $checked + "**")
  if ($failed -gt 0) {
    Write-Host "Failed: $failed" -ForegroundColor Red
  } 
  Write-Host ""
}

Cloud Resource Naming Convention (Azure)

In any organisation it is important to get a standard naming convention in place for most things, but especially with cloud-based resources.

As many types of cloud resources require globally unique names (due to platform DNS resolution), it’s important to have a strategy that will give you a good chance of achieving global uniqueness, but also as helpful as possible to human beings, as well as codifiable in DevOps CD pipelines.

Continue reading “Cloud Resource Naming Convention (Azure)”