Programming Journey Entry 8: Parallel Power Processing Pre-cursor in PS

dev log, nongamingprogramming, powershell, Programming 6 Minutes

All of these Programming Journey posts can be found in the associated category of this blog.

PowerShell, again

As mentioned in the last programming journey post, I started a simple-sounding PowerShell script to zip my Steam game library. And mostly finished up until it collapsed like a house of cards (checkmate) at which time I decided to start over but more correctly.

The right way

I’m almost positive the “right” way of writing such scripts is to plan out as much of the structure of the program first before even the first line is written. And only then, when details are defined and structured plan is in place, is the code implemented.

So I’m going to instead use a modified version of this approach that I think works better for me.

But before even that I will try and properly structure a new GitHub repo to be compatible with both incremental testing of the script as well as an automatic version increment system. Which I’ve never done before.

Technically the first step is finding a name, so here is what I came up with: Game Library Auto Archiver. Maybe not the best name but it’s what I’m going with. Actually I should probably search GitHub to make sure the name isn’t used already…I couldn’t find anything with that exact name.

Lets get started with Game Library Auto Archiver

I asked an LLM about the structure of the git repository and this is what it spit out. And since it seems right this is what I’m going with.

gamelibautoarchiver/
│── .github/workflows/     # (For GitHub Actions in the future)
│── Tests/                 # (Pester tests go here)
│   ├── gamelibautoarchiver.Tests.ps1
│── gamelibautoarchiver/ # (Your module folder)
│   ├── gamelibautoarchiver.psm1
│   ├── gamelibautoarchiver.psd1
│── .gitignore             # (Ignore unnecessary files)
│── README.md              # (Project documentation)
│── LICENSE                # (Optional, for open-source licensing)

I also learned of tool called Pester that can used with PS script in particular. So I installed Pester with the line:

Install-Module Pester -Scope CurrentUser -Force

which apparently succeeded (having installed version 5.7.1).

I doubt anybody cares about this part, but my method of creating a new repository is to start on GitHub and then clone that empty repository to my local computer.

Once the repo is on GitHub, I use the button on GitHub to copy the http URL of the repo to the clipboard and go to my ‘repo’ folder on PC, right click and select ‘clone repository’. This brings up a prompt with TortoiseGit with the clipboard copied repo address and a created local path where the repo will be stored. I then get to watch an animation of a turtle tortoise (I assume created in 1996) for a second while the repository is brought down.

I actually used an LLM for a couple things so far: I just copy/pasted some bullet points and short line about the function of the script for the read me. And an emoji-laced bullet list of eventual features.

I also pasted in the contents of the .gitignore file since it was there. This isn’t violating my no-LLM policy because I don’t care that much about the readme and .gitignore are going to contain what they are regardless of the source (me, LLM or a git client).

I haven’t added the ci.yml file yet, apparently for github automated testing and linting with github actions. I’m only vaguely aware of what that means. Looking at the contents of the suggested ci.yml file I don’t think it will cause any horrible messes yet so I’m going to put it in. I don’t have anything to test yet so it’s probably fine.

It seems I also need a third party module called PSScriptAnalyzer. Which I will install with:

Install-Module PSScriptAnalyzer -Force -Scope CurrentUser

This was successful.

Then invoke to confirm it works, or as some call it “verify the linter is working.

Invoke-ScriptAnalyzer -Path .\GameLibAutoArchiver.psm1 -Recurse

This did not produce any errors or apparently modify either of the files so I’m going to assume that’s good and the linter works.

Then I went to the Tests folder and modified gamelibautoarchiver.Tests.ps1 for the following:

Describe "GameLibAutoArchiver Module" {
    It "Module should exist" {
        Test-Path "$PSScriptRoot\..\GameLibAutoArchiver.psm1" | Should -Be $true
    }
}

I than ran this to confirm Pester is ready:

Invoke-Pester -Path .\tests

Well I couldn’t get it to work because my PS inexplicably has no value for the system variable $PSScriptRoot. So I’m not getting the results I should be getting from pester per the LLM. That’s alright. I’ll find a work around.

I managed to fix the issue with the $PSScriptRoot. I think the only issue is that I had a typo some where in my predefined variable paths. It took an LLM to point that out.

Developing a fresh workflow

I was going to develop a work flow and then a test script for getting to know write-progress. But before those two steps i needed a “reset test environment” type script. Which I didn’t think would take that long or be hard. But it did end up taking a while.

The good news is that this script is now working: first I found a game folder of roughly one gigabyte in size (horizon chase just happened to be there) and made a bunch of copies of that folder then renamed them to various game names. Then created two empty folders with some sort of game sounding names. So ten total folders, 8 of which are ~1 gigabyte in size. Those I stored in a “large games folders ref” folder.

If you’re wondering why I did that it’s so I can run tests over and over again. In other words if I have conditions under which a zip file will be deleted do I want to zip another folder manually to run the script again?

I did the same thing for destination reference folder but using a bunch of 0 byte “stub” files as I call them. These are zips with various date codes in the file names.

These reference folders are supposed to remain static perpetually, so I can populate and re-populate my actual source and destination folders the script will operate out of.

That’s where this setup-test-env (working file name) comes in: it deletes the contents of the folders, copies over the large folders, copies over the zips and adjusts the last write date of all the source folders so the script will be tested on dates.

If none of this making sense yet, it will be explained over time as I get to it. Actually the way it comes up with those last write dates is kind of clever in that it uses (Get-Date) and also math on that like (Get-Date).AddDays(5) in a for loop and get-item gets piped to set-itemproperty. For some reason I just thought that was clever.

Pre-progress Progress

I have finished the reset environment script or at least it’s good enough for now.

I also have a script for testing multi-zipping of folders. It’s relatively generic, not worrying about a naming scheme or anything. But after all that I’m not going to implement it right away as I had originally planned, I’m going to implement the it later. And also the LLM wrote it and even then it took a while. I asked it about using jobs instead of parallel and also using some kind of custom event to try and track each job of zipping a folder.

I also learned several more things about zipping lots of folders at once: even though the OS reports the my CPU count as 8, that doesn’t mean I can zip 8 folders at once or that it’s good great idea. I also learned the LLM won’t put in a “cancel all ” escape sequence unless specifically asked. Which is kind of important when developing such a script as an unsupervised multi-zip operation can lockup the script, PowerShell and possibly the OS.

So I ended up settling on 4 zip operations at once and that seems to work sufficiently well without overwhelming the hardware. And out of that came a formula for how many zip operations to default to relative to the number of OS reported processors available. So it wasn’t a waste of time as I ended up learning a lot I don’t think I would have learned had been doing this through a VM or just talking to an LLM without actually running on real world hardware.

Now, lets write a module

So to summarize, I created the repo with the directory structure, install two tools for testing scripts, created a script to reset environments back to a default for re-running the scripts and experimented with multi-zipping on real-world hardware.

The next entry will involve writing of an actual script.

Game Library Auto Archiver
GitHub Repo:
https://github.com/tildesarecool/Game-Library-Auto-Archiver

For those who may also be trying to learn PowerShell, I wanted to point out this the sticky thread on the PowerShell subreddit, What have you done with PowerShell this month?. You can go back through the months worth of these threads and find a lot of tricks and scripts you would not have otherwise come across. I don’t know if there’s a listing of them some place. The “beginner resources” page is pretty great, too.

Reference links:

Unknown's avatar

Published by tildesarecool

Published

2 thoughts on “Programming Journey Entry 8: Parallel Power Processing Pre-cursor in PS”

  1. Pingback: Programming Journey Entry 9: Planning the scrip out (an attempt was made) – Tildes on the side
  2. Pingback: Programming Journey Entry 9: Planning the script out (an attempt was made) – Tildes on the side

Leave a comment