Harvesting Web Data using PowerShell

Grab the PowerShell and then you can type Get-Top100Songs 1983, passing in the year you’d like to see, and here’s the song list

The PowerShell

Harvesting the data is a two step process. The first step is using Invoke-WebRequest to grab the html we want to wrangle. You hit the pop culture website to get the top 100 songs for the year you want. Invoke-WebRequest has a property AllElements which returns, well, all the elements on that page. Piping this list of elements we can use Where to find only elements that have the TagName of tbody. The innerhtml property returns a string of html.

Now you can shift gears to the second and final step, parse the actual html using example based parsing. The $t variable contains is the template for parsing the html. This template is passed to the new PowerShell v5.0 ConvertFrom-String. The songs returned from the web site are structured. Each song sits in a <TR></TR>, then the position, group and song sit in <TD> tags. Using the ConvertFrom-String template language, you mark the data so the example can be used to parse the html. This is marking the example {pos:} {group:} and {song:}. Also included in the example template, are records that are different. This lets ConvertFrom-String properly construct a domain specific language to parse the data.

Post Messages to Slack with PowerShell

There are several PowerShell approaches out there that let you post messages to a Slack channel.

Posting a “Deployment Complete” notifications to a Slack channel is an extremely efficient way to communicate to the team.

That’s why I built a PowerShell Slack DSC Provider

Using PowerShell’s Desired State Configuration is a key automation strategy. Manual efforts for testing, environment creation, and release management can significantly delay delivering value to customers.

Now, with 5 lines of PowerShell, you can post messages to your Slack channels at any point in your DSC pipeline.

Grab the Provider

If you’re running PowerShell v5.0? Grab it from the PowerShell Gallery

Install-Module -Name SlackDSCResource

Or check it out on GitHub

Create a PowerShell object from $PSBoundParameters

The $PSBoundParameters automatic variable contains all of the parameter names and values that are used when a script or function is run.

Sometimes you want to convert it to a object and have all those parameters (with values) added to it.

Here is a one liner. @{} is an empty hashtable and you add it to the $PSBoundParameters. Cast this is using [pscustomobject], and you’re done.

Prints

Update

Brian Scholer tweeted me saying this approach doesn’t work when you use it in a constrained runspace. No worries, here’s the solution:

Rather than cast it to PSCustomObject use New-Object PSObject -Property

Catch up with Brian on his blog or twitter.

PowerShell ConvertFrom-String: Serious Text wrangling

There’s a new PowerShell cmdlet ConvertFrom-String released with PowerShell v5.0. There are a bunch of write ups on using this cmdlet and I want to show how it makes quick work of HTML source.

HTML Source

Sometimes you’ll get html that looks like the snippet below (remember, it could be hundreds an hundreds of lines of HTML). So editing/transforming it by hand would take quite some time.

The Transform

Let’s say we wanted to go from the HTML above, to this:

I’ve written code (or used a macro recorder in a text editor) to find the first ‘>’, delete the text to the left, find the ‘(‘ grab the text I want, etc.

The challenge is, not all the people I work with know how to do this. Plus, there are many other (mundane) text reformatting tasks that people go through every day.

Enter ConvertFrom-String

The key here is the $template on starting on line 17. I’m using ConvertFrom-String to do example-driven parsing. The template provides the example (hints to ConvertFrom-String on what I want extracted).

I put curly braces around the data I want to extract, and give it a name Item and Count. The * tells ConvertFrom-String this should result in multiple records.

image

image

The data is piped to ConvertFrom-String, parsed an then piped to ForEach which does the final transform.

That, is slick and easy.

Note: I’m providing the data and template in the code. Both the data and template can be in separate external files so script could be run to do transforms over many inputs.

Check Out ConvertFrom-String Buddy

I created a GUI (Using PowerShell and WPF), you can get the script HERE.

It lets you quickly and easily experiment with ConvertFrom-String.

Paste the data you want to transform in the data text box (on the left). Start typing the example template in the template text box (on the right). As you type, you’ll immediately see results in the result text box.

Plus, it generates the PowerShell code as you go. You can copy that to the clipboard and save it as a script for later.

PowerShell and The Github Atom Editor

Yet another editor. Easy to install, run and start customizing.

At GitHub, we’re building the text editor we’ve always wanted. A tool you can customize to do anything, but also use productively on the first day without ever touching a config file. Atom is modern, approachable, and hackable to the core. We can’t wait to see what you build with it.

Install & Configure

The Atom Editor

Atom from Chocolatey (cinst Atom). PowerShell v5.0 preview ships with an early version of OneGet, which has a Chocolatey package provider, but it fails to install the Atom editor.

The Atom Runner

The Atom Editor comes with the Atom Package Manger (apm). With it, you can install more packages. The first one is the atom runner.

image

Configure The Runner

Edit the config.cson file and add these three lines (note: this is 32bit PowerShell):

‘runner’:
‘extensions’:
‘ps1’: ‘powershell’

image

The PowerShell Syntax Highlighter

Atom has a package manger, apm, use it to install the PowerShell syntax highlighter.

image

More Atom Packages

Here is the web page, https://atom.io/packages.

image

PowerShell and Atom Editor in Action

From PowerShell, type atom and press enter.

image

Add some PowerShell 1..5 | % {$_ * 2} and save the file (i.e testAtom.ps1).

image

Next, press Ctrl+Shift+P and type r

image

Then, press Enter

image

Wrap Up

  • We installed the Atom Editor
  • Then, installed two packages with apm. Atom-Runner and Language-PowerShell
  • This enabled syntax highlighting and the ability to run PowerShell scripts in the editor

These are the early days but the editor is looking interesting and worth keeping an eye on.

Day one, in very little time, PowerShell integration was up an running.

Bonus

Steve Murawski, Chris Hunt and I were tweeting about running 64bit PowerShell, Pester Tests and PowerShell snippets.

64 bit

‘runner’:
‘extensions’:
‘ps1′:’c:\\windows\\sysnative\\windowspowershell\\v1.0\\powershell.exe –file’

Running PowerShell Pester Tests

‘runner’:
‘extensions’:

‘Tests.ps1’:’c:\\windows\\sysnative\\windowspowershell\\v1.0\\powershell.exe -command Invoke-Pester -path ‘

Atom Snippets

Atom PowerShell snippets on GitHub.

Part II – PowerShell: Shifting Arrays

ShiftingArrays

In my first post, PowerShell – Shuffling and Shifting Arrays, I demoed shifting an array in the following way:

Jim Truher co-creator of PowerShell

Jim Truher  saw the post and upped the anti. (He tried to post this as a comment but my blog didn’t work).

He used PowerShell’s  extended type system to add the shiftunshiftpop and push methods to arraylists.

For bonus points, he sent Pester Unit Tests too. This made it a snap for me to tweak the shift and pop methods to handle an array as well as single items.

Examples

Shifting:

UnShifting:

PowerShell Script and Tests

Grab the PowerShell scripts and BDD tests on GitHub

PowerShell – Shuffling and Shifting Arrays

image

I was looking at some ruby code and bumped up against these shuffle and shift. I wanted to reproduce this in PowerShell.

Shuffling

Ruby

First I implemented Fisher-Yates shuffle and then came across this version: Easiest Way to Shuffle an Array in PowerShell.

You learn something new everyday.

As a function

Shifting

The shift method removes the first element of the array, returns it, shifting all other elements down by one. It returns null if the array is empty.

Ruby

The while loop executes, pulling one entry off at a time until there are no more items, then it will break.

Here it is in PowerShell:

Line 3 is great. It pulls off the first element and puts it in $d and returns the rest of the array, placing it back in $dirs. In PowerShell we need the if check to see if we need to break.

PowerShell Show-Map

This PowerShell function launches a map in the browser using an address from the command line or the clipboard.

Plus, it checks which version of PowerShell is running and if it is v5 or later, will use the new Get-Clipboard cmdlet. If it is an older version of PowerShell, it will use .NET to get the text from the clipboard.

The default map url is Google, specify the -UseBing switch parameter to launch the map using Bing.