PowerShell to build AWS Lambda Zip for Upload

I’m not sure why I wrote this in PowerShell instead of Python, as I’m using to package-up a Python program for AWS Lambda. To deploy code to Lambda, you have to create a zip file to upload. I found in the past that the Powershell zip for some odd reason is not compatible with AWS Lambda, so I use 7-zip.

So basically, I test my AWS program on a local machine (outside of the Lambda environment), then I run one of the scripts below to build the deploy. It also puts the date/time on the zip, so you have an archive of each of your past deploys (something AWS doesn’t seem to provide as of now). Thus if you need to go back to a prior version, you still have the zipped source.

I’m using the c:\Backup directory for the location of the .zip files to upload.

Certain libraries like Numpy, Pandas (which uses Numpy), and PYTZ (time zone) were apparently written in C (not Python, even though they are “Python libraries”, and since AWS runs on Unix, you have to zip and upload the correct Unix version of these libraries. To get those versions, see “aws-data-wrangler”

I exclude the ./data directory. When testing locally, I often output large test files there, and there is no need to upload them to AWS Lambda. This is done by the exclusion on the 7z.exe command (-x!data\*).

Here’s a simpler script when you don’t need the Unix compiled libraries:


<pre>#$ZipFileName = "c:\Users\nwalt\Downloads\TDAmeritradeGetQuotes.zip"
$fileDateTime = $(get-date -f "MM_dd_yyyy__HH_mm_ss")
$ZipFileName = "c:\Backup\PolygonDataUpdateIntervalsCurrent_Lambda_$fileDateTime.zip"
C:
cd c:\Apps\LambdaProcessData
#&amp; "E:\Program Files\7-Zip\7z.exe" a -r $ZipFileName index.js package.json client_secret.json GoogleCalendarAPI.js node_modules .credentials calendar-nodejs-quickstart.json
&amp; "C:\Program Files\7-Zip\7z.exe" a -x!data\* -r $ZipFileName *.* 
</pre>

Here’s a more involved script when you need Pandas, Numpy, and/or PYTZ which require the Unix compiled versions. “aws-data-wrangler” is a github library with the compiled versions that you need to download.

Files are copied to a staging area, certain files or directories are deleted, and then the AWS-Wrangler files are copied in. The “stagingArea” directory has date/time on it, which should typically avoid conflict with two people running it at the same time (on a server for example).

<pre>
cls
#
# Numpy must be the Linux version for deployment to AWS
# So we have to do some special tricks to package it. 
#
$fileDateTime = $(get-date -f "yyyy_MM_dd__HH_mm_ss")
write-host("ScriptRoot=" + $PSScriptRoot) 
$pathpartsArray = $PSScriptRoot -split "\\"  # escape character before \ 
write-host("Len=" + $pathpartsArray.Length) 
write-host($pathpartsArray)
$lowestDirName = $pathpartsArray[$pathpartsArray.Length-1]
write-host("lowestDirName=" + $lowestDirName) 
$ZipFileName = "c:\Backup\" + $lowestDirName + "_" + $fileDateTime + ".zip"
write-host("ZipFileName="  + $ZipFileName) 
#Remove-Item -Path c:\Apps\LambdaS3TriggerQuotesJsonToCsv_DeployStageArea -Recurse
$stageArea = $PSScriptRoot + "_StageArea_" + $fileDateTime
Write-Host "stageArea=$stageArea"
New-Item $stageArea -ItemType Directory  

# we need the /* on this one to copy the files/subfolders, but not the folder name itself
Copy-Item -Path $PSScriptRoot\* -Destination $stageArea -Recurse -Force
Write "Copied Source Code"

Remove-Item $stageArea\venv -Recurse      # this is hug, like 38MB, and apparently relates to Conda or other virtual environment manager, AWS doesn't allow more than 50 MB .zip file to be uploaded to Lambda 
Remove-Item $stageArea\numpy -Recurse
Remove-Item $stageArea\pandas -Recurse
Remove-Item $stageArea\pytz -Recurse

Copy-Item -Path "c:\Apps\awswrangler-layer-1.9.6-py3.8\python\numpy" -Destination $stageArea -Recurse  -Force
Copy-Item -Path "c:\Apps\awswrangler-layer-1.9.6-py3.8\python\numpy.libs" -Destination $stageArea -Recurse -Force
Copy-Item -Path "c:\Apps\awswrangler-layer-1.9.6-py3.8\python\numpy-1.19.2.dist-info" -Destination $stageArea -Recurse  -Force
Write "Copied Numpy"

Copy-Item -Path "c:\Apps\awswrangler-layer-1.9.6-py3.8\python\pandas" -Destination $stageArea -Recurse  -Force
Copy-Item -Path "c:\Apps\awswrangler-layer-1.9.6-py3.8\python\pandas-1.1.3.dist-info" -Destination $stageArea -Recurse  -Force
Write-Host "Copied Pandas" 

Copy-Item -Path "c:\Apps\awswrangler-layer-1.9.6-py3.8\python\pytz" -Destination $stageArea -Recurse  -Force
Copy-Item -Path "c:\Apps\awswrangler-layer-1.9.6-py3.8\python\pytz-2020.1.dist-info" -Destination $stageArea -Recurse  -Force
Write-Host "Copied Pytz" 

c: 
# go to the deployment directory that has the linux version of Numpy 
cd $stageArea 
dir 
# cd $PSScriptRoot # we need to be in the directory where the script is running 
# powershell zip doesn't work with Lambda, but 7zip does 
&amp; "C:\Program Files\7-Zip\7z.exe" a -x!data\* -r $ZipFileName *.* 

cd .. 
Write-Host "Cleaningup $stageArea"
Remove-Item $stageArea -Recurse

Write-Host "Completed: Created file: $ZipFileName"

</pre>

Leave a Reply