Sunday, 14 February 2016

Bulk Upload/copy a Folder Structure and Files to Amazon S3 Bucket Using Windows Powershell

There are two ways to solve this problem:

Solution 1 - Write your own recursive function that iterate throughout all of your file and folder in your specified folder structure.
Solution 2 - Use a simple tool called "AWS Command Line Interface," provided by Amazon.

Lets now discuss both of these solutions in more  detail:

Solution 1.

In my previous post there is a perfect example of how you can write your own Recursive Function to List All Files in Folders and Sub-folders using Windows PowerShell. You can use the code as a base concept and create a functional code for uploading the folder structure to Amazon S3 bucket.

Quick tip here. According to AWS documentation "Amazon S3 has a flat structure with no hierarchy like you would see in a typical file system. However, for the sake of organizational simplicity, the Amazon S3 console supports the folder concept as a means of grouping objects. Amazon S3 does this by using key name prefixes for objects."

So you don't have to create folder in AWS S3 bucket before uploading the files. All you need to do is specify path to file e.g photos/abc.png and AWS will automatically create folder against the file abc.png

The downside of this approach is that it is a very slow process. For my folder (containing 2 155 Files, 87 Folders total size 32,8 MB), it took 41 minutes to upload everything on my AWS S3 bucket.

Solution 2.

Before we go through this approach we need to know, "What Is the AWS Command Line Interface?"

According to AWS CLI documentation "The AWS Command Line Interface is a unified tool to manage your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts."

In this Tutorial we will use AWS CLI tools (recommend approach).

Before we go further you need to keep in mind few things

- We will be using Microsoft Windows 7 or above.
- Make sure you download and install  AWS CLI tools in your Windows environment.

Here is the documentation for the S3 API via CLI:
http://docs.aws.amazon.com/cli/latest/reference/s3/index.html
If you read through the documentation, here is how our statement should look
aws s3 cp /tmp/foo/ s3://bucket/ --recursive --exclude "*" --include "*.jpg"

If you try do everything according to the documentation you will end up with the problem of setting up Environment variables (that's what happened to me at least). You need environment variables so setup AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION

Tip: The way its describe to set Environment Variables in AWS documentation is for example
AWS_ACCESS_KEY_ID =AK0BJUTDXP2PPTT
and this approach didn't work . So i end up using the different approach to set Environment Variables
[Environment]::SetEnvironmentVariable("AWS_ACCESS_KEY_ID","AK0BJUTDXP2PPTT ")

Final code
$region = "AWS-BUCKET-REGION"
$folder = "c:\MyDirectory\"
$bucket = "AWS-BUCKET"
$accessKey = "AWS-ACCESS-KEY"
$secretKey = "AWS-SECRET-KEY"
$version = "1.3"

[Environment]::SetEnvironmentVariable("AWS_ACCESS_KEY_ID",$accessKey)
[Environment]::SetEnvironmentVariable("AWS_SECRET_ACCESS_KEY",$secretKey)
[Environment]::SetEnvironmentVariable("AWS_DEFAULT_REGION",$region)

aws s3 cp $folder s3://$bucket/$version/ --recursive

/Adnan

Wednesday, 30 December 2015

Batch Rename All Files in a Folder to Lowercase in MS Windows

If you are not interested in external programs to renames all files in a folder to lowercase, there is a simple command for you.

- Open Command Prompt (cmd.exe) in Windows
- Go to the directory and run the following command

for /f "Tokens=*" %f in ('dir /l/b/a-d') do (rename "%f" "%f")

Note: This is not a recursive function, it will rename files to lowercase only on the directory where you will run the command.

Sunday, 8 November 2015

Wednesday, 7 October 2015

Wednesday, 9 September 2015

Saturday, 16 May 2015

How to Get Random Items from Array or List in C#

The simple way to get random item from an Array is to use the return value from random.next(0, array.length) as index to get value from the array.

var randomIndex = random.Next(0, Array.Length);
Console.Write(Array[randomIndex]);
The downside of the above code is it might return you item multiple times (repetition) as we don't keep track of items that we are getting from the source Array.

The easy approach is to consider Array as a deck of cards. We want the items to be 'shuffled' similar to a deck of cards, meaning avoiding any repetition. So we will use  a List<> for the source items, grab them at random and push them to a Stack<> to create the deck of items.

You can create a Stack from anything that is IEnumerable
var stack = new Stack(myList);
See MSDN: http://msdn.microsoft.com/en-us/library/76atxd68.aspx

However, the stack constructor will be using a loop internally, you just don't see it. So for understanding the purpose I will use an example to create Stack with loop.

public static Stack CreateShuffledDeck(IEnumerable values) {

var random = new Random();  var list = new List(values); var stack = new Stack();  while (list.Count > 0) {  // Get the next item at random. var randomIndex = random .Next(0, list.Count); var randomItem = list[randomIndex];  // Remove the item from the list and push it to the top of the deck. list.RemoveAt(randomIndex); stack.Push(randomItem ); }  return stack; } 
Now we have a solution to create a Shuffled Deck. We can now get random items out using Stack.Pop method . Popping something from the stack means "taking the top 'thing'" off the stack.

public static string[] RandomArrayEntries(string[] arrayItems, int count) {
var listToReturn = new List();

if (arrayItems.Length != count) {
var deck = CreateShuffledDeck(arrayItems);

for (var i = 0; i < count; i++) {
var arrayItems= deck.Pop();
listToReturn .Add(item);
}

return listToReturn .ToArray();
}

return arrayItems;
}
We can execute the above code as following:

var countriesArray = new string[] { "Sweden", "Pakistan", "United Kingdom", "Denmark", "Norway", "Finland" };

var newRandomAraay = RandomArrayEntries(countriesArray, 3);

/Adnan

Tuesday, 9 December 2014