Ojitha Hewa Kumanayaka

This blog will build AWS Virtual Private Cloud (VPC) creation using AWS Cloudformation template, step by step. However, it is necessary to plan your network with Classless Inter-Domain Routing (CIDR). Simplest VPC Planning Attach AWS Internet Gateway and Route Table Testing the VPC Simplest VPC Here this simplest workable CFN: Fig 1: Creates only VPC This creates only the VPC, as shown in the above diagram. AWSTemplateFormatVersion: "2010-09-09" Description: My VPC example Parameters: EnvironmentName: Description: prefix for the resources Type: String Default: oj-test Resources: VPC: Type: AWS::EC2::VPC Properties: CidrBlock: 10.192.0.0/24 EnableDnsSupport: true EnableDnsHostnames: true Tags: - Key: Name Value: !Ref EnvironmentName As shown in the CFN, the CIDR block is the same as Network Address Block in Fig 2. To create the stack aws cloudformation create-stack --template-body file://test.y...

Create a fundamental Glue workflow using the AWS Cloudformation template. The Glue workflow replaces the use of the Step functions, which have been used to maintain Glue flow states. However, if you plan to automate your build deployment, here is the blog post 1 to help you. In this post, I completely ignore the AWS BuildPipeline, which is the recommended CI/CD pipeline explained in the above post. AWS Cloudformation for workflow Run the workflow Query in Athena Cleanup AWS Cloudformation for workflow CFN stack with the workflow As shown in the above diagram, trigger action the Glue Crawler. The CFN template is as follows: AWSTemplateFormatVersion: '2010-09-09' # Sample CFN YAML to demonstrate creating a crawler # # Parameters section contains names that are substituted in the Resources section # These parameters are the names the resources created in the Data Catalog Parameters: GlueWorkflowName: Type: String Description: workflow...

To programmatically connect to the AWS Services, need to use endpoints. The AWS SDK and the CLI use the default endpoint of the region to access the service. The regional endpoint is protocol://service-code.region-code.amazonaws.com standard. But services such as IAM not support the region. Some example of calling 1 to AWS services via API calls. All CLI calls are signed by Signature version 4 2 . To list all the active regions: (if you want to sort add the | sort to the end. aws ssm get-parameters-by-path --path /aws/service/global-infrastructure/regions --query Parameters[].Name In the above command regions is the important word. This command will return the following: --------------------------------------------------------------- | GetParametersByPath | +-------------------------------------------------------------+ | /aws/service/global-infrastructure/regions/ap-northeast-1 | | /aws/service/global-infrastructure/regions/eu-c...

Here the first example using decorator in python 2.7 import unittest import random import mock def myrandom(p): return random.random() > p class Test(unittest.TestCase): @mock.patch('random.random') def test_myrandom(self, mock_random): mock_random.return_value = 0.1 val = myrandom(0.0) assert val > 0 assert mock_random.call_count == 1 if __name__ == '__main__': unittest.main() Here the example for assert_callled_with() function import unittest import mock import example class Test(unittest.TestCase): @mock.patch('example.hello') def test1(self,mock_hello): x = 'Oj' example.hello(x) # Uses patched example.func mock_hello.assert_called_with(x) if __name__ == '__main__': unittest.main() Above test can be ran using context manager: import unittest import mock import example class Test(unittest.TestCase): def test1(self): x = ...

You can simply create a Administrators group as follows in the cli aws iam create-group --group-name Administrators aws iam list-groups aws iam list-attached-group-policies --group-name Administrators You can login using account alias: aws iam create-account-alias --account-alias ojitha-yahoo ws iam list-account-aliases cheat kinesis stream aws kinesis create-stream --stream-name AccessLogStream --shard-count 1 create EMR cluster aws emr create-cluster --name demo --instance-type m3.xlarge --instance-count 2 --release-label emr-4.6.0 --ec2-attributes KeyName=ojitha-yahoo --use-default-roles --applications Name=Hive Name=Spark create a Redshift cluster: aws redshift create-cluster --cluster-identifier demo --db-name demo --node-type dc1.large --cluster-type single-node --master-username master --master-user-password Redshift123 --publicly-accessible --port 8192 download appender wget http://emr-kinesis.s3.amazonaws.com/publisher/kinesis-log4j-appender-1.0.0.jar ...

Intro security Layers: Routing NACL (Network Access Control Lists) security Groups Host based Firewall, IDS and IPS Routing security no support for " edge to edge routing "  NACL (Network Aecess control list) Applied to subnet as whole stateless (Don't remeber TCP sessions) As a result must specify ingress and egress rules When you create a vpc, Aws will create a default NALC allow all * Allow or Deny specify - Protocol (Tcp, UDP) - source destination IP Range - source destination PORT range (Eg: Ingress 80 and 22 VPN Egress 1024- 65535) Security Groups Applied to the instance. statefull therefore can specify ingress and egress rules: but not need to specify both. security Groups. all outbound ⇒ allowed, inbound ⇒ deny therefore only allow rules, no deny specify: - protocol (TCP, UD.P) . - Source IP and Port range for ingress - Destination EP and Port rang only for engress When VPC is created, default security group is cre...

I would like to recommend IP explained before reading this. How the internet is working This is an introduction only 1 . Take the name of the services and convert to physical locations on the internet. URL For example, http://ojitha.blogspot.com need to be read from right to the left. Is this case com in the domain and blog spot is the subdomain. There are 13 root services configured in these recursive servers such as .net .  The Anycast address is a simple IP address that represents a bunch of servers appear on same Ip address. you can find all the server addresses in the http://iana.org , but to find the locations of root servers visit http://root-servers.org , If the domain doesn't exist, then resolve to NxDOMAIN in the authoritative server which will be cashed: this is called negative cache. The edge providers, content providers, provide recursive servers, transit providers, and public institutions.  Internet Exchange point These are the most invisible ...

IP explained Review of IPV 4 32 bits this 32 bits are divided in to 8 sections such as 32 / 8 = 4. Review of IPV6 128 bits  undecillion Leading zeros can be dropped (2134::AD1:..) :: can be used to represent collection of zeros (o:O:o) classless inter demain Routing Notation (CIDR) half of the possibilities cut down when leading bit is increased by 1.  In this figure, upper part shows the \(2^{16}\) possible ip address and the bottom part shows the \(2^{8}\) which is the smallest. For example in the Aws VPC subnets largest is 16 and smallest range is 28. Private Network Ranges As a standard 8, 12 and 16 Leadiing bits are reserved for the internal used. 10.0.0.0 to 10.255.255.255 for 10.0.0.0/8 172.16.0.0 to 172.31. 255.25 for 172.16.0.0/12 192.168.0.0 to 192.168.255.255 for 192.168.0.0/16