GitHub

Ojitha Hewa Kumanayaka

Lua filters for Pandoc

Lua filter used in Pandoc 3.6.3. This blog has solutions for: Creating Glossary for ePub ver 3 book GitHub style alerts

AWS PITR Explained

PITR stands for Point-in-Time Recovery, which is a feature offered by several AWS services to provide continuous data protection and the ability to restore data to a specific point in time.

Spark - create database and table

This is a short note to create a Hive meta store using Spark 3.3.1.

Semantic search with ELSER in Elasticsearch

Elastic Learned Sparse EncodeR(ELSER) is a retrieval model trained by Elastic that enables you to perform semantic search to retrieve more relevant search results. install ELSER v2: Only once (DevOPs will do for your) Create source index where you can insert all your documents Create target index Create ingestion pipeline Reindex process to create embeddings Ready to do semantic search using text expansion queries I created this blog post on docker to demonstrate Linux-optimised ELSER v2. The Elasticsearch version is 8.11.1.

Kafka PySpark streaming example

The diagram shows that the Kafka producer reads from Wikimedia and writes to the Kafka topic. Then Kafka Spark consumer pulls the data from the Kafka topic and writes the steam batches to disk.

Terraform For each iteration

This is to explain Terraform for each looping technique. In this example, 3 buckets are created to demonstrate the looping idea. In the first step, we will create the above 3 buckets starting from 0.

Spark to create a table in AWS Redshift

In this post, Spark reads the data from a CSV file to a DateFrame and saves that DataFrame as a Redshift table. In addition to that, I’ve explained how to create a table in Postgres, use Jupyter magics and plot a diagram.

Spark Streaming Basics

This is a very basic example created to explain Spark streaming. Spark run on the AWS Glue container locally.

Spark Kafka Docker Configuration

This is the continuation of the Spark Streaming Basics. I explained the basic stream example, which runs only on one AWS Glue container. The stream producer was Netcat, and the sink was a text file. In this post, the stream producer is still Netcat, but the sink is Kafka. Both Kafka and Spark running on Docker containers.

Introduction to Lambda Calculus

This is a short description of lambda calculus. Lambda calculus is the smallest programming language that is capable of variable substitution and a single function definition scheme. Haskell is the functional programming language based on lambda calculus, which I will explore. I already explained how to use VSCode for Haskell Development to support the code listed here.

Comments

Post a Comment

commented your blog

Popular posts from this blog

How To: GitHub projects in Spring Tool Suite

Image
The first step is sign up in the GitHub and create a  new project. For example I’ve created a new project Spring3part7 in the GitHub. Before add the projects you need to configure STS for GitHub access. For example, you need to add configuration as shown in the following picture Git local repository also important Now you are ready to pull the project from the GitHub, in the STS import menu select the following Now type the project name and click the ‘Search’ as shown in the following. However, when you select the found project click ‘Finish’ nothing will change in the STS interface. Now you are going to create real project. Here I am going to create simple utility project. This template project need to be created inside the imported GitHub project You have to give the same project name of the GitHub project as shown in the following project Now your project is ready to push. Before that you need to add ignore flag to all the folders and files except pom...
Continue »

Parse the namespace based XML using Python

In this blog, I am considering how to parser and modify the xml file using python. For example, I need to parser the following xml 1 (slightly modified for this blog) and need to write the modified xml to out.xml file. Here the country.xml <?xml version="1.0"?> <actors xmlns:fictional="http://characters.example.com" xmlns="http://people.example.com"> <actor type='T1'> <name>John Cleese</name> <fictional:character>Lancelot</fictional:character> <fictional:character>Archie Leach</fictional:character> </actor> <actor type='T2'> <name>Eric Idle</name> <fictional:character>Sir Robin</fictional:character> <fictional:character>Gunther</fictional:character> <fictional:character>Commander Clement</fictional:character> </actor> </actors> In ...
Continue »

EJB 3 JUnit testing with OpenEJB 3.1.4

This is quick example of how to create JUnit test to test the EJB 3.0 session bean. The approach is new innovation using @LocalClient. This will create EJB client for Java SE. The interface of the calculator class is, package com.ojitha.calculator; import javax.ejb.Local; @Local public interface CalculatorLocal { public int add(int a, int b); } This is the implement of the Calculator package com.ojitha.calculator; import javax.ejb.Stateless; /** * Session Bean implementation class Calculator */ @Stateless public class Calculator implements CalculatorLocal { /** * Default constructor. */ public Calculator() { } @Override public int add(int a, int b) { return a + b; } } Let's look at the openEJB based JUnit test class package com.ojitha.calculator; import static org.junit.Assert.*; import java.util.Properties; import javax.ejb.EJB; import javax.naming.InitialContext; import org.apache.openejb.api.LocalClient; import org.junit.After; import o...
Continue »