# Harness Engineering for Java and Spring Boot Applications


I want to accelerate development using AI coding agents, but I don't want to give up on code quality to get there. Generating code faster is useful only if I can keep the application understandable, maintainable, and correct as it grows.

An agent can implement a feature, generate tests, and refactor several classes in a few minutes. Reviewing all those changes carefully still takes time. If every review requires me to point out formatting issues, explain package boundaries again, and ask for missing authorization tests, I have moved the bottleneck rather than removed it.

So I set up a harness to take care of as much code quality verification as possible. Formatting, coverage checks, architecture rules, and behavioral tests become part of the development workflow. The agent gets feedback from executable checks, and I can spend more of my review time on product behavior and design decisions.

{{< box info >}}
**Reference application: FeedbackHub**

I'll use [FeedbackHub](https://github.com/sivaprasadreddy/feedback-hub), a Spring Boot application for employee feedback and discussions, to show what this looks like. It has user management, messages, replies, voting, moderation, and AI-assisted content analysis. That gives us meaningful application with boundaries and business rules to protect.
{{< /box >}}

For my Java projects, the basic expectation is concrete:

```shell
./mvnw spotless:apply <-- should automatically format the code
./mvnw verify         <-- should verify coding standards, architecture violations and run behaviour tests
```

These commands don't prove that a change is perfect. They provide a repeatable baseline that every change must meet.

## Agent Skills: Teach Your Coding Standards to AI Agents

Before any check runs, the agent has to know how I like to build applications. Package structure, how to write REST APIs, how to test them, which build plugins to configure: I don't want to explain these in every prompt.

**Agent Skills** are a good fit. A skill is a folder with a `SKILL.md` file, and optionally supporting files, that the agent loads when the task matches its description. I keep my coding standards and architecture guidelines in skills, and the agent imports them when it works on a Spring Boot application.

I use the Spring Boot skills from [sivalabs-agent-skills](https://github.com/sivaprasadreddy/sivalabs-agent-skills). They capture how I structure packages, build modular monoliths, write REST APIs and persistence code, and test them. Install them once and every project starts from the same baseline instead of a prompt I rewrite from memory.

### Guidelines, Not Rules

Skills are **guidelines**, not rules. A skill is text that goes into the agent's context, and the agent decides how much weight to give it. Most of the time it follows the guidance. Sometimes it doesn't:

- The skill is never loaded because the agent didn't think the task matched its description.
- The agent loads it, then drifts from it in the middle of a long task as the context fills up.
- It follows the spirit of a guideline but not the letter. For example, it makes a repository package-private in one module and public in another.
- A direct instruction in the prompt wins over a skill, even when that instruction contradicts it.

**Nothing in a skill forces the agent to comply. That is the main difference between a skill and the build checks. A failing test blocks the build. A skill that says "keep repositories package-private" only makes it more likely.**

So I use both:

```text
Agent Skills        -> shape the first draft so fewer things need fixing
Executable  checks  -> catch what the agent ignored, misread, or forgot
```

A skill reduces the number of violations the agent produces. [Spotless](https://github.com/diffplug/spotless), [ArchUnit](https://www.archunit.org/), [Taikai](https://github.com/enofex/taikai), [Spring Modulith](https://spring.io/projects/spring-modulith), and the coverage gate are what verify the result. 

{{< box important >}}
**Golden Rule**

If a guideline matters enough that you would reject a pull request for breaking it, you should turn it into an executable rule.
{{< /box >}}

I also still review the diff against these standards. A skill saves me from repeating myself, but it doesn't remove my responsibility for the result.

## Give the Agent a Feedback Loop

**Harness Engineering** means designing the instructions, tools, checks, and feedback loops around an AI coding agent so that it can produce changes we can verify.

A prompt such as “follow best practices” expresses an intention. It doesn't tell us whether the resulting code follows our architecture or preserves an authorization rule.

A useful harness makes those expectations observable:

```text
Read the use case and repository conventions
    -> implement a focused change
    -> format the code
    -> run architecture and behavioral tests
    -> run coverage and quality checks
    -> inspect failures and repair the change
    -> review the final diff
```

Each part has a different job. Instructions explain how to work. Tests and build checks enforce the rules we can automate. Runtime tools help investigate behavior that requires a running application. Human review evaluates whether the solution makes sense.

We already have much of the tooling in the Java ecosystem. The interesting work is connecting it into a loop that an agent can use without waiting for a developer to interpret every routine failure.

## Stop Reviewing Linting/Formatting Changes

Formatting is an easy place to start because the expected result is deterministic. I don't want either a developer or an AI agent spending time arguing about import order and line wrapping.

FeedbackHub uses [Spotless](https://github.com/diffplug/spotless) with [Palantir Java Format](https://github.com/palantir/palantir-java-format) to automatically verify whether code formatted properly or not:

```xml
<plugin>
    <groupId>com.diffplug.spotless</groupId>
    <artifactId>spotless-maven-plugin</artifactId>
    <version>3.10.4</version>
    <configuration>
        <java>
            <importOrder/>
            <removeUnusedImports/>
            <formatAnnotations/>
            <palantirJavaFormat>
                <version>2.103.0</version>
            </palantirJavaFormat>
        </java>
    </configuration>
    <executions>
        <execution>
            <phase>compile</phase>
            <goals>
                <goal>check</goal>
            </goals>
        </execution>
    </executions>
</plugin>
```

With the plugin configured, you can use the following goals:

```shell
./mvnw spotless:apply   <-- formats the code
./mvnw spotless:check   <-- report violations and fails the build without fixing them
```
Binding the `check` goal to `compile` phase means builds enforce the formatting policy early.

## JaCoCo: Make Missing Test Coverage Visible

AI can generate tests quickly. That doesn't mean the tests exercise the important behavior.

FeedbackHub uses [JaCoCo](https://www.jacoco.org/jacoco/) to collect coverage, generate a report, and enforce a minimum of 80% line coverage across the application bundle.

The plugin binds the `report` and coverage `check` goals to `verify` phase:

```xml
<plugin>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <version>0.8.16</version>
    <executions>
        <execution>
            <goals>
                <goal>prepare-agent</goal>
            </goals>
        </execution>
        <execution>
            <id>report</id>
            <phase>verify</phase>
            <goals>
                <goal>report</goal>
            </goals>
        </execution>
        <execution>
            <id>check</id>
            <phase>verify</phase>
            <goals>
                <goal>check</goal>
            </goals>
            <configuration>
                <rules>
                    <rule>
                        <element>BUNDLE</element>
                        <limits>
                            <limit>
                                <counter>LINE</counter>
                                <value>COVEREDRATIO</value>
                                <minimum>80%</minimum>
                            </limit>
                        </limits>
                    </rule>
                </rules>
            </configuration>
        </execution>
    </executions>
</plugin>
```

This distinction matters when giving an agent verification instructions:

```shell
# Runs tests, but does not reach the verify-phase coverage gate.
./mvnw test

# Runs tests, and also generates the coverage report and checks the threshold.
./mvnw verify
```

Coverage is a useful signal, not a correctness score. A test can execute an authorization branch without checking whether the wrong user was rejected. A bundle-wide threshold can also hide an under-tested feature behind well-covered code elsewhere. I use coverage to identify gaps, then look at the assertions and acceptance criteria to judge whether those gaps matter.


## ArchUnit: Turn Architecture Decisions into Tests

Architecture tends to drift through small, convenient changes. A controller reaches directly into another module's repository. A domain service imports a web type. An entity becomes public because it is easier to pass around.
These are easy changes for an agent to make when its immediate goal is to get a feature working. 

[ArchUnit](https://www.archunit.org/) lets us encode boundaries as executable rules.

FeedbackHub's `ArchUnitTests` includes these rules:

```java
@AnalyzeClasses(packages = "dev.sivalabs.feedbackhub")
class ArchUnitTests {

    @ArchTest
    static final ArchRule repositoriesShouldNotBePublic =
            classes().that()
                    .areAssignableTo(Repository.class)
                    .should().notBePublic();

    @ArchTest
    static final ArchRule entitiesShouldNotBePublic =
            classes().that()
                    .areAnnotatedWith(Entity.class)
                    .should().notBePublic();

    @ArchTest
    static final ArchRule controllersShouldNotBeAccessedByDomain =
            noClasses().that()
                    .resideInAPackage("..domain..")
                    .should().accessClassesThat()
                    .resideInAPackage("..web..");

    // Services are public (accessible to inbound handlers in the same module),
    // but must only be accessed from within their own module — other modules must go through *API facades.
    @ArchTest
    static final ArchRule servicesShouldNotBeUsedFromOtherModules = classes()
            .that()
            .areAnnotatedWith(Service.class)
            .should(new ArchCondition<>("only be accessed from same module") {
                @Override
                public void check(JavaClass service, ConditionEvents events) {
                    String serviceModule = moduleOf(service);
                    for (Dependency dependency : service.getDirectDependenciesToSelf()) {
                        JavaClass origin = dependency.getOriginClass();
                        String originModule = moduleOf(origin);
                        if (!serviceModule.equals(originModule)) {
                            String message =
                                    origin.getName() + " accesses " + service.getName() + " from another module";
                            events.add(SimpleConditionEvent.violated(dependency, message));
                        }
                    }
                }
            });

    private static String moduleOf(JavaClass clazz) {
        String pkg = clazz.getPackageName();
        String[] parts = pkg.split("\\.");
        return parts[BASE_PACKAGE.split("\\.").length];
    }
}
```

The examples omit imports for readability. The test also has a custom rule preventing services from being accessed from other modules. Cross-module callers must use the module's API facade.

That design is visible in the users module. Here is an excerpt from `UsersAPI`:

```java
package dev.sivalabs.feedbackhub.users;

@Component
public class UsersAPI {
    private final UserService userService;

    UsersAPI(UserService userService) {
        this.userService = userService;
    }

    public Optional<UserDto> findNameById(Long userId) {
        return userService.findById(userId);
    }
}
```

Other modules can ask for a user's info through this facade. They don't need access to the repository or entity.

ArchUnit also supports layered and onion architecture rules. You can use those rules if you want to follow those architecture style for your application. 

```java
@ArchTest
static final ArchRule shouldFollowLayeredArchitecture = 
        layeredArchitecture()
            .consideringAllDependencies()
            .layer("Controller").definedBy("..controller..")
            .layer("Service").definedBy("..service..")
            .layer("Persistence").definedBy("..persistence..")
        
            .whereLayer("Controller").mayNotBeAccessedByAnyLayer()
            .whereLayer("Service").mayOnlyBeAccessedByLayers("Controller")
            .whereLayer("Persistence").mayOnlyBeAccessedByLayers("Service");
    
@ArchTest
static final ArchRule shouldFollowOnionArchitecture = 
        onionArchitecture()
            .domainModels("com.myapp.domain.model..")
            .domainServices("com.myapp.domain.service..")
            .applicationServices("com.myapp.application..")
            .adapter("cli", "com.myapp.adapter.cli..")
            .adapter("persistence", "com.myapp.adapter.persistence..")
            .adapter("rest", "com.myapp.adapter.rest..");
```

## Use Taikai for Recurring Java and Spring Conventions

[Taikai](https://github.com/enofex/taikai), built on top of ArchUnit, provides a fluent API for many recurring Java, testing, JPA, and Spring conventions.

The following is a shortened selection of rules from FeedbackHub's `TaikaiTests`:

```java
class TaikaiTests {

    @Test
    void shouldFulfillConstraints() {
        Taikai.builder()
            .namespace("dev.sivalabs.feedbackhub")
            .java(java -> java
                .noUsageOfDeprecatedAPIs()
                .methodsShouldNotDeclareGenericExceptions())
            .test(test -> test.junit(junit -> junit
                .classesShouldNotBeAnnotatedWithDisabled()
                .methodsShouldNotBeAnnotatedWithDisabled()))
            .jpa(jpa -> jpa.entities(entities -> entities
                .shouldBePackagePrivate()
                .shouldHaveFieldAnnotatedWithVersion()
                .namesShouldMatch(".+Entity")))
            .spring(spring -> spring
                .noAutowiredFields()
                .controllers(controllers -> controllers
                    .namesShouldEndWithController()
                    .shouldNotDependOnOtherControllers()
                    .shouldBePackagePrivate())
                .repositories(repositories -> repositories
                    .shouldNotDependOnServices()
                    .namesShouldEndWithRepository()
                    .shouldBePackagePrivate())
                .transactional(transactional -> transactional
                    .methodsShouldBePublic()
                    .shouldNotBeUsedInControllers()))
            .build()
            .checkAll();
    }
}
```

These checks reduce recurring review comments: use constructor injection, keep transactions out of controllers, preserve repository boundaries, and don't disable tests to make the build green.

Some rules are team preferences rather than universal truths. FeedbackHub's full test also checks logger conventions, entity identifier types, naming, and date types. Adopt a rule because you can explain its value for your application. A harness full of arbitrary restrictions creates work without improving the design.

## Spring Modulith: Verify Module Boundaries 

[Spring Modulith](https://spring.io/projects/spring-modulith) adds another level of verification: the relationships between application modules.

By default, it discovers modules in the direct subpackages of the application class's package. FeedbackHub has business areas such as `users` and `messages`, alongside packages such as `config`, and `shared`.

The core verification test is small:

```java
class ModularityTests {
    private final ApplicationModules modules = ApplicationModules.of(Application.class);

    @Test
    void verifyModuleStructure() {
        modules.verify();
    }
}
```

`verify()` checks for cycles between modules and access to internal packages. It can also enforce explicitly allowed dependencies when modules declare them.

Java visibility alone cannot express all of this. A service may need to be public so that a controller in another package of the same module can call it. That doesn't mean every other module should be allowed to use it.

FeedbackHub makes selected model types available through a named interface:

```java
@NamedInterface("users-models")
package dev.sivalabs.feedbackhub.users.domain.models;

import org.springframework.modulith.NamedInterface;
```

It also explicitly opens the shared module:

```java
@ApplicationModule(type = ApplicationModule.Type.OPEN)
package dev.sivalabs.feedbackhub.shared;

import org.springframework.modulith.ApplicationModule;
```

These declarations document deliberate exposure. An **open** module allows access to its internal packages, so it deserves a conscious decision. Making every module **open** would remove much of the protection we wanted.

Together, module verification and the custom ArchUnit service rule help preserve the distinction between a module's implementation and its supported entry points.

## Test the Behavior the User Cares About

Formatting and architecture checks can all pass while a feature remains wrong. Business behavior needs its own evidence.

For example, deleting a message in FeedbackHub means changing its status and hiding its content from normal views while retaining the original database record. It also requires checking ownership.

This test from `DeleteMessageTests` verifies a meaningful failure case:

```java
@Test
void anotherUserCannotDeleteMessage() {
    var otherUserSession = session(login("siva@gmail.com", "secret"));

    assertThat(deleteMessage(otherUserSession, ADMIN_MESSAGE_ID))
            .hasStatus(HttpStatus.FORBIDDEN)
            .hasViewName("error/403");
    assertThat(statusOf(ADMIN_MESSAGE_ID)).isEqualTo("ACTIVE");
}
```

The response assertion checks that the request was rejected. The database assertion checks that rejection didn't leave behind an unauthorized state change. Both matter.

The test class also verifies that an owner can soft-delete a message, that its original content remains stored, that normal views hide the content, and that a deleted message cannot be deleted again.

These tests inherit the shared Spring Boot test setup in `BaseIT`. 
The application uses [Testcontainers](https://testcontainers.com/) for PostgreSQL:

```java
@Bean
@ServiceConnection
PostgreSQLContainer postgres() {
    return new PostgreSQLContainer(parse("postgres:18-alpine"));
}
```

This exercises the application against PostgreSQL rather than assuming an in-memory substitute behaves identically.

When asking an agent for tests, I want acceptance criteria translated into observable outcomes: authorization, validation, persistence, privacy, and failure behavior. The number of test methods is much less interesting.

## Code Analysis using JaCoCo and SonarQube

Consistent formatting makes a diff easier to read. It doesn't tell us whether a method contains a likely bug, whether code is duplicated, or whether a security-sensitive implementation deserves attention. JaCoCo tells us what the tests executed. We still need something that looks at the code itself.

[Checkstyle](https://checkstyle.sourceforge.io/), [SpotBugs](https://spotbugs.github.io/), and [PMD](https://pmd.github.io/) offer automated checks for coding conventions and potential defects. FeedbackHub uses [SonarQube](https://www.sonarsource.com/products/sonarqube/) because it combines several kinds of feedback in one place.

### What SonarQube Can Catch

SonarQube analyzes the source code against a rule set called a quality profile. Each finding is classified so that we can decide how urgent it is:

- **Bugs**: code that is likely wrong, such as a possible `NullPointerException`, a resource that is never closed, or a comparison that is always false.
- **Vulnerabilities and security hotspots**: SQL built by string concatenation, hard-coded credentials, weak cryptography, or disabled CSRF protection.
- **Code smells**: maintainability problems such as high cognitive complexity, long methods, duplicated blocks, unused fields, empty catch blocks, and `System.out` instead of a logger.
- **Coverage and duplication**: the JaCoCo report is imported, so SonarQube can also flag new code that has too little test coverage.

The Java rules also cover Spring-specific practices, for example field injection with `@Autowired`, and `@Transactional` on methods that Spring's proxies can't intercept. This complements ArchUnit and Taikai. They enforce the conventions we write for our application. SonarQube brings a large library of general Java rules we don't have to write ourselves.

A **quality gate** turns these findings into a pass or fail decision. The default gate looks at *new code* only: no new bugs, no new vulnerabilities, sufficient coverage on new code, and limited duplication. That is useful with AI-generated changes because it doesn't require us to fix years of legacy findings before we can get a clean signal on today's diff.

### Integrating SonarQube with the Application

The integration has three parts: a SonarQube server, a project configuration, and a scan step.

For local experiments, a Community Build server runs in Docker:

```shell
docker run -d --name sonarqube -p 9000:9000 sonarqube:community
```

Open `http://localhost:9000`, log in, create a project, and generate a token. For a team, use a shared SonarQube server or SonarQube Cloud instead and keep the token in a secret store.

The scan reads the reports that the build already produces, so the project configuration only has to point at them:

```properties
# feedbackhub/sonar-project.properties
sonar.sources=src/main/java
sonar.tests=src/test/java
sonar.coverage.jacoco.xmlReportPaths=target/site/jacoco/jacoco.xml
sonar.junit.reportPaths=target/surefire-reports
```

Run the verification build first, so the JaCoCo and Surefire reports exist, then run the scan:

```shell
./mvnw clean verify

# Run from feedbackhub/ after the verification build.
mvn initialize sonar:sonar \
  -Dsonar.host.url=http://localhost:9000 \
  -Dsonar.token="$SONAR_TOKEN"
```

The CI workflow runs the same scan on the main branch after verification, with `SONAR_TOKEN` supplied through the environment.

Publishing analysis and enforcing a quality gate are separate concerns. The current workflow contains the scan step. It doesn't explicitly wait for the server's quality gate result. If a project requires the gate before merging, wire the result into the required CI checks as well. For example, by setting `sonar.qualitygate.wait=true` so that the scan fails when the gate fails.

### Let the Agent Fix Violations with the SonarQube MCP Server

Opening the SonarQube dashboard and copying issues into a prompt is the kind of manual step a harness should remove. The SonarQube MCP server lets the agent query SonarQube directly, so it can read the findings, understand the rule behind each one, and repair the code.

Register the server with the agent. For Claude Code, a configuration looks like this:

```shell
claude mcp add sonarqube \
  --env SONARQUBE_TOKEN="$SONAR_TOKEN" \
  --env SONARQUBE_URL=http://localhost:9000 \
  -- docker run -i --rm -e SONARQUBE_TOKEN -e SONARQUBE_URL mcp/sonarqube
```

For SonarQube Cloud, supply `SONARQUBE_ORG` instead of `SONARQUBE_URL`. The exact environment variables and available tools depend on the server version, so check the [SonarQube MCP Server documentation](https://github.com/SonarSource/sonarqube-mcp-server) for the one you install. Use a token with only the access the agent needs.

The server exposes tools to search issues in a project, read the details of a rule, check the quality gate status, and analyze code snippets. With those, the agent can follow a repeatable loop:

```text
Run verify and the Sonar scan
    -> fetch the open issues and the quality gate status
    -> read the rule behind each issue to understand why it matters
    -> fix the code and add or update tests where behavior is affected
    -> run spotless:apply and verify again
    -> re-scan and confirm the issues are gone
```

A task instruction for this can be short:

```text
Use the SonarQube MCP server to list the open issues for the feedbackhub project
on this branch, starting with bugs and vulnerabilities.
For each issue, read the rule description before changing code.
Fix the root cause, don't suppress the warning.
Run ./mvnw spotless:apply and ./mvnw verify,
then re-run the scan and report the quality gate status.
```

Some guardrails keep this safe:

- **Fix, don't silence.** Don't allow `@SuppressWarnings`, `// NOSONAR`, or marking issues as "won't fix" to make the gate pass. Treat those as policy changes that need a human decision.
- **Review the fixes.** Some rules are fixed mechanically, such as unused imports or a missing `final`. Others, like reducing cognitive complexity, change the structure of the code. The existing tests and architecture rules should catch regressions, and the final diff still needs a review.
- **Judge false positives.** Not every finding applies to our application. Let the agent explain why it thinks an issue is a false positive and leave the decision to you.
- **Check the scan results.** The agent should confirm the fix by running the analysis again, not by assuming the change worked.

For the agent, analysis findings are another source of actionable feedback. For the reviewer, they remove some repetitive inspection work. Neither replaces understanding the changed behavior.

## Put the Workflow into Agent Skills

Build checks tell the agent whether a change satisfies executable rules. Agent skills explain how to approach the work before those checks run.

FeedbackHub includes project-specific skills for turning a PRD into requirements, implementing a use case, synchronizing requirements with existing behavior, and reviewing Java changes.

The implementation skill uses one documented use case as the scope contract. Its workflow can be summarized as:

```text
Read the selected UC-### and its dependencies
Resolve significant ambiguity before editing
Change its status to IN_PROGRESS
Implement its acceptance criteria and focused tests
Run relevant verification and review the diff
Change its status to COMPLETED only after verification passes
```

That puts useful structure around the requirement-to-implementation loop. For a pending use case, an invocation looks like:

```text
Use $implement-usecase to implement UC-008 from docs/requirements.md.
```

## BootUI: Give the Agent Access to Runtime Evidence

Some problems only become clear when the application runs. [BootUI](https://www.julien-dubois.com/boot-ui/) provides a developer console for Spring Boot and Quarkus, with advisors for inspecting best-practice violations and MCP access to findings and runtime diagnostics.

FeedbackHub includes the BootUI starter:

```xml
<dependency>
    <groupId>com.julien-dubois.bootui</groupId>
    <artifactId>bootui-spring-boot-starter</artifactId>
    <version>${bootui.version}</version>
</dependency>
```

The application keeps both the console and its MCP support off by default:

```properties
bootui.enabled=${BOOTUI_ENABLED:off}
bootui.mcp.enabled=${BOOTUI_MCP_ENABLED:off}
```

For a local development session, those switches can be enabled when starting the application:

```shell
BOOTUI_ENABLED=on BOOTUI_MCP_ENABLED=on \
  ./mvnw spring-boot:run
```

Now, configure the BootUI MCP server for your AI Agent:

```json
{
  "mcpServers": {
    "bootui": {
      "type": "http",
      "url": "http://localhost:8080/bootui/api/mcp"
    }
  }
}
```
With the corresponding MCP connection configured, the agent can use runtime findings as evidence for investigation.

{{< youtube 9V5IeBmGI14 >}}

With BootUI MMCP configured, you can ask AI Agent to fetch advisor findings and fix them.

The same feedback principle applies to the SonarQube MCP setup. A developer can ask the agent to inspect open issues and repair relevant findings. This extends the harness beyond the compiler's view of the code while keeping automated build checks as a separate enforcement layer.

## Run Architecture Verification Tests Early

A harness is easier to use when the developer, the agent, and CI share the same verification commands.

To prioritize architecture checks, configure JUnit's test class orderer in `src/test/resources/junit-platform.properties`:

```properties
junit.jupiter.testclass.order.default=org.junit.jupiter.api.ClassOrderer$OrderAnnotation
```

This makes JUnit honor class-level `@Order` annotations. FeedbackHub assigns the following priorities to its architecture tests; the class bodies are omitted here:

```java
import org.junit.jupiter.api.Order;

@Order(1)
class ArchUnitTests {
    // Project-specific architecture rules
}

@Order(2)
class TaikaiTests {
    // Java, Spring, and JPA conventions
}

@Order(3)
class ModularityTests {
    // Module boundary verification and documentation
}
```

Lower values run first. Classes without `@Order` use JUnit's default priority, which places them after these explicitly ordered classes. `ArchUnitTests` and `TaikaiTests` share a priority, so their relative order is not specified by these annotations.

Within a single sequential JUnit execution, this puts architecture checks ahead of the broader behavioral suite. It gives the agent earlier feedback when a change breaks structural rules. Class ordering doesn't coordinate separate test executions or guarantee completion order when parallel execution is enabled, and it doesn't stop the suite after a failure. Keep tests independent; the ordering is there to improve feedback speed.

Now when maven run verification checks using `./mvnw verify`, the agent receives feedback on architecture violations before the broader behavioral suite runs.

The feedback should be specific enough that the agent can act on it:

```text
Formatting violation -> apply the formatter
Architecture violation -> repair the dependency or visibility
Behavioral test failure -> inspect the failed expectation
Coverage gate failure -> identify meaningful untested behavior
Analysis finding -> assess and fix the relevant issue
```

One expectation I would put directly in `AGENTS.md` development workflow instructions: 

```
DON'T LOWER THRESHOLDS, DISABLE TESTS, OR WEAKEN ARCHITECTURE RULES MERELY TO MAKE A CHANGE PASS.
```

Changes to the quality policy deserve their own explanation and review.

## Start with the Rules You Keep Repeating

I would build this incrementally. First, automate formatting and establish a reliable verification command. Then add architecture rules for the boundaries that matter most, behavioral tests for risky workflows, and coverage checks that expose missing evidence. Add analysis and runtime tools where they answer recurring questions.

A practical task instruction for this project could be:

```text
Implement the selected use case using the existing module structure.
Add tests for its success, authorization, and failure behavior.
Run ./mvnw spotless:apply.
Run ./mvnw verify.
Fix relevant failures without weakening the checks.
Review the final diff and report any verification that could not run.
```

I still need to review whether a requirement is right, an API is well designed, a migration is safe, or an edge case is missing. Those decisions benefit from the time recovered by automating routine checks.

The payoff I want from AI is faster delivery of changes I can trust. Every important convention I turn into an executable check makes that goal easier to sustain as the application grows.


# Spring Boot Tips: Easiest Way To Provide Environment Variables


Spring Boot has excellent support for [externalized configuration](https://docs.spring.io/spring-boot/reference/features/external-config.html). 
You can supply configuration through properties files, YAML files, environment variables, command-line arguments, and several other sources.

{{< box info >}}
**Spring Boot Configuration Management Best Practices**

Explore [Spring Boot Configuration Management Best Practices](https://blog.jetbrains.com/idea/2026/08/spring-boot-configuration-management-best-practices/) to learn more.
{{< /box >}}

That flexibility is useful, but local development still presents a small annoyance: where should you define environment variables?

You could export them from `.bashrc` or `.zshrc`, but then they affect every terminal session. 
You could configure them in your IDE, but each run configuration and each developer may need the same setup. 


For local development, I usually prefer a small file in the project directory.

<!--more-->

## Import a local properties file

Suppose the application needs a Telegram bot token and chat ID. 
Add the following to `application.properties`:

```properties
spring.config.import=optional:file:.env.properties

telegram.bot-token=${TELEGRAM_BOT_TOKEN}
telegram.chat-id=${TELEGRAM_CHAT_ID}
```

Then create `.env.properties` in the project root:

```properties
TELEGRAM_BOT_TOKEN=111111111111:ABCJDHRIJFL_NHFjnJKG
TELEGRAM_CHAT_ID=258745896
```

The `spring.config.import` property tells Spring Boot to load additional configuration from that file. 
The `file:` prefix means it is read from the filesystem, relative to the directory from which the application is started.
You can also use `classpath:.env.properties` if you want to place `.env.properties` in the classpath (ex: `src/main/resources`).

The `optional:` prefix is important. Without it, Spring Boot fails to start when the file is missing. 
With it, the application can still use environment variables supplied by your IDE, shell, container, or deployment platform.

## Prefer the conventional `.env` name?

Spring Boot normally uses a file extension to determine the configuration format. 
Because `.env` has no recognized extension, add a format hint:

```properties
spring.config.import=optional:file:.env[.properties]
```

The `[.properties]` suffix tells Spring Boot to parse `.env` as a Java properties file. 
Its contents remain the same:

```properties
TELEGRAM_BOT_TOKEN=111111111111:ABCJDHRIJFL_NHFjnJKG
TELEGRAM_CHAT_ID=258745896
```

This gives every developer one obvious place for local values while leaving production free to provide real environment variables.

{{< youtube bpQJpFXLFBk >}}

## One important precaution

These files often contain credentials. Do not commit them. 
Add the file you chose to `.gitignore`:

```gitignore
.env
.env.properties
```

If teammates need a template, commit an `.env.example` containing only placeholder values. 
That documents the required variables without leaking secrets.


# Spring AI : Advisors API

Once an AI feature moves beyond a demo, the model call is rarely the whole story.

You need to log requests, restore conversation history, retrieve relevant documents, reject unsafe input, record metrics, and perhaps validate the model's output.
Putting all of that code directly around every `ChatClient` call works for a while. Then a second endpoint arrives, followed by streaming, and suddenly every model call has its own slightly different pile of plumbing.

<!--more-->

{{< box info >}}
**Sample Code Repository**

You can find the sample code for this article in the [GitHub repository](https://github.com/sivaprasadreddy/sivalabs-blog-samples-code/tree/master/spring-ai-tutorial/04-advisors)

{{< /box >}}

Spring AI's **Advisors** provide a cleaner boundary for that plumbing.
An advisor can inspect or change a request before it reaches the model, inspect or change the response on the way back, or stop the chain without calling the model at all.

If you have worked with Servlet filters or Spring AOP advices, the idea will feel familiar.
Advisors form an ordered chain around a `ChatClient` execution. The difference is scope: a Servlet filter surrounds HTTP traffic, and an AOP advice surrounds method calls; an advisor surrounds an AI request and response.

## Intercept AI Request and Responses using Advisors

Imagine that our application starts with this innocent-looking call:

```java
String response = chatClient
        .prompt(question.question())
        .call()
        .content();
```

Now suppose every request must pass a sensitive-content check and every successful model interaction must be logged.
We could add both concerns to the controller, but neither belongs to HTTP request handling.
We would also have to remember to repeat them in every place that uses the `ChatClient`.

With advisors, the execution becomes a chain:

```text
Request  -> SafetyCheckAdvisor -> MyLoggingAdvisor -> ChatModel
Response <- SafetyCheckAdvisor <- MyLoggingAdvisor <- ChatModel
```

Each advisor decides whether to delegate to the next link.
That small detail is what makes the abstraction useful: an advisor is not limited to observing a call. It can enrich it, replace its response, or short-circuit it.

{{< figure src="/images/spring-ai-advisors-detailed.webp" >}}

## Building an advisor to log LLM request and response

Our first advisor logs the request, calls the rest of the chain, and then logs the response:

```java
class MyLoggingAdvisor implements CallAdvisor {
    private static final Logger logger = LoggerFactory.getLogger(MyLoggingAdvisor.class);

    @Override
    @NonNull
    public ChatClientResponse adviseCall(
            ChatClientRequest chatClientRequest,
            CallAdvisorChain callAdvisorChain) {

        logger.info("request: {}", chatClientRequest);

        ChatClientResponse chatClientResponse =
                callAdvisorChain.nextCall(chatClientRequest);

        logger.info("response: {}", chatClientResponse.chatResponse());

        return chatClientResponse;
    }

    @Override
    public String getName() {
        return this.getClass().getSimpleName();
    }

    @Override
    public int getOrder() {
        return 20;
    }
}
```

`ChatClientRequest` contains the prompt and an advisor context map. `ChatClientResponse` contains the model response and the context that has travelled through the chain.
That context is useful when two advisors need to collaborate without stuffing internal state into the prompt.

The interesting line is this one:

```java
callAdvisorChain.nextCall(chatClientRequest);
```

Calling `nextCall(...)` hands control to the next advisor. Eventually the chain reaches the chat model.
When the model returns, control comes back through the advisors in reverse order.
That gives `MyLoggingAdvisor` an around-style view of the execution: code before `nextCall(...)` handles the outgoing request; code after it handles the incoming response.

This same structure can do more than logging. An advisor can create a new `ChatClientRequest` to enrich a prompt, attach information to the context, or return a modified `ChatClientResponse`.

Now you can add `MyLoggingAdvisor` as a default advisor for `ChatClient` as follows:

```java
@RestController
class ChatController {
    private final ChatClient chatClient;

    ChatController(ChatClient.Builder builder) {
        this.chatClient = builder
                .defaultAdvisors(new MyLoggingAdvisor())
                .build();
    }
    
    //....
}
```

The important habit is to keep an advisor focused on one cross-cutting concern. A single "do everything AI advisor" becomes just another controller in disguise.

Also, prompts and model responses frequently contain personal or confidential data. Logging the entire request is convenient while learning, but it is a risky production default.
Redact sensitive fields, restrict log access, and prefer metadata such as latency, token usage, model name, and correlation IDs when the full content is not essential.

## SafetyCheckAdvisor: Blocking a request without calling the model

Here is a deliberately small safety advisor. It checks the prompt for configured words and returns a synthetic assistant response when it finds one:

```java
class SafetyCheckAdvisor implements CallAdvisor {
    private static final Logger logger = LoggerFactory.getLogger(SafetyCheckAdvisor.class);

    private final List<String> sensitiveWords;

    SafetyCheckAdvisor(List<String> sensitiveWords) {
        this.sensitiveWords = sensitiveWords;
    }

    @Override
    public ChatClientResponse adviseCall(
            ChatClientRequest chatClientRequest,
            CallAdvisorChain callAdvisorChain) {

        logger.info("Checking for sensitive words");

        boolean containsSensitiveWord = !CollectionUtils.isEmpty(sensitiveWords)
                && sensitiveWords.stream().anyMatch(word ->
                    chatClientRequest.prompt().getContents().contains(word));

        if (containsSensitiveWord) {
            logger.info("Found sensitive words in the request. Abort further processing");
            return createFailureResponse(chatClientRequest);
        }

        return callAdvisorChain.nextCall(chatClientRequest);
    }

    private ChatClientResponse createFailureResponse(ChatClientRequest chatClientRequest) {

        String failureResponse =
                "I'm unable to respond to that due to sensitive content.";

        return ChatClientResponse.builder()
                .chatResponse(ChatResponse.builder()
                        .generations(List.of(new Generation(
                            new AssistantMessage(failureResponse))))
                        .build())
                .context(Map.copyOf(chatClientRequest.context()))
                .build();
    }

    @Override
    public String getName() {
        return this.getClass().getSimpleName();
    }

    @Override
    public int getOrder() {
        return 10;
    }
}
```

Notice what the rejection branch does **not** contain: `callAdvisorChain.nextCall(...)`. The advisor creates a normal-looking `ChatClientResponse` and returns immediately, so downstream advisors and the model are skipped.

This example is useful for understanding short-circuiting, but a substring list is not a production moderation system.
It is case-sensitive, can match part of an innocent word, and is easy to evade with spacing or spelling changes.
Depending on the risk, a real application might normalize input, use a dedicated moderation model, enforce tenant-specific policy, and validate model output as well as user input.

### Testing the short circuit

The sample sends a request containing a configured sensitive word and verifies that the endpoint returns the advisor's response:

```java
@Test
void chatWithSensitiveWordsShouldNotBeSentToLLM() {
    MvcTestResult testResult = mockMvcTester.post().uri("/ai/chat")
            .contentType(MediaType.APPLICATION_JSON)
            .content("""
                    {
                        "question": "Why javascript ecosystem moves at a fucking crazy speed?"
                    }
                    """)
            .exchange();

    assertThat(testResult)
            .hasStatusOk()
            .bodyJson()
            .convertTo(ChatController.Answer.class)
            .satisfies(answer -> assertThat(answer.answer())
                    .contains("I'm unable to respond to that due to sensitive content"));
}
```

This is a useful end-to-end check of the HTTP response and advisor configuration.
In a production codebase, I would add a focused test with a fake or mocked `ChatModel` and explicitly verify that it received no call.
That turns the test name "should not be sent to LLM" into a directly verified contract rather than an inference from the returned text.

## Advisor Ordering is Important

All advisors are ordered. A lower `getOrder()` value has higher precedence and sees the request earlier.
Responses unwind through the same chain in the opposite direction.

Our safety advisor returns `10`, while the logging advisor returns `20`:

```java
@Override
public int getOrder() {
    return 10;
}
```

That gives us this behavior:

```text
Successful request:
Safety before -> Logging before -> Model -> Logging after -> Safety after

Blocked request:
Safety before -> synthetic response
```

Because the safety check runs first and does not delegate when it rejects a prompt, neither `MyLoggingAdvisor` nor the model sees that prompt.
That is a useful privacy property in this sample.

If we gave the logging advisor the lower order instead, it would wrap the safety advisor and log blocked requests too.
Neither choice is universally correct. The point is that advisor order changes system behavior, so choose it deliberately and cover important chains with tests.

## `CallAdvisor` and `StreamAdvisor`

Spring AI separates non-streaming and streaming interception:

```java
public interface CallAdvisor extends Advisor {
    ChatClientResponse adviseCall(
            ChatClientRequest request,
            CallAdvisorChain chain);
}

public interface StreamAdvisor extends Advisor {
    Flux<ChatClientResponse> adviseStream(
            ChatClientRequest request,
            StreamAdvisorChain chain);
}
```

> **Note:** Implement `CallAdvisor` for `chatClient.prompt(...).call()` and `StreamAdvisor` for `chatClient.prompt(...).stream()`. If the same concern must apply to both execution styles, implement both interfaces or use a built-in advisor that supports both. Streaming is not merely a synchronous response wrapped in a `Flux`; response-side work must account for multiple emissions and completion of the stream.

## Default vs Per-request Advisors

We can register both advisors as defaults while building the `ChatClient`:

```java
@RestController
class ChatController {
    private final ChatClient chatClient;

    ChatController(ChatClient.Builder builder) {
        this.chatClient = builder
                .defaultAdvisors(
                        new MyLoggingAdvisor(),
                        new SafetyCheckAdvisor(List.of("fuck", "screw")))
                .build();
    }

    @PostMapping("/ai/chat")
    Answer chat(@RequestBody @Valid Question question) {
        String response = chatClient
                .prompt(question.question())
                .call()
                .content();

        return new Answer(response);
    }

    record Question(@NotBlank String question) {}

    record Answer(String answer) {}
}
```

The `defaultAdvisors(...)` applies the advisors to every request made by this client.
That is the right choice for policies such as mandatory safety checks, observability, or conversation memory that define how a particular client behaves.

For a concern needed by only one request, add it to the request specification instead:

```java
String answer = chatClient.prompt("Explain virtual threads")
        .advisors(new MyLoggingAdvisor())
        .call()
        .content();
```

I prefer defaults for invariants and per-request advisors for optional behavior.
If a safety rule is important enough to protect every call, relying on each developer to remember `.advisors(...)` is an optimistic security design.

## Spring AI Built-in Advisors

Custom advisors are straightforward, but Spring AI already provides advisors for common jobs.
The set evolves, so check the reference documentation for the version you use.
In Spring AI 2.0.0, useful options include:

- `SimpleLoggerAdvisor` logs requests and responses and supports call and streaming flows. Treat full prompt logging as sensitive data handling, not harmless diagnostics.
- `SafeGuardAdvisor` provides a basic sensitive-word guard. Like our example, it is a guardrail rather than a complete moderation strategy.
- `MessageChatMemoryAdvisor` adds conversation memory as messages. `VectorStoreChatMemoryAdvisor` retrieves semantically relevant memories from a vector store.
- `QuestionAnswerAdvisor` implements a straightforward vector-store question-answering flow, while `RetrievalAugmentationAdvisor` supports more modular RAG pipelines.
- `ReReadingAdvisor` applies a re-reading prompting technique to improve reasoning on some tasks.
- `ToolCallingAdvisor` coordinates model tool calls, and `StructuredOutputValidationAdvisor` validates structured model output.

The names are less important than the pattern. Memory, retrieval, safety, logging, tool execution, and output validation are all concerns that surround a model call.
Advisors let those concerns compose without forcing the controller to become the AI equivalent of a junk drawer.

Be especially careful when combining stateful advisors. For example, whether memory runs before or after retrieval can change the query sent to the vector store.
Write down the intended request and response path, assign order values with room between them, and test the composition—not only each advisor in isolation.

## Summary

I would recommend starting with built-in advisors for logging, memory, and RAG, then write a custom advisor only when the application's policy is genuinely domain-specific.
I would keep mandatory advisors on a dedicated, centrally configured `ChatClient`, reserve per-request advisors for optional behavior, and treat order values as part of the design.

# Spring AI : Chat Memory

Large language models are stateless. Send two independent requests to a model, and the second request knows nothing about the first one.
That is perfectly fine for one-shot questions, but it makes a chatbot feel surprisingly forgetful:

<!--more-->

```text
User: My name is Siva.
Assistant: Nice to meet you, Siva.
User: What is my name?
Assistant: I don't know.
```

{{< figure src="/images/spring-ai-chat-memory-say-my-name.webp" >}}

The model has not forgotten anything, the application simply did not send the earlier messages again.
Spring AI's chat memory support handles that plumbing for us.
It stores the relevant messages and adds them to each new prompt, giving the model enough context to continue the conversation.

{{< box info >}}
**Sample Code Repository**

You can find the sample code for this article in the [GitHub repository](https://github.com/sivaprasadreddy/sivalabs-blog-samples-code/tree/master/spring-ai-tutorial/03-chat-memory)

{{< /box >}}


## Introducing ChatMemory
The ChatMemory abstraction allows you to implement memory to support conversation-style interactions with LLMs.

- **Chat memory** is the context sent to the model so that it can answer the current request.
- **Chat history** is the complete, durable record of everything said in a conversation.

`ChatMemory` is designed for the first job. A memory strategy may deliberately discard old messages to stay within a useful context window.

## Adding ChatMemory to the application

Add the chat-memory starter alongside the starter for your model provider:

```xml
<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-chat-memory</artifactId>
</dependency>
```

Spring Boot then auto-configures a `ChatMemory` bean. Unless another repository is present, that bean uses:

- `MessageWindowChatMemory` to decide which messages remain in context.
- `InMemoryChatMemoryRepository` to store those messages in the application process.

This separation is useful. `ChatMemory` owns the retention strategy, while `ChatMemoryRepository` only stores and retrieves messages.
We can change the storage without changing how the context window behaves.

### Keeping a useful message window

`MessageWindowChatMemory` keeps a sliding window of messages. Its default limit is 20, and older messages are evicted when that limit is exceeded.
`SystemMessage` instances are preserved.

You can configure it explicitly when the default is not suitable:

```java
@Bean
ChatMemory chatMemory(ChatMemoryRepository repository) {
    return MessageWindowChatMemory.builder()
            .chatMemoryRepository(repository)
            .maxMessages(50)
            .build();
}
```

The default `InMemoryChatMemoryRepository` stores data in a `ConcurrentHashMap`:
It is ideal for learning and tests, but all conversations disappear when the application restarts.
It also cannot share memory between multiple application instances. We will fix that shortly with PostgreSQL.

## Letting `ChatClient` manage the messages

The `MessageChatMemoryAdvisor` connects `ChatClient` to `ChatMemory`.
Before a model call, it retrieves the conversation's messages and adds them to the prompt.
After the call, it saves the new user and assistant messages.

Configure the advisor while building the client:

```java
@RestController
@RequestMapping("/")
class ChatController {
    private final ChatClient chatClient;

    ChatController(ChatClient.Builder builder, ChatMemory chatMemory) {
        this.chatClient = builder
                .defaultAdvisors(
                    MessageChatMemoryAdvisor.builder(chatMemory).build(),
                    new SimpleLoggerAdvisor()
                )
                .build();
    }
}
```

Every call using a memory advisor must provide a conversation ID.
There is no default; omitting it results in an `IllegalArgumentException`.

{{< box info >}}
**Spring AI Advisors**

We will explore more about Spring AI Advisors in the next article.
{{< /box >}}

To see the behavior with the least possible code, we can start with a static ID:

```java
@PostMapping("/ai/simple-chat")
ResponseEntity<Answer> chat(@RequestBody @Valid Question question) {
    String conversationId = "demo";

    String response = chatClient.prompt()
            .user(question.question())
            .advisors(advisor -> advisor.param(
                    ChatMemory.CONVERSATION_ID, conversationId))
            .call()
            .content();

    return ResponseEntity.ok(new Answer(response));
}

record Question(@NotBlank String question) {}
record Answer(String answer) {}
```

Send a first request "My name is Siva" and then in the second request ask "What is my name?".
The second request receives the earlier exchange as context.

```shell
$ curl --request POST http://localhost:8080/ai/simple-chat \
  --header 'Content-Type: application/json' \
  --data '{"question":"My name is Siva"}'

Nice to meet you, Siva! How can I help you today?

$ curl --request POST http://localhost:8080/ai/simple-chat \
  --header 'Content-Type: application/json' \
  --data '{"question":"What is my name?"}'
  
Your name is Siva.
```

The `SimpleLoggerAdvisor` output makes the memory behavior visible.
The first model request contains only the message sent by the caller:

```text
messages=[
  UserMessage{content='My name is Siva', ...}
]
context={chat_memory_conversation_id=demo}
```

After the model replies, `MessageChatMemoryAdvisor` stores both the user message and the assistant response under the `demo` conversation ID.
When the second request arrives, the advisor retrieves that exchange and adds it before the new question:

```text
messages=[
  UserMessage{content='My name is Siva', ...},
  AssistantMessage [textContent=Nice to meet you, Siva! How can I help you today?, ...],
  UserMessage{content='What is my name?', ...}
]
context={chat_memory_conversation_id=demo}
```

Notice that the second `curl` command sends only `What is my name?`.
The client does not resend the conversation.
The advisor rebuilds the prompt on the server from the messages associated with the conversation ID, which gives the model enough context to answer `Your name is Siva.`

There is one obvious catch: every caller uses `demo`, so every caller also shares the same memory.
A static ID is useful for demonstrating the feature, not for a real application.

## Choosing where memory lives

Spring AI 2.0 includes repositories for in-memory storage, JDBC databases, Cassandra, Neo4j, MongoDB, and Redis.
Azure Cosmos DB support is maintained as an external module.
You can also implement `ChatMemoryRepository` for another store.

The choice is mostly operational:

- Use in-memory storage for local development and isolated tests.
- Use JDBC when a relational database is already part of the application.
- Consider Redis for low-latency access and TTL-based expiry.
- Choose Cassandra, MongoDB, or Neo4j when those systems already match your workload and operations.

For a typical Spring Boot application that already uses PostgreSQL, JDBC is the least surprising place to start.

## Persisting chat memory in PostgreSQL

Add the JDBC, JDBC memory repository, PostgreSQL driver, and Spring Boot's Docker Compose support:

```xml
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-chat-memory-repository-jdbc</artifactId>
</dependency>
<dependency>
    <groupId>org.postgresql</groupId>
    <artifactId>postgresql</artifactId>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-docker-compose</artifactId>
    <scope>runtime</scope>
    <optional>true</optional>
</dependency>
```

Then create `compose.yaml` in the application directory:

```yaml
services:
  postgres:
    image: postgres:18-alpine
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=postgres
    ports:
      - "5432:5432"
```

By default, Spring AI initializes the JDBC schema only for embedded databases.
PostgreSQL is not embedded, so enable initialization while developing locally:

```properties
spring.ai.chat.memory.repository.jdbc.initialize-schema=always
```

This creates the `SPRING_AI_CHAT_MEMORY` table using the PostgreSQL-specific script. The supported values are:

- `embedded` — initialize only embedded databases; this is the default.
- `always` — initialize on every application startup.
- `never` — do not initialize the schema.

{{< box tip >}}
**Recommended Schema Initialization**

For production, I prefer `never` and a versioned Flyway or Liquibase migration.
Database schemas deserve the same review and deployment discipline as application code.
{{< /box >}}

When the application starts, Spring Boot launches the service and creates the database connection.
Spring AI detects JDBC and auto-configures `JdbcChatMemoryRepository`; the existing auto-configured `MessageWindowChatMemory` uses it automatically.

The JDBC repository supports PostgreSQL, MySQL/MariaDB, SQL Server, HSQLDB, and Oracle Database out of the box.
It stores message order using a `sequence_id` column and exposes each message's creation time in metadata.

## Giving each browser its own conversation

A conversation ID links all the messages within a conversation. 
In an authenticated application, a user ID plus a chat/session ID is often the best choice.
For this small demo, a generated UUID set as a cookie is enough:

```java
@PostMapping("/ai/chat")
ResponseEntity<Answer> chat(
        @RequestBody @Valid Question question,
        @CookieValue(name = "X-CONV-ID", required = false) String convId) {

    String conversationId = convId == null
            ? UUID.randomUUID().toString()
            : convId;

    String response = chatClient.prompt()
            .user(question.question())
            .advisors(advisor -> advisor.param(
                    ChatMemory.CONVERSATION_ID, conversationId))
            .call()
            .content();

    ResponseCookie cookie = ResponseCookie
            .from("X-CONV-ID", conversationId)
            .path("/")
            .maxAge(3600)
            .build();

    return ResponseEntity.ok()
            .header(HttpHeaders.SET_COOKIE, cookie.toString())
            .body(new Answer(response));
}
```

On the first request, the server generates a UUID and returns it as `X-CONV-ID`.
The browser sends that cookie with later requests, allowing the advisor to load the correct memory.
Another browser receives a different ID and therefore a different conversation.

For a public application, treat this ID as untrusted input: use a secure, HTTP-only cookie, 
validate access on the server, expire abandoned conversations, and never let knowledge of an ID grant access to another user's data.

## What I would use in a real application

Start with `MessageWindowChatMemory` and a modest window, then measure token usage and answer quality before tuning it.
Use the in-memory repository only while developing; move to a shared store before running multiple instances.
Most importantly, keep chat memory and chat history as separate concerns.
The model needs a carefully selected context window, while your users and auditors may need the full conversation.

Chat memory is not the model becoming stateful. 
It is the application consistently rebuilding context, and Spring AI makes that easy to do without scattering message-management code across every endpoint.


# Spring AI : Structured Output


Chat models are great at producing prose, but most applications don't want prose — they want a Java object, a `List<String>`, or a `Map` they can bind to a response and return from a REST API.

Spring AI's **Structured Output Converters** bridge that gap: they turn plain-text chat responses into typed Java objects, without you having to hand-write a JSON parser or repeat "please respond with valid JSON" in every prompt.
<!--more-->

{{< box info >}}
**Sample Code Repository**

You can find the sample code for this article in the [GitHub repository](https://github.com/sivaprasadreddy/sivalabs-blog-samples-code/tree/master/spring-ai-tutorial/02-structured-output)

{{< /box >}}

## The Need for Structured Output

An LLM only knows how to return text. If you ask it for "3 blog title suggestions", you get back something like:

```text
Supercharge your Spring Boot development, Unlock hidden productivity with Spring Boot, Ship faster with Spring Boot
```

To use that in your application, someone has to:

1. Tell the model *exactly* what shape the response should be in (JSON, CSV, a schema, etc.)
2. Parse the raw text response back into a Java type
3. Handle the inevitable edge cases — markdown code fences around JSON, trailing text, etc.

Spring AI does all three for you via `StructuredOutputConverter` implementations — `BeanOutputConverter`, `ListOutputConverter`, and `MapOutputConverter`, and via the `ChatClient.entity(...)` convenience method built on top of them.

## Generating a Tweet — the Easy Way

Let's say we want the AI model to generate a tweet with some content and a list of hashtags:

```java
record Tweet(String content, List<String> hashtags) {}
```

Here's the whole endpoint:

```java
@PostMapping("/ai/gen-tweet")
Tweet generateTweet(@RequestBody @Valid Input input) {
    Tweet tweet = chatClient
            .prompt()
            .system("""
                    You are an expert software developer and experienced content creator.
                    Your job is to generate interesting tweet based on the given input.
                    
                    * Use a catchy first line to convey the essence of the input.
                    * Keep it concise and engaging.
                    * Maintain a professional tone.
                    * Use bullet points to list key features or benefits
                    * Use emojis where appropriate.
                    * Include relevant hashtags at the end.
                    
                    Don't use Markdown syntax. Use plain text format.
                    """)
            .user(u ->
                    u.text("""
                        Generate a tweet for the following content:

                        {content}
                        """)
                    .param("content", input.prompt())
            )
            .call()
            .entity(Tweet.class);

    log.info("Generated tweet: {}", tweet);
    return tweet;
}
```

That one line `.entity(Tweet.class)` is doing all the heavy lifting. 

Behind the scenes, Spring AI:

1. Builds a `BeanOutputConverter<Tweet>` from the `Tweet` record
2. Appends its JSON format instructions to the prompt
3. Sends the prompt to the model
4. Parses the JSON response back into a `Tweet` instance

Calling it with the following prompt:

```text
IntelliJ IDEA 2026.2 is released with a lot of new features such as Logpoints, Dependency completion in Gradle, Spring Security Inlays and Unlocking, Docker Compose improvements. Checkout https://www.jetbrains.com/idea/whatsnew/2026-2/ for more details.
```

generated the response which is converted into a `Tweet` instance:

```text
Tweet[content=Ship faster with IntelliJ IDEA 2026.2 🚀
• 🐞 Logpoints for live debugging
• 🔗 Gradle dependency completion
• 🔐 Spring Security inlays & unlocking
• 🐳 Improved Docker Compose
More: https://www.jetbrains.com/idea/whatsnew/2026-2/
#IntelliJIDEA #JetBrains #Java #Gradle #Spring #Docker #IDE, hashtags=[#IntelliJIDEA, #JetBrains, #Java, #Gradle, #Spring, #Docker, #IDE]]
```

No manual parsing anywhere — a plain Java record came straight out of `.entity(Tweet.class)`.

{{< box tip >}}
**Using a Classpath Resource as the System Prompt**

The system prompt tells the model how to behave, and can grow long.
Instead of embedding it as a Java text block, load it from a resource file:

```java
ChatController(ChatClient.Builder builder,
               @Value("classpath:/prompts/tweet-system-message.md")
               Resource tweetSystemMsgResource) {
    this.chatClient = builder.build();
    this.tweetSystemMsgResource = tweetSystemMsgResource;
}
```

The `chatClient.prompt().system(tweetSystemMsgResource)` accepts a `Resource` directly, so the prompt lives in its own file — easy to read, edit, and version separately from the controller code.

{{< /box >}}

## BeanOutputConverter — Behind the Scenes

The `.entity(Tweet.class)` is a shortcut. Here's what it's doing under the hood, spelled out explicitly:

```java
@PostMapping("/ai/gen-tweet-2")
Tweet generateTweetV2(@RequestBody @Valid Input input) {
    PromptTemplate pt = new PromptTemplate("""
    Generate a tweet for the following content:

    {content}

    {format}
    """);

    BeanOutputConverter<Tweet> beanOutputConverter = new BeanOutputConverter<>(Tweet.class);
    String format = beanOutputConverter.getFormat();
    Map<String, Object> vars = Map.of("content", input.prompt(), "format", format);

    SystemMessage systemMessage = new SystemMessage(tweetSystemMsgResource);
    Message userMessage = pt.createMessage(vars);

    Prompt prompt = new Prompt(List.of(systemMessage, userMessage));
    String response = chatClient.prompt(prompt).call().content();

    Tweet tweet = beanOutputConverter.convert(response);
    log.info("Generated tweet: {}", tweet);
    return tweet;
}
```

The `beanOutputConverter.getFormat()` generates a JSON Schema for `Tweet` and wraps it in instructions. Here's the actual text it produces:

```text
Your response should be in JSON format.
Do not include any explanations, only provide a RFC8259 compliant JSON response following this format without deviation.
Do not include markdown code blocks in your response.
Remove the ```json markdown from the output.
Here is the JSON Schema instance your output must adhere to:
```{
  "$schema" : "https://json-schema.org/draft/2020-12/schema",
  "type" : "object",
  "properties" : {
    "content" : { "type" : "string" },
    "hashtags" : { "type" : "array", "items" : { "type" : "string" } }
  },
  "required" : [ "content", "hashtags" ],
  "additionalProperties" : false
}```
```

That text gets substituted into `{format}` in the prompt template, sent to the model as part of the user message, and the raw text response is handed to `beanOutputConverter.convert(response)`, which parses the JSON into a `Tweet`.

The `.entity(Tweet.class)` does exactly this: same format string, same converter, same `convert()` call — just without the boilerplate.

## Inspecting Requests and Responses with SimpleLoggerAdvisor

Since the format instructions get silently appended to your prompt, it's useful to see exactly what's being sent to the model.
`SimpleLoggerAdvisor` logs the full request and response:

```java
this.chatClient = builder
        .defaultAdvisors(new SimpleLoggerAdvisor())
        .build();
```

Enable `DEBUG` logging for it in `application.properties`:

```properties
logging.level.org.springframework.ai.chat.client.advisor=DEBUG
```

With that in place, the tweet request logs the system message, the user message, and — separately — the injected format instructions as prompt *context*:

```text
request: ChatClientRequest[prompt=Prompt{messages=[
  SystemMessage{textContent='You are an expert software developer...'},
  UserMessage{content='Generate a tweet for the following content: ...'}
], ...},
context={spring.ai.chat.client.output.format=Your response should be in JSON format. ...}]
```

And the response log shows the raw JSON the model returned, before it's converted into a `Tweet`:

```text
response: {
  ...
  "output" : {
    "text" : "{\n  \"content\": \"Ship faster with IntelliJ IDEA 2026.2 🚀...\",\n  \"hashtags\": [ \"#IntelliJIDEA\", \"#Java\", ... ]\n}"
  }
}
```

This is the fastest way to debug a structured output that isn't parsing the way you expect — you can see precisely what instructions were sent and what the model actually replied with.

{{< box info >}}
**Spring AI Advisors**

We will explore more about Spring AI Advisors in the upcoming articles.
{{< /box >}}

## ListOutputConverter

For a simple list of strings, you don't need a whole record type — `ListOutputConverter` does the job:

```java
@PostMapping("/ai/suggest-titles")
TitleSuggestionsResponse suggestTitles(@RequestBody @Valid TitleSuggestionsRequest req) {
    ListOutputConverter outputConverter = new ListOutputConverter();

    PromptTemplate pt = new PromptTemplate("""
    I would like to give a presentation about the following:

    {topic}

    Give me {count} title suggestions for this topic.

    Make sure the title is relevant to the topic and it should be a single short sentence.

    {format}
    """);

    Map<String, Object> vars = Map.of("topic", req.topic(), "count", req.count(),
            "format", outputConverter.getFormat());
    Message message = pt.createMessage(vars);
    String response = chatClient.prompt().messages(message).call().content();

    List<String> titles = outputConverter.convert(response);

    return new TitleSuggestionsResponse(titles);
}
```

Its format instructions are much simpler than `BeanOutputConverter`'s JSON schema — it just asks for comma-separated values:

```text
Respond with only a list of comma-separated values, without any leading or trailing text.
Example format: foo, bar, baz
```

Calling `POST /ai/suggest-titles` with `{"topic": "Spring Boot Tips and Tricks", "count": 3}` returns:

```json
{
  "titles": [
    "Supercharge your Spring Boot development",
    "Unlock hidden productivity with Spring Boot",
    "Ship faster with Spring Boot"
  ]
}
```

`ListOutputConverter.convert(response)` simply splits that comma-separated text into a `List<String>`.

## MapOutputConverter

`MapOutputConverter` works the same way, but targets a `Map<String, Object>`:

```java
@GetMapping("/ai/langs")
Map<String, Object> languages() {
    MapOutputConverter outputConverter = new MapOutputConverter();

    PromptTemplate pt = new PromptTemplate("""
    Return all popular programming languages and their inception year.

    {format}
    """);

    Map<String, Object> vars = Map.of("format", outputConverter.getFormat());
    Message message = pt.createMessage(vars);
    String response = chatClient.prompt().messages(message).call().content();

    Map<String, Object> languages = outputConverter.convert(response);
    log.info("Languages: {}", languages);
    return languages;
}
```

Its format instructions target a `HashMap` shape specifically:

```text
Your response should be in JSON format.
The data structure for the JSON should match this Java class: java.util.HashMap
Do not include any explanations, only provide a RFC8259 compliant JSON response following this format without deviation.
Remove the ```json markdown surrounding the output including the trailing "```".
```

`GET /ai/langs` returns a real JSON map straight from the model:

```json
{
  "Assembly": 1949,
  "Fortran": 1957,
  "Lisp": 1958,
  "ALGOL": 1958,
  "Python": 1991,
  "Java": 1995,
  "JavaScript": 1995,
  "Kotlin": 2011,
  "Rust": 2010,
  "TypeScript": 2012,
  "Zig": 2016
}
```
*(trimmed — the model actually returns 60+ entries)*

## Native Structured Output

Everything so far relies on **prompt engineering**: the format instructions are just text appended to the prompt, and the model is trusted to follow them. That works with any model, but it's still the model *choosing* to comply.

Some providers — including OpenAI — support **structured output at the API level**: you send a JSON Schema as part of the request itself, and the provider enforces it server-side, without needing extra prompt text.

Spring AI exposes this through `ChatClient.EntityParamSpec.useProviderStructuredOutput()`:

```java
Tweet tweet = chatClient
        .prompt()
        .system(tweetSystemMsgResource)
        .user(u -> u.text("""
                Generate a tweet for the following content:

                {content}
                """)
                .param("content", input.prompt()))
        .call()
        .entity(Tweet.class, spec -> spec.useProviderStructuredOutput());
```

With this flag set, Spring AI sends the `Tweet` JSON Schema as an API-level constraint (OpenAI's `response_format.json_schema`) instead of stuffing it into the prompt text — no extra tokens spent on format instructions, and a stronger guarantee that the response actually matches the schema.

It's **not enabled by default**, because support varies by provider and model:

- **OpenAI**'s Structured Outputs API doesn't accept a top-level JSON *array* schema — so a bare `List<T>` won't work in native mode. Wrap it in a container record, or keep using `ListOutputConverter`.
- Some **Ollama** models with a built-in reasoning/thinking mode can return plain text instead of JSON when this is enabled, breaking deserialization.

Given those caveats, the default prompt-based converters remain the safer, provider-agnostic choice — reach for `useProviderStructuredOutput()` when you know your target model supports it and want the stricter guarantee.

## Summary

| Converter                | Target type           | Format instructions     |
|--------------------------|-----------------------|-------------------------|
| `BeanOutputConverter<T>` | Any Java bean/record  | Full JSON Schema        |
| `ListOutputConverter`    | `List<String>`        | Comma-separated values  |
| `MapOutputConverter`     | `Map<String, Object>` | JSON matching `HashMap` |

- `.entity(Class)` on `ChatClient` is a convenient shortcut over these converters — use it unless you need custom prompt wiring.
- `SimpleLoggerAdvisor` + `DEBUG` logging is the quickest way to see exactly what format instructions and responses are flowing through your prompts.
- For providers that support it, `spec.useProviderStructuredOutput()` moves schema enforcement from the prompt into the API call itself.

# Spring AI : Chat with LLMs


Calling an LLM is just an HTTP request. The trouble is that every provider has its own API,
configuration, and response format.

Spring AI handles that provider-specific plumbing and gives us a familiar Spring API. In this article,
we will use `ChatClient` to talk to OpenAI's GPT-5 model and then make our prompts reusable with
`PromptTemplate`.

<!--more-->

{{< box info >}}
**Sample Code Repository**

You can find the sample code for this article in the [GitHub repository](https://github.com/sivaprasadreddy/sivalabs-blog-samples-code/tree/master/spring-ai-tutorial/01-chat-openai)

{{< /box >}}

## Configure OpenAI

First, create an API key from the [OpenAI Platform](https://platform.openai.com/) and expose it as an
environment variable:

```shell
export OPENAI_API_KEY=your-api-key
```

Do not commit the key to Git. Also note that ChatGPT subscriptions and OpenAI API billing are separate.

Add the **Spring Web**, **Validation**, and **OpenAI** model starter to the Spring Boot application:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
	https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>4.1.0</version>
        <relativePath/>
    </parent>
    <groupId>com.sivalabs</groupId>
    <artifactId>01-chat-openai</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>01-chat-openai</name>

    <properties>
        <java.version>25</java.version>
        <spring-ai.version>2.0.0</spring-ai.version>
    </properties>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.ai</groupId>
                <artifactId>spring-ai-bom</artifactId>
                <version>${spring-ai.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
    
    <dependencies>
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-starter-webmvc</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-starter-validation</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-starter-model-openai</artifactId>
        </dependency>
    </dependencies>
    <!-- other configuration -->
</project>
```

Configure the API key and model in `application.properties`:

```properties
spring.ai.openai.api-key=${OPENAI_API_KEY}
spring.ai.openai.chat.model=gpt-5
```

Spring Boot now autoconfigures the OpenAI chat model and provides a `ChatClient.Builder` bean.

## Create a `ChatClient`

Inject the builder and create a `ChatClient`:

```java
@RestController
class ChatController {
    private final ChatClient chatClient;

    ChatController(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }

}
```

We will reuse this client for all our chat requests.

## Send your first prompt

Let's start with the smallest example: accept a question, send it to GPT-5, and return its answer.

```java
@PostMapping("/ai/chat")
Answer chat(@RequestBody @Valid Question question) {
    log.info("Prompt: {}", question.question());

    String response = chatClient
            .prompt(question.question())
            .call()
            .content();

    log.info("LLM Response: {}", response);
    return new Answer(response);
}

record Question(@NotBlank String question) {}

record Answer(String answer) {}
```

The API reads almost like a sentence:

* `prompt(question)` supplies the user input.
* `call()` sends a request to the model.
* `content()` extracts the generated text.

Try the endpoint:

```shell
curl --request POST http://localhost:8080/ai/chat \
  --header 'Content-Type: application/json' \
  --data '{"question":"Explain dependency injection in one paragraph"}'
```

## Add system and user messages

The previous example sends only the user's question. Usually, the application should also tell the
model how to behave. For example, we may want it to be friendly and professional.

```java
@PostMapping("/ai/chat2")
Answer chat2(@RequestBody @Valid Question question) {
    String response = chatClient
            .prompt()
            .system("You are a friendly, helpful assistant. You always respond professionally")
            .user(question.question())
            .call()
            .content();

    return new Answer(response);
}
```

The two messages serve different purposes:

* The **system message** defines the assistant's role and behaviour.
* The **user message** contains the user's question.

For most chat requests, this fluent style is the simplest option.

## Build a `Prompt` explicitly

Sometimes messages are created in another service or assembled dynamically. In that case, create the
message objects and `Prompt` directly.

```java
@PostMapping("/ai/chat3")
Answer chat3(@RequestBody @Valid Question question) {
    String systemPrompt = """
            You are a friendly, helpful assistant.
            You always respond professionally.
            """;

    SystemMessage systemMessage = new SystemMessage(systemPrompt);
    UserMessage userMessage = new UserMessage(question.question());

    Prompt prompt = new Prompt(List.of(systemMessage, userMessage));

    String response = chatClient
            .prompt(prompt)
            .call()
            .content();

    return new Answer(response);
}
```

`SystemMessage` and `UserMessage` represent messages with specific roles. `Prompt` groups those
messages into one model request.

This produces the same kind of request as the fluent example. Use it when you need direct access to
the message objects; otherwise, the fluent API is easier to read.

## Why do we need `PromptTemplate`?

Now suppose we want AI to suggest conference presentation titles. The instruction stays the same, but the topic
and number of titles change with every request.

Instead of building that prompt with string concatenation, we can use named placeholders:

```java
PromptTemplate promptTemplate = new PromptTemplate("""
        I would like to give a presentation about the following:

        {topic}

        Give me {count} title suggestions for this topic.

        Make sure the title is relevant to the topic and it should be a single short sentence.
        """);
```

Provide the placeholder values as a map and create a message:

```java
Map<String, Object> variables = Map.of(
        "topic", req.topic(),
        "count", req.count()
);

Message message = promptTemplate.createMessage(variables);

String response = chatClient
        .prompt()
        .messages(message)
        .call()
        .content();
```

`createMessage(variables)` replaces `{topic}` and `{count}` and returns a `UserMessage` ready to send.

A `PromptTemplate` can produce different results depending on what we need:

```java
String text = promptTemplate.render(variables);
Message message = promptTemplate.createMessage(variables);
Prompt prompt = promptTemplate.create(variables);
```

Use `render()` for text, `createMessage()` when adding it to other messages, and `create()` when the
template represents the complete prompt.

The endpoint accepts this request:

```java
record TitleSuggestionsRequest(
        @NotBlank String topic,
        @NotNull Integer count
) {}
```

```shell
curl --request POST http://localhost:8080/ai/suggest-titles \
  --header 'Content-Type: application/json' \
  --data '{"topic":"Building modular monoliths with Spring Boot","count":5}'
```

## Use an inline PromptTemplate

For a template used in only one request, `ChatClient` provides a shorter option:

```java
@PostMapping("/ai/suggest-titles2")
Answer suggestTitles2(@RequestBody @Valid TitleSuggestionsRequest req) {
    String response = chatClient
            .prompt()
            .system("You are a friendly, helpful assistant. You always respond professionally")
            .user(user -> user
                    .text("""
                            I would like to give a presentation about the following:

                            {topic}

                            Give me {count} title suggestions for this topic.

                            Make sure the title is relevant to the topic and it should be a single short sentence.
                            """)
                    .param("topic", req.topic())
                    .param("count", req.count())
            )
            .call()
            .content();

    return new Answer(response);
}
```

`text()` defines the template, while `param()` binds each placeholder. The instruction and its values
remain close to the model call, which makes short prompts easy to understand.

For long or shared prompts, move the text to a classpath resource and create `PromptTemplate` using that
Spring `Resource`. This keeps large prompts out of Java code.

## Which approach should you use?

| Requirement                      | API                                 |
|----------------------------------|-------------------------------------|
| Send one user question           | `chatClient.prompt(text)`           |
| Add system and user instructions | `prompt().system(...).user(...)`    |
| Assemble messages dynamically    | `new Prompt(messages)`              |
| Reuse a prompt with variables    | `PromptTemplate`                    |
| Use a short template once        | `user(u -> u.text(...).param(...))` |
| Store a long or shared template  | Resource-backed `PromptTemplate`    |


My preference is to start with the fluent `ChatClient` API. Use explicit `Message`, `Prompt`, and
`PromptTemplate` objects only when the prompt needs to be assembled, reused, or tested separately.

With these basics in place, we can move on to structured output, chat memory, tool calling,
and RAG in future articles.

# Support Me


If my articles, videos, or tutorials helped you solve a problem or learn something new, consider buying me a coffee. Your support helps me keep creating free tutorials, videos, and open-source projects for the Java community.

# Books


I have written books focused on helping developers learn Java and Spring Boot through practical examples.

# Conferences


I speak at developer conferences, Java user groups, meetups, and internal engineering events on Java, Spring Boot, 
Architecture, Testing, and Developer Productivity.

## Speaker
* [GIDS 2026 - Spring AI + MCP: Building the Missing Bridge Between Java Apps and AI Agents](https://www.youtube.com/watch?v=xWz-vXnKvEo)
* [Spring I/O 2026 - Skyrocket Developer Productivity with Spring Boot & IntelliJ IDEA](https://2026.springio.net/sessions/skyrocket-developer-productivity-with-spring-boot-and-intellij-idea/)
* [Naija JUG - Spring Boot : "From Beginner to Pro"](https://www.youtube.com/watch?v=ZFCbYLmLRKs)
* [GIDS 2025 - Building Modular Monoliths using Spring Modulith](https://www.youtube.com/watch?v=LOrTLDpVbAA)
* [Livestream: Productivity Hacks for Spring Boot Application Developers](https://www.youtube.com/watch?v=YCOXns1l_Us)
* [Podcast with Josh Long](https://www.youtube.com/watch?v=r0GckYxLugQ)
* [Podcast with Dan Vega and DaShaun Carter on Spring Office Hours](https://www.youtube.com/watch?v=i0pokvrl4xc)
* [Livestream: Developing Spring Boot Applications With Joy](https://www.youtube.com/watch?v=VvWtoaeHQUQ)
* [Bangalore JUG - Modern Spring Boot Application Development](https://www.meetup.com/bangalorejug/events/298925096/)
* [Bangalore JUG - Modern Spring Boot Application development with JDK 21](https://www.meetup.com/bangalorejug/events/300267722/)
* [Hyderabad JUG - Harnessing the Power of Java 21 and Spring Boot 3 for Modern Software Development](https://www.meetup.com/jughyderabad/events/300108982/)

## Topics

* Modern Spring Boot application development
* Building modular monoliths and microservices
* Testing Spring Boot applications with Testcontainers
* Practical architecture for backend systems
* Developer productivity and engineering practices
* Getting started with AI-assisted development

If you would like me to speak at your event, reach out through the social links on this site.

# Courses


Free video courses from the SivaLabs YouTube channel.

# Projects


## [1. testcontainers-jooq-codegen-maven-plugin](https://github.com/sivalabs/testcontainers-jooq-codegen-maven-plugin)

A Maven plugin to run jOOQ code generation using Testcontainers, so database schema-based code generation can run consistently without requiring a locally installed database.

## [2. sivalabs-agent-skills](https://github.com/sivaprasadreddy/sivalabs-agent-skills)

A collection of Agent Skills and guidelines for Java/Spring Boot application development.

## [3. sdd-skills](https://github.com/sivaprasadreddy/sdd-skills)

Agent Skills supporting Spec Driven Development with AI Agents.

## [4. progen](https://github.com/sivaprasadreddy/progen)

A project generator CLI tool for quickly creating Spring Boot applications with common best practices baked in.

## [5. generator-springboot](https://github.com/sivaprasadreddy/generator-springboot)
A Yeoman generator for generating MicroServices with Spring Boot. 

# AI Agents Are Doing The Heavy Lifting, Then Why Am I So Tired?

I love writing code.

Not just because it pays the bills.

I genuinely enjoy the process.

Give me a use case, let me think about the design, write a failing test, make it pass, refactor, move classes around, 
rename things, delete half the code I wrote an hour ago, run the test suite again, smile because everything is still green... 
and I can happily spend an entire day doing that.

That's my happy place.

It feels like this:

{{< figure src="/raw-images/nicolas-cage-fresh-air.gif" width="350" height="300" >}}

That feeling when you finally find the right abstraction after trying three terrible ones? Beautiful.

Then AI coding agents arrived. And like everyone else, I jumped in.

I spent a good amount of time setting things up properly. Spring Boot Agent Skills. MCP servers. Project rules. Coding conventions. Reusable prompts. The whole harness.

I wasn't doing vibe coding. I still wanted architectural control.

So instead of saying **"Build me an ecommerce application."** I would write detailed functional requirements, explain the boundaries, define the constraints, and let the agent do the mechanical work.

The results were... honestly impressive.

Feature after feature.

Tests. Documentation. Refactoring.

Everything kept appearing faster than I could have written it myself.

It wasn't just a little faster. It was significantly faster.

I remember thinking

_"Okay... this is it. I've officially become one of those ‘10x developer because of AI' people."_

Life was good.

Then every evening I noticed something strange.

I was exhausted.

Not "I've been debugging production at 2 AM" exhausted.

Just mentally drained.

Which made absolutely no sense.

The AI was reading the requirements.

The AI was designing the solution.

The AI was writing the code.

The AI was fixing compilation errors.

The AI was even updating the tests.

So...

**What exactly was I getting tired of?**

It took me a few days to figure it out.

Before AI, every feature started the same way.

I'd stare at the screen for five minutes pretending to think.

(Okay... actually thinking.)

Eventually an approach would emerge.

Then I'd implement it one small step at a time.

Because I had already decided the architecture, every new class made sense.

Every method existed because I had created it.

Every abstraction reflected a decision I had already made.

By the time I finished the feature, I didn't need to "understand" the code.

I already understood it.

I had lived through every line.

AI completely changes that experience.

Now I write a prompt.

The agent disappears for a minute.

Then returns with enough code to keep me busy for half an hour.

I glance through it.

Looks reasonable.

{{< figure src="/raw-images/lame-code-review.gif" width="350" height="300" >}}

Merge.

Next use case.

Looks reasonable.

Merge.

Next use case.

Looks reasonable.

Merge.

After about ten use cases, I noticed something small.

Just a tiny refactoring.

"This helper method could probably be reused."

I opened the module.

Made the change.

Then I noticed another method doing almost the same thing.

"Hmm..."

Then I found validations split between the controller and the service.

"Interesting..."

Then every test class had invented its own way of creating test data.

"That's... creative."

Then there were two utility methods that nobody called anymore.

Then there were three slightly different implementations of exactly the same logic.

At some point I stopped refactoring.

I had accidentally started cleaning an entire module.

Then another.

Then another.

And somewhere around hour three, I realized something uncomfortable.

I hadn't actually reviewed the AI-generated code.

I had only checked whether it looked reasonable.

Those are not the same thing.

We all love saying,

> "Always review every line before it goes to production."

Absolutely.

We should.

But let's also admit something.

Most developers don't wake up excited to review someone else's code.

Writing code is creative.

Reviewing code is archaeology.

When you review someone else's implementation, you're digging through layers trying to reconstruct their thinking.

_"Why did they introduce this abstraction?"_

_"What problem were they trying to solve?"_

_"Why is this validation here instead of there?"_

_"Was this intentional... or just Tuesday afternoon?"_

You're constantly rebuilding another person's mental model inside your own head.

That's surprisingly exhausting.

Now replace "someone else" with an AI that produces code ten times faster than you can read it.

Congratulations.

You've become a professional code reviewer.

That was the moment everything clicked.

The AI wasn't making me tired by writing code.

It was making me review far more code than I ever had before.

So I changed my workflow.

No more quick scans.

From that day on, every generated feature got a proper review before moving to the next task.

Which introduced a new problem.

The AI can generate code much faster than I can thoroughly review it.

By lunchtime, it happily has three more features waiting.

Meanwhile, I'm still tracing through the second service wondering,

_"Wait... where does this DTO actually get validated?"_

The bottleneck had moved.

It wasn't writing code anymore.

It was understanding code.

Ironically, AI had automated the part I enjoy the most and left me spending most of my day doing the part I enjoy the least.

Now, can this be improved?

Absolutely.

Every time I find a recurring mistake, I teach the agent not to make it again.

Sometimes it's another Agent Skill.

Sometimes it's a better project rule.

Sometimes it's another guardrail.

Sometimes it's a reusable prompt.

The quality keeps improving.

And that's genuinely satisfying.

But even with a solid harness (Agent Skills, MCP servers, project rules, detailed prompts, coding standards), the agent occasionally decides that today is a good day to invent a brand-new pattern that exists nowhere else in the project.

AI has a remarkable ability to be consistently excellent...

...and occasionally spectacularly creative.

Which means human review is still necessary.

At least for now.

I often see people saying,

> "I'm running twenty agents simultaneously."
> 
> "I'm a hundred times more productive."

Maybe they are.

Maybe their work parallelizes beautifully.

Maybe they're simply much better at context switching than I am.

If that's true, honestly, hats off.

Me?

I can deeply focus on maybe two things before my brain starts filing formal complaints.

So no.

I may probably never become a 100x developer.

What AI has given me is something different.

It lets me produce far more software.

It removes a huge amount of mechanical work.

It catches plenty of mistakes.

It remembers things I forget.

It absolutely makes me more productive.

But it hasn't removed my responsibility to understand the code that's about to ship.

And understanding code that somebody or something else wrote is still hard work.

One approach that I've found helpful is asking the AI to work in much smaller increments.

Generate one service.

Review it.

Generate one repository.

Review it.

Generate one controller.

Review it.

The reviews become dramatically easier because the changes are small.

The downside is obvious.

You spend much more time talking to the AI instead of letting it disappear for fifteen minutes and build an entire feature.

Like everything else in software engineering...

It's another trade-off.

So these days, whenever someone tells me

"AI lets me build software ten times faster."

I completely believe them.

But I also wonder something else.

**Who's reviewing all that code?**

Because after spending the last few weeks with AI coding agents, I've become convinced of one thing.

Writing code is fun.

Reading code is work.

AI is incredibly good at making us do a lot more of the second.

Now I'm curious.

How are you dealing with AI fatigue?

Have you found a workflow that keeps code reviews manageable without sacrificing quality? Or have you simply accepted that reviewing AI-generated code is now a significant part of the job?

I'd love to hear your experience.
# Confessions of an AI Agent FOMO Survivor

Every morning, I open X/Twitter, LinkedIn, or YouTube and learn about a brand-new AI engineering discipline that apparently everyone else has already mastered.

A few months ago it was Prompt Engineering.

Then came Context Engineering.

Soon after, people started talking about **Agent Skills, Harness Engineering, Loop Engineering, Ralph Wiggum Loops, Sub-Agent Architectures, Agentic Workflows, Cognitive Tooling**, and probably three more terms while I was writing this sentence.

At this point I fully expect to wake up tomorrow and discover that "Recursive Quantum Harness Context Loop Engineering" is the new must-have skill for AI developers.

And honestly?

It can feel overwhelming.

Every new buzzword comes with an implicit message:

> "If you're not doing this, you're not leveraging AI capabilities fully."

Which immediately triggers a mild form of AI FOMO.

Maybe my setup is outdated.

Maybe I'm missing the secret sauce.

Maybe there is a GitHub repository with 47 agent skills, 12 MCP servers, and a five-level hierarchy of sub-agents that would finally unlock AGI.

So naturally, I spend an hour reading about it instead of actually building software.

## The Surprising Discovery
Over the last several months of working with AI coding agents, I've discovered something that feels almost disappointing:

**A minimal setup works surprisingly well.**

Not because all the new ideas are useless.

Many of them are genuinely valuable.

But I've found that for day-to-day software development, especially on Java/Spring Boot projects, I don't need a giant agent infrastructure to be productive.

Most of the time, I just need enough structure for the agent to understand the project and enough guidance to keep it from doing creative things that nobody asked for.

## My Typical Setup
When starting a new Spring Boot project, I almost never ask an AI agent to generate the initial project structure.

Instead, I use a project generator. Spring Initializr exists for a reason.

The generated project uses the latest versions, recommended defaults, and current conventions.

I'd rather start with a project created by the people who maintain the framework than have an AI confidently generate a Maven configuration from six months ago.

Once the project is created, I manually implement one complete end-to-end flow.

For example:

* Controller
* Service
* Repository
* Entity
* DTOs
* Tests

A complete vertical slice.

This is probably the most important thing I do.

That flow becomes the living example of the conventions I want the agent to follow.

Naming conventions.

Package structure.

Error handling.

Testing style.

Mapping patterns.

Validation approach.

Instead of writing a ten-page document explaining architectural preferences, I give the AI a working example.

Developers learn from examples.

AI agents do too.

## The Tiny AGENTS.md
Another thing I've learned is that a bigger context is not automatically better context.

My AGENTS.md (or CLAUDE.md) files are intentionally small.

I include only things that are genuinely useful:

* Project overview
* Build commands
* Test commands
* A few important conventions

That's it.

Not fifty pages of documentation.

Not an encyclopedia of company history.

Not a manifesto on software craftsmanship.

Every extra token has a cost.

And every additional instruction creates another opportunity for conflicts, confusion, or context dilution.

Sometimes I see people stuffing everything imaginable into agent instruction files.

The AI equivalent of:

> "Just in case, let me attach the entire internet."

Meanwhile, the agent is trying to figure out how to add a new REST endpoint.

## Agent Skills: Useful, But Selectively
I do use agent skills.

For Spring Boot projects, I often use the Spring Boot Agent Skills from:

https://github.com/sivaprasadreddy/sivalabs-agent-skills

The key word is relevant.

I choose skills that directly help with the type of software I'm building.

Not because they are trending.

Not because somebody's GitHub repository got ten thousand stars.

Not because a YouTube thumbnail promised "10X Better Agents."

Just because they're useful.

That's a surprisingly effective selection criterion.

## MCP Servers: Less Is More
My MCP server setup is equally boring.

Typically:

* Context7 for looking up the latest documentation
* Playwright for UI testing and fixes

That's usually enough.

Every MCP server introduces additional capabilities, complexity, and potential distractions.

Sometimes I see screenshots where people have twenty MCP servers connected.

The agent has access to databases, Slack, Jira, GitHub, browsers, calendars, weather reports, stock prices, smart refrigerators, and possibly the International Space Station.

Meanwhile, I'm just trying to implement a pagination endpoint.

## Specs or Detailed Prompts
Sometimes I use Spec Driven Development.

Sometimes I don't.

When the task is large or requires multiple iterations, a specification document works well.

For smaller tasks, I often write a detailed prompt in a Markdown file and use that as input.

```markdown
# REQ_001 - User Registration

## Requirement Description
Create a new user account with basic profile and credentials.

## Preconditions
- Request payload should contain non-blank `name`, valid `email`, and non-blank `password`.
- Email must be unique in persistence layer.

## Postconditions
- A new user record is persisted.
- Password is stored in encoded form.
- User role is set to `ROLE_USER`.

## API
- HTTP Method: `POST`
- URL: `/api/users`
- Headers:
  - `Content-Type: application/json`

## Request Body

{
  "name": "TestUser",
  "email": "testuser@gmail.com",
  "password": "Secret@1234"
}

## Response Body
- Status: `201 Created`

{
  "name": "TestUser",
  "email": "testuser@gmail.com",
  "role": "ROLE_USER"
}

## Error Responses
- `400 Bad Request` (validation failure)
    - Title: `Validation Error`
    - `errors` includes messages like:
        - `Name is required`
        - `Invalid email address`
        - `Password is required`
- `500 Internal Server Error` (e.g., persistence constraint/unhandled errors)
```

I've found that the format matters less than the clarity.

If the following are clearly defined:

* Inputs
* Outputs
* Constraints
* Acceptance criteria

The agent usually does a decent job.

Not perfect.

But surprisingly good.

Occasionally I need to make corrections.

Sometimes I need to redirect it.

Sometimes it invents abstractions that nobody requested.

But overall, the results are often much better than people assume.

## Things I Try to Avoid
One of the biggest temptations in the AI ecosystem is collecting agent configurations like Pokémon cards.

A new repository appears.

Everyone stars it.

Someone posts:

> "This changed everything."

Five minutes later you've cloned:

* 37 agent skills
* 12 sub-agent configurations
* 8 workflow frameworks
* 4 prompt libraries
* 1 mysterious folder that nobody understands

The assumption is:

> More agent stuff = better agents.

In my experience, that's not always true.

Every skill adds instructions.

Every configuration adds opinions.

Every framework adds assumptions.

Eventually, the agent receives conflicting guidance from multiple sources and starts behaving like a committee trying to design a horse.

You asked for a simple CRUD endpoint.

_The agent delivers a distributed event-driven hexagonal architecture with CQRS and three design patterns nobody can explain._

There's also the security aspect.

Pulling random agent skills from the internet and giving them access to your development workflow deserves at least the same scrutiny you'd apply to any other dependency.

Maybe more.

## About All The Loop Engineering Stuff
Now, before the internet gets upset, I'm not saying the newer agent techniques are useless.

Far from it.

I think approaches like Ralph Wiggum loops, loop engineering, multi-agent systems, autonomous task execution, and similar ideas can absolutely improve agent capabilities.

Especially for long-running tasks and highly autonomous workflows.

If your goal is:

> "Here is a backlog. Wake me up when it's done."

Then sophisticated orchestration may be exactly what you need.

But let's be honest.

Not everyone has unlimited AI budgets.

Not everyone wants agents consuming tokens all night while recursively discussing architecture with themselves.

Not everyone needs a digital employee working unsupervised for eight hours.

Many of us are simply trying to build software faster and better during the workday.

For that use case, a relatively simple setup often delivers most of the value.

## Simpler Than Expected
The longer I work with AI coding agents, the more I appreciate boring solutions.

A clean project.

One well-implemented example.

A small instruction file.

A handful of relevant skills.

A couple of useful MCP servers.

Clear requirements.

That's it.

No revolutionary framework.

No secret prompt.

No agent enlightenment ceremony.

Just enough structure for the agent to succeed.

And surprisingly often, that works really well.

Maybe I'll eventually adopt the latest loop engineering technique.

Maybe next year's buzzword will genuinely change everything.

Or maybe we'll discover that software engineering's oldest lesson still applies:

> **Simplicity scales surprisingly far.**

I'm curious what kind of setup you're using.

Do you have a minimal agent harness?

Are you running a fleet of specialized subagents?

What has worked well for you?

And perhaps more importantly...

What didn't work?

# Don't Pollute Your Spring Boot Main EntryPoint Class


Most Spring Boot applications start with a beautifully boring main class.

Then one day we need caching. We add `@EnableCaching`.

Then we need async processing. We add `@EnableAsync`.

Then scheduling. Then JPA auditing. Then maybe something else.

Before we know it, the main class has become the place where every framework feature goes to live.

At first glance, this feels harmless. The application starts. The feature works. Everybody moves on.

This works great... until a slice test gets involved.

Let's look at why adding every `@EnableXXX` annotation to the Spring Boot main class can make your tests more fragile, and what I prefer to do instead.

<!--more-->

## The Main Class Should Be Boring

In a typical Spring Boot application, the main class looks like this:

```java
package dev.sivalabs.demo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class Application {

    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }

}
```

This class has one job: start the application.

That is it.

But the quickest way to enable many Spring features is to add an annotation directly here:

```java
package dev.sivalabs.demo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cache.annotation.EnableCaching;
import org.springframework.data.jpa.repository.config.EnableJpaAuditing;
import org.springframework.scheduling.annotation.EnableAsync;
import org.springframework.scheduling.annotation.EnableScheduling;

@SpringBootApplication
@EnableCaching
@EnableAsync
@EnableScheduling
@EnableJpaAuditing
public class Application {

    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }

}
```

You will see this style in many projects. I have written this style too.

And to be fair, it works.

The problem is not that the application fails to start. The problem is that the main class is now carrying configuration for multiple technical concerns:

* caching
* async execution
* scheduling
* persistence auditing

Those concerns may not all be needed in every Spring ApplicationContext you create.

And tests create many different kinds of ApplicationContexts.

## Here's Where The Trouble Starts

Let's use a small example with Spring Data JPA auditing.

Suppose we have a `User` entity with `createdAt` and `updatedAt` fields:

```java
import jakarta.persistence.EntityListeners;
import org.springframework.data.annotation.CreatedDate;
import org.springframework.data.annotation.LastModifiedDate;
import org.springframework.data.jpa.domain.support.AuditingEntityListener;

@Entity
@Table(name = "users")
@EntityListeners(AuditingEntityListener.class)
class User {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @Column(nullable = false)
    private String name;

    @Column(nullable = false, unique = true)
    private String email;

    @Column(name = "created_at")
    @CreatedDate
    protected LocalDateTime createdAt;

    @Column(name = "updated_at")
    @LastModifiedDate
    protected LocalDateTime updatedAt;

    //setters & getters

}
```

The entity uses Spring Data JPA auditing through `@CreatedDate` and `@LastModifiedDate`.

Then we create a repository:

```java
interface UserRepository extends JpaRepository<User, Long> {
}
```

And a simple REST controller:

```java
@RestController
@RequestMapping("/api/users")
class UserController {

    private final UserRepository userRepository;

    UserController(UserRepository userRepository) {
        this.userRepository = userRepository;
    }

    @GetMapping
    List<User> getAll() {
        return userRepository.findAll();
    }
}
```

To make auditing work, we need to enable it using `@EnableJpaAuditing`.

The easy option is to put it on the main class:

```java
@SpringBootApplication
@EnableJpaAuditing
public class Application {

    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }

}
```

Again, the application starts fine.

But now let's write a focused test for the web layer:

```java
@WebMvcTest(controllers = UserController.class)
class UserControllerTests {

    @MockitoBean
    UserRepository userRepository;

    @Autowired
    protected MockMvcTester mvc;

    @Test
    void shouldGetAllUsers() {
        given(userRepository.findAll()).willReturn(List.of());

        var testResult = mvc.get().uri("/api/users").exchange();

        assertThat(testResult).hasStatusOk();
    }
}
```

`@WebMvcTest` is intentionally narrow. It loads the MVC layer, not the entire application.

That is exactly why we use it. We want to test request mapping, validation, JSON serialization, HTTP status codes, exception handling, and controller behavior without starting the database layer.

We also provide the controller dependency using a mock:

```java
@MockitoBean
UserRepository userRepository;
```

So, in theory, JPA should not matter here.

But if `@EnableJpaAuditing` is sitting on the main class, the test can fail with an error like this:

```shell
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jpaAuditingHandler': Cannot resolve reference to bean 'jpaMappingContext' while setting constructor argument
....
....
....
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jpaMappingContext': JPA metamodel must not be empty
....
....
```

This is the annoying part.

We are testing a controller. We mocked the repository. We did not ask Spring to start JPA.

But JPA auditing still entered the room because we attached it to the main application configuration.

## So What's Actually Going On?

Spring Boot test slices like `@WebMvcTest` still need to find the application's boot configuration.

In a normal project, that usually means the class annotated with `@SpringBootApplication`.

If that class only says "this is my Spring Boot application", life is simple.

But if that class also enables JPA auditing, scheduling, async processing, caching, or other infrastructure, those choices become part of the boot configuration that test slices have to deal with.

In this specific case, `@EnableJpaAuditing` tries to create auditing infrastructure. That infrastructure expects JPA mapping support to be available. A web slice does not load the full JPA stack, so Spring ends up trying to create a persistence-related bean in a context where persistence was never supposed to exist.

That is how you get a JPA error from a controller test.

The framework is not being unreasonable. We gave it mixed signals.

## Move Feature Configuration To Its Own Place

The fix is simple: move `@EnableJpaAuditing` into a dedicated configuration class.

```java
import org.springframework.context.annotation.Configuration;
import org.springframework.data.jpa.repository.config.EnableJpaAuditing;

@Configuration
@EnableJpaAuditing
public class PersistenceConfig {
    
}
```

And keep the main class clean:

```java
package dev.sivalabs.demo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class Application {

    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }

}
```

Now the application still enables JPA auditing during a normal application startup, because `PersistenceConfig` is part of component scanning.

But the main class no longer forces every test slice to carry that persistence concern as part of the boot configuration.

Run the `@WebMvcTest` again, and it should pass.

## This Is Not Just About JPA Auditing

JPA auditing is a good example because it fails loudly.

But the same smell appears with other annotations too:

```java
@SpringBootApplication
@EnableCaching
@EnableAsync
@EnableScheduling
@EnableJpaAuditing
public class Application {
}
```

Each annotation may be perfectly valid.

The question is: should it live on the main class?

In most business applications, I prefer grouping these concerns into small configuration classes:

```java
@Configuration
@EnableCaching
class CacheConfig {
}
```

```java
@Configuration
@EnableAsync
class AsyncConfig {
}
```

```java
@Configuration
@EnableScheduling
class SchedulingConfig {
}
```

```java
@Configuration
@EnableJpaAuditing
class PersistenceConfig {
}
```

This is not about creating configuration classes for the sake of architecture theater.

It is about making each technical concern explicit, named, and easier to include or exclude when needed.

The main class should be the front door of the application.

Don't turn it into the storage room.

# A Few Hard Learned Lessons On Structuring Spring Boot Applications

One of the most common questions developers ask when starting a Spring Boot project is: 

**How should I structure my code?**

If you spend enough time in Java circles, you'll quickly run into recommendations like **Clean Architecture, Hexagonal Architecture, Onion Architecture, Ports and Adapters**, and a dozen variations of the same idea.

Now, before anyone gets upset: I think those architectures solve real problems.

The issue is that many teams adopt them on day one, long before they actually have those problems.

Suddenly a simple CRUD application has ports, adapters, facades, mappers, DTOs, command handlers, query handlers, and enough boilerplate to make you wonder whether you're building software or preparing tax documents.

Over the years I've gravitated toward a much simpler approach.

It gives me most of the benefits I care about:

* Clear module boundaries
* Separation of concerns
* Testable business logic
* Maintainable code

...without drowning the codebase in ceremony.

And now that AI coding agents are becoming part of everyday development, having a simple and consistent set of conventions matters even more.

Let me share the conventions I use and how they can be turned into Agent Skills so AI-generated code also follows the same structure.

## Why Are My Files Scattered Across 20 Packages?
The traditional Spring Boot structure organizes code by technical role:

```shell
dev.sivalabs.bookstore/
├── controllers/
│   ├── UserController.java
│   ├── OrderController.java
├── services/
│   ├── UserService.java
│   ├── OrderService.java
├── repositories/
│   ├── UserRepository.java
│   ├── OrderRepository.java
```

At first glance it looks neat and organized.

Until you need to change one feature.

Then adding a single order field turns into a mini treasure hunt across **controllers, services, repositories**, 
and probably a **mapper** package lurking somewhere.

Even worse, this structure does nothing to protect module boundaries.

Nothing stops **OrderController** from calling **UserRepository** directly.

The compiler shrugs. Spring happily wires it.

You find out six months later during a refactoring session that starts with optimism and ends with bargaining.

That's why I prefer organizing code by business feature rather than technical layer.

**Package by feature** groups everything belonging to a business domain together:

```shell
dev.sivalabs.bookstore/
├── Application.java
├── shared/                     # Cross-cutting utilities
├── users/                      # Users bounded context
│   ├── config/
│   ├── domain/
│   │   ├── models/
│   │   ├── UserEntity.java
│   │   ├── UserRepository.java
│   │   ├── UserMapper.java
│   │   └── UserService.java
│   ├── api/
│   │   ├── UserController.java
│   │   ├── CreateUserRequest.java
│   │   └── CreateUserResponse.java
│   └── UsersAPI.java
├── orders/
├── catalog/
└── config/
    ├── WebMvcConfig.java
    ├── SecurityConfig.java
    └── GlobalExceptionHandler.java
```

Now all the code related to users lives together.

If I need to understand the Users module, I don't have to play hide-and-seek across the entire codebase. Everything is right there inside the users package.

## Why Not Keep Everything Flat Inside the Module?
This is a question I get quite often.

After all, if everything belongs to the **Users** module, why not put everything directly under `users/`?

Because developers are human.

Give them access to an HTTP request DTO, and eventually someone will pass it directly into the service layer.

Give them access to an HTTP-specific type and eventually it leaks into business logic.

Good package structures don't just organize code.

They make the wrong thing harder to do.

If **UserController**, **CreateUserRequest**, **CreateUserResponse**, and **UserService** all live in users/ together, 
the request/response DTOs are visible to **UserService** and **UserRepository**.

Nothing in the language prevents a developer from accidentally leaking an HttpServletRequest-bound DTO into a service method signature.

Separating **domain** from **api** makes this impossible: **UserService** cannot even see **CreateUserRequest** because it lives inside the api package with package-private visibility.

## The Business Rules Shouldn't Care How the Request Arrived
Today the request might arrive through a REST endpoint. Tomorrow it might come from a Kafka event handler, a scheduled job, a CLI command, or something else entirely.

The business rules shouldn't change just because the delivery mechanism changed.

That's why I like keeping inbound request handlers thin and pushing all business logic into the domain layer.

The job of an inbound request handler (HTTP Request Handler, Kafka Listener, Scheduled Job, etc.) is surprisingly simple:

* Accept input
* Validate input
* Convert input into a command or query
* Call business logic
* Return a response

That's it.

```java
// api/UserController.java  — Inbound HTTP request handler
@RestController
@RequestMapping("/api/users")
class UserController {

    private final UserService userService;

    UserController(UserService userService) {
        this.userService = userService;
    }

    @PostMapping
    ResponseEntity<CreateUserResponse> createUser(@RequestBody @Valid CreateUserRequest request) {
        var cmd = new CreateUserCmd(request.name(), request.email());
        var result = userService.createUser(cmd);
        return ResponseEntity.status(HttpStatus.CREATED).body(new CreateUserResponse(result.userId()));
    }
}
```

Meanwhile, the service orchestrates the actual business rules:

```java
// domain/UserService.java  — business logic
@Service
@RequiredArgsConstructor
public class UserService {

    private final UserRepository userRepository;
    private final UserMapper userMapper;

    @Transactional
    public RegistrationResult createUser(CreateUserCmd cmd) {
        if (userRepository.existsByEmail(cmd.email())) {
            throw new UserAlreadyExistsException(cmd.email());
        }
        var entity = userMapper.toEntity(cmd);
        var saved = userRepository.save(entity);
        return new RegistrationResult(saved.getId());
    }
}
```

Keeping these responsibilities separate pays off in two ways:

First, testing becomes dramatically simpler.

Second, adding a new delivery mechanism becomes almost trivial because your business logic is already isolated.

## public, public, public Everywhere... Please Stop It
One thing I see in many Spring Boot applications is this:

Everything is **public**.

Controllers? Public.

Services? Public.

Repositories? Public.

Entities? Public.

Mappers? Public.

Utility classes? Also public.

{{< figure src="/images/public-everywhere.webp" >}}

At that point your module boundaries aren't really boundaries anymore.

They're just decorations.

Java gave us package-private visibility for a reason.

We should use it.

My default rule is simple:

**If a class does not need to be visible outside the module, don't make it public.**

That single rule dramatically reduces accidental coupling.

It also makes refactoring much safer because fewer parts of the system can depend on internal implementation details.

```markdown
| Class type                        | Visibility      | Reason                                                                |
|-----------------------------------|-----------------|-----------------------------------------------------------------------|
| UserEntity                        | package-private | Direct entity access couples modules to the DB schema                 |
| UserRepository                    | package-private | Data access belongs to the owning module                              |
| UserMapper                        | package-private | Internal conversion detail                                            |
| UserController                    | package-private | Registered by Spring via component scan; no external reference needed |
| UserService                       | public          | Exposed to inbound request handlers within the same module            |
| CreateUserCmd, UserVM, UserId ... | public          | Shared domain types placed in `domain/models/`                        |
```

## One Front Door Per Module
A service often exposes many methods.

Some are intended for internal use.

Others are safe for other modules to call.

**The mistake is exposing everything.**

Imagine a company where every visitor can walk directly into every office.

That sounds ridiculous.

Most companies have a reception desk.

Your modules should too.

That's what the ModuleAPI facade is.

Other modules interact with the facade.

The facade decides what is available.

Everything else stays internal.

```java
// users/UsersAPI.java
@Service
public class UsersAPI {

    private final UserService userService;

    UsersAPI(UserService userService) {
        this.userService = userService;
    }

    public RegistrationResult createUser(CreateUserCmd cmd) {
        return userService.createUser(cmd);
    }

    public Optional<UserVM> findUserById(Long id) {
        return userService.findUserById(id);
    }
}
```

This keeps dependencies intentional instead of accidental.

## Trust Developers, But Verify With Tests
Every team starts with good intentions.

Someone creates an architecture document.

Everyone agrees it looks great.

The wiki gets published.

People nod approvingly.

{{< figure src="/images/follow-guidelines-right.webp" >}}

Six months later someone injects **UserRepository** directly into an unrelated module, and nobody remembers when the architecture document was last opened.

The uncomfortable truth is this:

**If the build doesn't fail, it's probably just a suggestion.**

That's why I like automated architecture verification.

## ArchUnit: Turning "Don't Do That" Into a Build Failure
Guidelines written in a wiki erode over time. Not because people are careless. Because people are busy, deadlines exist, and "temporary" code has a surprisingly long lifespan.

The only reliable way to keep architecture rules alive is to make violations fail the build.

[ArchUnit](https://www.archunit.org/) lets you convert architecture guidelines into executable tests.

Instead of saying:

**"Repositories should not be accessed outside the module."**

you can make the build fail when somebody does exactly that.

Suddenly, architecture stops being a PowerPoint slide and becomes part of your CI pipeline.

Example tests:

```java
@AnalyzeClasses(packages = "dev.sivalabs.bookstore")
class ArchitectureTests {

    // Controllers must not be called by service or repository classes
    @ArchTest
    static final ArchRule controllers_should_not_be_accessed_by_domain =
        noClasses()
            .that().resideInAPackage("..domain..")
            .should().accessClassesThat()
            .resideInAPackage("..api..");

    // Services must not import anything from the api/web layer
    @ArchTest
    static final ArchRule services_must_not_depend_on_controllers =
        noClasses()
            .that().haveNameMatching(".*Service")
            .should().dependOnClassesThat()
            .haveNameMatching(".*Controller");

    // Repositories must not be accessed from outside their own module
    @ArchTest
    static final ArchRule repositories_are_not_public =
        noClasses()
            .that().haveNameMatching(".*Repository")
            .should().bePublic();

    // Entities must not be public
    @ArchTest
    static final ArchRule entities_are_not_public =
        noClasses()
            .that().haveNameMatching(".*Entity")
            .should().bePublic();
}
```

You can check out the [Taikai](https://github.com/enofex/taikai) library that provides a comprehensive suite of predefined rules for Spring Boot.

## Spring Modulith: Don't Just Define Boundaries, Verify Them
ArchUnit is excellent for enforcing custom rules.

[Spring Modulith](https://spring.io/projects/spring-modulith) goes one step further by understanding your module structure directly.

It can verify dependencies, detect cycles, validate boundaries, and even generate documentation from your application structure.

That's a pretty good deal for a dependency you add to your test scope.

```xml
<dependency>
    <groupId>org.springframework.experimental</groupId>
    <artifactId>spring-modulith-test</artifactId>
    <scope>test</scope>
</dependency>
```

Modulith verification test:

```java
@Test
void verify_module_structure() {
    ApplicationModules.of(Application.class).verify();
}
```

**ApplicationModules.verify()** checks the things we usually hope people remember:

* No module accesses internal (package-private) types of another module
* There are no dependency cycles between modules
* Each module's public API is used for cross-module communication


{{< box info >}}
**Migrating to Modular Monolith using Spring Modulith and IntelliJ IDEA**

Check out [Migrating to Modular Monolith using Spring Modulith and IntelliJ IDEA](https://blog.jetbrains.com/idea/2026/02/migrating-to-modular-monolith-using-spring-modulith-and-intellij-idea/) if you want more details on how to use Spring Modulith.
{{< /box >}}

## Teaching AI Your Architecture Rules Instead of Hoping for the Best
Here's something I've noticed while working with AI coding agents:

They're basically junior developers with infinite energy and zero context.

Ask an AI agent to:

**"Create a user management feature."**

and unless you've provided guidance, you'll probably get:

```shell
controllers/
entities/
services/
repositories/
```

With every class marked **public**.

The AI isn't being careless. It simply doesn't know your conventions.

That's where [Agent Skills](https://agentskills.io/) become incredibly useful.

Instead of repeating the same architectural guidance in every prompt, you encode the rules once and let the agent apply them consistently.

I have created [Spring Boot Skills](https://github.com/sivaprasadreddy/sivalabs-agent-skills/tree/main/skills/spring-boot) which also include how to [organize code](https://github.com/sivaprasadreddy/sivalabs-agent-skills/blob/main/skills/spring-boot/references/code-organization.md) as described above.

The interesting thing is that the conventions work especially well as Agent Skills because they're simple.

Humans can remember them.

AI agents can follow them.

And architecture tests can verify them.

That's a surprisingly effective combination.

{{< box info >}}
**AI-Assisted Java Application Development with Agent Skills**

Check out [AI-Assisted Java Application Development with Agent Skills](https://blog.jetbrains.com/idea/2026/03/ai-assisted-java-application-development-with-agent-skills/) to explore how to create Agent Skills.
{{< /box >}}

## The Rules I Keep Coming Back To
After building Spring Boot applications for years, these are the conventions I keep returning to:

```markdown
| Principle                  | Why I Like It                   |
|----------------------------| ------------------------------- |
| Package by feature         | Related code stays together     |
| Separation of concerns     | Business logic remains reusable |
| Minimize What You Expose   | Boundaries become real          |
| ArchUnit + Modulith        | The build enforces the rules    |
```

Do you need full-blown Clean Architecture or Hexagonal Architecture to build a maintainable Spring Boot application?

Sometimes.

Most of the time?

Probably not.

A small set of well-enforced conventions gets you surprisingly far.

* Package code by business feature.
* Keep controllers thin.
* Expose less.

Verify the rules automatically.

That's the structure I've found easiest to understand, easiest to maintain, and easiest to teach, to both developers and AI agents.

Because these conventions are simple and consistent, you can hand them to your AI coding agent as an Agent Skill. Then the code it generates follows the same structure from the first prompt, instead of after five rounds of "not like that".

Try building a Spring Boot application using [Spring Boot Skills](https://github.com/sivaprasadreddy/sivalabs-agent-skills/tree/main/skills/spring-boot). 
Or, refactor an existing application which is following layered architecture to use the recommended code organization.

# Before You Judge, Get Curious


A few years ago, I was working at a fintech company in Germany. Our business unit planned to build a new offering for customers in Germany and the UK. 
Our team — product owner, business analyst, developers, and QA — gathered at a location and ran brainstorming sessions. 
Since we were also using the product ourselves, we were essentially both the builders and the end users.

<!--more-->

After a full day of discussions, we collectively came up with a list of features for the first MVP. 
The BA took brief notes, and later, together with the developers, prepared JIRA cards for all those features. 
We then expanded on the requirements to flesh out the use case details and added acceptance criteria. 
In that company, we (developers) worked closely with the product owner and BA throughout the entire requirements process.

---

A few years later, I worked at a client site on a banking project. 
The culture was very different. Business folks with deep domain knowledge were always busy and could only spare limited time for requirement-gathering sessions. 
Typically, the BA would talk to the business team, prepare requirement documents, and get approval from the business owner — only then would developers enter the picture. 
That was simply how that organization operated.

---

Now consider a solo entrepreneur building a SaaS product. They have to wear many hats: market research, validating the business idea, writing requirements, implementing them, testing, and releasing the product. 
In this context, the developer does it all — from preparing requirement specs to shipping the software.

---

The bottom line is that different organizations have different ways of working. In some, the BA owns the requirement specs; in others, the PO, BA, developers, and QA all collaborate on them together. 
And in the solo-entrepreneur scenario, the developer steps into the roles of business owner and end user to prepare the specs themselves.

To truly appreciate this diversity, you either need to have worked across organizations with different cultures, or have the open-mindedness to accept that your way isn't the only way.

What if a developer spent their entire career at companies where fully-prepared requirement specs were handed to them, and developers simply started from the implementation phase — and then decided that's the *only* right way to build software?

Some people live in a bubble, too close-minded or arrogant to even consider that other organizations might operate differently. For them, the very idea of a developer taking a rough requirement and fleshing out the details and acceptance criteria seems unthinkable. 
But if they stepped outside that bubble and talked to people from other organizations — or to solo SaaS builders — they might gain a very different perspective.

---

Recently, I came across a LinkedIn post asking: *"In the age of AI-assisted development, do you care about code quality?"*

While most people replied "Yes, code quality is important," one CTO responded:

> We used to care a lot about code quality and put a lot of quality gates in the delivery process.
> But now we are optimizing for the quality and speed of the outcome, not for code quality.

The original post author then went: *"Our friend XYZ thinks code quality doesn't matter. Let's help him"* — subtly implying, "You are wrong and you need help."

My knee-jerk reaction to hearing "code quality doesn't matter" would be "That's ridiculous — of course code quality matters."

But what if, instead of being judgmental, we actually tried to understand *why* he feels that way?

After all, he is a CTO — responsible for leading his team and making sure things are going well. If he's saying they manage to deliver a solid product without obsessing over code quality, isn't it worth asking *how* he pulls that off?

We may strongly believe code quality matters, and we likely have painful past experiences to back that up. So how about asking the CTO how they handle the problems that typically come with lower code quality? By simply asking those questions, one of three things happens: you understand how they manage without it, the CTO realizes they hadn't considered your points, or you realize the CTO is simply bluffing.

Either way, asking questions to gain clarity is far better than deciding someone is wrong and needs saving.

---

There's no shortage of judgmental people who live in a bubble and believe only they know what's right.

If you think you can change their minds by explaining your point of view, you're most likely wasting your time. They're incapable of seeing things from your perspective. Clarify one point, and they'll immediately find another angle to prove you wrong.

---

In this fast-paced world, your time and mental peace are precious. If there's a disagreement with a teammate or colleague, it makes sense to invest time in understanding each other and reaching a common ground — that's worth it.

But spending energy explaining yourself to a random judgmental stranger on the internet who has zero interest in asking questions or understanding where you're coming from? That's just draining yourself for nothing.

Be open to constructive criticism. But if the other person is only interested in proving you wrong because you hold a different opinion — with no interest in genuine dialogue — don't engage. Life is too short.

If you disagree with someone, ask questions first. Don't be so quick to judge.

---

In today's world of AI-assisted development, much has changed. The way we write code, the way we use tools, and the way we learn — all of it is evolving at a pace we haven't seen before. While some core principles remain as valid as ever, many long-held assumptions deserve a fresh look.

There's a fine balance to strike. Those who ignore the lessons of the past are doomed to repeat the same mistakes. But those who cling too tightly to old ideas will never innovate. The sweet spot is being grounded in fundamentals while staying genuinely open to experimentation.

We are all figuring this out together. Some new tools and approaches will prove transformative; others won't pan out. That's how progress works — and that's okay.

The real superpower in this era isn't having all the answers. It's setting aside your preconceived opinions and being willing to try something that looks ridiculous on the surface. You never know what you might discover.

Cheers.

# Feeling Left Behind in Tech? This Is Your 90-Day Comeback Plan


You've been working for years. You know your systems inside out. 
You've solved real problems, handled pressure, and delivered results.

But now you open a job description… and it feels like you're reading a different language.

New frameworks. New tools. New expectations.

And the thought creeps in: *"Have I fallen behind?"*

<!--more-->

Here's the truth most people won't tell you: **Your old tech stack isn't your biggest problem. Your approach to upskilling is.**

You don't need to learn *everything*. You don't need to start from scratch. 
And you definitely don't need to compete by out-learning someone with 2 years of experience.

What you *do* need is a focused, practical strategy.

By the end of this article, you'll know exactly:

* What to learn (and what to ignore)
* How to learn quickly without burning out
* How to position yourself to actually crack interviews

{{< figure src="/images/upskilling-for-interviews.webp" alt="Upskilling for interviews"
width="350" height="300" >}}

## 1. The Reality Check: Your Old Tech Stack Isn't the Real Problem

It's easy to blame your situation on outdated technology.

But hiring managers don't just hire tools, they hire **problem solvers**.

If you've spent years:

* Debugging complex systems
* Working with production environments
* Handling real-world constraints

You already have something many candidates don't: **experience that matters**

The gap isn't your ability. It's **relevance and presentation**.

## 2. Shift Your Mindset: You're Not Starting From Zero

This is where most people go wrong.

They think:

> "I need to learn everything from scratch"

No, you don't.

You're not a beginner. You're a **translator**.

Your job is to:

* Map what you already know → to modern tools
* Reuse your fundamentals → in new contexts

For example:

* If you know one backend language, you can learn another faster
* If you understand system design, tools are just implementation details

**Your experience is your shortcut if you use it properly.**

## 3. Step 1: Identify the Right Skills (Not All Skills)

This is where most people waste months.

They try to learn:

* Every trending framework
* Every tool in job descriptions
* Every buzzword they see

Instead, do this:

### Look at 10–15 job descriptions

Find patterns:

* Which skills repeat?
* Which are "must-have" vs "nice-to-have"?

### Focus on 3 layers only:

1. **Core language** (e.g., Java, Python, JavaScript)
2. **One relevant framework** (not five)
3. **Basic system design / architecture understanding**

That's it.

You're not trying to become an expert in everything, you're trying to become **interview-ready**.

## 4. Step 2: Learn Strategically, Not Exhaustively

Here's a mistake you might be making:

> "I'll complete this full course, then start applying."

That approach kills momentum.

Instead, follow this cycle:

### Learn → Apply → Repeat

* Learn a concept (short and focused)
* Immediately apply it in a small project
* Move on

### Set a constraint:

* 2–3 hours per day
* 6–8 weeks of focused effort

That's enough if you stay focused.

You don't need mastery. You need **working knowledge + confidence**.

## 5. Step 3: Build Proof (Projects That Actually Matter)

Interviewers don't trust "I've learned this."

They trust **"I've built this."**

But here's the key:
Your project doesn't need to be big. It needs to be **relevant**.

### Good project examples:

* A simple CRUD application
* A small API with authentication
* A basic system that shows real-world thinking

### What matters more than complexity:

* Clean structure
* Clear logic
* Ability to explain your decisions

Think of your project as:

> "A conversation starter in interviews"

## 6. Step 4: Repackage Your Experience for Interviews

This is where many experienced professionals lose opportunities.

They say things like:

* "I've been working on legacy systems"
* "My experience isn't relevant"

That's the wrong framing.

### Instead, translate your experience:

Old way:

> "Worked on legacy application maintenance"

Better:

> "Maintained and optimised a high-traffic production system, improving performance and reducing downtime"

See the difference?

You're not selling tools.
You're selling **impact and thinking**.

## 7. Step 5: Prepare for Interviews Smartly

You don't need to prepare for everything. Focus on:

### 1. Core concepts

* Data structures (basics, not extreme depth)
* Problem-solving patterns

### 2. Practical knowledge

* Your chosen tech stack
* Your project

### 3. Communication

* Explaining your decisions
* Talking through problems clearly

### Golden rule:

> If you can explain it simply, you understand it well enough.

## 8. Common Mistakes to Avoid While Upskilling

Let's call these out directly:

* **Trying to learn too many things at once**
* **Waiting too long before applying**
* **Comparing yourself to others constantly**
* **Ignoring your existing experience**
* **Over-consuming content without building anything**

If you recognize yourself in any of these, adjust now.

## 9. A Simple 60–90 Day Upskilling Plan

Here's a realistic structure you can follow:

### **Weeks 1–3:**

* Pick one language + one framework
* Learn fundamentals
* Start small exercises

### **Weeks 4–6:**

* Build 1–2 small projects
* Focus on real-world use cases

### **Weeks 7–9:**

* Start applying for jobs
* Prepare for interviews
* Refine your resume

### **Weeks 10–12 (if needed):**

* Improve weak areas
* Continue interviews
* Iterate based on feedback

## **10. Final Thoughts: You're Competing Differently (Not Worse)**

It might feel like you're behind.

But you're not competing on the same terms as someone early in their career.

They have:

* New tools

You have:

* Experience
* Context
* Real-world problem solving

When you combine that with **focused upskilling**, you become a very strong candidate.

So don't aim to catch up with everyone.

Aim to be:

> **The experienced professional who adapted and proved it.**

That's who gets hired.

# Stepping into Crazy AI World


If you are feeling overwhelmed by the rapid pace of technological advancements, especially in the field of AI, know that you are not alone. 
The AI landscape is evolving at an unbelievable speed, with new ideas and tools emerging every day. 
<!--more-->

This can be both exciting and daunting, as it challenges our understanding of what's possible and forces us to adapt quickly.

You may encounter conflicting narratives: a CEO claims that in 9 months a 90% percentage of code 
will be written by AI, YouTubers who became AI Experts overnight tell you that "AI will take over the world by 2030".
And then suddenly, you come across a Tweet or a LinkedIn post saying AI is not yet good enough for the real world projects.

If you are spending too much time on social media, it's easy to get confused and become anxious and overwhelmed. 

Remember that while the AI revolution is happening, content creators and CEOs often have reasons to exaggerate.

So, chill a little bit.

Based on what I read/heard from various sources, none of the AI companies are making any profits as of now.
And there are rumors that if they can't figure out a way to make money in a few years, then it's going to be an economic disaster.
But I think there is no going back now. So, it is best to accept the new reality and adapt to it.

* Software development is going to be heavily depends on AI
* The "coding" part of software development is going to be delegated to AI
* Learning how to use the AI tools effectively is going to be an essential skill

Whether AI is going to replace human developers is debatable and mostly not in our hands.

So, instead of worrying about the things that are not in our control, focus on what you can control.

We can start exploring how to leverage AI agents for coding, and learn how to use the AI tools effectively.

## Explore available AI tools
* You can start using **ChatGPT**, **Claude** and **Gemini** to learn new technologies or concepts.
* You can start using **OpenAI's Codex**, **Anthropic's Claude Code**, **Google's Gemini CLI** to learn how to use AI Coding Agents
* Learn how to configure the guidelines using [AGENTS.md](https://agents.md/), **CLAUDE.md**, **GEMINI.md**, etc to instruct the AI agents about how to work with the project
* Learn how to use [Agent Skills](https://agentskills.io/) to tune AI Agents generate code according to your preference
* Learn how to create custom commands to automate the repetitive tasks like commiting code, reviewing PRs, enhancing usecase, etc.
* Learn how to use AI tools integrated with IDEs like IntelliJ IDEA leveraging the integrated [AI capabilities](https://www.jetbrains.com/ai/) using [Junie](https://www.jetbrains.com/junie/), [ACP](https://blog.jetbrains.com/ai/2025/12/bring-your-own-ai-agent-to-jetbrains-ides/), etc 

## Spec Driven Development
The latest silver bullet in the age of AI is the Spec Driven Development (SDD). 
In SDD, the main focus is on writing clear and concise specifications that define the desired behavior of the software. 
The AI agents can be used to generate code based on these specifications, ensuring that the code meets the required standards and functionality.

IMO, SDD together with AI Coding Agents combined with custom Agents, Agent Skills, and custom Commands can revolutionize the software development process.

{{< figure src="/images/ai-powered-swe.webp" alt="AI powered Software Development" >}}

## Conclusion
As I mentioned, these are not normal times. The world is changing at an unprecedented pace to catch up.
So, don't be too hard on yourself thinking only you are not able to keep up with the pace. 

Embrace the change and leverage AI tools to enhance your productivity and deliver high-quality software faster than ever before.

Peace!





# The Power of Value Objects


A **Value Object** is a domain concept defined by its values rather than by identity.
For example, we can represent **EventId**, **EventCode** or **Email** as a value object.
In this article, let's explore the benefits of using value objects over primitive types.

<!--more-->

## Value Objects
In Java, typically, we can use records to create value objects.

```java
public record EventId(Long id) {
    public EventId {
        if (id == null || id < 0) {
            throw new IllegalArgumentException("Event id cannot be null");
        }
    }

    public static EventId of(Long id) {
        return new EventId(id);
    }
}
```

Now we can create an **EventId** object using **EventId.of(1L)** method.

Now the question is, **why do we need to create a value object?
Why can't we simply use primitive types?**

Let's explore the benefits of using value objects.

## Benefits of using Value Objects
By using value objects, we are defining domain concepts as a first-class citizen.
But in practice, let's see how using value objects improves/simplifies our code.

### No need for defensive programming
Typically, in a web application or REST API, we receive input data from the client.
And, we validate the input data and pass it to the business logic layer.

But sometimes, we might forget to validate the input data or the same method is being called from multiple places.
So, to be safe, developers keep adding validation checks all over the place.

```java
@RestController
class EventController {
    @GetMapping("/api/events/{id}")
    public void getEvent(@PathVariable Long id) {
        if(id == null || id < 0) {
            throw new IllegalArgumentException("Invalid event id: " + id);
        }
        var event = eventService.getEvent(id);
        // ...
    }
}

@RestController
class RegistrationController {
    @GetMapping("/api/events/{id}/registrations")
    public void getEvent(@PathVariable Long eventId) {
        //not validating eventId
        var event = eventService.getEvent(id);
        // ...
    }
}

@Service
class EventService {
    public Event getEvent(Long id) {
        if(id == null || id < 0) {
            throw new IllegalArgumentException("Invalid event id: " + id);
        }
        // ...
    }
}
```

As you can see in the above example, if the method is called from multiple places,
just to be safe, developers keep adding validation checks.

But if we use value objects, the validation logic is centralized in the value object.
**If the value object is constructed successfully, that means it has valid data.**

```java
@RestController
class EventController {
    @GetMapping("/api/events/{id}")
    public void getEvent(@PathVariable Long id) {
        var event = eventService.getEvent(EventId.of(id));
        // ...
    }
}

@RestController
class RegistrationController {
    @GetMapping("/api/events/{id}/registrations")
    public void getEvent(@PathVariable Long eventId) {
        //not validating eventId
        var event = eventService.getEvent(EventId.of(id));
        // ...
    }
}

@Service
class EventService {
    public Event getEvent(EventId id) {
        //no need to validate id
        // ...
    }
}
```

The input validation logic is centralized in the value object.
So, we don't need to validate the input data in the controller or service layer.
The input validation happens in the controller while constructing the value object.
If the input is invalid, then it will throw an exception, and we could have a Global ExceptionHandler to handle it.

{{< box tip >}}
**What if someone passes `null` as `EventId` itself?**

This is where [jSpecify can help you](https://www.youtube.com/watch?v=p3v8wJG3-zo).
{{< /box >}}

### No more accidentally passing invalid data
Imagine we have a method as follows:

```java
public void register(Long eventId, Long userId) {
    //...
}
```

While calling this method, we might accidentally call **register(request.getUserId(), request.getEventId())**.
As the method accepts any **Long** value, it might be possible to call the method with wrong inputs.
But if we use value objects (i.e **UserId** and **EventId**), then we can't accidentally pass invalid data.

```java
public void register(EventId eventId, UserId userId) {
    //...
}
```

Assume **request.getUserId()** returns **UserId** and **request.getEventId()** returns **EventId** type values.
Calling the method as **register(request.getUserId(), request.getEventId())** will be a compiler error.

So, by using Value Objects, you can prevent accidentally passing invalid data.

### Using Value Objects with JPA
You can use Value Objects with JPA as **Embeddable** objects.

```java
//Omitting validation logic for brevity
record EventId(Long id) {}
record Address(String street, String city, String state){}

@Entity
@Table(name = "events")
class Event {
    @EmbeddedId
    @AttributeOverride(name = "id", column = @Column(name = "id", nullable = false))
    private EventId id;

    @Embedded
    @AttributeOverrides({
            @AttributeOverride(name = "street", column = @Column(name = "street", nullable = false)),
            @AttributeOverride(name = "city", column = @Column(name = "city")),
            @AttributeOverride(name = "state", column = @Column(name = "state"))
    })
    private Address address;

}
```

You can create a Spring Data JPA repository for `Event` as follows:

```java
interface EventRepository extends JpaRepository<Event, EventId> { //EventId as primary key instead of Long
}
```

Now we can use `EventRepository` as follows:

```java
@Service
class EventService {
    private final EventRepository eventRepository;
    public EventService(EventRepository eventRepository) {
        this.eventRepository = eventRepository;
    }
    public Event getEvent(EventId id) {
        return eventRepository.findById(id).orElseThrow(); //using EventId instead of Long
    }

}
```

### Automatic Binding of PathVariable/RequestParam to Value Objects
If you are using Spring MVC, we can create **Converters** to bind input values to Value Objects.

```java
import org.springframework.core.convert.converter.Converter;
import org.springframework.stereotype.Component;

@Component
public class StringToEventIdConverter implements Converter<String, EventId> {

    @Override
    public EventId convert(String source) {
        return new EventId(Long.parseLong(source));
    }
}
```

This allows Spring MVC to automatically convert path variables or request parameters like **/{eventId}** from String to **EventId**:

```java
@GetMapping("/{eventId}")
ResponseEntity<Event> findEvent(@PathVariable EventId eventId) {
    // eventId is already an EventId object
}
```

### Binding primitives to Request Bodies with Value Objects
By default, if you marshall an Event object with EventId property to JSON using Jackson, it will produce the following JSON:

```json
{
  "id": {
    "id": 123
  }
}
```

We can use the **@JsonValue** annotation to serialize the value instead of object strcuture.
Similarly, we can use **@JsonCreator** annotation to specify which constructor or factory method to use while deserializing the JSON into an object.

**EventCode Value Object:**

```java
import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonValue;
import jakarta.validation.constraints.NotBlank;

public record EventCode(
        @JsonValue 
        @NotBlank(message = "Event code cannot be null or empty")
        String code
) {
    @JsonCreator
    public EventCode {
        if (code == null || code.trim().isEmpty()) {
            throw new IllegalArgumentException("Event code cannot be null");
        }
    }

    public static EventCode of(String code) {
        return new EventCode(code);
    }
}
```

**CreateEventRequest Request Payload:**

```java
record CreateEventRequest(
        @Valid EventCode code
        // ... other properties
) {
}
```

Now Spring MVC will automatically bind the **code** property from the JSON payload to **EventCode** object.

```json
{
  "code": "ABSHDJFSD",
  "property-1": "value-1",
  "property-n": "value-n"
}
```

### Binding flattened JSON to Nested Objects
We may want to bind flattened JSON to nested objects.
In those scenarios, we can use **@JsonUnwrapped** and **@JsonCreator(mode = JsonCreator.Mode.PROPERTIES)** annotations 
to map flattened JSON to nested objects.

```java
import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;
import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.Pattern;
import jakarta.validation.constraints.Size;

public record EventDetails(
        @NotBlank(message = "Title is required")
        @Size(min = 3, max = 200, message = "Title must be between 3 and 200 characters")
        String title,

        @NotBlank(message = "Description is required")
        @Size(max = 10000, message = "Description cannot exceed 10000 characters")
        String description,

        @Size(max = 500, message = "Image URL cannot exceed 500 characters")
        @Pattern(regexp = "^https?://.*", message = "Image URL must be a valid HTTP/HTTPS URL")
        String imageUrl) {

    @JsonCreator(mode = JsonCreator.Mode.PROPERTIES)
    public EventDetails(
            @JsonProperty("title") String title,
            @JsonProperty("description") String description,
            @JsonProperty("imageUrl") String imageUrl
    ) {
        this.title = AssertUtil.requireNotNull(title, "title cannot be null");
        this.description = AssertUtil.requireNotNull(description, "description cannot be null");
        this.imageUrl = imageUrl;
    }

    public static EventDetails of(String title, String description, String imageUrl) {
        return new EventDetails(title, description, imageUrl);
    }
}
```

**CreateEventRequest Request Payload:**

```java
record CreateEventRequest(
        @Valid EventCode code,
        @JsonUnwrapped @Valid EventDetails details
        // ... other properties
) {
}
```

Now Spring MVC will automatically bind the **title**, **description** and **imageUrl** property values
from the JSON payload to **EventDetails** object.

```json
{
  "code": "ABSHDJFSD",
  "title": "Spring Boot Workshop",
  "description": "Learn Spring Boot best practices",
  "imageUrl": "https://example.com/image.jpg",
  "property-1": "value-1",
  "property-n": "value-n"
}
```

## Conclusion
In this article, we explored the benefits of using value objects over primitive types.
We saw how using value objects improves our code by centralizing domain concepts.
We also saw how we can streamline using value objects from Spring controllers to JPA repositories.

# Announcing FeatureTracker Application


As a Java Developer Advocate at JetBrains, I frequently demonstrate IntelliJ IDEA features through videos and articles. To do this, I usually create small demo applications focused on specific features. While this works well initially, the number of such applications grows quickly. Soon, it becomes difficult to remember which app was built for what purpose, turning it into a maintenance headache.

<!--more-->

Some of the newer features in IntelliJ IDEA, like [Spring Debugger](https://www.jetbrains.com/help/idea/spring-debugger.html) and [Workspaces](https://blog.jetbrains.com/idea/2024/08/workspaces-in-intellij-idea/), require more complex applications to showcase their capabilities effectively.

* [Spring Debugger Features Preview in Tweets](https://x.com/intellijidea/status/1927686185247215911)
* [Workspaces in IntelliJ IDEA](https://blog.jetbrains.com/idea/2024/08/workspaces-in-intellij-idea/)
* [IDE Workspaces Development: Challenges and Plans](https://blog.jetbrains.com/idea/2025/03/ide-workspaces-development-challenges-and-plans/)

Although PetClinic is a popular demo application, I find it too small and limiting for my needs.

So, we decided to build a more realistic application using a wide range of technologies.

## Introducing Feature Tracker
Feature Tracker is a centralized platform for product teams and users to track the implementation status of features across multiple products, monitor release cycles, and identify areas for improvement. It also collects user feedback and provides insights into which features are most appreciated by users.

### Technical Architecture
Feature Tracker application will be developed following Microservices architecture principles and best practices. 

The following are some of the key services identified so far for the feature-tracker application:

#### Infrastructure Components
* **Keycloak**: Keycloak will be used as an OAuth 2.0 Authorization Server for microservices.
* **config-server**: Spring Cloud Config Server to provide centralized configuration management for microservices.
* **api-gateway**: Spring Cloud Gateway that will act as an API Gateway for all microservices.

#### Microservices
* **User Service**: Manages user profiles and fetches user information from Keycloak.

  **TechStack:** Java, Spring Boot, Spring Security OAuth, Keycloak, Maven

* **Feature Service**: Manages products, releases, and features.

  **TechStack:** Java, Spring Boot, PostgreSQL, Spring Data JPA, Flyway, Redis, Kafka, Spring Security OAuth, Testcontainers, Maven

* **Notifications Service**: Listens for events and sends notifications via email.

  **TechStack:** Kotlin, Spring Boot, Kafka, Email, Testcontainers, Gradle

* **Search Service**: Stores searchable data and provides APIs for faceted search.

  **TechStack:** Kotlin, Spring Boot, ElasticSearch, Testcontainers, Gradle

* **Media Service**: Manages storage for media files like images and videos.

  **TechStack:** Java, Spring Boot, Spring Security OAuth, AWS S3, LocalStack, Testcontainers, Gradle

* **Analytics Service**: Tracks metrics such as product usage and user engagement.

  **Tech Stack:** Java, Spring Boot, WebFlux, Kafka, Timeseries DB, Spring Security OAuth, Testcontainers, Maven

* **Angular Frontend**: Frontend UI using Angular SPA.

  **TechStack:** Angular, Keycloak-js, PrimeNG, TailwindCSS

## Getting Started with Feature Tracker
Feature Tracker is an open source application under **active development phase**.

The first version v0.0.1 of the Feature Tracker application is released.
All the code is available at https://github.com/feature-tracker.

This release includes:

* Initial setup of keycloak, config-server, api-gateway
* Initial setup of feature-service, user-service, notification-service
* Initial setup of Angular Frontend

To get started with the Feature Tracker application, please follow the instructions at https://feature-tracker.github.io/.

**Demo of Feature Tracker 0.0.1**

{{< youtube we1afMhOLAQ >}}

## How to contribute?
* Run the application and let us know if you face any issues
* Review the code and create issues if there is something to be improved
* Create issues to request new features or share your ideas

Contributions are most welcome :)

# Protect Your Mental Health While Being On Social Media


Social media is a double-edged sword. If used properly, it can be immensely helpful.
Personally, being on social media, especially on Twitter, helped me to connect with amazing people across the world.
This in turn gave me a lot of opportunities such as learning from the experts, job opportunities, a chance to write books, speaking at conferences, etc.

<!--more-->

Having said that, being on social media is also very stressful. Not everyone online is kind or polite.
Some people feel unhappy or frustrated in their lives, and they use the internet to take out their anger on strangers.

{{< figure src="/images/social-media-overwhelmed.webp" alt="Overwhelmed by Social Media"
 width="350" height="300" >}}


I have been in my share of debates and heated discussions on social media. I've tried quitting Twitter many times, 
but I always come back because I don't want to lose the benefits over some bad experiences. 
Instead, I've learned how to handle those people and use social media in a positive way.

A couple of months ago, I was talking to a friend, and he said:

> You suddenly became so calm, not getting into any arguments at all on Twitter. What happened?

I have my own opinions, and sometimes they clash with those of people I admire.
Usually, we talk respectfully and agree to disagree while staying friends.
But some people use personal insults, play cheap tricks to get a response, or keep provoking you over past disagreements.

These debates and heated discussions suck a lot of your time and energy.
I got tired of these angry people and want it to stop.

So I thought about it, studied how these arguments start, and came up with tactics to handle them. 

In this article, I would like to share my learnings.

## Think about where to spend your time and energy
I have limited time and energy each day, so I must choose wisely.
Heated debates drain a lot of energy, no matter who wins.
Before jumping in, I ask myself, **Is this necessary?** and **Is it worth my time?**
If I still feel it's important, I wait 15 minutes before replying. Usually, I decide to let it go.

We usually get into these debates to defend something we believe in, and we may think it is important to take a stand for it.

> Remember, for most people these debates are nothing but a popcorn time.

## Not everyone deserves a reply
Every tweet/post/message has context, but it can be lost or misread.

If someone asks genuine questions to understand your point, feel free to explain.
If they only want to provoke or prove you wrong, don't engage.

For example, let's say I tweeted **I prefer Maven over Gradle**.

If someone replies:
* **May I know why?**
* **Care to share what problems you are facing with Gradle?**
* **I have been using Gradle, and I am happy with it. May I know what features you have in Maven, but not in Gradle?** 
 
**Good, reply to them**. These are good replies that are trying to understand what you said, and you can engage in those conversations.

If someone replies:

* **Convince me why I should use Maven over Gradle in 2025**
* **Maven sucks, I don't believe why anyone still uses it**

**Bad, ignore them**. These are not the people who want to understand what you said, or has any interest in having a meaningful conversation. Don't reply to them.

Some people ask even very basic questions which they can figure out by a simple Google search. 
But they want you to do that Google search for them. 
Don't reply to them.

> Respect your time.

## Don't engage in conversations with bullies
Some people build their empire over time by sharing their knowledge, opinions, and sometimes infusing fear into the public.
If someone doesn't agree 100% with their ideology, they can't take it.
They go all the way to twist your words and use their cult followers to bully you.

Simply reply at once to clearly state what you said and end the conversation from your side.

> "Never argue with stupid people, they will drag you down to their level and then beat you with experience." - Mark Twain

You can switch "stupid people" with "snake oil salesman" and this rule still applies :)

## Use mute and block freely
As I mentioned already, there is no shortage of people who are frustrated in their lives, and they just want to vent their anger on someone.

If someone is bothering you again and again, feel free to mute or block them.

You still welcome polite, professional criticism. But there is no need to tolerate jerks.

> Take constructive criticism, but mute/block jerks.

## Don't push your ideology on others
Our experience shapes our opinions, and it's fine to share them.
But we shouldn't assume our way is the only right way.

For example, again as I mentioned earlier, I do not have good experience with Gradle and Maven worked flawlessly for me over the years.

But imagine, somebody is developing Android applications, and they are using Gradle tool.
They mastered Gradle, developed their own plugins, and their team is very happy and productive with Gradle.

Then imagine me telling them, **You should not use Gradle**.

If they are happy and productive with it, who the hell I am to tell them what build tool they should use?

> You can have your opinions and preferences, but don't push them on others.

This mindset simply eliminates a lot of arguments.

* **You like Lombok.** - Great 
* **You hate Lombok and you never want to use it.** - Fine, good for you.
* **You don't have any problem with using Field Injection.** - Sure.
* **Hexagonal Architecture working great for you.** Fantastic. Please share your experience.
* **You like to write Mocks by hand instead of using Mockito**. I am happy for you.

> Never impose your ideology on others unless you are going to bear the consequences.

## Summary
Every day I have many things to take care of.
There are family duties, work timelines to meet, and if you are 40+, like me, there are health concerns to deal with.

Taking care of all of these itself is a challenge, and there is no need to add more chaos.

Knowing how to use social media wisely and managing unpleasant things/people makes a big difference.

I hope some of the points I shared might help you.

# You Can Code Offline With Local AI(Ollama)

ChatGPT and other Generative AI tools took the world by a storm.
People are using these AI tools for various purposes such as to explore a topic, 
to seek answers to their questions or to get help in coding, etc.
ChatGPT and Google's Gemini are very popular AI tools that are available for free with some usage limitations.

<!--more-->

**Coding without help from Google, StackOverflow and blogs is challenging.
But what if you had to code without internet connectivity?**

**No need to worry.**

**Now there are some FREE AI tools that you can run locally on your laptop so that you can get help even offline(without internet connection).
Also, by using these local AI tools, you won't risk sharing your details with public AI tool vendors.**

In this article, we will explore:
* Running Ollama locally
* Using GUI Clients to interact with Ollama
* Using Ollama as a Java Developer

## Running Ollama locally
[Ollama](https://ollama.com/) is an open-source project that provides LLM models that you can run locally.
Download and install Ollama from [here](https://ollama.com/download).

Once installed, you can start Ollama and run any available [Models](https://ollama.com/library).

You can perform various actions using `ollama` as follows:

```shell
$ ollama list // <- to list downloaded models
$ ollama ps // <- to list running models
$ ollama run <model> // <- to download and run the model
$ ollama show llama3.2 // <- to show information about llama3.2 model
$ ollama help // <- shows all available commands
```

Let's run [llama3.2](https://ollama.com/library/llama3.2) model by running the following command:

```shell
$ ollama run llama3.2
>>> Send a message (/? for help)
```

When you run it for the first time, it will download the model and start the server.
Once the server is started, you can check the status by visiting [http://localhost:11434](http://localhost:11434)
which should display **Ollama is running**.

You can download as many models you want and run multiple models at the same time.

Now you can ask any question from the Terminal/Console:

```shell
>>> Explain what is LLM in layman terms

LLM stands for Large Language Model. In simple words, it's a type of artificial intelligence (AI) that can understand and generate human-like language.

Imagine you have a huge library with an almost endless number of books. Each book contains lots of information and knowledge about the world. A Large Language 
Model is like this library, but instead of physical books, it's made up of digital words and phrases.
....
....
>>>
```

While you can interact with Ollama from CLI, it would be nice to have a GUI Client.

## Using GUI Clients to interact with Ollama
You can see a list of [Web and Desktop GUI Clients for Ollama](https://github.com/ollama/ollama?tab=readme-ov-file#web--desktop) 
in Ollama's GitHub Readme file.

I tried some of those available options, and I like [Enchanted(MacOs native client)](https://github.com/gluonfield/enchanted) and
[Open WebUI](https://github.com/open-webui/open-webui).

You can install them following their installation instructions in their README files.

Here is the Enchanted GUI Client.

{{< figure src="/raw-images/ollama-enchanted-demo.gif" >}}

## Using Ollama as a Java Developer
We can use Ollama with a model that is optimized for coding 
such as [codellama](https://ollama.com/library/codellama), 
[qwen2.5-coder](https://ollama.com/library/qwen2.5-coder),
[codegemma](https://ollama.com/library/codegemma)
to get help in coding related tasks.

Some coding tasks that you can try:

**1. Generate classes or records**

**Prompt:** Generate a Java class to store user information with equals(), hashCode() and toString() methods.

{{< figure src="/raw-images/ollama-generate-class.gif" >}}

**Prompt:** Generate a Java record User with id, name, email, password and phone fields.

{{< figure src="/raw-images/ollama-generate-record.gif" >}}

**2. Generate SQL Table Schema**

**Prompt:** Generate SQL script to create users table that supports user registration, user activation, and forgot password features.

{{< figure src="/raw-images/ollama-generate-sql-script.gif" >}}

**Prompt:** Given the following table structure, generate a query that returns the employee names and their manager names.

```sql
create table employees (
  emp_id int primary key,
  name varchar(100) not null,
  manager_id int references employees(id)
);
```

{{< figure src="/raw-images/ollama-generate-sql-query.gif" >}}

**3. Ask to explain code snippet**

You can ask to explain some code snippet.

**Prompt:** Explain the following code snippet:

```java
@GetMapping("/{slug}")
@Operation(summary = "Find post by slug")
@ApiResponses({
    @ApiResponse(responseCode = "200", description = "Returns post for the given slug"),
    @ApiResponse(responseCode = "404", description = "Post doesn't exists for the given slug"),
})
ResponseEntity<PostDto> getPostBySlug(@PathVariable(value = "slug") String slug) {
    var post = postService
            .findPostBySlug(slug)
            .map(blogMapper::toPostDto)
            .orElseThrow(() -> new ResourceNotFoundException("Post with slug '" + slug + "' not found"));
    return ResponseEntity.ok(post);
}
```

{{< figure src="/raw-images/ollama-explain-code.gif" >}}

**4. Generate tests for existing code**

You can generate tests for existing code.

**Prompt:** Write integration test for the following code snippet:

```java
@GetMapping("/{slug}")
ResponseEntity<PostDto> getPostBySlug(@PathVariable(value = "slug") String slug) {
    var post = postService
            .findPostBySlug(slug)
            .map(blogMapper::toPostDto)
            .orElseThrow(() -> new ResourceNotFoundException("Post with slug '" + slug + "' not found"));
    return ResponseEntity.ok(post);
}
```

{{< figure src="/raw-images/ollama-generate-test.gif" >}}

**5. Explore pros and cons of competing technologies/approaches**

You can explore the differences of various approaches and ask to give pros and cons of them.

**Prompt:** Explain the differences between REST and GraphQL and also give pros and cons of each.

{{< figure src="/raw-images/ollama-generate-pros-and-cons.gif" >}}

## Summary
The Generative AI tools can help us to be more productive by offloading the boring tasks to AI.
The tools like Ollama and various freely available LLM models enable us to run LLMs locally.

If you are new to Generative AI, using the Gen AI chat would be a first step.
We can be much more productive by using the AI-powered IDEs that use AI Agents to generate or modify code.

I have shown some basic examples of how you can use Ollama for coding related tasks.
You can leverage these AI tools to be more productive by letting them do mundane boring tasks.
Also, we can brainstorm ideas, ask to write documentation, and do many more things.

Happy coding :-)

# Spring Boot + Testcontainers Tests at Jet Speed


Spring Boot 3.1.0 introduced [support for Testcontainers](https://docs.spring.io/spring-boot/reference/testing/testcontainers.html) 
to simplify local development and testing. 
[Testcontainers](https://testcontainers.com/) helps in writing tests using real dependencies instead of mocks, but it may also increase the test execution time.

<!--more-->

In this article, I will share some insights on **how to reduce test execution time while using Testcontainers**.

## Sample Spring Boot application
Let's assume you have created a Spring Boot project using [Spring Initializr](https://start.spring.io/#!type=maven-project&dependencies=web,data-jpa,flyway,testcontainers,postgresql) 
by selecting **Spring Web**, **Spring Data JPA**, **PostgreSQL**, **Flyway** and **Testcontainers**.

And, assume that you have implemented two API endpoints **GET /api/bookmarks** and **GET /api/users** by creating the following components:

**Bookmark JPA entity**

```java
@Entity
@Table(name = "bookmarks")
public class Bookmark {
    @Id
    private Long id;
    private String title;
    private String url;
    // code omitted for brevity
}
```

**BookmarkRepository (Spring Data JPA Repository)**

```java
public interface BookmarkRepository extends JpaRepository<Bookmark, Long> {
  List<Bookmark> findAllByOrderByCreatedAtDesc();
}
```

**BookmarkController**

```java
@RestController
@RequestMapping("/api/bookmarks")
class BookmarkController {
    private final BookmarkRepository bookmarkRepository;
    BookmarkController(BookmarkRepository bookmarkRepository) {
        this.bookmarkRepository = bookmarkRepository;
    }

    @GetMapping
    List<Bookmark> getBookmarks() {
        return bookmarkRepository.findAllByOrderByCreatedAtDesc();
    }
}
```

**User JPA entity**

```java
@Entity
@Table(name = "users")
public class User {
    @Id
    private Long id;
    private String name;
    private String email;
    // code omitted for brevity
}
```

**UserRepository (Spring Data JPA Repository)**

```java
public interface UserRepository extends JpaRepository<User, Long> {
    List<User> findAllByOrderByNameAsc();
}
```

**UserController**

```java
@RestController
@RequestMapping("/api/users")
class UserController {
    private final UserRepository userRepository;
    public UserController(UserRepository userRepository) {
        this.userRepository = userRepository;
    }

    @GetMapping
    List<User> list() {
        return userRepository.findAllByOrderByNameAsc();
    }
}
```

We would like to write tests for:

* Testing the custom methods in repositories 
* Testing end-to-end APIs 

Now, let's explore different ways of writing tests using Testcontainers and understand the pros and cons of each approach.

## Inefficient use of ServiceConnection support

The Spring Boot's **@ServiceConnection** support simplifies registering the container connection properties obtained from 
the containers spun up by Testcontainers.

We can write four test classes using **@ServiceConnection** support as follows.

**BookmarkRepositoryTest.java**

```java
@DataJpaTest
@Testcontainers
class BookmarkRepositoryTest {
    @Container
    @ServiceConnection
    static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>(DockerImageName.parse("postgres:17"));

    @Autowired
    BookmarkRepository bookmarkRepository;

    @BeforeEach
    void setUp() {
        bookmarkRepository.deleteAllInBatch();
    }

    @Test
    void shouldGetAllBookmarksOrderByCreatedAtDesc() {
        bookmarkRepository.save(new Bookmark("JetBrains Blog","https://blog.jetbrains.com"));
        bookmarkRepository.save(new Bookmark("IntelliJ IDEA Blog","https://blog.jetbrains.com/idea/"));

        List<Bookmark> bookmarks = bookmarkRepository.findAllByOrderByCreatedAtDesc();

        assertThat(bookmarks).hasSize(2);
        assertThat(bookmarks.get(0).getTitle()).isEqualTo("IntelliJ IDEA Blog");
        assertThat(bookmarks.get(1).getTitle()).isEqualTo("JetBrains Blog");
    }
}
```

**BookmarkControllerTest.java**

```java
@SpringBootTest(webEnvironment = RANDOM_PORT)
@Testcontainers
class BookmarkControllerTest {

    @Container
    @ServiceConnection
    static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>(DockerImageName.parse("postgres:17"));

    @Autowired
    private TestRestTemplate restTemplate;

    @Autowired
    private BookmarkRepository bookmarkRepository;

    @BeforeEach
    void setUp() {
        bookmarkRepository.deleteAllInBatch();
    }

    @Test
    void shouldGetAllBookmarks() {
        bookmarkRepository.save(new Bookmark("JetBrains Blog","https://blog.jetbrains.com"));
        bookmarkRepository.save(new Bookmark("IntelliJ IDEA Blog","https://blog.jetbrains.com/idea/"));

        Bookmark[] bookmarks = restTemplate.getForObject("/api/bookmarks", Bookmark[].class);

        assertThat(bookmarks.length).isEqualTo(2);
        assertThat(bookmarks[0].getTitle()).isEqualTo("IntelliJ IDEA Blog");
        assertThat(bookmarks[1].getTitle()).isEqualTo("JetBrains Blog");
    }
}
```

**UserRepositoryTest.java**

```java
@DataJpaTest
@Testcontainers
class UserRepositoryTest {

    @Container
    @ServiceConnection
    static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>(DockerImageName.parse("postgres:17"));

    @Autowired
    UserRepository userRepository;

    @BeforeEach
    void setUp() {
        userRepository.deleteAllInBatch();
    }

    @Test
    void shouldGetAllUsersOrderByNameAsc() {
        userRepository.save(new User("Paul","paul@gmail.com"));
        userRepository.save(new User("John","john@gmail.com"));

        List<User> users = userRepository.findAllByOrderByNameAsc();

        assertThat(users).hasSize(2);
        assertThat(users.get(0).getName()).isEqualTo("John");
        assertThat(users.get(1).getName()).isEqualTo("Paul");
    }
}
```

**UserControllerTest.java**

```java
@SpringBootTest(webEnvironment = RANDOM_PORT)
@Testcontainers
class UserControllerTest {
    @Container
    @ServiceConnection
    static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>(DockerImageName.parse("postgres:17"));

    @Autowired
    private TestRestTemplate restTemplate;

    @Autowired
    private UserRepository userRepository;

    @BeforeEach
    void setUp() {
        userRepository.deleteAllInBatch();
    }

    @Test
    void shouldGetAllUsers() {
        userRepository.save(new User("Paul","paul@gmail.com"));
        userRepository.save(new User("John","john@gmail.com"));
        User[] users = restTemplate.getForObject("/api/users", User[].class);

        assertThat(users).hasSize(2);
        assertThat(users[0].getName()).isEqualTo("John");
        assertThat(users[1].getName()).isEqualTo("Paul");
    }
}
```

Each test class defines the **PostgreSQLContainer** and uses **@Testcontainers**, **@Container** and **@ServiceConnection** 
annotations to start a PostgreSQL container and register the datasource properties pointing to that database container.

Now if you run all the tests from console using **./mvnw test** you will see output similar to the following:

```shell
....
....
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.jetbrains.bookmarks.UserControllerTest
11:45:32.545 [main] INFO org.springframework.test.context.support.AnnotationConfigContextLoaderUtils -- Could not detect default configuration classes for test class [com.jetbrains.bookmarks.UserControllerTest]: UserControllerTest does not declare any static, non-private, non-final, nested classes annotated with @Configuration.
11:45:32.567 [main] INFO org.testcontainers.images.PullPolicy -- Image pull policy will be performed by: DefaultPullPolicy()
11:45:32.567 [main] INFO org.testcontainers.utility.ImageNameSubstitutor -- Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
11:45:32.585 [main] INFO org.testcontainers.DockerClientFactory -- Testcontainers version: 1.20.4
11:45:32.668 [main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy -- Loaded org.testcontainers.dockerclient.UnixSocketClientProviderStrategy from ~/.testcontainers.properties, will try it first
11:45:32.771 [main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy -- Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
11:45:32.772 [main] INFO org.testcontainers.DockerClientFactory -- Docker host IP address is localhost
11:45:32.785 [main] INFO org.testcontainers.DockerClientFactory -- Connected to docker: 
  Server Version: 27.3.1
  API Version: 1.47
  Operating System: OrbStack
  Total Memory: 16032 MB
11:45:32.846 [main] INFO tc.testcontainers/ryuk:0.11.0 -- Creating container for image: testcontainers/ryuk:0.11.0
11:45:32.982 [main] INFO tc.testcontainers/ryuk:0.11.0 -- Container testcontainers/ryuk:0.11.0 is starting: 9b677f097ae41635e81a77a47d20e8ef1ced81738dfe7f92e5bb600c7127b8ad
11:45:33.174 [main] INFO tc.testcontainers/ryuk:0.11.0 -- Container testcontainers/ryuk:0.11.0 started in PT0.327729S
11:45:33.177 [main] INFO org.testcontainers.utility.RyukResourceReaper -- Ryuk started - will monitor and terminate Testcontainers containers on JVM exit
11:45:33.177 [main] INFO org.testcontainers.DockerClientFactory -- Checking the system...
11:45:33.177 [main] INFO org.testcontainers.DockerClientFactory -- ✔︎ Docker server version should be at least 1.6.0
11:45:33.226 [main] INFO org.springframework.boot.test.context.SpringBootTestContextBootstrapper -- Found @SpringBootConfiguration com.jetbrains.bookmarks.BookmarksApplication for test class com.jetbrains.bookmarks.UserControllerTest
11:45:33.258 [main] INFO tc.postgres:17 -- Creating container for image: postgres:17
11:45:33.299 [main] INFO tc.postgres:17 -- Container postgres:17 is starting: ea792d9e7384db1e76a3a02d3e29e5a1001cc93dd42ef9135da05d8a46ea5015
11:45:34.433 [main] INFO tc.postgres:17 -- Container postgres:17 started in PT1.175057S
11:45:34.435 [main] INFO tc.postgres:17 -- Container is started (JDBC URL: jdbc:postgresql://localhost:33247/test?loggerLevel=OFF)
...
...
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.383 s -- in com.jetbrains.bookmarks.UserControllerTest
[INFO] Running com.jetbrains.bookmarks.BookmarkRepositoryTest
2025-01-02T11:45:36.871+05:30  INFO 43508 --- [bookmarks] [           main] t.c.s.AnnotationConfigContextLoaderUtils : Could not detect default configuration classes for test class [com.jetbrains.bookmarks.BookmarkRepositoryTest]: BookmarkRepositoryTest does not declare any static, non-private, non-final, nested classes annotated with @Configuration.
2025-01-02T11:45:36.882+05:30  INFO 43508 --- [bookmarks] [           main] .b.t.c.SpringBootTestContextBootstrapper : Found @SpringBootConfiguration com.jetbrains.bookmarks.BookmarksApplication for test class com.jetbrains.bookmarks.BookmarkRepositoryTest
2025-01-02T11:45:36.883+05:30  INFO 43508 --- [bookmarks] [           main] tc.postgres:17                           : Creating container for image: postgres:17
2025-01-02T11:45:36.926+05:30  INFO 43508 --- [bookmarks] [           main] tc.postgres:17                           : Container postgres:17 is starting: 7036d558b85bc7461b3c6e13ce8854cc51f9f1454c2de050ea0b8eef97cc23b2
2025-01-02T11:45:37.586+05:30  INFO 43508 --- [bookmarks] [           main] tc.postgres:17                           : Container postgres:17 started in PT0.70296S
....
....
[INFO] Running com.jetbrains.bookmarks.BookmarkControllerTest
2025-01-02T11:45:38.031+05:30  INFO 43508 --- [bookmarks] [           main] t.c.s.AnnotationConfigContextLoaderUtils : Could not detect default configuration classes for test class [com.jetbrains.bookmarks.BookmarkControllerTest]: BookmarkControllerTest does not declare any static, non-private, non-final, nested classes annotated with @Configuration.
2025-01-02T11:45:38.034+05:30  INFO 43508 --- [bookmarks] [           main] .b.t.c.SpringBootTestContextBootstrapper : Found @SpringBootConfiguration com.jetbrains.bookmarks.BookmarksApplication for test class com.jetbrains.bookmarks.BookmarkControllerTest
2025-01-02T11:45:38.035+05:30  INFO 43508 --- [bookmarks] [           main] tc.postgres:17                           : Creating container for image: postgres:17
2025-01-02T11:45:38.075+05:30  INFO 43508 --- [bookmarks] [           main] tc.postgres:17                           : Container postgres:17 is starting: 8ad96e283aedbfacef546788a070c1d993f105b55fdc25191a89451d5bf86c15
2025-01-02T11:45:39.218+05:30  INFO 43508 --- [bookmarks] [           main] tc.postgres:17                           : Container postgres:17 started in PT1.183377S
...
...
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.673 s -- in com.jetbrains.bookmarks.BookmarkControllerTest
[INFO] Running com.jetbrains.bookmarks.UserRepositoryTest
2025-01-02T11:45:39.705+05:30  INFO 43508 --- [bookmarks] [           main] t.c.s.AnnotationConfigContextLoaderUtils : Could not detect default configuration classes for test class [com.jetbrains.bookmarks.UserRepositoryTest]: UserRepositoryTest does not declare any static, non-private, non-final, nested classes annotated with @Configuration.
2025-01-02T11:45:39.709+05:30  INFO 43508 --- [bookmarks] [           main] .b.t.c.SpringBootTestContextBootstrapper : Found @SpringBootConfiguration com.jetbrains.bookmarks.BookmarksApplication for test class com.jetbrains.bookmarks.UserRepositoryTest
2025-01-02T11:45:39.710+05:30  INFO 43508 --- [bookmarks] [           main] tc.postgres:17                           : Creating container for image: postgres:17
2025-01-02T11:45:39.753+05:30  INFO 43508 --- [bookmarks] [           main] tc.postgres:17                           : Container postgres:17 is starting: 6b7d9471caf985f15e47710b9e4331897a1915a353a3a922f4ec4fd643d1fe30
2025-01-02T11:45:41.102+05:30  INFO 43508 --- [bookmarks] [           main] tc.postgres:17                           : Container postgres:17 started in PT1.392548S
...
...
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.760 s -- in com.jetbrains.bookmarks.UserRepositoryTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
```

**There are few problems with this approach:**

* We are duplicating the container definition in each class. If we need to upgrade postgresql container version to a newer version, you will need to update in a lot of places.
* If you search for **'Creating container for image: postgres:17'**, you will notice **four** occurrences, meaning each test class spun up a new postgresql docker container.
  As the project grows, there could be 100's of repositories and controllers.
  If a new container is started for every test class, then the test execution will take a very long time.

## Reuse Testcontainers configuration and leverage Spring Test Context cache
When we create the Spring Boot project with PostgreSQL and Testcontainers starters, 
the following **TestcontainersConfiguration** class is also created under **src/test/java**.

**TestcontainersConfiguration.java**

```java
@TestConfiguration(proxyBeanMethods = false)
class TestcontainersConfiguration {

	@Bean
	@ServiceConnection
	PostgreSQLContainer<?> postgresContainer() {
		return new PostgreSQLContainer<>(DockerImageName.parse("postgres:17"));
	}
}
```

Instead of defining **PostgreSQLContainer** in each test class, we can reuse the container definition in **TestcontainersConfiguration.java** as follows:

**BookmarkRepositoryTest.java**

```java
@DataJpaTest
@Import(TestcontainersConfiguration.class)
class BookmarkRepositoryTest {
    @Autowired
    BookmarkRepository bookmarkRepository;

    @BeforeEach
    void setUp() {
        bookmarkRepository.deleteAllInBatch();
    }

    @Test
    void shouldGetAllBookmarksOrderByCreatedAtDesc() {
        //...
    }
}
```

**BookmarkControllerTest.java**

```java
@SpringBootTest(webEnvironment = RANDOM_PORT)
@Import(TestcontainersConfiguration.class)
class BookmarkControllerTest {

    @Autowired
    private TestRestTemplate restTemplate;

    @Autowired
    private BookmarkRepository bookmarkRepository;

    @BeforeEach
    void setUp() {
        bookmarkRepository.deleteAllInBatch();
    }

    @Test
    void shouldGetAllBookmarks() {
        //...
    }
}
```

**UserRepositoryTest.java**

```java
@DataJpaTest
@Import(TestcontainersConfiguration.class)
class UserRepositoryTest {

    @Autowired
    UserRepository userRepository;

    @BeforeEach
    void setUp() {
        userRepository.deleteAllInBatch();
    }

    @Test
    void shouldGetAllUsersOrderByNameAsc() {
        //...
    }
}
```

**UserControllerTest.java**

```java
@SpringBootTest(webEnvironment = RANDOM_PORT)
@Import(TestcontainersConfiguration.class)
class UserControllerTest {
    @Autowired
    UserRepository userRepository;

    @BeforeEach
    void setUp() {
        userRepository.deleteAllInBatch();
    }

    @Test
    void shouldGetAllUsers() {
        //...
    }
}
```

In this approach, we avoided duplicating the container definition by importing the same **TestcontainersConfiguration**
using **@Import(TestcontainersConfiguration.class)**.

Now, if you run all the tests again using **./mvnw test**, and search for **Creating container for image: postgres:17**, 
you will see only **two** occurrences of this log statement.

The reason for postgresql container being created only 2 times is [Spring Test Context Caching](https://docs.spring.io/spring-framework/reference/testing/testcontext-framework/ctx-management/caching.html).

We are using the same context configuration for both the controller tests, hence the context created for the first controller test is reused for the second controller test as well.
Similarly, the first repository test will create the context loading all the JPA entities and repositories, and the same context is cached and reused by the second repository test class as well.

You can set the following logging property to see how many contexts have been created.

```properties
logging.level.org.springframework.test.context.cache=debug
```

Now if you run the tests using **./mvnw test**, you can see the following log statements showing how many contexts are created.

```shell
2025-01-02T18:44:40.386+05:30  INFO 7182 --- [bookmarks] [           main] c.j.bookmarks.UserControllerTest         : Started UserControllerTest in 3.766 seconds (process running for 4.253)
2025-01-02T18:44:40.387+05:30 DEBUG 7182 --- [bookmarks] [           main] org.springframework.test.context.cache   : Spring test ApplicationContext cache statistics: [DefaultContextCache@55a4a0ab size = 1, maxSize = 32, parentContextCount = 0, hitCount = 0, missCount = 1, failureCount = 0]
...
...
2025-01-02T18:44:42.551+05:30  INFO 7182 --- [bookmarks] [           main] c.j.bookmarks.BookmarkRepositoryTest     : Started BookmarkRepositoryTest in 1.589 seconds (process running for 6.418)
2025-01-02T18:44:42.551+05:30 DEBUG 7182 --- [bookmarks] [           main] org.springframework.test.context.cache   : Spring test ApplicationContext cache statistics: [DefaultContextCache@55a4a0ab size = 2, maxSize = 32, parentContextCount = 0, hitCount = 14, missCount = 2, failureCount = 0]
...
...
[INFO] Running com.jetbrains.bookmarks.BookmarkControllerTest
2025-01-02T18:44:42.567+05:30  INFO 7182 --- [bookmarks] [           main] t.c.s.AnnotationConfigContextLoaderUtils : Could not detect default configuration classes for test class [com.jetbrains.bookmarks.BookmarkControllerTest]: BookmarkControllerTest does not declare any static, non-private, non-final, nested classes annotated with @Configuration.
2025-01-02T18:44:42.569+05:30  INFO 7182 --- [bookmarks] [           main] .b.t.c.SpringBootTestContextBootstrapper : Found @SpringBootConfiguration com.jetbrains.bookmarks.BookmarksApplication for test class com.jetbrains.bookmarks.BookmarkControllerTest
2025-01-02T18:44:42.569+05:30 DEBUG 7182 --- [bookmarks] [           main] org.springframework.test.context.cache   : Spring test ApplicationContext cache statistics: [DefaultContextCache@55a4a0ab size = 2, maxSize = 32, parentContextCount = 0, hitCount = 30, missCount = 2, failureCount = 0]
...
...
[INFO] Running com.jetbrains.bookmarks.UserRepositoryTest
2025-01-02T18:44:42.590+05:30  INFO 7182 --- [bookmarks] [           main] t.c.s.AnnotationConfigContextLoaderUtils : Could not detect default configuration classes for test class [com.jetbrains.bookmarks.UserRepositoryTest]: UserRepositoryTest does not declare any static, non-private, non-final, nested classes annotated with @Configuration.
2025-01-02T18:44:42.594+05:30  INFO 7182 --- [bookmarks] [           main] .b.t.c.SpringBootTestContextBootstrapper : Found @SpringBootConfiguration com.jetbrains.bookmarks.BookmarksApplication for test class com.jetbrains.bookmarks.UserRepositoryTest
2025-01-02T18:44:42.594+05:30 DEBUG 7182 --- [bookmarks] [           main] org.springframework.test.context.cache   : Spring test ApplicationContext cache statistics: [DefaultContextCache@55a4a0ab size = 2, maxSize = 32, parentContextCount = 0, hitCount = 46, missCount = 2, failureCount = 0]
```

If you use different configurations for different tests, then Spring can't reuse the same cached context and will create a new context.

You can explore this behavior by activating a specific profile or injected Mock bean(s) only in one of the tests.

Update the **UserControllerTest.java** as follows:

```java
@SpringBootTest(webEnvironment = RANDOM_PORT)
@Import(TestcontainersConfiguration.class)
class UserControllerTest {

    @Autowired
    private TestRestTemplate restTemplate;

    @Autowired
    private UserRepository userRepository;

    @MockitoBean
    private BookmarkRepository bookmarkRepository;

    // code omitted for brevity

}
```

In this **UserControllerTest** class, we are injecting a mock **BookmarkRepository** bean, 
whereas we are injecting a real **BookmarkRepository** bean in **BookmarkControllerTest**.java.
So, both the tests can't use the same context. Now if you run all the tests, 
you will notice that **three** postgresql containers will be created.

## Using Testcontainers Singleton Containers Pattern
Spring's Test Context Caching helps to reduce the number of contexts being created, which in turn reduces the number of containers being created.
We can further reduce the number of times the containers are created using [Testcontainers Singleton Containers Pattern](https://java.testcontainers.org/test_framework_integration/manual_lifecycle_control/#singleton-containers).

With singleton containers pattern, the containers will be created only once per a JVM instance(basically static fields) and can be reused by all the test classes.

To use the singleton containers pattern with Spring Boot's ServiceConnection support, 
update the **TestcontainersConfiguration.java** as follows:

```java
@TestConfiguration(proxyBeanMethods = false)
class TestcontainersConfiguration {

	static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>(DockerImageName.parse("postgres:17"));

	@Bean
	@ServiceConnection
	PostgreSQLContainer<?> postgresContainer() {
		return postgres;
	}
}
```

Here the **PostgreSQLContainer** is defined as a **static** field and is returning the same instance from the **@ServiceConnection** bean definition method.
So, Spring Boot will reuse the same container for all the tests that imports **TestcontainersConfiguration**.

Now, if you run all the tests again using **./mvnw test**, and search for **Creating container for image: postgres:17**,
you will see only **one** occurrence of this log statement.

This greatly reduces the test execution time. However, we need to be mindful about a few things.

* Using singleton containers with parallel test execution might result in flaky test behavior. 
  As the same container(s) will be used by multiple tests in parallel, the test data setup can be corrupted by other tests running in parallel.
* The tests should be implemented with proper test data setup and cleanup, leaving the containers in a predictable state. 
  Anyway, this is a good approach to be followed whether we use the singleton container pattern or not.

## Summary
Testcontainers greatly simplifies writing tests using real services as test dependencies instead of relying on mocks.
But a common complaint is using Testcontainers will increase the test execution time as it needs to spin up docker containers.

I would say even if it takes a few more (milli)seconds, it is better to write tests that give more confidence in our implementation. 
Writing ultrafast unit tests with mocks which doesn't ensure your code works as expected with real dependencies is not very useful.

I hope this article helps you to understand how to use Testcontainers more efficiently and reduce the time taken for test suite execution.

# My First Impression of Google's NotebookLM


I came across [NotebookLM](https://notebooklm.google/) from my social media feed, and it looks interesting.

With NotebookLM, you can upload documents, videos, web URLs and let the AI agents talk about the uploaded material.
That sounds intriguing.

Recently I wrote an article about software complexity and shared my viewpoint.
Many people appreciated it and some people didn't. That is fine to have different opinions.

<!--more-->

I thought, let's see how NotebookLM interprets what I said.

So, I went to https://notebooklm.google/ and used the Webpage URL of my article
**Is Software Development Hard? Or, We are making it complicated?** (https://www.sivaprasadreddy.com/is-software-development-hard/)
as input.

First, it showed me the following Summary:

{{< figure src="/raw-images/software-complexity-notebooklm-summary.png" alt="software-complexity-notebooklm-summary" >}}

I am already spellbound looking at the summary.

I didn't expect that AI can summarize something so accurately.

A few minutes later, it generated the audio transcription of 2 AI agents talking about my article.

### Audio Transcript

{{< audio
path="audio/Is-Software-Development-Hard.mp3"
caption="Is Software Development Hard"
>}}

Listening to this audio, I was blown away.
I can only imagine two people can sit together and talk rationally, just like in this audio.

My first impression of NotebookLM is :heart_eyes:

# Running your own Spring Initializr and using it from IntelliJ IDEA

If you ever worked with **Spring Boot**, then you are probably aware of [Spring Initializr](https://start.spring.io/).
The Spring Initializr is a web application that you can use to create a Spring Boot application.

Do you know Spring Initializr itself is an [open-source Spring Boot application](https://github.com/spring-io/start.spring.io)?
You can fork it, customize it, deploy on your infrastructure and use it to generate Spring Boot applications.

<!--more-->

The next question would be, what kind of customizations can I do? Why should I bother to host it myself?

Imagine you are working for a large enterprise organization that uses Spring Boot for building various applications.
To improve developer productivity, you might have created your own spring-boot starters that provide your domain-specific functionality.
Now, you may want to make those custom starters available in the Spring Initializr.
This is where you can customize the Spring Initializr to make those custom starters available in the starter list.
This is only one use case, but you can customize other aspects of code generation as you see fit for your needs.

Let me demonstrate how you can customize the Spring Initializr to add a new dependency support.
While building Spring Boot applications, I often use [RestAssured](https://rest-assured.io/) for testing REST APIs.
Let's customize the Spring Initializr to add RestAssured dependency support.

## Run Spring Initializr Locally

1. Clone the start.spring.io GitHub repository
    ```shell
    $ git clone https://github.com/spring-io/start.spring.io.git
    ```
2. Build and run the application as described in the README.
    ```shell
    $ ./mvnw clean install
    $ cd start-site
    $ ../mvnw spring-boot:run
    ```
3. Now you should be able to access the locally running Spring Initializr at [http://localhost:8080](http://localhost:8080).
You can generate a Spring Boot application using this local service just like you would with https://start.spring.io.

{{< box warning >}}
**Slow Response for First Requests**

When you run the Spring Initializr locally and try to generate the application for the first time,
it may take some time (10+ seconds). From the next time onwards, it will be faster.
{{< /box >}}

## Add RestAssured Dependency Support
All the dependencies (starters) that we see when we click on the **ADD DEPENDENCIES** button
are configured in the **start-site/src/main/resources/application.yml** file.

Now, let's add the following **RestAssured** dependency configuration under **Testing** group after **Testcontainers** configuration.

```yaml
...
...
- name: Testcontainers
  id: testcontainers
  description: Provide lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container.
  groupId: org.testcontainers
  artifactId: junit-jupiter
  scope: test
  starter: false
  links:
    - rel: reference
      href: https://java.testcontainers.org/
- name: RestAssured
  id: rest-assured
  description: TestAssured for testing REST APIs
  groupId: io.rest-assured
  artifactId: rest-assured
  scope: test
  starter: false
  links:
    - rel: reference
      href: https://rest-assured.io/
...
...
```

Stop the running application and restart:

```shell
$ ../mvnw spring-boot:run
```

Now you should be able to see **RestAssured** dependency under **Testing** category.

{{< figure src="/images/local-spring-initializr.webp" alt="Spring Initializer" >}}

{{< box tip >}}
**Clear Browser Cache**

Your browser might have cached the previous responses and doesn't show you **RestAssured** dependency.
From your browser, try to **Empty Cache and Hard Reload**.
{{< /box >}}

Now, if you generate a Spring Boot application with **RestAssured** dependency selected, 
then your build file should have the **RestAssured** dependency added.

```xml
<!-- Maven -->
<dependency>
   <groupId>io.rest-assured</groupId>
   <artifactId>rest-assured</artifactId>
   <scope>test</scope>
</dependency>

<!-- Gradle -->
testImplementation 'io.rest-assured:rest-assured'
```

## Using Custom Spring Initializr from IntelliJ IDEA Ultimate Edition
IntelliJ IDEA Ultimate Edition provides [extensive support for Spring Boot](https://www.jetbrains.com/idea/spring/), including a dedicated project wizard.
IntelliJ IDEA under the hood uses https://start.spring.io to create Spring Boot applications.
We can customize the default **Server URL** to **http://localhost:8080**.

{{< figure src="/images/spring-default-server-url.webp" alt="Spring Initializer Server URL" >}}

{{< figure src="/images/spring-local-server-url.webp" alt="Spring Initializer Local URL" >}}

Now we should be able to see **RestAssured** dependency as follows:

{{< figure src="/images/rest-assured-starter.webp" alt="Spring Initializer RestAssured starter" >}}

## Summary
Spring Initializr is a Spring Boot application that powers the widely used https://start.spring.io.
You can customize and extend its functionality to support your needs.
If your organization has custom starters, want to generate the initial application with some customizations,
you can host your own Spring Initializr.
You can also configure your IntelliJ IDEA to use your own custom Spring Initializr to generate Spring Boot applications.

# Master the Art of Requesting Technical Help: A Handy Email Template

As a software developer, at the beginning of my career, I thought it was more important to learn one more programming language or framework or library.
But as years passed by, I realized it is also equally important to learn how to interact with people.
Now remote work is becoming new normal and social media platforms like Twitter, LinkedIn, etc. 
are a common place to interact with people across the globe. 

<!--more-->

No matter how skilled you are, you will need help from others.
It is important to learn how to ask for help in a polite and professional manner
so that you have a better chance of getting help.

I occasionally get emails and DMs on Twitter and LinkedIn asking for some technical help.
Most of them are polite and very professional.
But once in a while, I get requests in such a way that I wonder, don't they even know how to ask for help??!!! 
When I receive requests for help, I don't look for polished English.
But I look for the right attitude and some basic manners.

In this article, I would like to share some tips on asking for technical help in a more professional way.
If you follow these tips, I cannot guarantee that you will get a reply, but there could be a better chance of getting a reply.

First, let's look at some examples of how not to ask for help.

## Some examples of how NOT to ask for help
Here are some examples of how NOT to ask for help.

**Example 1**

```shell
Subject: hi

Im getting an error in Spring Security. plz help, its urgent.
```

First, looking at the title "hi" I don't know whether it is a genuine email or a spam mail trying to sell viagra pills.

Second, nobody has time to come up with a questionnaire asking for more details about what you are working on, what is the use case, what is the Exception you are getting, etc.
You are supposed to provide that.

**Example 2**

```shell
Subject: Urgent

hi bro,
I need urgent help in Spring Boot. Here is my phone no: 988888998. Call me on WhatsApp. Its very urgent.
```

So, you need help from someone for free, and you are telling him/her to call you urgently. Not gonna happen.

**[In Clint Eastwood's voice]: What is this bro shit anyway.**

I know its common using "bro" in this modern generation, 
but I would refrain from using that when talking to a person for the first time. 

Once you get to know each other, you can call bro, dude, buddy, or whatever. 

**Example 3**

```shell
Subject: Regarding Spring Security Issue

Hi,
I see you have published some videos on Spring Boot and Testcontainers.
They are good, but I need more in-depth videos. 
Make more in-depth videos and publish them regularly.
```

This is better than previous examples, but can be improved.

**The phrase "I need more in-depth videos" is very vague.** Are you looking for something specific?
Any specific complex use case you have in mind? 
Giving more details might help them plan for the videos covering advanced concepts.

Second, using a simple word **"Please"** would go a long way.

The sentence **"Make more in-depth videos and publish them regularly"** sounds like you are passing an order.
Instead, write **"Could you please make videos covering those advanced use cases. 
That would be really helpful. Looking forward to more videos in this series"**.

**Example 4**

```shell
Subject: DDD + Kafka + MongoDB

hey
whatsup bro?

I suggest you make videos on developing Spring Boot application following DDD using mongodb and kafka.

We are planning to use these technologies in our application.
can you also explain how to deploy this application in kubernetes?
```

You are not asking for some help. You are asking someone to build a real project, 
including the deployment process which generally takes a few months for a 5-member team.

It is highly unlikely you are going to get a reply.

## Email Template Asking for Technical Help

Imagine a situation I am in:

> I am working with a framework, and something is not working.
I tried many ways to fix the issue, but I couldn't figure out how to resolve the issue.
I have googled and read through many blogs and watched videos. Still no luck.
>
> I know "John," who has a blog and a YouTube channel with lots of content about the same framework.
I have watched some of his videos and read some of his blog posts.
They are very helpful. But I couldn't find content related to my current problem.
Knowing John's expertise on the subject, I believe he could help me to resolve the issue I am facing.
>
> I found John's Email address and Twitter handle from his blog.

As Twitter/LinkedIn DMs are not great for sharing code snippets,
I would like to write an email to John asking for help.

**If I am interacting with someone for the first time asking for help,
this is how I would write the email:**


{{< box info >}}
**Title: Request for help on [Technology]**

**[Salutation]**

Hi John,

**[Introduce yourself]**

My name is Siva. I am a software developer from India.
I have been working with Java and Spring technologies since 2006.

**[Tell him/her how do you know them]**

I have been a follower of your blog and YouTube channel for a few years.
I learned many things from your videos and blog posts.
You explain concepts very clearly with practical examples.

**[Give some background on what you are working on and gist of the problem]**

I am working on a project that uses **[combination of Technologies]**.
I am facing a problem, and I am unable to figure out what is the root cause of it or how to resolve this issue.

**[Explain what problem you are facing]**

Here is the use case I am implementing: **[Use case description here]**

This is how I configured the application: **[Key configuration relevant to the problem]**

This is how I am invoking this feature and this is the error I am getting: **[Exception stacktrace]**

**[Share what you have already tried]**

I have already tried the following:

1. Approach 1
2. Approach 2

But still the problem persists.

Here is a minimal reproducer of the problem: **[source code repository link]**

**[Appreciate their help]**

If you could take a look at this issue and provide me some guidance or
point me in the right direction, that would be of great help for me.

**[Thank him/her for sharing their knowledge]**

Once again, thanks a lot for sharing your knowledge via blog posts and videos.
They are very helpful to understand the concepts in depth.

**[Signature]**

Thanks,

Siva
{{< /box >}}

This might look too formal, but this also conveys my admiration to John and my professional attitude.
As I am giving enough details about the problem, what I have already tried, John understands that I am not asking for help just casually.
It conveys that I did my homework before reaching out for help.

Also, as I am giving a minimal reproducer,
there is a better chance that John might take a look at the issue and try to help me.

## Important things to understand
When we are asking someone for help, we need to understand that they might be busy in their life.

* They might be fully occupied with their official work.
* They might be facing some personal family related issues.
* They might be going through tough times.

So, we shouldn't expect a reply immediately.

**Just because someone is active on social media doesn't mean
they have nothing better to do than solving YOUR problems immediately.**

## Summary
Modern software development involves using a wide range of technologies.
We may not have expert level knowledge on all those technologies.
We definitely need help from others who have more expertise in those areas.

Thankfully, with social media platforms, we have the luxury of reaching out to people living anywhere in the world.
But knowing how to ask for help is crucial. 
Asking politely and in a professional manner increases your chances of getting a response.

I hope this article helps you to seek help in a better way.

# About Me


## K. Siva Prasad Reddy

I am a software engineer, architect, trainer, and technical content creator. I have been building software professionally since 2006, with most of my work centered around Java, Spring Boot, distributed systems, and backend engineering.

I enjoy taking complex technical ideas and explaining them through practical examples. That is the main reason behind SivaLabs: sharing tutorials, opinions, and learning resources that help developers build better applications with confidence.

## What I Work On

* Building scalable backend applications with **Java**, **Kotlin**, **Go**, **Spring Boot**, and **Spring Cloud**.
* Designing **microservices**, **modular monoliths**, REST APIs, and event-driven systems.
* Working with **Kafka**, **RabbitMQ**, **AWS SQS**, Docker, Kubernetes, and cloud-native development practices.
* Automating delivery and infrastructure using **Jenkins**, **Terraform**, **AWS CDK**, and **Pulumi**.
* Exploring AI-assisted development with tools, agents, **Spring AI**, and **LangChain4j**.

## Teaching And Writing

I write tutorials and articles for both beginners and experienced developers. I also create video tutorials and training material around Java, Spring Boot, microservices, testing, architecture, and developer productivity.

I have authored books and long-form technical material, and I regularly share practical lessons from real-world software development.

## Contact

* Email: **sivalabs.in@gmail.com**
* Blog: [sivalabs.in](https://www.sivalabs.in)
* YouTube: [youtube.com/sivalabs](https://www.youtube.com/sivalabs)
* LinkedIn: [K. Siva Prasad Reddy](https://www.linkedin.com/in/ksivaprasadreddy/)
* GitHub: [sivaprasadreddy](https://github.com/sivaprasadreddy)
* Twitter/X: [@sivalabs](https://twitter.com/sivalabs)
* Bluesky: [@sivalabs.in](https://bsky.app/profile/sivalabs.in)
* Mastodon: [@sivalabs](https://mastodon.social/@sivalabs)

{{< box info >}}
**Looking for a trainer or freelancer?**

If you are looking for a trainer or freelancer for your next project, please contact me at **sivalabs.in@gmail.com**.
{{< /box >}}

# Mastering Spring Boot in 5 Stages


[Spring Boot](https://spring.io/) is the most popular framework in the Java world to build enterprise applications.
Also, Spring Boot is the most sought-after skill to get hired as a Java developer.

Here is my recommended approach to learn Spring Boot.

<!--more-->

## 1. Prerequisites: What you should already know
1. If you are completely new to Java, then directly jumping into Spring Boot is NOT recommended.
2. First, learn Core Java and get familiar with the Java ecosystem.
3. Get familiar with the installation of JDK and using the build tools like Maven or Gradle.
4. Get familiar with using IDEs like IntelliJ IDEA, Eclipse/STS, VS Code

## 2. If you are new to Spring itself
Spring Boot is an opinionated framework built on top of various Spring portfolio projects.

So, if you are completely new to Spring itself, I would recommend first to learn a few Spring Core concepts:

* Dependency Injection
* Spring Bean Configuration
* Bean Scopes
* Aspect Oriented Programming (AOP)

**Resources:**
* [Spring Boot Missing Guide : What you need to know before starting with Spring Boot](https://www.youtube.com/playlist?list=PLuNxlOYbv61jZL1IiciTgWezZoqEp4WXh)

You don't have to master every detail of these concepts in-depth.
But having a decent understanding of these concepts will make it easy to understand Spring Boot.

## 3. You are familiar with Spring framework and want to learn Spring Boot
Once you are familiar with Spring core concepts, then you can begin your journey to learn Spring Boot.

{{< box tip >}}
**Important Note:**

Spring Boot is an opinionated framework built on top of various other Spring projects with convention-over-configuration approach.
So, if you are not familiar with underlying technologies such as JPA/Hibernate, Kafka, etc then Spring Boot looks like magic.

While learning as a beginner, you may have to proceed further without understanding how exactly things work behind the scenes.
Once you got it working, you may have to go back and explore further how Spring Boot auto-configuration makes it work.
{{< /box >}}
 

### Stage 1: Master Spring Boot Basics
In Stage 1, you learn the fundamentals of Spring Boot.

* [Getting Started with Spring Boot](https://www.sivalabs.in/getting-started-with-spring-boot/): Learn how to create a Spring Boot application and
  build a basic HelloWorld style REST API. Then learn how to run the Spring Boot application.
* **Build a basic CRUD REST API:** Create a basic REST API performing CRUD(Create, Read, Update, Delete) operations using a HashMap based storage.
  The goals of this exercise are:
    * Get familiar with how to implement API handler methods for different types of HTTP methods using
      **@GetMapping, @PostMapping, @PutMapping, @DeleteMapping** annotations.
    * Understand how to use **@PathVariable, @RequestBody** to work with URL path variables and HTTP request body.
    * To keep it simple, store the data in a HashMap with key as primaryKey and object as the value.
* **Understand Spring Boot Core Features**:
  Spring Boot provides some features out of the box which are commonly used in building web and enterprise applications.

    * **Spring Boot Logging**: Spring Boot automatically provides logging support with SLF4J and Logback.
      You may want to configure it to suit your application needs. So, you need to understand how to customize the default logging configuration.
        
      * [Spring Boot Logging Tutorial](https://www.sivalabs.in/spring-boot-logging-tutorial/)
      * [Spring Boot Tips : Part 3 - How to implement Logging in Spring Boot applications](https://www.youtube.com/watch?v=tmj6QphzAPo)

    * **Spring Boot Application Configuration**: Spring Boot provides support for providing the application configuration properties in a wide variety of ways.
      But you mostly like will be using three or four approaches to provide default configuration and overriding the configuration.
        
      * [Spring Boot Application Configuration Tutorial](https://www.sivalabs.in/spring-boot-application-configuration-tutorial/)
      * [Spring Boot Tips : Part 2 - Managing Application Configuration Properties In The Right Way](https://www.youtube.com/watch?v=4OX7hbAGuRE)

    * **Spring Boot Profiles**: Profiles is a concept in Spring framework which helps in configuring the application differently for different environments.
      In real projects, you typically want to configure your application with different property values for different environments (dev, QA, prod).
      You can use Spring profiles to customize/configure your application per-environment and enable the desired profile when starting the application.
      Spring Boot makes is easy to use profiles with some default conventions.
      
      * [Spring Boot Profiles Tutorial](https://www.sivalabs.in/spring-boot-profiles-tutorial/)

    * **Spring Boot Testing**: Testing is an integral part of any professional software development.
      Spring Boot provides excellent support for testing.
      
      * [Spring Boot Testing Tutorial](https://www.sivalabs.in/spring-boot-testing-tutorial/): This tutorial will give an overview of Spring Boot's testing support.
      But when you use databases or Kafka, etc., then you can use the [Testcontainers](https://testcontainers.com/) library for testing. You can learn this in the next stage.

After completing these tutorials, you should have a good grasp on Spring Boot basics.

### Stage 2: Learn to build REST APIs using Database
In Stage 2, you learn how to work with SQL databases, how to handle database transactions,
how to test your application when using a database.
You should also learn how to handle exceptions gracefully.
Finally, you will learn some best practices while building REST APIs using Spring Boot.

**Prerequisites:**
* You should be familiar with JDBC if you want to use **JdbcTemplate** or **JdbcClient**
* You should have at least some basic knowledge of JPA if you want to use **Spring Data JPA**

Spring Boot provides higher-level abstractions to make it easy to work with databases.
You can use newer **JdbcClient** or older **JdbcTemplate** to execute SQL queries without writing low-level boilerplate code.

* [Spring Boot JdbcClient Tutorial](https://www.sivalabs.in/spring-boot-jdbcclient-tutorial/)
* [Spring Boot JdbcTemplate Tutorial](https://www.sivalabs.in/spring-boot-jdbctemplate-tutorial/)

Database transaction management is a crucial part of any real-world application using an SQL database.
Spring Boot makes it straightforward to apply transaction boundaries using **@Transactional** annotation.
But it is crucial to understand how the transaction management works to avoid some common gotchas.

* [Spring Boot Database Transaction Management Tutorial](https://www.sivalabs.in/spring-boot-database-transaction-management-tutorial/)

In real-world projects, it is highly recommended to use database migration tools like **Flyway** or **Liquibase** to apply any database changes.
Again, Spring Boot makes it very straightforward to use those database migration tools with its auto-configuration support.

* [Spring Boot Flyway Database Migration Tutorial](https://www.sivalabs.in/spring-boot-flyway-database-migration-tutorial/)

JPA/Hibernate is the most popular ORM library in Java.
Spring Data JPA provides a higher-level abstraction on top of JPA that simplifies performing CRUD operations, pagination, etc.

Before learning Spring Data JPA, I highly recommend learning JPA basics of how to configure entities.
To keep it simple, you can focus on working with a single entity without any OneToMany, ManyToOne, ManyToMany relations.

This is the right time to learn some best practices while building REST APIs such as following REST API conventions,
using proper status codes, etc.
Also, now is the right time to learn how to test REST APIs by writing integration tests or repository tests by using Testcontainers.

* [Spring Boot REST API Best Practices - Part 1 : Implementing Get Collection API](https://www.sivalabs.in/spring-boot-rest-api-best-practices-part-1/)
* [Spring Boot REST API Best Practices - Part 2 : Implementing Create and Update APIs](https://www.sivalabs.in/spring-boot-rest-api-best-practices-part-2/)
* [Spring Boot REST API Best Practices - Part 3 : Implementing FindById and DeleteById APIs](https://www.sivalabs.in/spring-boot-rest-api-best-practices-part-3/)

It's very important to handle exceptions and return meaningful responses in case of failures.
Spring Boot provides different approaches to handle exceptions.

* [Spring Boot REST API Best Practices - Part 4 : Exception Handling in REST APIs](https://www.sivalabs.in/spring-boot-rest-api-best-practices-part-4/)

{{< youtube VvWtoaeHQUQ >}}

To learn more about testing the Spring Boot applications, you can refer the following resources:
* [Java Testing Made Easy](https://www.youtube.com/playlist?list=PLuNxlOYbv61jtHHFHBOc9N7Dg5jn013ix)
* [Philip Riecks Blog](https://rieckpil.de/)
* [Testcontainers Guides](https://testcontainers.com/guides/)

To learn more about JPA/Hibernate and Spring Data JPA:

* [Vlad Mihalcea Blog](https://vladmihalcea.com/blog/)
* [Thorben Janssen Blog](https://thorben-janssen.com/)

By now, you should have good enough knowledge to build a REST API backing with a database in a professional way.

### Stage 3: Learn Spring Security
In Stage 3, you can learn Spring Security, Event Handling, and Job Scheduling.

Securing the web application or REST APIs is a very common requirement.
Spring Security is a highly configurable, customizable framework for implementing security.
The "highly configurable and customizable" nature of Spring Security also comes with a steep learning curve.

* The official [Securing a Web Application Guide](https://spring.io/guides/gs/securing-web) helps you to be familiar with Spring Security basics.
* The [Getting started with Spring Security and Spring Boot](https://reflectoring.io/spring-security/) article by [Tom Hombergs](https://x.com/TomHombergs) is an excellent comprehensive tutorial to learn Spring Security.
* An excellent article on [How to Secure your REST APIs with Spring Security & JSON Web Tokens (JWTs)](https://www.danvega.dev/blog/spring-security-jwt) by [Dan Vega](https://x.com/therealdanvega)

{{< box tip >}}
**TIP:**

Securing microservices using OAuth 2 is an advanced concept which you can learn in the next stage(s).
{{< /box >}}

Publishing and consuming events is a good practice to decouple the producer and consumer.
Spring provides support for publishing and consuming events using an annotation-based programming model.

* [Spring Boot Application Events Explained](https://reflectoring.io/spring-boot-application-events-explained/)

Running background jobs is another common requirement in enterprise applications.
Spring Boot provides job scheduling feature out of the box, but also supports integration with Quartz and JobRunr.

* [Scheduling Tasks](https://spring.io/guides/gs/scheduling-tasks)
* [Quartz Scheduler](https://docs.spring.io/spring-boot/reference/io/quartz.html)
* [JobRunr with Spring Boot](https://www.jobrunr.io/en/documentation/configuration/spring/)

### Stage 4: Learn building microservices and securing them with OAuth 2
In Stage 4, you can focus on learning how to build microservices and secure them with OAuth 2.

By now you know how to build a monolithic application using Spring Boot.
Now, you can focus on building microservices using Spring Boot, Spring Cloud, etc.

* **Resources:**
    * [Spring Security OAuth 2.0 Series](https://www.sivalabs.in/spring-security-oauth2-tutorial-introduction/)
    * [Spring Boot Microservices Course](https://www.youtube.com/playlist?list=PLuNxlOYbv61g_ytin-wgkecfWDKVCEDmB)
    * [Spring Boot and OAuth2 Guide](https://spring.io/guides/tutorials/spring-boot-oauth2)
    * [Securing a REST API with OAuth 2.0 Spring Academy Course](https://spring.academy/courses/spring-academy-secure-rest-api-oauth2)

### Stage 5: Explore how things work behind the scenes
In stage 5, you can try to understand how things work behind the scenes.
The most important part of Spring Boot is "Auto Configuration", which many people call as "magic".

Spring Boot auto-configuration is nothing but @Conditional feature on steroids.
You can read [How Spring Boot AutoConfiguration magic works?](https://www.sivalabs.in/how-springboot-autoconfiguration-magic/) article to understand Spring Boot AutoConfiguration.

At this point of time, you can focus on what is required for your project and be able to use them with Spring Boot.
Your project may be using Kafka, RabbitMQ, Spring Batch, etc.
With the knowledge you gained by now, you shouldn't have a problem in learning how to integrate them with Spring Boot.

At this stage, I would recommend focusing on learning Architecture concepts.

* [Spring Modulith](https://docs.spring.io/spring-modulith/reference/): An approach to building modular monolithic applications using Spring Boot.

    * [Spring Modulith Crash Course : Building Modular Monoliths using Spring Boot](https://www.youtube.com/watch?v=FkP2aZiBrhg)
    * [The Modern Monolith, with Spring Modulith](https://www.youtube.com/watch?v=Pae2D4XcEIg)

* **Domain Driven Design (DDD)**
    * [Architecturally evident Spring applications with jMolecules](https://www.youtube.com/watch?v=-I7KiV_6f-s)
    * [Modular monoliths](https://www.youtube.com/watch?v=kbKxmEeuvc4)
    * [Implementing Domain Driven Design with Spring](https://www.youtube.com/watch?v=VGhg6Tfxb60)
    * [Clean Architecture with Spring](https://www.youtube.com/watch?v=cPH5AiqLQTo)

* [Microservices.io](https://microservices.io/)

## Learning Resources
* [Spring Website](https://spring.io)
* [Spring Academy](https://spring.academy/)
* [Spring Boot Guides](https://spring.io/guides)
* [Spring IO YouTube channel](https://www.youtube.com/@SpringIOConference)
* [Spring Developer YouTube channel](https://www.youtube.com/@SpringSourceDev)
* [Josh Long YouTube channel](https://www.youtube.com/@coffeesoftware)
* [SivaLabs Blog](https://sivalabs.in)
* [SivaLabs YouTube channel](https://www.youtube.com/@sivalabs)
* [Thomas Vitale Blog](https://www.thomasvitale.com/)
* [Reflectoring Blog](https://reflectoring.io/)
* [Marco Behler Blog](https://www.marcobehler.com/)
* [Vlad Mihalcea Blog](https://vladmihalcea.com/blog/)
* [Philip Riecks Blog](https://rieckpil.de/)
* [Piotr\'s TechBlog](https://piotrminkowski.com/)

## Summary
Spring Boot is a powerful framework aimed to make Java enterprise application development easy.
Spring Boot provides integration support for everything under the sun either by official starters or community implemented starters.

So, expecting to master Spring Boot in a few hours or in a couple of days is unrealistic.
It's going to take some time.
Luckily, you don't have to learn everything that Spring Boot offers.

If you reach Stage 3, you are good to apply for jobs looking for Spring Boot skills and learn more on the job.

I hope these resources and guidelines help you to learn Spring Boot.

Happy learning!
# Thymeleaf Layouts using Fragment Expressions in Spring Boot GraalVM Native Image


Typically, in Spring Boot + Thymeleaf applications, 
we use [thymeleaf-layout-dialect](https://github.com/ultraq/thymeleaf-layout-dialect) 
to define the common layout of the web pages and it works fine. 

But when we compile the Spring Boot application to GraalVM native image,
it is failing due to [this error](https://github.com/ultraq/thymeleaf-layout-dialect/issues/235).
I tried many suggestions mentioned in the above issue, but none of them worked for me. 

<!--more-->

Then [Oliver Drotbohm](https://twitter.com/odrotbohm) suggested me [Flexible layouts](https://www.thymeleaf.org/doc/tutorials/3.1/usingthymeleaf.html#flexible-layouts-beyond-mere-fragment-insertion) 
approach to create layouts support natively provided by Thymeleaf itself.
This approach works fine with GraalVM native image as well.

In this article, we will see how to create layouts using Thymeleaf without using thymeleaf-layout-dialect.

## Create a Spring Boot Application
Generate a Spring Boot application with **Web**, **Thymeleaf** and **GraalVM Native Support** dependencies using [Spring Initializr](https://start.spring.io/).

## Create Layout Template

Create a layout template `layout.html` in `src/main/resources/templates` directory.

```html
<!DOCTYPE html>
<html lang="en"
      xmlns:th="http://www.thymeleaf.org"
      th:fragment="layout (title, content, pageScripts)" >
<head>
    <meta charset="utf-8"/>
    <meta http-equiv="X-UA-Compatible" content="IE=edge"/>
    <title>MyApp - <th:block th:insert="${title}"/></title>
</head>
<body>
<main>
    <div id="app" class="container">
        <div th:block th:replace="${content}">
            <p>Layout content</p>
        </div>
    </div>
</main>
<th:block th:replace="${pageScripts}">
</th:block>
</body>
</html>
```

**Few things to note here:**

* The root html tag has **th:fragment="layout (title, content, pageScripts)"** attribute, which gives a name to the fragment and defines the parameters it accepts.
* The layout has three parameters: **title**, **content**, and **pageScripts**.
  * The **title** parameter is used to set the title of the page.
  * The **content** parameter is used to include the content of the page.
  * The **pageScripts** parameter is used to include the page-specific scripts.
* Some applications follow a pattern for the **title** as **"AppName - PageName"**. So, I have used the same pattern here.

## Create a Page Template
Let's create a page template `home.html` in `src/main/resources/templates` directory which uses the layout.

```html
<!DOCTYPE html>
<html lang="en"
      xmlns="http://www.w3.org/1999/xhtml"
      th:replace="~{layout :: layout(title=~{::title/text()},
                    content=~{::#content}, pageScripts=~{})}">
<head>
    <title>Home</title>
</head>
<body>
<div id="content">
    <h1>This is Home Page</h1>
    <a href="/">Home Page</a>
    <a href="/about">About Page</a>
</div>
</body>
</html>
```

**Few things to note here:**

* The root html tag has **th:replace="~{layout :: layout(...)}"** attribute, 
  which is using the fragment expression to specify which fragment to replace with.
  Here we are referencing the filename `layout.html` and the fragment name `layout`.
* Then, we are passing all the parameter values with named arguments.
* The **title** parameter is specified using the fragment expression **~{::title/text()}**.
  This refers to the text content of the `<title>` element in the current page template.
* The **content** parameter is specified using the fragment expression **~{::#content}**.
  This refers to the content of the `<div id="content">` element in the current page template.
* The **pageScripts** parameter is specified using the fragment expression **~{}**.
  This refers to an empty fragment that can be used for specifying no markup.

Instead of specifying the parameter values using named-arguments, you can also specify them using default positional arguments as follows:

```html
<html lang="en"
      xmlns="http://www.w3.org/1999/xhtml"
      th:replace="~{layout :: layout(~{::title/text()}, ~{::#content},~{})}">
...
</html>
```

## Create Another Page Template
Let's create another page template `about.html` in `src/main/resources/templates` directory which uses the layout.

```html
<!DOCTYPE html>
<html lang="en"
      xmlns="http://www.w3.org/1999/xhtml"
      th:replace="~{layout :: layout(~{::title/text()}, ~{::#content},~{::#pageScripts})}">
<head>
    <title>About</title>
</head>
<body>
<div id="content">
    <h1>This is About Page</h1>
    <a href="/">Home Page</a>
    <a href="/about">About Page</a>
</div>
<th:block id="pageScripts">
<script src="https://code.jquery.com/jquery-3.7.1.min.js"></script>
</th:block>
</body>
</html>
```

**Few things to note here:**

* We are passing the layout fragment parameters using positional arguments.
* The **pageScripts** parameter is specified using the fragment expression **~{::#pageScripts}**.
  This refers to the content of the `<div id="pageScripts">` element in the current page template.
  For this specific page, we are including a script tag that loads jQuery.

## Create controller handler methods
Finally, let's create the Controller to handle requests for `"/"` and `"/about"` URLs.

```java
package com.sivalabs.demo;

import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.GetMapping;

@Controller
class DemoController {

    @GetMapping
    String home() {
        return "home";
    }

    @GetMapping("/about")
    String about() {
        return "about";
    }
}
```

## Run the Application
First, let's verify that the application is working fine without any issues.
Run the Spring Boot application and access the home page at `http://localhost:8080/`.

If you are using an Apple MacBook M1/M2/M3, then you can use the `dashaun/builder:tiny` builder to support ARM64 architecture.

```xml
<plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
    <configuration>
        <image>
            <builder>dashaun/builder:tiny</builder>
        </image>
    </configuration>
</plugin>
```

If you are using Gradle:

```groovy
tasks.named("bootBuildImage") {
	builder = "dashaun/builder:tiny"
}
```

Now, lets compile the application to GraalVM native image and run it.

```shell
#Maven
./mvnw spring-boot:build-image -Pnative -DskipTests -Dspring-boot.build-image.imageName=thymeleaf-demo

#Gradle
./gradlew bootBuildImage -Pnative -x test --imageName=thymeleaf-demo

docker run -p 8080:8080 thymeleaf-demo
```

Access the home page at `http://localhost:8080/` and about page at `http://localhost:8080/about`.
You should be able to see the pages without any issues.

## Conclusion
Though creating layouts using fragment expressions is a bit verbose compared to **thymeleaf-layout-dialect**,
it works fine with GraalVM native image.

One of the main problems with **thymeleaf-layout-dialect** is that it uses Groovy under the hood, which has some problems with GraalVM native image.
There is an attempt to port **thymeleaf-layout-dialect** to Java, https://github.com/zhanhb/thymeleaf-layout-dialect, 
You can try it if you are interested.

# Spring AI RAG using Embedding Models and Vector Databases

In this article, we will explore the following:

* Introduction to Embedding Models.
* Loading data using DocumentReaders.
* Storing embeddings in VectorStores.
* Implementing RAG (Retrieval-Augmented Generation), a.k.a. Prompt Stuffing.

Let's get started.

<!--more-->

## Introduction

{{< box info >}}
**Sample Code Repository**

You can find the sample code for this article in the [GitHub repository](https://github.com/sivaprasadreddy/spring-ai-samples/tree/main/spring-ai-openai)

{{< /box >}}

Large Language Models (LLMs) like OpenAI, Azure OpenAI, Google Vertex, etc. are
trained on large datasets. But those models are not trained on your private data,
so they may not be able to answer questions specific to your domain.
But training a model on your private data may be expensive and time-consuming.
So, how can we use these LLMs to answer questions specific to our domain?

One way to do this is to use RAG (Retrieval-Augmented Generation), a.k.a. Prompt Stuffing.
Using RAG, we will retrieve relevant documents from a datastore and pass that to the LLM to generate the answer.
In this process, we will use an embedding model to convert the documents into embeddings and store them in a vector database.

## Understand Retrieval-Augmented Generation (RAG)
Your business might store structured data in relational databases, unstructured data in NoSQL databases, or even in files.
You will be able to effectively query relational databases using SQL, NoSQL databases using their query languages.
You can also use full-text search engines like Elasticsearch, Solr, etc., to query unstructured data.

However, you might want to retrieve data using natural language with semantic meaning.

For example, "I love Java programming language" and "Java is always my go-to language" have the same semantic meaning, but uses different words.
Trying to retrieve data using the exact words might not be effective.

This is where **Embeddings** comes into play. Embeddings are vector representations of words, sentences, or documents.
You can use these embeddings to retrieve data using natural language.

You can convert your structured and unstructured data into embeddings and store them in a **Vector Database**.
You can then query the **Vector Database** using natural language and retrieve relevant data.
Then, you can query the AI models passing along the relevant data to get the responses.

> **Retrieval-Augmented Generation (RAG)** is the process of optimizing the output of a LLM,
by using additional knowledge base in addition to its training data before generating a response.

## Embedding APIs
The Embedding APIs can convert words, sentences, documents, or images into embeddings. 
An embedding is a vector representation of words, sentences, or documents.

For example, A word "Apple" might be represented as a vector [0.1, 0.2, 0.3, 0.4, 0.5]. 
A sentence "I love Apple" might be represented as a vector [0.1, 10.3, -10.2, 90.3, 2.4, -0.5].

Spring AI provides an **EmbeddingModel** interface to convert text or documents into embeddings.
You can use any of the supported **EmbeddingModel** implementations 
like **OpenAiEmbeddingModel**, **OllamaEmbeddingModel**, 
**AzureOpenAiEmbeddingModel**, **VertexAiEmbeddingModel**, etc.

Depending on which implementation you want to use, you can add the corresponding dependency 
and configure the properties in **application.properties** file.

For example, if you want to use OpenAI's EmbeddingModel, you can add the following dependency to your **pom.xml** file.

```xml
<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-openai-spring-boot-starter</artifactId>
    <version>1.0.0-M1</version>
</dependency>
```

Configure the properties in **application.properties** file.

```properties
spring.ai.openai.api-key=${OPENAI_API_KEY}

# You can override the above common api-key for embedding using the following property
spring.ai.openai.embedding.api-key=${OPENAI_API_KEY}
```

With the above configuration, you can inject **EmbeddingModel** and 
convert text or documents into embeddings as follows:

```java
@Component
class MyComponent {
    private final EmbeddingModel embeddingModel;
    
    public MyComponent(EmbeddingModel embeddingModel) {
        this.embeddingModel = embeddingModel;
    }
    
    public void convertTextToEmbedding() {
        //Example 1: Convert text to embeddings
        List<Double> embeddings1 = embeddingModel.embed("I like Spring Boot");
        
        //Example 2: Convert document to embeddings
        List<Double> embeddings2 = embeddingModel.embed(new Document("I like Spring Boot"));
        
        //Example 3: Convert text to embeddings using options
        EmbeddingRequest embeddingRequest =
                new EmbeddingRequest(List.of("I like Spring Boot"),
                        OpenAiEmbeddingOptions.builder()
                                .withModel("text-davinci-003")
                                .build());
        EmbeddingResponse embeddingResponse = embeddingModel.call(embeddingRequest);
        List<Double> embeddings3 = embeddingResponse.getResult().getOutput();
    }
}
```

## Vector Database
A Vector Database is a database that stores embeddings.
You can store embeddings of words, sentences, or documents in a Vector Database.
You can use the Vector Database to query the embeddings using natural language and retrieve relevant data.

Spring AI provides a **VectorStore** interface to store and retrieve embeddings.
Currently, Spring AI provides **VectorStore** implementations like **SimpleVectorStore**, 
**ChromaVectorStore**, **Neo4jVectorStore**, **PgVectorStore**, **RedisVectorStore**, etc.

Let us see how to use **SimpleVectorStore** to store and retrieve embeddings.

```java
@Configuration
class AppConfig {
    @Bean
    VectorStore vectorStore(EmbeddingModel embeddingModel) {
        return new SimpleVectorStore(embeddingModel);
    }
}

@Component
class MyComponent {
    private final VectorStore vectorStore;
    
    public MyComponent(VectorStore vectorStore) {
        this.vectorStore = vectorStore;
    }
    
    public void storeAndRetrieveEmbeddings() {
        // Store embeddings
        List<Document> documents = 
                List.of(new Document("I like Spring Boot"),
                        new Document("I love Java programming language"));
        vectorStore.add(documents);
        
        // Retrieve embeddings
        SearchRequest query = SearchRequest.query("Spring Boot").withTopK(2);
        List<Document> similarDocuments = vectorStore.similaritySearch(query);
        String relevantData = similarDocuments.stream()
                            .map(Document::getContent)
                            .collect(Collectors.joining(System.lineSeparator()));
    }
}
```

In the above code, we are adding the Documents to the VectorStore,
which internally converts the documents into embeddings using the EmbeddingClient
and stores them in the Vector Database.

We are then querying the VectorStore using natural language and retrieving relevant data.
We specified the maximum number of similar documents to return using the **withTopK()** method.

## DocumentReader and DocumentWriter
In the above examples, we directly constructed a **Document** instance from a String to represent the text or document.
But in real-world applications, you might want to read the documents from a file, database, or any other source.

Spring AI provides **DocumentReader** and **DocumentWriter** interfaces to read and write documents from and to different sources.

As of now, Spring AI provides **DocumentReader** implementations like **JsonReader**, 
**TextReader**, **PagePdfDocumentReader**, etc.

The **VectorStore** interface extends the **DocumentWriter** interface, 
so you can use any **VectorStore** implementation as a **DocumentWriter**.

Let us see how to use **TextReader** to read a text document and store it in the VectorStore.

```java
@Component
class MyComponent {
    private final VectorStore vectorStore;
    
    @Value("classpath:myfile.txt")
    private Resource resource;
    
    public MyComponent(VectorStore vectorStore) {
        this.vectorStore = vectorStore;
    }
    
    public void storeEmbeddingsFromTextFile() {
        var textReader = new TextReader(resource);
        textReader.setCharset(Charset.defaultCharset());
        List<Document> documents = textReader.get();

        vectorStore.add(documents);
    }
}
```

In the above example, we are reading the text from a classpath file and storing it in the VectorStore.

## Implementing RAG (Retrieval-Augmented Generation)
Now that we have seen how to convert documents into embeddings and store them in the Vector Database,
and how to retrieve relevant documents using natural language, let us see how to implement RAG.

```java
@RestController
class RAGController {
    private final ChatClient chatClient;
    private final VectorStore vectorStore;

    RAGController(ChatClient.Builder chatClientBuilder, VectorStore vectorStore) {
        this.chatClient = chatClientBuilder.build();
        this.vectorStore = vectorStore;
    }
    
    // Assume that we have already read the documents from files containing information about people 
    // and stored them in the VectorStore as described in the previous section.
    
    @GetMapping("/ai/rag/people")
    Person chatWithRag(@RequestParam String name) {
        // Querying the VectorStore using natural language looking for the information about a person.
        List<Document> similarDocuments = 
                vectorStore.similaritySearch(SearchRequest.query(name).withTopK(2));
        String information = similarDocuments.stream()
                .map(Document::getContent)
                .collect(Collectors.joining(System.lineSeparator()));
        
        // Constructing the systemMessage to indicate the AI model to use the passed information
        // to answer the question.
        var systemPromptTemplate = new SystemPromptTemplate("""
              You are a helpful assistant.
              
              Use the following information to answer the question:
              {information}
              """);
        var systemMessage = systemPromptTemplate.createMessage(
                Map.of("information", information));

        // Using BeanOutputConverter to parse the response into an instance of Person.
        var outputConverter = new BeanOutputConverter<>(Person.class);
        
        // Constructing the userMessage to ask the AI model to tell about the person.
        PromptTemplate userMessagePromptTemplate = new PromptTemplate("""
        Tell me about {name} as if current date is {current_date}.

        {format}
        """);
        Map<String,Object> model = Map.of("name", name,
                "current_date", LocalDate.now(),
                "format", outputConverter.getFormat());
        var userMessage = new UserMessage(userMessagePromptTemplate.create(model).getContents());

        var prompt = new Prompt(List.of(systemMessage, userMessage));

        var response = chatClient.prompt(prompt).call().content();

        return outputConverter.convert(response);
    }
}

record Person(String name,
              String dateOfBirth,
              int experienceInYears,
              List<String> books) {
}
```

The explanation of the above code is included in the comments.

Overall, the RAG process involves the following steps:
* Load the documents from different sources using **DocumentReaders**.
* Convert the documents into embeddings using **EmbeddingModel** and store them in the **VectorStore**.
* Query the VectorStore using natural language and retrieve relevant documents.
* Construct the **SystemMessage** to indicate the AI model to use the passed information to answer the question.
* Construct the **UserMessage** to ask the AI model for the information.
* Construct the prompt and call the AI model to get the response.
* Parse the response into the desired format using **OutputConverters**.
* Return the response.

## Conclusion
In this article, we have seen how to use Embedding APIs to convert text or documents into embeddings.
We have also seen how to use Vector Database to store and retrieve embeddings.
We have implemented RAG (Retrieval-Augmented Generation) to use the retrieved information to answer questions using AI models.

# Getting Started with Spring AI and OpenAI

In this article, we will explore the following:

* Introduction to Spring AI.
* Interacting with OpenAI using Spring AI.
* Using **PromptTemplates**.
* Using **OutputConverters**.

Let's get started.

<!--more-->

{{< box info >}}
**Sample Code Repository**

You can find the sample code for this article in the [GitHub repository](https://github.com/sivaprasadreddy/spring-ai-samples/tree/main/spring-ai-openai)

{{< /box >}}

## Introduction to OpenAI and Spring AI
ChatGPT took the world by storm when it was released by OpenAI.
It was the first time a language model was able to generate human-like responses to prompts. 
OpenAI has since released several other models, including DALL-E, which can generate images from textual prompts.

[Spring AI](https://spring.io/projects/spring-ai) is a Java library that provides a simple and easy-to-use interface to interact with LLM models.
Spring AI provides higher-level abstractions to interact with various LLMs such as 
**OpenAI**, **Azure OpenAI**, **Hugging Face**, **Google Vertex**, **Ollama**, **Amazon Bedrock**, etc.

In this article, we are going to explore how to interact with OpenAI using Spring AI.

First, we need to create an account in OpenAI and get the API key.
* Go to [OpenAI Platform](https://platform.openai.com/) and create an account.
* In the Dashboard, click on **API Keys** from the left navigation menu and create a new API key.

> **If you are creating a new account, you will be granted some free credits to use the OpenAI APIs.
Otherwise, you need to buy credits to use the OpenAI APIs.**

Once you have the API key, set the environment variable `OPENAI_API_KEY` with the API key.

```bash
export OPENAI_API_KEY=<your-api-key>
```
## Create a Spring AI Project
Let's create a new Spring Boot project using Spring Initializr.

* Go to [Spring Initializr](https://start.spring.io/)
* Select **Web**, and **OpenAI** starters

## Interacting with OpenAI using ChatClient
Spring AI provides **ChatClient** abstraction to interact with different types of LLMs 
without coupling with the actual LLM model.

For example, we can use **ChatClient** to interact with OpenAI as follows:

```java
@RestController
class ChatController {

    private final ChatClient chatClient;

    ChatController(ChatClient.Builder chatClientBuilder) {
        this.chatClient = chatClientBuilder.build();
    }

    @GetMapping("/ai/chat")
    Map<String, String> chat(@RequestParam String question) {
        String response = chatClient.prompt().user(question).call().content();
        return Map.of("question", question, "answer", response);
    }
}
```

In the above code, there is nothing coupled to OpenAI.

We can configure the **ChatClient** to use OpenAI by providing the API key and other parameters in the **application.properties** file.

```properties
spring.ai.openai.api-key=${OPENAI_API_KEY}
spring.ai.openai.chat.model=gpt-3.5-turbo
spring.ai.openai.chat.temperature=0.7
```

Now we can run the application and test the chat API.

```bash
curl --location 'http://localhost:8080/ai/chat?question=Tell%20me%20about%20SpringBoot'

//OUTPUT:
{
  "question":"Tell me about Spring Boot",
  "answer":"Spring Boot is an open-source Java-based framework used for building and 
            deploying stand-alone, production-ready applications. It is a part of the 
            larger Spring ecosystem and provides a simpler and faster way to set up and 
            configure Spring applications.\n\nSpring Boot eliminates the need for 
            manual configuration by providing default settings for most Spring projects, 
            allowing developers to quickly get started with their application development. 
            It also offers a wide range of features, such as embedded servers, metrics, 
            health checks, and security, that are pre-configured and ready to use out of the box."
}
```

## Using PromptTemplates
We can use **PromptTemplates** to provide a set of predefined prompts to the **ChatClient**.

```java
@RestController
class ChatController {

    private final JokeService jokeService;

    ChatController(JokeService jokeService) {
        this.jokeService = jokeService;
    }

    @GetMapping("/ai/chat-with-prompt")
    Map<String,String> chatWithPrompt(@RequestParam String subject) {
        String answer = jokeService.getJoke(subject);
        return Map.of("answer", answer);
    }
}

@Service
class JokeService {
    private final ChatClient chatClient;

    JokeService(ChatClient.Builder chatClientBuilder) {
        this.chatClient = chatClientBuilder.build();
    }

    String getJoke(String subject) {
        PromptTemplate promptTemplate = new PromptTemplate("Tell me a joke about {subject}");
        Prompt prompt = promptTemplate.create(Map.of("subject", subject));
        return chatClient.prompt(prompt).call().content();
    }
}
```

By using PromptTemplates, we can hide the complexity of creating prompts and provide a simple interface to the users.

In the above example, we created a Prompt that represents the user message.
We can use **SystemMessage** to indicate the role of the LLM in the conversation.

```java
@Service
class JokeService {
    private final ChatClient chatClient;

    JokeService(ChatClient.Builder chatClientBuilder) {
        this.chatClient = chatClientBuilder.build();
    }

    String getJoke(String subject) {
        SystemMessage systemMessage = new SystemMessage("You are a helpful and funny chat bot");
        UserMessage userMessage = new UserMessage("Tell me a joke about " + subject);
        Prompt prompt = new Prompt(List.of(systemMessage, userMessage));
        return  chatClient.prompt(prompt).call().content();
    }
}
```

In the above example, we created a **SystemMessage** and **UserMessage** to represent the conversation between the user and the LLM.
By using **SystemMessage**, we can define the role and provide additional context to the LLM.

## Using OutputConverters
In the previous examples, we get the response from LLMs as Strings.
We can use **OutputConverters** to parse the response and extract the required information in the desired format.

As of now, Spring AI provides the following type of **OutputConverters**:
* **BeanOutputConverter** - To parse the response and convert into a Java Bean.
* **MapOutputConverter** - To parse the response and convert into a Map.
* **ListOutputConverter** - To parse the response and convert into a List.

Let's create a new controller called **MovieController** to get the list of movies directed by a director.

```java
@RestController
class MovieController {
    private final ChatClient chatClient;

    MovieController(ChatClient.Builder chatClientBuilder) {
        this.chatClient = chatClientBuilder.build();
    }

    private static final String PROMPT_TEMPLATE = """
            What are the best movies directed by {director}?
                    
            {format}
            """;
    //...
}
```

Now, let's see how to use **BeanOutputConverter** to parse the response and convert it into a Java Bean.

```java
record DirectorResponse(String director, List<String> movies) {}

@RestController
class MovieController {
    //...

    @GetMapping("/ai/chat/movies")
    DirectorResponse chat(@RequestParam String director) {
        var outputConverter = new BeanOutputConverter<>(DirectorResponse.class);
        var userPromptTemplate = new PromptTemplate(PROMPT_TEMPLATE);
        Map<String, Object> model = Map.of("director", director, "format", outputConverter.getFormat());
        var prompt = userPromptTemplate.create(model);
        var response = chatClient.prompt(prompt).call().content();
        return outputConverter.convert(response);
    }
}
```

In the above example, we created a Java Bean called **DirectorResponse** to represent the response from the LLM.
The **BeanOutputConverter** will parse the response and convert it into **DirectorResponse** object.

Similarly, we can use **MapOutputConverter** and **ListOutputConverter** to parse the response and convert it into a Map and List respectively.

```java
@RestController
class MovieController {
    //...

    @GetMapping("/ai/chat/movies-as-map")
    Map<String, Object> chatWithMapOutput(@RequestParam String director) {
        var outputConverter = new MapOutputConverter();
        var userPromptTemplate = new PromptTemplate(PROMPT_TEMPLATE);
        Map<String, Object> model = Map.of("director", director, "format", outputConverter.getFormat());
        var prompt = userPromptTemplate.create(model);
        var response = chatClient.prompt(prompt).call().content();
        return outputConverter.convert(response);
    }

    @GetMapping("/ai/chat/movies-as-list")
    List<String> chatWithListOutput(@RequestParam String director) {
        var outputConverter = new ListOutputConverter(new DefaultConversionService());
        var userPromptTemplate = new PromptTemplate(PROMPT_TEMPLATE);
        Map<String, Object> model = Map.of("director", director, "format", outputConverter.getFormat());
        var prompt = userPromptTemplate.create(model);
        var response = chatClient.prompt(prompt).call().content();
        return outputConverter.convert(response);
    }
}
```

We can test the APIs as follows:

```bash
curl --location 'http://localhost:8080/ai/chat/movies?director=Quentin%20Tarantino'

//OUTPUT:
{"director":"Quentin Tarantino","movies":["Pulp Fiction","Inglourious Basterds","Django Unchained","Kill Bill: Volume 1","Kill Bill: Volume 2"]}

curl --location 'http://localhost:8080/ai/chat/movies-as-map?director=Quentin%20Tarantino'

//OUTPUT:
{"best_movies":[{"title":"Pulp Fiction","year":1994},{"title":"Inglourious Basterds","year":2009},{"title":"Kill Bill: Volume 1","year":2003},{"title":"Kill Bill: Volume 2","year":2004},{"title":"Django Unchained","year":2012}]}

curl --location 'http://localhost:8080/ai/chat/movies-as-list?director=Quentin%20Tarantino'

//OUTPUT:
["Pulp Fiction","Kill Bill: Volume 1","Inglourious Basterds","Django Unchained","Once Upon a Time in Hollywood"]
```

You need to use the appropriate **OutputConverter** based on the response from the LLM and in which format you want to convert to.

## Conclusion
In this article, we have seen how to interact with OpenAI using Spring AI.
In the next article, we will learn about Embedding Models, VectorStores and how to implement 
RAG (Retrieval-Augmented Generation) using Spring AI.

# LangChain4j Retrieval-Augmented Generation (RAG) Tutorial

In this article, we will explore the following:

* Understand the need for **Retrieval-Augmented Generation (RAG)**.
* Understand **EmbeddingModel**, **EmbeddingStore**, **DocumentLoaders**, **EmbeddingStoreIngestor**.
* Working with different EmbeddingModels and EmbeddingStores.
* Ingesting data into EmbeddingStore.
* Querying LLMs with data from EmbeddingStore.

Let's get started.

<!--more-->


{{< box info >}}
**Sample Code Repository**

You can find the sample code for this article in the [GitHub repository](https://github.com/sivaprasadreddy/langchain4j-demos)

{{< /box >}}

{{< box info >}}
**LangChain4j Tutorial Series**

You can check out the other articles in this series:

* [Part 1: Getting Started with Generative AI using Java, LangChain4j, OpenAI and Ollama]({{< relref "2024-02-19-getting-started-with-generative-ai-using-java-langchain4j.md" >}})
* [Part 2: Generative AI Conversations using LangChain4j ChatMemory]({{< relref "2024-02-21-generative-ai-conversations-using-langchain4j-chat-memory.md" >}})
* [Part 3: LangChain4j AiServices Tutorial]({{< relref "2024-02-23-langchain4j-ai-services.md" >}})
* [Part 4: LangChain4j Retrieval-Augmented Generation (RAG) Tutorial]({{< relref "2024-02-26-langchain4j-rag.md" >}})
{{< /box >}}

## Understand the need for Retrieval-Augmented Generation (RAG)
In the previous articles, we have seen how to ask questions and get responses from the AI models. 
But while building AI powered applications for your business, the AI models don't have the context of your business data.

Your business might store structured data in relational databases, unstructured data in NoSQL databases, or even in files.
You will be able to effectively query relational databases using SQL, NoSQL databases using their query languages.
You can also use full-text search engines like Elasticsearch, Solr, etc., to query unstructured data.

However, you might want to retrieve data using natural language with semantic meaning.

For example, "I love Java programming language" and "Java is always my go-to language" have the same semantic meaning, but uses different words.
Trying to retrieve data using the exact words might not be effective.

This is where **Embeddings** comes into play. Embeddings are vector representations of words, sentences, or documents.
You can use these embeddings to retrieve data using natural language.

You can convert your structured and unstructured data into embeddings and store them in an **EmbeddingStore**.
You can then query the **EmbeddingStore** using natural language and retrieve relevant data.
Then, you can query the AI models passing along the relevant data to get the responses.

> **Retrieval-Augmented Generation (RAG)** is the process of optimizing the output of a LLM, 
by using additional knowledge base in addition to its training data before generating a response.

## Understand EmbeddingModel, EmbeddingStore, EmbeddingStoreIngestor
First, let us understand the various components involved in RAG.

### EmbeddingModel
An **EmbeddingModel** is a model that can convert words, sentences, or documents into embeddings.
An embedding is a vector representation of words, sentences, or documents.

For example, A word "Apple" might be represented as a vector `[0.1, 0.2, 0.3, 0.4, 0.5]`.
A sentence "I love Apple" might be represented as a vector `[0.1, 10.3, -10.2, 90.3, 2.4, -0.5]`.

LangChain4j provides an interface `EmbeddingModel` and support various implementations:
* **AllMiniLmL6V2EmbeddingModel**: An embedding model that runs within your Java application's process
* **OpenAiEmbeddingModel**: Represents an [OpenAI embedding model](https://platform.openai.com/docs/guides/embeddings)
* **AzureOpenAiEmbeddingModel**: Represents an [Azure OpenAI embedding model](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/understand-embeddings)
* **OllamaEmbeddingModel**: Represents any of the [Ollama supported models](https://ollama.com/library) embedding model
* **VertexAiEmbeddingModel**: Represents a Google's [Vertex AI embedding model](https://cloud.google.com/vertex-ai/docs/generative-ai/embeddings/get-text-embeddings)
* **HuggingFaceEmbeddingModel**: Represents a [HuggingFace embedding models](https://huggingface.co/models?other=embeddings)

We can create an instance of the **EmbeddingModel** as follows:

```java 
EmbeddingModel embeddingModel = new AllMiniLmL6V2EmbeddingModel();
// or
EmbeddingModel embeddingModel = OpenAiEmbeddingModel.withApiKey("demo");
// or
EmbeddingModel embeddingModel = OllamaEmbeddingModel.builder()
        .baseUrl("http://localhost:11434")
        .modelName("llama2")
        .build();
```

### EmbeddingStore
An **EmbeddingStore** is a datastore that can store embeddings of words, sentences, or documents.
There are various Vector databases available in the market like **Neo4j**, **ElasticSearch**, 
**Pgvector**, **Redis**, **Chroma**, **Milvus**, etc., that can store embeddings.

LangChain4j provides an interface **EmbeddingStore** and supports various implementations:
* InMemoryEmbeddingStore
* Neo4jEmbeddingStore
* OpenSearchEmbeddingStore
* ChromaEmbeddingStore
* PineconeEmbeddingStore
* MilvusContainer
* and more...

You can create an instance of the **EmbeddingStore** as follows:

```java
EmbeddingStore embeddingStore = new InMemoryEmbeddingStore();
// or
EmbeddingStore embeddingStore = ChromaEmbeddingStore.builder()
        .baseUrl("http://localhost:8000")
        .collectionName("my-collection")
        .build();
// similarly, you can use any other implementation
```

### DocumentLoaders
A **DocumentLoader** is a component that can load data from various sources like files, URLs, etc., into a **Document**.

LangChain4j provides various **DocumentLoader** implementations:
* **FileSystemDocumentLoader**: Loads data from files into a **Document**
* **UrlDocumentLoader**: Loads data from URLs into a **Document**

```java
Path path = Paths.get("/path/to/some/file.txt");
Document document = FileSystemDocumentLoader.loadDocument(path, new TextDocumentParser());
```

You can load a Document from Web URLs as follows:

```java
URL url = new URL("https://www.sivalabs.in/about-me/");
Document htmlDocument = UrlDocumentLoader.load(url, new TextDocumentParser());
HtmlTextExtractor transformer = new HtmlTextExtractor(null, null, true);
Document document = transformer.transform(htmlDocument);
```

### EmbeddingStoreIngestor
An **EmbeddingStoreIngestor** is a component that can ingest data into the **EmbeddingStore**.

We can convert text data into embeddings using the **EmbeddingModel** and store it in 
an **EmbeddingStore** as follows:

```java
TextSegment segment = TextSegment.from("your text goes here");
Embedding embedding = embeddingModel.embed(segment).content();
embeddingStore.add(embedding, segment);
```

Our data could be in various file formats like TXT, CSV, PDF, Web Pages, etc.
We can load the data from files into Documents, split the documents into text segments, 
convert the text segments into embeddings, and store them in the EmbeddingStore.

{{< figure src="/images/rag-ingestion.webp" alt="RAG Ingestion" >}}
Source: https://docs.langchain4j.dev/tutorials/rag

Let's create a sample data file **siva.txt** in **src/main/resources** folder 
with the following content:

```txt
Siva, born on 25 June 1983, is a software architect working in India.
He started his career as a Java developer on 26 Oct 2006 and worked with other languages like Kotlin, Go, JavaScript, Python too.

He authored "Beginning Spring Boot 3" book with Apress Publishers.
He has also written "PrimeFaces Beginners Guide" and "Java Persistence with MyBatis 3" books with PacktPub.
```

Now we can load the data from the file and store the embeddings in the **EmbeddingStore** as follows:

```java
URL fileUrl = RAGDemo.class.getResource("/siva.txt");
Path path = Paths.get(fileUrl.toURI()); 

Document document = FileSystemDocumentLoader.loadDocument(path, new TextDocumentParser());

DocumentSplitter splitter = DocumentSplitters.recursive(600, 0);
List<TextSegment> segments = splitter.split(document);
List<Embedding> embeddings = embeddingModel.embedAll(segments).content();
embeddingStore.addAll(embeddings, segments);
```

You can also use the **EmbeddingStoreIngestor** to ingest data into the **EmbeddingStore** as follows:

```java
URL fileUrl = RAGDemo.class.getResource("/siva.txt");
Path path = Paths.get(fileUrl.toURI());

Document document = FileSystemDocumentLoader.loadDocument(path, new TextDocumentParser());

DocumentSplitter splitter = DocumentSplitters.recursive(600, 0);

EmbeddingStoreIngestor ingestor = EmbeddingStoreIngestor.builder()
                .documentSplitter(splitter)
                .embeddingModel(embeddingModel)
                .embeddingStore(embeddingStore)
                .build();
ingestor.ingest(document);
```

## Querying LLMs with data from EmbeddingStore
Once we have ingested data into the **EmbeddingStore**, we can query the **EmbeddingStore** 
using natural language and retrieve relevant data.
Then, we can query the AI models passing along the relevant data to get the responses.

{{< figure src="/images/rag-retrieval.webp" alt="RAG retrieval" >}}
Source: https://docs.langchain4j.dev/tutorials/rag

```java
Embedding queryEmbedding = embeddingModel.embed("Tell me about Siva?").content();
List<EmbeddingMatch<TextSegment>> relevant = embeddingStore.findRelevant(queryEmbedding, 1);
EmbeddingMatch<TextSegment> embeddingMatch = relevant.getFirst();
String information = embeddingMatch.embedded().text();

Prompt prompt = PromptTemplate.from("""
                Tell me about {{name}}?
                
                Use the following information to answer the question:
                {{information}}
                """).apply(Map.of("name", "Siva","information", information));
String answer = model.generate(prompt.toUserMessage()).content().text();
System.out.println("Answer:"+answer);

// Output:
// Answer:Siva is a software architect born on 25 June 1983 in India. 
// He began his career as a Java developer on 26 Oct 2006 and has since worked with other languages 
// such as Kotlin, Go, JavaScript, and Python. Siva has authored several books including 
// "Beginning Spring Boot 3" with Apress Publishers, "PrimeFaces Beginners Guide" and 
// "Java Persistence with MyBatis 3" with PacktPub.
```

{{< box warning >}}
**WARNING**

One thing to be mindful of is the length of the relevant data that will be passed to the AI models.
The AI models might have a limit on the number of tokens they can process at a time.
Also, the LLMs might charge you based on the number of tokens processed.
{{< /box >}}

While this works, we can simplify this using **AiServices** with **ContentRetriever** as follows:

```java
record Person(String name,
              LocalDate dateOfBirth,
              int experienceInYears,
              List<String> books) {
}

interface PersonDataExtractor {
    @UserMessage("Get information about {{it}} as of {{current_date}}")
    Person getInfoAbout(String name);
}

ContentRetriever contentRetriever = EmbeddingStoreContentRetriever.builder()
                .embeddingStore(embeddingStore)
                .embeddingModel(embeddingModel)
                .maxResults(1)
                .build();
PersonDataExtractor extractor =
        AiServices.builder(PersonDataExtractor.class)
                .chatLanguageModel(model)
                .contentRetriever(contentRetriever)
                .build();
Person person = extractor.getInfoAbout("Siva");
System.out.println(person);
// Output:
// Person[name=Siva, dateOfBirth=1983-06-25, experienceInYears=17, books=[Beginning Spring Boot 3, PrimeFaces Beginners Guide, Java Persistence with MyBatis 3]]
``` 
We have created an instance of **EmbeddingStoreContentRetriever**
and get an instance of **PersonDataExtractor** using **AiServices** by passing the **ContentRetriever**. 

Now, when we can call the **getInfoAbout()** method, it will get the relevant data from the **EmbeddingStore**
and pass it to the AI model to get the response.

## Conclusion
In this article, we have seen how we can augment the LLM capabilities with our own data using RAG.

To learn more advanced RAG capabilities provided by LangChain4j, see https://docs.langchain4j.dev/tutorials/rag.

# LangChain4j AiServices Tutorial

In this article, we will explore the following:

* Using LangChain4j **AiServices** to interact with LLMs.
* How to ask questions and map responses to different formats?
* Summarizing the given text in different formats.
* Analyzing the sentiment of the given text.

Let's get started.

<!--more-->


{{< box info >}}
**Sample Code Repository**

You can find the sample code for this article in the [GitHub repository](https://github.com/sivaprasadreddy/langchain4j-demos)

{{< /box >}}

In the [previous article]({{< relref "2024-02-21-generative-ai-conversations-using-langchain4j-chat-memory.md" >}}), 
we have seen how to have a conversation using LangChain4j **ChatMemory** and **ConversationalChain**.
As mentioned in the **ConversationalChain** JavaDocs, let us see how to use **AiServices** which is more flexible and feature rich.

{{< box info >}}
**LangChain4j Tutorial Series**

You can check out the other articles in this series:

* [Part 1: Getting Started with Generative AI using Java, LangChain4j, OpenAI and Ollama]({{< relref "2024-02-19-getting-started-with-generative-ai-using-java-langchain4j.md" >}})
* [Part 2: Generative AI Conversations using LangChain4j ChatMemory]({{< relref "2024-02-21-generative-ai-conversations-using-langchain4j-chat-memory.md" >}})
* [Part 3: LangChain4j AiServices Tutorial]({{< relref "2024-02-23-langchain4j-ai-services.md" >}})
* [Part 4: LangChain4j Retrieval-Augmented Generation (RAG) Tutorial]({{< relref "2024-02-26-langchain4j-rag.md" >}})
{{< /box >}}

So far we have seen asking questions using the **chatLanguageModel.generate(question)** method which returns a response as a **String**.
But, using **AiServices**, we can create a Java interface that takes input and returns output in a specific format.

## Using AiServices to interact with LLMs
Let's see how to use **AiServices** to interact with LLMs.

```java
import dev.langchain4j.model.chat.ChatLanguageModel;
import dev.langchain4j.model.openai.OpenAiChatModel;

public class AiServicesDemo {

    interface Assistant {
        String chat(String message);
    }
    
    public static void main(String[] args) {
        String openAiKey = "demo";
        //String openAiKey = System.getenv("OPENAI_API_KEY");
        ChatLanguageModel model = OpenAiChatModel.withApiKey(openAiKey);

        ChatMemory chatMemory = MessageWindowChatMemory.withMaxMessages(2);
        Assistant assistant = AiServices.builder(Assistant.class)
                                        .chatLanguageModel(model)
                                        .chatMemory(chatMemory)
                                        .build();
        String question = "What are all the movies directed by Quentin Tarantino?";
        String answer = assistant.chat(question);
        System.out.println("Answer: " + answer);
    }
}
```

We have created an interface called **Assistant** with a method **chat** which takes a message and returns a response.
We have used **AiServices.builder()** to create an instance of **Assistant** and used **ChatMemory** to remember the context of the conversation.
LangChain4j dynamically provides an implementation for the **Assistant** interface.

So far, this doesn't look much better from the previous approach. 
But the real power of **AiServices** comes when we want to map the request and response to a specific format.

Let's see another example.

```java
interface FunnyAssistant {
    @UserMessage("Tell me a joke about {{it}}")
    String tellMeAJokeAbout(String subject);
}

FunnyAssistant funnyAssistant = AiServices.builder(FunnyAssistant.class)
        .chatLanguageModel(model)
        .build();
String answer = funnyAssistant.tellMeAJokeAbout("Python");
System.out.println("Joke about [Python]: " + answer);
//Joke about [Python]: Why do programmers prefer Python over C++? Because they don't like snakes... or brackets!
```

In this example, we are passing only the subject as input, and the complete question is formed using the **@UserMessage** annotation.
When we look at the client code **funnyAssistant.tellMeAJokeAbout("Python")**, 
it just looks like a regular method call without any AI specific logic.

As there is only one parameter, and we didn't name it explicitly with **@V("name")**, it is named as "it" by default.

Let's see a few more examples.

```java
interface FunnyAssistant {
    @UserMessage("Tell me a joke about {{it}}")
    String tellMeAJokeAbout(String subject);

    //@SystemMessage("You are a friendly, polite chat assistant")
    @SystemMessage("You are a sarcastic and funny chat assistant")
    String chat(String message);

    @SystemMessage("You are an IT consultant who just replies \"It depends\" to every question")
    String ask(String question);
}

FunnyAssistant funnyAssistant = AiServices.builder(FunnyAssistant.class)
        .chatLanguageModel(model).build();

String answer = funnyAssistant.chat("Do you think you are smarter than humans?");
System.out.println("Answer: " + answer);
//Answer: Oh, definitely not. I mean, I may know a lot of random facts and have access to vast amounts of information, but I still can't tie my own shoelaces. So, I think humans have the upper hand on that one.

answer = funnyAssistant.ask("Do we need to use Microservices?");
System.out.println("Answer: " + answer);
//Answer: It depends

answer = funnyAssistant.ask("Is Node.js better than Python?");
System.out.println("Answer: " + answer);
//Answer: It depends
```

That was fun :-) Mocking the IT Consultants and making the Chat Assistant sarcastic.

But the point is, you can use **@SystemMessage** to tell the AI model about the role of the assistant and how to respond.
You can also use both **@UserMessage** and **@SystemMessage** on the same method.

## Summarizing the given text
A good use case for Generative AI is analyzing large text and summarizing it.
Let's see how we can summarize the given text using AiServices.

```java
interface Summarizer {
    @UserMessage("Give a summary of {{name}} in 3 bullet points using the following information:\n\n {{info}}")
    String summarize(@V("name") String name, @V("info") String info);
}

Summarizer summarizer = AiServices.builder(Summarizer.class).chatLanguageModel(model).build();

String aboutSiva = """
            Siva, born on 25 June 1983, is a software architect working in India.
            He started his career as a Java developer on 26 Oct 2006 and worked with other languages like
            Kotlin, Go, JavaScript, Python too.
                            
            He authored "Beginning Spring Boot 3" book with Apress Publishers.
            He has also written "PrimeFaces Beginners Guide" and "Java Persistence with MyBatis 3" books with PacktPub.
            """;
String answer = summarizer.summarize("Siva", aboutSiva);
System.out.println("[About Siva]: \n" + answer);

//[About Siva]: 
//- Siva is a software architect born on 25 June 1983 in India, who started his career as a Java developer in 2006 and has experience working with other languages like Kotlin, Go, JavaScript, and Python.
//- He authored the book "Beginning Spring Boot 3" with Apress Publishers, as well as "PrimeFaces Beginners Guide" and "Java Persistence with MyBatis 3" with PacktPub.
//- Siva has a strong background in software development and has contributed to multiple books in the field of programming and software development.
```

In this example, the given text is summarized, but to me, it doesn't look very impressive.

Let's see how to summarize the given text in a specific format.

```java
interface Summarizer {

    @UserMessage("""
            Give a person summary in the following format:
             Name: ...
             Date of Birth: ...
             Profession: ...
             Books Authored: ...
            
            Use the following information:
            {{info}}
            """)
    String summarizeInFormat(@V("info") String info);
}

answer = summarizer.summarizeInFormat(aboutSiva);
System.out.println("[About Siva]:\n" + answer);
```

This gave me the following response:

```shell
[About Siva]:
Name: Siva
Date of Birth: 25 June 1983
Profession: Software Architect
Books Authored:
    1. "Beginning Spring Boot 3" with Apress Publishers
    2. "PrimeFaces Beginners Guide" with PacktPub
    3. "Java Persistence with MyBatis 3" with PacktPub
```

This looks nice. We can ask the AI model to summarize the given text in a specific format.

Let's try to summarize the text in JSON format.

```java
interface Summarizer {

    @UserMessage("""
            Summarize the the following information in JSON format having name, date of birth,
            experience in years and books authored as keys:
            {{info}}
            """)
    String summarizeAsJSON(@V("info") String info);
}

answer = summarizer.summarizeAsJSON(aboutSiva);
System.out.println("[About Siva]:\n" + answer);
```

This gave me the following response:

```shell
[About Siva in JSON Format]: {
  "name": "Siva",
  "date_of_birth": "25 June 1983",
  "experience_in_years": 15,
  "books_authored": [
    "Beginning Spring Boot 3 (Apress Publishers)",
    "PrimeFaces Beginners Guide (PacktPub)",
    "Java Persistence with MyBatis 3 (PacktPub)"
  ]
}
```

This is great, but so far we are getting responses in String format.
How about getting the response as a Java object? Let's try that.

```java 
record Person(String name, 
              LocalDate dateOfBirth, 
              int experienceInYears, 
              List<String> booksAuthored) {
}

interface Summarizer {
    @UserMessage("""
        Give a person summary as of {{current_date}}
                
        Using the following information:
                
        {{info}}
        """)
    Person summarizeAsBean(@V("info") String info);
}

Person person = summarizer.summarizeAsBean(aboutSiva);
System.out.println("[About Siva]:\n" + person);
```

This approach successfully extracts the details and maps the values into Person record and gave me the following response:

```shell
[About Siva]:
Person[name=Siva, dateOfBirth=1983-06-25, experienceInYears=17, booksAuthored=[Beginning Spring Boot 3, PrimeFaces Beginners Guide, Java Persistence with MyBatis 3]]
```

## Mapping Java object to UserMessage using StructuredPrompt
We have seen different ways to map the response to different formats.
We can also map the input passed as a Java object to a user message using **@StructuredPrompt** annotation as follows:

```java
@StructuredPrompt("Give summary of {{name}} as of {{current_date}} using the following information:\n\n{{info}}")
record PersonSummaryPrompt(String name, String info) {
}

interface Summarizer {
    Person summarize(PersonSummaryPrompt prompt);
}

PersonSummaryPrompt prompt = new PersonSummaryPrompt("Siva", aboutSiva);
Person person = summarizer.summarize(prompt);
System.out.println("[About Siva]\n: " + person);
```

This gave me the following response:

```shell
[About Siva]:
Person[name=Siva, dateOfBirth=1983-06-25, experienceInYears=17, booksAuthored=[Beginning Spring Boot 3, PrimeFaces Beginners Guide, Java Persistence with MyBatis 3]]
```

## Sentiment Analysis
Another usecase of Generative AI is sentiment analysis.
Let's see how to analyze the sentiment of the given text using AiServices.

```java
enum Sentiment {
    POSITIVE, NEGATIVE, SARCASTIC
}

interface SentimentAnalyzer {
    Sentiment analyze(String text);
}

SentimentAnalyzer sentimentAnalyzer = AiServices.builder(SentimentAnalyzer.class)
        .chatLanguageModel(model).build();
Sentiment sentiment = analyzer.analyze("I love Java programming language");
System.out.println("Sentiment: " + sentiment);
//Sentiment: POSITIVE

sentiment = analyzer.analyze("I am sick of writing if err != nil for every 3rd line of code");
System.out.println("Sentiment: " + sentiment);
//Sentiment: NEGATIVE

sentiment = analyzer.analyze("You seem to like ten layers of abstractions in your code, don't you?");
System.out.println("Sentiment: " + sentiment);
//Sentiment: SARCASTIC
```

We have seen how to use **AiServices** to interact with LLMs and how to map the request and response to different formats.
But there is a lot more to AiServices such as **RAG**, **Tools**, **Shared or per-user ChatMemory** and more.

## Conclusion
In this article, we have taken a static piece of text as input and summarized it in different formats.
But in the real world, we may need to analyze the large text stored in text files, web pages, or databases.

We will explore how to use LangChain4j **Retrieval-Augmented Generation (RAG)** to use our own knowledge base 
to answer questions in the next article.


# Generative AI Conversations using LangChain4j ChatMemory

In this article, we will explore the following:

* How to use LangChain4j **ChatMemory** and **ConversationalChain** to implement conversation style interaction?
* How to ask questions using **PromptTemplate**?

In the [previous article]({{< relref "2024-02-19-getting-started-with-generative-ai-using-java-langchain4j.md" >}}), we have seen how to interact with OpenAI using Java and LangChain4j.

Let's get started.

<!--more-->


{{< box info >}}
**LangChain4j Tutorial Series**

You can check out the other articles in this series:

* [Part 1: Getting Started with Generative AI using Java, LangChain4j, OpenAI and Ollama]({{< relref "2024-02-19-getting-started-with-generative-ai-using-java-langchain4j.md" >}})
* [Part 2: Generative AI Conversations using LangChain4j ChatMemory]({{< relref "2024-02-21-generative-ai-conversations-using-langchain4j-chat-memory.md" >}})
* [Part 3: LangChain4j AiServices Tutorial]({{< relref "2024-02-23-langchain4j-ai-services.md" >}})
* [Part 4: LangChain4j Retrieval-Augmented Generation (RAG) Tutorial]({{< relref "2024-02-26-langchain4j-rag.md" >}})
{{< /box >}}

{{< box info >}}
**Sample Code Repository**

You can find the sample code for this article in the [GitHub repository](https://github.com/sivaprasadreddy/langchain4j-demos)

{{< /box >}}

I will assume that you have created a Java project with the following dependency to your **pom.xml**:

```xml
<dependency>
    <groupId>dev.langchain4j</groupId>
    <artifactId>langchain4j</artifactId>
    <version>0.27.1</version>
</dependency>
<dependency>
    <groupId>dev.langchain4j</groupId>
    <artifactId>langchain4j-open-ai</artifactId>
    <version>0.27.1</version>
</dependency>
<dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-classic</artifactId>
    <version>1.5.0</version>
</dependency>
```

## What is the need for ChatMemory?
In general, while we, human beings, are having a conversation, we remember the context of the conversation and use it to continue the conversation.

Consider the following example:

```shell
Person1: What are all the movies directed by Quentin Tarantino?
Person2: Pulp Fiction, Kill Bill, etc.
Person1: How old is he?
Person2: He is 60 years old.
```

In the above conversation, Person2 remembers the context of the conversation 
and understands "he" in the question "How old is he?" refers to Quentin Tarantino.

Let's try to implement the above conversation using LangChain4j OpenAI LLM.


```java
import dev.langchain4j.model.chat.ChatLanguageModel;
import dev.langchain4j.model.openai.OpenAiChatModel;

public class OpenAIChatMemoryDemo {
    
    public static void main(String[] args) {
      String openAiKey = "demo";
      //String openAiKey = System.getenv("OPENAI_API_KEY");
      ChatLanguageModel model = OpenAiChatModel.withApiKey(openAiKey);

      String answer = model.generate("What are all the movies directed by Quentin Tarantino?");
      System.out.println(answer); // Pulp Fiction, Kill Bill, etc.

      answer = model.generate("How old is he?");
      System.out.println(answer);
    }
}
```

When you run the above code, you will get the outputs similar to the following:

```shell
Answer 1: I'm sorry, I cannot answer that question without more context or information about the person you are referring to.

Answer 2: I'm sorry, without more context I am unable to determine who "he" is or his age. Can you please provide more information?
...
```

As you can see, the **ChatLanguageModel** does not remember the context of the conversation.

LangChain4j provides a way to remember the context of the conversation 
using **ConversationalChain** and **ChatMemory**.

## How to use ConversationalChain with ChatMemory?
From the JavaDocs of **ConversationalChain**:

> A chain for conversing with a specified ChatLanguageModel while maintaining a memory of 
> the conversation. Includes a default ChatMemory (a message window with maximum 10 messages), 
> which can be overridden. It is recommended to use AiServices instead, as it is more powerful.

Let's see how to use **ConversationalChain** with **ChatMemory**.

```java
import dev.langchain4j.model.chat.ChatLanguageModel;
import dev.langchain4j.model.openai.OpenAiChatModel;

public class OpenAIChatMemoryDemo {
    
    public static void main(String[] args) {
      String openAiKey = "demo";
      //String openAiKey = System.getenv("OPENAI_API_KEY");
      ChatLanguageModel model = OpenAiChatModel.withApiKey(openAiKey);

      ChatMemory chatMemory = MessageWindowChatMemory.withMaxMessages(20);
      //ChatMemory chatMemory = TokenWindowChatMemory.withMaxTokens(300, new OpenAiTokenizer(GPT_3_5_TURBO));

      ConversationalChain chain = ConversationalChain.builder()
                                    .chatLanguageModel(model)
                                    .chatMemory(chatMemory)
                                    .build();
      String answer = chain.execute("What are all the movies directed by Quentin Tarantino?");
      System.out.println(answer); // Pulp Fiction, Kill Bill, etc.

      answer = chain.execute("How old is he?");
      System.out.println(answer); // Quentin Tarantino was born on March 27, 1963, so he is currently 58 years old.
    }
}
```

We have created an instance of **ChatMemory** using **MessageWindowChatMemory** with a maximum of 20 messages.
Instead of using **MessageWindowChatMemory**, you can also use **TokenWindowChatMemory** with a specified maximum limit of tokens.

Then, we have created an instance of **ConversationalChain** using the chat model and chat memory.
Now, when you run the above code, you will get the outputs similar to the following:

```shell
Reservoir Dogs (1992), Pulp Fiction (1994), Jackie Brown (1997), ...

Quentin Tarantino was born on March 27, 1963, so he is currently 58 years old.
```

So, the **ConversationalChain** remembers the context of the conversation and provides the correct answers.

If you are a keen observer, you might have noticed that the second question "How old is he?" is not providing the exact age of Quentin Tarantino.
It is calculating the age based on the birthdate and date of model training, not based on the current date.

You can refine the question by providing more context to get the exact age.

```java
answer = chain.execute("How old is he as of "+ LocalDate.now() + "?");
System.out.println(answer); //As of February 21, 2024, Quentin Tarantino would be 60 years old.
```

Now you will get the correct age of Quentin Tarantino.
While you can do String concatenation to provide the context, LangChain4j provides a better way to ask questions using **PromptTemplate**.

## How to ask questions using PromptTemplate?
**PromptTemplate** is a way to ask questions with a predefined template, optionally with placeholders.

```java
ConversationalChain chain = ConversationalChain.builder()
        .chatLanguageModel(model)
        .chatMemory(chatMemory)
        .build();
String answer = chain.execute("What are all the movies directed by Quentin Tarantino?");
System.out.println(answer); // Pulp Fiction, Kill Bill, etc.

Prompt prompt = PromptTemplate.from("How old is he as of {{current_date}}?").apply(Map.of());
answer = chain.execute(prompt.text());
System.out.println(answer); //As of February 21, 2024, Quentin Tarantino would be 60 years old.
```

We have created an instance of **Prompt** using **PromptTemplate** with a placeholder **{{current_date}}**.
However, we didn't pass the value for **current_date**, because special variables **{{current_date}}**, 
**{{current_time}}**, and **{{current_date_time}}** are automatically filled with **LocalDate.now()**, 
**LocalTime.now()**, and **LocalDateTime.now()** respectively.

If there are other placeholders, you can pass the values using **Map**.

```java
Prompt prompt = PromptTemplate
                    .from("How old is {{name}} as of {{current_date}}?")
                    .apply(Map.of("name","Quentin Tarantino"));
```

PromptTemplates provides many other features, and we will explore them in future articles.

## Manually adding messages to ChatMemory
You can also manually add messages to **ChatMemory** without using **ConversationalChain**. 

```java
ChatLanguageModel model = OpenAiChatModel.withApiKey(openAiKey);
ChatMemory chatMemory = MessageWindowChatMemory.withMaxMessages(20);

chatMemory.add(UserMessage.userMessage("What are all the movies directed by Quentin Tarantino?"));
AiMessage answer = model.generate(chatMemory.messages()).content();
System.out.println(answer.text()); // Pulp Fiction, Kill Bill, etc.
chatMemory.add(answer);

chatMemory.add(UserMessage.userMessage("How old is he?"));
AiMessage answer2 = model.generate(chatMemory.messages()).content();
System.out.println(answer2.text()); // Quentin Tarantino was born on March 27, 1963, so he is currently 58 years old.
chatMemory.add(answer2);
```

I can't think of a use case where you need to manually add messages to **ChatMemory** instead of 
using **ConversationalChain**, but there is such an option.

## Conclusion
In this article, we have seen how to use **ConversationalChain** with **ChatMemory** to remember 
the context of the conversation.

From the JavaDocs of **ConversationalChain**, it is recommended to use **AiServices** instead, as it is more powerful.
In the next article, we will explore **AiServices** and its features.

# Getting Started with Generative AI using Java, LangChain4j, OpenAI and Ollama

In this article, we will explore the following:

* Brief introduction to Generative AI?
* How to interact with OpenAI APIs using Java?
* How to use LangChain4j to interact with OpenAI?
* How to run a LLM model locally using Ollama?
* Working with Ollama using LangChain4j and Testcontainers.

Let's get started.

<!--more-->


{{< box info >}}
**LangChain4j Tutorial Series**

You can check out the other articles in this series:

* [Part 1: Getting Started with Generative AI using Java, LangChain4j, OpenAI and Ollama]({{< relref "2024-02-19-getting-started-with-generative-ai-using-java-langchain4j.md" >}})
* [Part 2: Generative AI Conversations using LangChain4j ChatMemory]({{< relref "2024-02-21-generative-ai-conversations-using-langchain4j-chat-memory.md" >}})
* [Part 3: LangChain4j AiServices Tutorial]({{< relref "2024-02-23-langchain4j-ai-services.md" >}})
* [Part 4: LangChain4j Retrieval-Augmented Generation (RAG) Tutorial]({{< relref "2024-02-26-langchain4j-rag.md" >}})
{{< /box >}}

## Introduction to Generative AI
Unless you are living under a rock, you might have heard about Generative AI. 
It is a fascinating field of AI that is used to generate new content, such as images, text, audio, etc. 
A few years ago, building a chat bot was a big deal, but now we are talking about generating human-like 
text, images, and audio using a simple API call.

In general, AI involves complex subjects like **Machine Learning**, **Deep Learning**, **Natural Language Processing (NLP)**, etc.
Building AI applications requires a good understanding of these subjects, and it is not easy to get started.
Thanks to the latest advancements in AI, we can build AI-powered applications without having a deep understanding of these complex subjects.
The experts in the AI field built Large Language Models (LLMs) that can generate human-like text, and these models are available as APIs.

A **Large Language Model (LLM)** refers to a type of artificial intelligence model designed to understand and 
generate human-like language on a large scale. These models are trained on massive datasets that include 
diverse examples of language use, allowing them to learn patterns, context, and semantics of language.

Generative AI has diverse use cases, including:

* **Content Generation:** Creating text, images, videos, and music.
* **Language Translation:** Improving machine translation models.
* **Chatbots:** Developing conversational agents for customer support.
* **Code Generation:** Writing code for software development.
* **Text Analysis:** Analyzing and summarizing large volumes of text.

There are a few popular LLMs available as APIs, such as **GPT-3/4 from OpenAI**, **Vertex from Google**, 
**Azure AI from Microsoft**, etc.

## Getting Started with OpenAI using Java
In this section, we will explore how to interact with OpenAI APIs using Java.

{{< box info >}}
**Sample Code Repository**

You can find the sample code for this article in the [GitHub repository](https://github.com/sivaprasadreddy/langchain4j-demos)

{{< /box >}}

First, we need to create an account in OpenAI and get the API key.
* Go to [OpenAI Platform](https://platform.openai.com/) and create an account.
* In the Dashboard, click on the **API Keys** from the left navigation menu and create a new API key.

> **If you are creating a new account, you will be granted some free credits to use the OpenAI APIs.
Otherwise, you need to buy credits to use the OpenAI APIs.**

Once you have the API key, you can use it to interact with OpenAI APIs.
You can find the OpenAI API Reference documentation [here](https://platform.openai.com/docs/api-reference/).

First, let us see how we can ask a question to OpenAI using its [Chat API endpoint](https://platform.openai.com/docs/api-reference/chat).

```shell
curl https://api.openai.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {
        "role": "user",
        "content": "List all the movies directed by Quentin Tarantino"
      }
    ]
  }'
```

You might get a response like this:

```json
{
  "id": "chatcmpl-8tvrQyjw3s28IREvkFl6kIhTxIUGZ",
  "object": "chat.completion",
  "created": 1708341148,
  "model": "gpt-3.5-turbo-0125",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1. Reservoir Dogs (1992)\n2. Pulp Fiction (1994)\n3. Jackie Brown (1997)\n4. Kill Bill: Vol. 1 (2003)\n5. Kill Bill: Vol. 2 (2004)\n6. Death Proof (2007)\n7. Inglourious Basterds (2009)\n8. Django Unchained (2012)\n9. The Hateful Eight (2015)\n10. Once Upon a Time in Hollywood (2019)"
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 16,
    "completion_tokens": 105,
    "total_tokens": 121
  },
  "system_fingerprint": "fp_69829325d0"
}
```

As you can see, it's a simple API call to get the response from OpenAI.

Now, let's see how we can use Java's built-in **HttpClient** to interact with OpenAI.

We are going to use the Jackson JSON library to serialize/deserialize the request and response.

```xml
<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.16.1</version>
</dependency>
```

First, we are going to create Java Record classes to represent the request and response JSON payloads.

```java
record Message(String role, String content) {}

record ChatRequest( String model, List<Message> messages, double temperature) {}

@JsonIgnoreProperties(ignoreUnknown = true)
record ChatUsage(
        @JsonProperty("prompt_tokens")
        int promptTokens,
        @JsonProperty("completion_tokens")
        int completionTokens,
        @JsonProperty("total_tokens")
        int totalTokens
) {}

@JsonIgnoreProperties(ignoreUnknown = true)
record ChatResponseChoice(
        int index,
        Message message,
        @JsonProperty("finish_reason")
        String finishReason
) {}

@JsonIgnoreProperties(ignoreUnknown = true)
record ChatResponse(
        String id, String object, long created,
        String model, ChatUsage usage,
        List<ChatResponseChoice> choices
) {}
```

We have created Java records to represent the request and response, including only the fields we are interested in.

Next, let's use Java's **HttpClient** to interact with OpenAI.

```java
package com.sivalabs.openai.demo;

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.databind.ObjectMapper;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.util.List;

public class ChatDemo {
    public static final String OPENAI_API_KEY = System.getenv("OPENAI_API_KEY");
    public static final String CHAT_URL = "https://api.openai.com/v1/chat/completions";
    public final static String MODEL = "gpt-3.5-turbo";
    public final static double TEMPERATURE = 0.7;

    static HttpClient client = HttpClient.newHttpClient();
    static ObjectMapper mapper = new ObjectMapper();

    public static void main(String[] args) throws Exception {
        ChatRequest chatRequest = new ChatRequest(
                MODEL, List.of(new Message("user", "List all the movies directed by Quentin Tarantino")),
                TEMPERATURE);
        String requestPayload = mapper.writeValueAsString(chatRequest);
        
        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create(CHAT_URL))
                .header("Authorization", "Bearer %s".formatted(OPENAI_API_KEY))
                .header("Content-Type", "application/json")
                .header("Accept", "application/json")
                .POST(HttpRequest.BodyPublishers.ofString(requestPayload))
                .build();
        HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
        
        String body = response.body();
        ChatResponse chatResponse = mapper.readValue(body, ChatResponse.class);
        System.out.println(chatResponse.choices().getFirst().message().content());
    }
}
```

We have created an instance of **ChatRequest** and serialized it to JSON string using Jackson.
Then we created an instance of **HttpRequest** and sent it using Java's **HttpClient**.
Finally, we deserialized the response JSON to **ChatResponse** using Jackson.

Few important things to note here:
* We are getting the OpenAI API key from the environment variable **OPENAI_API_KEY**.
* We are using **gp-3.5-turbo** model, you can see all available models in OpenAI [here](https://platform.openai.com/docs/models/overview).
* We are using a **temperature** of 0.7, which controls the randomness of the response. 
  A lower temperature will result in more deterministic responses, and a higher temperature will result in more random responses.

An important thing to note here is the **usage** field in the response, which gives you an idea of how many tokens are used for the request and response.

```json
{
  "..":"..",
  "usage": {
    "prompt_tokens": 16,
    "completion_tokens": 105,
    "total_tokens": 121
  }
}
```

Your OpenAI API usage is metered based on the number of tokens used.
You can learn more about token at [What are tokens and how to count them?](https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them).

We have seen how to interact with OpenAI using Java's **HttpClient**.
However, this is low-level and not very convenient to use.
In the next section, we will see how to use **LangChain4j** to interact with OpenAI.

## Using LangChain4j to interact with OpenAI
[LangChain4j](https://langchain4j.github.io/langchain4j/) is a Java port of the popular Python [LangChain](https://www.langchain.com/) library for interacting with LLMs.
LangChain4j provides a high-level API to interact with [LLMs](https://langchain4j.github.io/langchain4j/docs/category/language-models), 
including **OpenAI**, **Vertex AI**, **Ollama**, **HuggingFace**, etc.

Create a Java project and add the following dependency to your **pom.xml**:

```xml
<dependency>
    <groupId>dev.langchain4j</groupId>
    <artifactId>langchain4j</artifactId>
    <version>0.27.1</version>
</dependency>
<dependency>
    <groupId>dev.langchain4j</groupId>
    <artifactId>langchain4j-open-ai</artifactId>
    <version>0.27.1</version>
</dependency>
<dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-classic</artifactId>
    <version>1.5.0</version>
</dependency>
```

We have added the **langchain4j** core library and **langchain4j-open-ai** which provides integration with OpenAI.

Now, let's see how to use LangChain4j to interact with OpenAI.

```java
import dev.langchain4j.model.chat.ChatLanguageModel;
import dev.langchain4j.model.openai.OpenAiChatModel;

public class LangChain4jOpenAIDemo {
    
    public static void main(String[] args) {
        String openAiKey = System.getenv("OPENAI_API_KEY");
        ChatLanguageModel model = OpenAiChatModel.withApiKey(openAiKey);
        String answer = model.generate("List all the movies directed by Quentin Tarantino");
        System.out.println(answer);
    }
}
```

As you can see, using LangChain4j provides high-level abstraction and gives a simple API to interact with OpenAI.

{{< box tip >}}
**TIP**

You can also use the special LangChain4j OpenAI API key "demo" to test the OpenAI APIs without using your own API key. 

{{< /box >}}

We have just seen the basic usage of LangChain4j to interact with OpenAI.
In the future articles, we will explore more advanced features of LangChain4j such as using PromptTemplates, AiServices, RAG, etc.

OpenAI is a paid service, and you will be charged based on the number of tokens used.
While learning and experimenting, you might want to run the LLM model locally without incurring cost.
Luckily, there are a few open-source projects that provide LLM models that you can run locally.
In the next section, we will see how to run a LLM model locally using Ollama.

## Running a LLM model locally using Ollama
[Ollama](https://ollama.com/) is an open-source project that provides LLM models that you can run locally.
Download and install Ollama from [here](https://ollama.com/download).
Once installed, you can start Ollama and run any available [Models](https://ollama.com/library).
We are going to run [llama2](https://ollama.com/library/llama2) model.

```shell
ollama run llama2
```

When you run it for the first time, it will download the model and start the server.
Once the server is started, you can check the status by visiting [http://localhost:11434](http://localhost:11434)
which should display **Ollama is running**.

Now, let's see how to use LangChain4j to interact with Ollama.

Let's add **langchain4j-ollama** dependency to our `pom.xml`.

```xml
<dependency>
    <groupId>dev.langchain4j</groupId>
    <artifactId>langchain4j-ollama</artifactId>
    <version>0.27.1</version>
</dependency>
```

Now, let's use LangChain4j Ollama APIs to interact with Ollama LLM.

```java
import dev.langchain4j.model.chat.ChatLanguageModel;
import dev.langchain4j.model.ollama.OllamaChatModel;

public class OllamaChatDemo {

    public static void main(String[] args) {
        ChatLanguageModel model = OllamaChatModel.builder()
                .baseUrl("http://localhost:11434")
                .modelName("llama2")
                .build();
        String answer = model.generate("List all the movies directed by Quentin Tarantino");
        System.out.println(answer);
    }
}
```

You can see that using LangChain4j, we can interact with Ollama LLM in the same way as we did with OpenAI.

Instead of installing Ollama manually, you can use [Testcontainers](https://www.testcontainers.org/) to run Ollama in a Docker container.
In the next section, we will see how to use LangChain4j with Testcontainers to run Ollama in a Docker container.

## Working with Ollama using LangChain4j and Testcontainers
[Testcontainers](https://www.testcontainers.org/) is a Java library that can provide lightweight, throwaway instances of any Docker container.
LangChain4j provides various [Ollama Docker images](https://hub.docker.com/repositories/langchain4j) with different LLM models.

First, let's add the Testcontainers dependency to our `pom.xml`.

```xml
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>testcontainers</artifactId>
    <version>1.19.4</version>
    <scope>test</scope>
</dependency>
```

Let's see how to use LangChain4j with Testcontainers to run Ollama in a Docker container.

```java
public class LangChain4jTestcontainersDemo {
    static String MODEL_NAME = "llama2";
  
    public static void main(String[] args) {
      GenericContainer<?> ollama = new GenericContainer<>("langchain4j/ollama-" + MODEL_NAME + ":latest")
              .withExposedPorts(11434);
      ollama.start();
      
      String baseUrl = String.format("http://%s:%d", ollama.getHost(), ollama.getFirstMappedPort());
      ChatLanguageModel model = OllamaChatModel.builder()
              .baseUrl(baseUrl)
              .modelName(MODEL_NAME)
              .build();
      String answer = model.generate("List all the movies directed by Quentin Tarantino");
      System.out.println(answer);

      ollama.stop();
    }
}
```

Testcontainers will start the Ollama Docker container and expose the container port 11434 on a random available port on the host.
We can get the host and port using `ollama.getHost()` and `ollama.getFirstMappedPort()` respectively.

If you are building a Spring Boot, Quarkus, or Micronaut application, 
you can run Ollama in a Docker container for integration tests and running the application locally
using their out-of-the-box integration with Testcontainers.

## Conclusion
We have explored the basics of Generative AI and how to interact with OpenAI using Java and LangChain4j.
In the future articles, we will explore more advanced features of LangChain4j, such as using PromptTemplates, AiServices, RAG, etc.
# Should I use a framework or libraries?


In the software development world, trends come and go, and often we go through the same cycle again and again.
It seems 2024 is the year of **"Framework vs Libraries"** debate. I mean this debate is not new, but it is getting louder again.

<!--more-->

For example, most of the Go community prefers to use libraries instead of a framework.
The Java community is divided into two groups, one prefers to use Spring Boot or Quarkus or Micronaut, and the other prefers to use libraries.

I would like to share my thoughts on this topic.

## What is a Framework vs Libraries debate?
For building an application, you can use a framework which provides core foundation, 
and you can implement your application following the framework's guidelines.
The framework provides integration points with libraries to build your application.

Or you can use only the necessary libraries to build your application, and you are responsible to integrate all the libraries properly.

There are pros and cons to each approach.

### Using libraries
Some developers prefer to use the necessary libraries and integrate them to build an application.

**Pros:**
* You use only the required libraries for your application.
* You have better control over the application design.
* This approach might result in a smaller application size and memory footprint.

**Cons:**
* You are responsible for integrating all the libraries and making them work together properly.
* New team members need to learn how the libraries are integrated in each application.
* Badly integrated libraries can result in performance issues, memory leaks, security vulnerabilities, etc.

### Using a framework
Some developers prefer to use a framework that provides a core foundation and integration support with most commonly used libraries.

**Pros:**
* As the framework provides a core foundation, you can focus on building your application.
* The framework authors already spent time integrating the libraries, well tested and documented.
* Hiring/Onboarding new team members is relatively easy as they can leverage their existing knowledge of the framework.

**Cons:**
* The framework might have a lot of features that you don't need for your application, but will result in consuming more CPU/Memory resources.
* You need to follow the framework's way of doing things, which sometimes imposes restrictions on your application design.
* Customizing the framework to meet your application needs might require extensive knowledge on internals of the framework.

## My Preference
As always, the correct and not very useful answer to any **"this vs that"** question is **"It depends"**.

Personally, I prefer to use a framework for building applications unless the performance is the most important factor.

In my experience, most of the teams consist of developers with varying levels of experience and skills.
In such cases, using a framework helps to onboard new team members quickly and reduces the learning curve.
Unless the team is highly skilled in the libraries they are using, they are going to create a home-grown less efficient framework anyway.

Also, not every application has to cater to millions of users under sub-second performance requirements.
Most of the applications are built to solve a business problem, and the performance requirements are not that high.

Using a proven battle-tested framework helps to build applications quickly and focus on solving the business problem.
There will be enough resources to learn and get help from the community.

There is a flip side to this approach too. Some frameworks have a steep learning curve, and it might take some time to properly learn and use it.
I have seen many people horribly misusing the framework and creating a mess.

I prefer to invest some time in learning the framework and use it properly.
So, unless there is a strong reason to use libraries and glue things together myself, I prefer to use a framework.

What is your preference and why? Please share your thoughts in the comments section below.

# It's time to apply KISS principle to career


You might have heard from IT employees that there is no work-life balance in IT due to unrealistic deadlines, long working hours, etc.
I did work crazy hours in the first 7 or 8 years of my career, but not anymore.
Fortunately, I have been working in the companies that value work-life balance, and I never had to work beyond 8 hours in the recent past.

<!--more-->


However, I have been working on writing blogs, making videos, etc. in my free time.
Though I enjoy doing it, it is taking a toll on my health.
I am not going to blame anyone for this because I am the one willingly doing it out of passion for sharing my knowledge.

I have been working remotely from home for the past 4 years, and unknowingly, my screen time increased a lot.
As I have been enjoying doing this additional work (blogging, YouTube, etc), I didn't realize that I am spending too much time on the screen.
More importantly, I am not doing any physical activity, and I am not following good food habits.

Recently, me, my wife and kid all got sick at the same time, and it was a very stressful time for me and my family.
My wife and kid recovered within a week, but I took more time to recover. 
I got pneumonia and the lungs were severely infected. The normal CRP level is 0–6, and mine was 123.
At one point of time, I was not able to breathe properly, and my heart beat rate was 120+ (60-100 was normal range), and I was admitted to the hospital.
After being hospitalized for 10 days, I got recovered, but I am still not feeling 100% normal.
The doctor said it is due to stress, lack of physical activity and low immunity.

While lying on the hospital bed, I was thinking of what I have been doing wrong and what I should do to avoid this in the future.
In the process of retrospection, the following things came to my mind:

* **Lot of content creation:** In 2023, I have created quite a bit of content.
  * I have created [Spring Boot - The Missing Guide](https://www.youtube.com/watch?v=3tTS8rwV7zQ&list=PLuNxlOYbv61jZL1IiciTgWezZoqEp4WXh) YouTube series, which is 8 part series. I have explained how things work under the hood in Spring Boot.
    Someone commented that **the level of depth is not even covered in many paid courses and even the experienced Spring Boot developers will learn something from this series**.
  * I have created [Java Testing Made Easy: Learn to write Unit, Integration, E2E & Performance Tests](https://www.youtube.com/watch?v=FGoLvCc6BeI&list=PLuNxlOYbv61jtHHFHBOc9N7Dg5jn013ix) YouTube series, which is 12 part series.
    I have explained how to write different types of tests in Java.
  * I have also done a couple of LIVE Streams on YouTube.
  * I have written a 10-part [Spring Security OAuth 2 Tutorial](https://www.sivalabs.in/spring-security-oauth2-tutorial-introduction/) series on my blog.
  * I have written a 5-part [Spring Boot + jOOQ Tutorial](https://www.sivalabs.in/spring-boot-jooq-tutorial-getting-started/) series on my blog.
  
  IMO, it is a lot of content creation which involves a lot of research, preparation, recording, editing, etc.

* **Tried to understand negativity towards Java**:
  I have been using Java for the past 17+ years, and really like Java as a language and the ecosystem.
  But quite often I see people talking very negatively about Java, and say there are much better languages than Java.
  So, I thought of actually learning some of these languages and frameworks to understand what is so special about them.
  I have learned decent enough Go, Python (Django and FastAPI). Well, they have some nice features that Java doesn't have until recently.
  But when I look at the overall ecosystem, I still feel Java, and it's ecosystem is much better than these languages.
  I can take a 5 year old Java application and run it without any issues, but I can't say the same about Python or NodeJS.
  I am not saying Java is the best language, but it is a damn good language with a powerful ecosystem.

  I think many of these loudmouths who talk negatively about Java are either inexperienced or have some agenda(clicks and revenue).
  Any real experienced developer will acknowledge that every language has its pros and cons, and there is no one language that fits all.
  Especially, when it comes to large enterprises, Java is still the most popular language for a good reason.

* **Tried to understand is FP really better than OOP**:
  Another thing I have been seeing is people talking about Functional Programming and how it is better than OOP.
  I haven't really used FP in any of my projects, so I thought of learning it.
  I see the merits of some of the FP concepts such as pure functions, immutability, lazy evaluation, etc, but I don't think it is better than OOP.
  Specifically, I find mixing pure + side-effect functions using all those monads, monoids, functors, etc very confusing.
  I felt that pure FP code needs lot more unpacking in the mind than OOP or procedural code.
  I am not experienced enough in FP, so I can't say it's better than OOP or not. I just realized Pure FP is not my cup of tea.

* **Too much time on Twitter**:
  I really enjoyed interacting with people on Twitter, but I think I am spending too much time on Twitter.
  Especially in recent times, somehow people are becoming more judgmental and aggressive.
  After being on Twitter for several years, I learned not to engage with some known assholes, but occasionally I end up engaging with new people who turn out to be trolls or sad people venting their anger on random people on internet.
  In any case, I think I should reduce my time on Twitter.

After considering all this, I came to the conclusion that I should be doing LESS so that I can focus MORE on my personal life and health.
So, here are a few things I am planning to change in 2024:

* **Learn new things on need-basis**: It's common to have FOMO as things are rapidly evolving in the IT world.
  But, there is no way you can learn everything, so I will learn new things only when I need them.
  If you are really good at one programming language, you can learn another language quickly.
  So, I am going to continue with Java and Kotlin. Within Java itself there are many things I need to learn such as Virtual Threads, CRaC, etc.

* **Put aside the fancy stuff**: As I said earlier, I feel Pure FP is not my cup of tea (at least for now).
  Also, there is crazy hype around AI tools such as ChatGPT, Bard, LLMs, etc.
  While they are interesting, I am not sure how they are going to be used in large enterprises in the real world.
  I will wait and see how these things evolve and then decide whether to learn them or not.

* **Daily Routine changes**: I am going to change my daily routine to include some physical activity and better diet.
  * Don't check the phone till 9 AM.
  * Consistently do some physical activity such as walking, cycling, etc.
  * Less junk food, sweets, etc.


I know being aware of the problem is just the first step, and I need to take action to change my habits.
I really need to keep it simple by ignoring/avoiding all the unnecessary things and focus on what is important to me.

I got my first wakeup call in 2016, but I didn't change my habits.
Now, I got the second wakeup call, and I really want to change my habits.
I don't even want to think about the third wakeup call.

I need to often remind myself of the following quote:

> **"Take care of your body. It's the only place you have to live." – Jim Rohn**

I wish you all a very happy and healthy new year 2024.

# Go for Java/Spring Boot Developers

I have been using Java for more than 17 years, and I really like Java and its ecosystem.
Within the Java ecosystem, Spring Boot is my go-to framework for building applications.

I used the Go language for an official project a couple of years ago, and initially I had mixed feelings about it.
But the more I use it, the more I like it.

<!--more-->


I wanted to expand my skill set and decided to get more hands-on experience with Go.
Whenever I try to learn a new language or a framework, I try to map the concepts of the new framework/language
to the one I am already familiar with. This helps me to understand the new framework/language ecosystem faster.
But this sometimes also becomes a bottleneck to adapt to the new framework/language idioms quickly.

The best way to learn anything new is to build something with it.
So, in this article I will show how to build a REST API using Go covering various aspects such as
configuration management, logging, database access, etc.

I am not going to teach Go language basics of how to declare variables, loops, functions, etc.

Following are very good resources to learn the Go language basics.

* [GOLANGBOT Golang tutorial series](https://golangbot.com/learn-golang-series/)
* [CALLICODER Go Tutorials](https://www.callicoder.com/categories/golang/)
* [Learn Go with Tests](https://quii.gitbook.io/learn-go-with-tests/)
* [Go by Example](https://gobyexample.com/)

## What are we going to build?
Go doesn't have a Spring Boot-like framework. Usually, Go developers prefer to use the Standard Library and
add only the necessary libraries to build applications. We are going to follow the same approach in this tutorial.

We are going to build a REST API in Go using the following libraries:

* [Gin Web Framework](https://gin-gonic.com/) - Web framework
* [Viper](https://github.com/spf13/viper) - Configuration library
* [zap](https://github.com/uber-go/zap) - Logging library
* [pgx](https://github.com/jackc/pgx) - PostgreSQL driver and toolkit for Go
* [golang-migrate](https://github.com/golang-migrate/migrate) - Database Migrations

Let's get started.

## Install Go and Tools
You can download and install Go from https://go.dev/doc/install.
Once installed, add the Go bin directory to your **PATH** environment variable.

```shell
export GOPATH=$HOME/go
export PATH="$PATH:$GOPATH/bin"
```

You can use **VS Code**, **IntelliJ IDEA Ultimate with Go Plugin**, **GoLand**, or any other IDE of your choice for Go development.

## Project Setup
We are going to build a REST API for a simple bookmarking application exposing the CRUD endpoints.

Let's create a new project directory and initialize a Go module.

```shell
$ mkdir bookmarks
$ cd bookmarks
$ go mod init github.com/sivaprasadreddy/bookmarks
```

Here **github.com/sivaprasadreddy/bookmarks** is the module name/path.
This can be any valid name like just **bookmarks**, but the common practice is to use the project's source code repository name as module name.

Go doesn't have a central repository like **Maven Central** or **NPM Registry**.
Go modules are downloaded from the source code repository directly.
So, it's a good practice to use the source code repository name as the module name.

When you run the **go mod init** command, it will create a **go.mod** file with the following content:

```go
module github.com/sivaprasadreddy/bookmarks

go 1.21
```

Now, in the project root directory, create a file called **main.go** with the following content:

```go
package main

import "fmt"

func main() {
	fmt.Println("Hello World!")
}
```

In Go, the entry point of the application is the **main()** function in the **main** package.

Now, let's run the application using the following command:

```shell
$ go run main.go
Hello World!
```

You can also build the application to generate your OS-specific binary and use the binary to run the application as follows:

```shell
$ go build
$ ./bookmarks
Hello World!
```

You can also use **go build -o binary-name** to specify a different binary name.

## Run application as a server
Go standard library provides **net/http** module which you can use to build a HTTP server.
Update the **main.go** file as follows:

```go
package main

import (
	"fmt"
	"log"
	"net/http"
)

func main() {
	mux := &http.ServeMux{}
	mux.HandleFunc("/hello", hello)
	log.Fatal(http.ListenAndServe(":8080", mux))
}

func hello(w http.ResponseWriter, r *http.Request) {
	_, err := fmt.Fprintln(w, "Hello World!")
	if err != nil {
		log.Println("Error processing the request")
	}
}
```

Here we are using the **http.ServeMux** to register the request handlers and start the server on port 8080.

Though it's just a simple example, there are many things to note here:
* In Go, the visibility of a function or field of a struct is based on the first letter of the identifier. 
  If the first letter is in uppercase, it is exported and visible outside the package. 
  If the first letter is in lowercase, it is private and not visible outside the package.
  So, the **hello** function is not exported and is not visible outside the **main** package.
* Go functions can return multiple values. 
  In the above example, the **fmt.Fprintln()** function returns two values, the number of bytes written and an error.
  We are ignoring the number of bytes written and checking only the error.
* A common convention is returning the error as the last value from the function.
* Go doesn't have exceptions. So, you need to handle the errors explicitly. 

Now, let's run the application using **go run main.go** and access the URL [http://localhost:8080/hello](http://localhost:8080/hello) in the browser.
You should see the response **Hello World!**.

## Live Reload
Let's change the response text from **Hello World!** to **Hello Go!**.
To pick up the code changes, we need to restart the application. 
During the development, it would be annoying to restart the application for every code change.

There are few ways to achieve live reload in Go applications.
* [Air](https://github.com/cosmtrek/air) - Live reload for Go apps
* [Go development with hot reload using Taskfile](https://dev.to/vitorvargas/go-development-with-hot-reload-using-taskfile-5cdj)

I prefer to use Air. You can install Air using the following command:

```shell
$ go install github.com/cosmtrek/air@latest
$ air -v
```

We can create a default air configuration file using **air init** which will create a file called **.air.toml** 
in the project root directory with sensible defaults. 
Then you can simply run the command **air** to start the application.

```shell
$ air init
$ air
```

Now, go ahead and change the response text from **Hello World!** to **Hello Go!** and save the file.
Refresh your browser, and you should see the updated response.

## Using Gin Web Framework
While Go standard library package **net/http** is good enough to build simple HTTP servers, its features are limited.
So, we are going to use the [Gin Web Framework](https://github.com/gin-gonic/gin) which provides a lot of useful features such as 
**routing**, **JSON Validation**, **Error management**, etc.

There are few other lightweight alternatives also like [Echo](https://echo.labstack.com/), [Fiber](https://gofiber.io/), [Chi](https://github.com/go-chi/chi), etc.
But I prefer to use **Gin** as it is the most popular one and feature rich.

Let's add the **Gin** dependency to our project using the following command:

```shell
$ go get -u github.com/gin-gonic/gin
```

After running this command, the gin module will be downloaded and added to the **go.mod** file.

Update the **main.go** file as follows:

```go
package main

import (
	"github.com/gin-gonic/gin"
	"log"
	"net/http"
)

func main() {
	r := gin.Default()
	r.GET("/hello", hello)
	log.Fatal(r.Run(":8080"))
}

func hello(c *gin.Context) {
	c.JSON(http.StatusOK, gin.H{
		"message": "Hello World",
	})
}
```

Here we are using the **gin.Default()** to create a Gin router and attached a handler function to handle **GET /hello** requests.
In the handler, we are returning a JSON response using the **c.JSON()** method.
The **gin.H** is a shortcut for **map[string]interface{}**.

Before starting the application, run the **go mod tidy** command.
This command adds any missing module dependencies used in the code but not declared in the **go.mod** file.
If there are any unused dependencies, **go mod tidy** will remove those from **go.mod** accordingly.

If you take a look at the **go.mod** file, you can see the dependencies added in two sections.
The first **require** section includes the **direct dependencies** that are used by our application code.
The second **require** section includes the **indirect dependencies** used by the packages.

It will also create or update **go.sum** file which contains checksums for the exact contents of each dependency at the time it is added to your module.
You can think of it as **package-lock.json** file in Node.js.

```shell
$ go mod tidy
$ air
```

## Application Configuration Management using Viper
Any non-trivial application has some configuration such as database connection details, API keys, etc.
In Spring Boot, this is a solved problem. You can configure your properties in **application.properties** or **application.yml** file
and bind them to your objects by annotating them with **@ConfigurationProperties**.

In Go, there are many [3rd party libraries](https://awesome-go.com/configuration/) available for configuration management.
Some of the popular ones are [godotenv](https://github.com/joho/godotenv), [Viper](https://github.com/spf13/viper), [envconfig](https://github.com/kelseyhightower/envconfig), etc.
Among them, I like **Viper** as it is very flexible and feature-rich.

Let's add the **Viper** dependency to our project using the following command:

```shell
$ go get -u github.com/spf13/viper
```

I would like to have a default configuration file and be able to override the properties via environment variables.
Viper supports this out of the box and can also work with different file formats like json, yaml, etc.

I prefer to use **JSON** for configuration files. 
So, let's create a file called **config.json** in the project root directory with the following content:

```json
{
  "environment": "dev",
  "server_port": 8080,
  "logging": {
    "filename": "bookmarks.log",
    "level": "debug"
  },
  "db": {
    "host": "localhost",
    "port": 15432,
    "username": "postgres",
    "password": "postgres",
    "database": "postgres"
  }
}
```

Now, let's create a file called **config.go** in the **internal/config** directory with the following content:

```go
package config

import (
	"github.com/spf13/viper"
	"log"
	"strings"
)

type AppConfig struct {
	Environment string   `mapstructure:"environment"`
	ServerPort  int      `mapstructure:"server_port"`
	Logging     Logging  `mapstructure:"logging"`
	Db          DbConfig `mapstructure:"db"`
}

type Logging struct {
	FileName string `mapstructure:"filename"`
	Level    string `mapstructure:"level"`
}

type DbConfig struct {
	Host     string `mapstructure:"host"`
	Port     int    `mapstructure:"port"`
	UserName string `mapstructure:"username"`
	Password string `mapstructure:"password"`
	Database string `mapstructure:"database"`
}

func GetConfig(configFilePath string) (AppConfig, error) {
	log.Printf("Config File Path: %s\n", configFilePath)
	conf := viper.New()
	conf.SetConfigFile(configFilePath)
	replacer := strings.NewReplacer(".", "_")
	conf.SetEnvKeyReplacer(replacer)
	conf.AutomaticEnv()

	err := conf.ReadInConfig()
	if err != nil {
		log.Printf("error reading config file: %v\n", err)
	}
	var cfg AppConfig

	err = conf.Unmarshal(&cfg)
	if err != nil {
		log.Printf("configuration unmarshalling failed!. Error: %v\n", err)
		return cfg, err
	}
	return cfg, nil
}
```

* Go doesn't have classes. Instead, it has **structs** which are used to define data structures.
* We have created a struct called **AppConfig** which represents the application configuration.
* We are using the **mapstructure** tags to map the json property paths to the **AppConfig** struct fields.
* We have configured viper in such a way that we can replace **db.host** property value with **DB_HOST** environment variable.
* The **AutomaticEnv()** method will automatically read in environment variables.
* We are using the **conf.Unmarshal()** method to unmarshal(bind) the configuration values into the **AppConfig** struct.
* Finally, the method name **GetConfig()** is exported and is visible outside the **config** package.

{{< box info >}}
**Go internal packages**

One important thing to remember is, in Go some package names have special meaning.
If you name a package as **internal**, it means that the package is only visible to the other packages inside the same module.
See [Internal packages](https://golang.org/doc/go1.4#internalpackages) for more details.
{{< /box >}}

Now, let's update the **main.go** file as follows:

```go
package main

import (
	"fmt"
	"github.com/gin-gonic/gin"
	"github.com/sivaprasadreddy/bookmarks/internal/config"
	"log"
	"net/http"
)

func main() {
	cfg, err := config.GetConfig("config.json")
	if err != nil {
		log.Fatal(err)
	}
	r := gin.Default()
	r.GET("/hello", hello)
	log.Fatal(r.Run(fmt.Sprintf(":%d", cfg.ServerPort)))
}

func hello(c *gin.Context) {
	c.JSON(http.StatusOK, gin.H{
		"message": "Hello World",
	})
}
```

Instead of hard coding the port number, we are using the value from **AppConfig** struct.

Now if you change the port number in **config.json** file, you might expect air to automatically restart the application.
But you need to add the **json** extension to the **include_ext** array in **.air.toml** file as follows:

```toml
  include_ext = ["go", "tpl", "tmpl", "html", "json"]
```

Now you need to restart the application manually for air to pick up the new configuration from **.air.toml** file.

Okay, we have the configuration management in place. Let's move on to the next topic.

## Logging using zap
Again, in Spring Boot, this is a solved problem.
Spring Boot by default automatically configures logging using **Slf4j** and **Logback**.
If you want to switch to a different logging implementation like log4j2, then it is simply a matter of excluding the default logging implementation and adding the new one.
And, you can configure the logging using **application.properties** or **application.yml** file.

Go also has a standard library package called **log** which can be used for logging.
However, it is very basic and doesn't have many features. 
There are many 3rd party logging libraries available for Go like [zap](https://github.com/uber-go/zap), [zerolog](https://github.com/rs/zerolog), etc.
Inspiring from these libraries, Go 1.21 introduced a new package called **slog** to support structured logging.

But **Zap** is a very popular logging library which is widely used and provides a lot of features. 
So, we are going to use it in our application.

We are going to configure logging to log to a file and also to the console.
Also, we are going to use [lumberjack](https://github.com/natefinch/lumberjack) library for log rotation.

Let's add the **zap** and **lumberjack** dependencies to our project using the following commands:

```shell
$ go get -u go.uber.org/zap
$ go get -u gopkg.in/natefinch/lumberjack.v2
```

Now, let's create a file called **logger.go** in the **internal/logger** directory with the following content:

```go
package config

import (
	"os"

	"go.uber.org/zap"
	"go.uber.org/zap/zapcore"
	"gopkg.in/natefinch/lumberjack.v2"
)

type Logger struct {
	*zap.SugaredLogger
}

func NewLogger(cfg AppConfig) *Logger {
	logFile := cfg.Logging.FileName
	logLevel, err := zap.ParseAtomicLevel(cfg.Logging.Level)
	if err != nil {
		logLevel = zap.NewAtomicLevelAt(zap.InfoLevel)
	}
	hook := lumberjack.Logger{
		Filename:   logFile,
		MaxSize:    1024,
		MaxBackups: 30,
		MaxAge:     7,
		Compress:   true,
	}

	encoder := getEncoder()
	core := zapcore.NewCore(
		encoder,
		zapcore.NewMultiWriteSyncer(zapcore.AddSync(os.Stdout), zapcore.AddSync(&hook)),
		logLevel)
	options := []zap.Option{
		zap.AddCaller(),
		zap.AddStacktrace(zap.ErrorLevel),
	}
	if cfg.Environment != "prod" {
		options = append(options, zap.Development())
	}
	sugaredLogger := zap.New(core, options...).With(zap.String("env", cfg.Environment)).Sugar()
	return &Logger{sugaredLogger}
}

func getEncoder() zapcore.Encoder {
	return zapcore.NewJSONEncoder(zapcore.EncoderConfig{
		TimeKey:        "ts",
		LevelKey:       "level",
		NameKey:        "logger",
		CallerKey:      "caller",
		FunctionKey:    zapcore.OmitKey,
		MessageKey:     "msg",
		StacktraceKey:  "stacktrace",
		LineEnding:     zapcore.DefaultLineEnding,
		EncodeLevel:    zapcore.LowercaseLevelEncoder,
		EncodeTime:     zapcore.ISO8601TimeEncoder,
		EncodeDuration: zapcore.SecondsDurationEncoder,
		EncodeCaller:   zapcore.ShortCallerEncoder,
	})
}
```

Though it looks like a lot of code, it's pretty much configuring the encoder what details to include in the log.
Also, we used log filename and loglevel from the **AppConfig** struct.

Now update the **main.go** to use the **logger** as follows:

```go
package main

import (
	"fmt"
	"github.com/gin-gonic/gin"
	"github.com/sivaprasadreddy/bookmarks/internal/config"
	"log"
	"net/http"
)

func main() {
	cfg, err := config.GetConfig("config.json")
	if err != nil {
		log.Fatal(err)
	}
	logger := config.NewLogger(cfg)
	logger.Infof("Application is running on %d", cfg.ServerPort)
	r := gin.Default()
	r.GET("/hello", hello)
	log.Fatal(r.Run(fmt.Sprintf(":%d", cfg.ServerPort)))
}

func hello(c *gin.Context) {
	c.JSON(http.StatusOK, gin.H{
		"message": "Hello World",
	})
}
```

Now, if you run the application, you should see the following log message in the console and also in the **bookmarks.log** file.

```shell
{"level":"info","ts":"2023-11-18T12:11:51.091+0530","caller":"bookmarks/main.go:17","msg":"Application is running on 8080","env":"dev"}
```

Next, let's setup database integration.

## Database Integration using pgx
Go standard library provides **database/sql** package to access relational databases.
We are going to use PostgreSQL as our database, and we are going to use [pgx](https://github.com/jackc/pgx) driver.

You can use the following **docker-compose.yml** file to start the PostgreSQL database:

```yaml
version: '3.8'
services:

  bookmarks-db:
    image: postgres:16-alpine
    container_name: bookmarks-db
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=postgres
    ports:
      - "15432:5432"
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready -U postgres" ]
      interval: 10s
      timeout: 5s
      retries: 5
```

Start the database container using **docker compose up -d** command, 
connect to the database and create **bookmarks** table and sample data using the following script:

```sql
create table bookmarks
(
    id         bigserial primary key,
    title      varchar   not null,
    url        varchar   not null,
    created_at timestamp
);

INSERT INTO bookmarks (title, url, created_at) 
VALUES ('SivaLabs Blog', 'https://sivalabs.in', CURRENT_TIMESTAMP);
```

Let's add the **pgx** dependency to our project using the following command:

```shell
$ go get -u github.com/jackc/pgx/v5
```

First, let's create a file called **db.go** in the **internal/config** directory with the following content:

```go
package config

import (
	"context"
	"fmt"
	"github.com/jackc/pgx/v5"
	"log"
)

func GetDb(config AppConfig) *pgx.Conn {
	connStr := fmt.Sprintf("host=%s port=%d user=%s password=%s dbname=%s sslmode=disable",
		config.Db.Host, config.Db.Port, config.Db.UserName, config.Db.Password, config.Db.Database)
	conn, err := pgx.Connect(context.Background(), connStr)
	if err != nil {
		log.Fatal(err)
	}
	return conn
}
```

Nothing groundbreaking here. We are passing the **AppConfig** struct and creating a connection string using the database configuration.
Then we are using the **pgx.Connect()** method to create a connection to the database.
If it fails to connect to the database, we are logging the error and exiting the application.

Next, in **main.go** file let's create a struct to represent a **Bookmark** as follows:

```go
type Bookmark struct {
	ID        int
	Title     string
	Url       string
	CreatedAt time.Time
}
```

Now, in the **main.go** file, let's implement a function to fetch all the bookmarks from the database as follows:

```go
func getAll(ctx context.Context, db *pgx.Conn) ([]Bookmark, error) {
	query := `select id, title, url, created_at FROM bookmarks`
	rows, err := db.Query(ctx, query)
	if err != nil {
		return nil, err
	}
	defer rows.Close()

	var bookmarks []Bookmark
	for rows.Next() {
		var bookmark = Bookmark{}
		err = rows.Scan(&bookmark.ID, &bookmark.Title, &bookmark.Url, &bookmark.CreatedAt)
		if err != nil {
			return nil, err
		}
		bookmarks = append(bookmarks, bookmark)
	}
	if err := rows.Err(); err != nil {
		return nil, err
	}
	return bookmarks, nil
}
```

Those of us who got habituated to using **Spring Data JPA** and simply calling **bookmarkRepository.findAll()** method might find this code a bit verbose.
It took me a while to get used to this style of coding in Go.

* We are using the **pgx.Conn** object to execute the query and get the result set.
* We are using the **rows.Next()** method to iterate over the result set.
* We are using the **rows.Scan()** method to map the result set to the **Bookmark** struct.
* We are using the **rows.Err()** method to check for any errors while iterating over the result set.
* We are using the **defer** keyword to close the result set at the end of the function execution.
* We are returning the **[]Bookmark** slice and an error from the function.
* A bunch of error checks and handling them by returning **nil** value for **[]Bookmark** and the error value.

Verbose, but understandable.

I think the following picture would fit here.

{{< figure src="/images/java-is-verbose.webp" >}}

Sorry, can't resist adding this meme. 😄

Now, let's update the **main.go** file to add a handler to GET **/api/bookmarks** endpoint as follows:

```go
package main

import (
	"context"
	"fmt"
	"github.com/gin-gonic/gin"
	"github.com/jackc/pgx/v5"
	"github.com/sivaprasadreddy/bookmarks/internal/config"
	"log"
	"net/http"
	"time"
)

func main() {
	cfg, err := config.GetConfig("config.json")
	if err != nil {
		log.Fatal(err)
	}
	logger := config.NewLogger(cfg)
	db := config.GetDb(cfg)

	r := gin.Default()
	r.GET("/api/bookmarks", getAllBookmarks(db, logger))
	log.Fatal(r.Run(fmt.Sprintf(":%d", cfg.ServerPort)))
}

func getAllBookmarks(db *pgx.Conn, logger *config.Logger) gin.HandlerFunc {
	return func(c *gin.Context) {
		ctx := c.Request.Context()
		bookmarks, err := getAll(ctx, db)
		if err != nil {
			logger.Errorf("Error fetching bookmarks from db: %v", err)
			c.JSON(http.StatusInternalServerError, gin.H{
				"error": "Failing to fetch bookmarks",
			})
		}
		c.JSON(http.StatusOK, bookmarks)
	}
}

// Bookmark struct
// func getAll(ctx context.Context, db *pgx.Conn) ([]Bookmark, error) 
```

Here, the key part to understand is the **getAllBookmarks** function.
Usually we create gin handler function with signature as **func(c \*gin.Context)** and 
attach it as handler using **r.GET("/api/bookmarks", getAllBookmarks)**.

However, we need to pass the **db** and **logger** objects to the handler function.
So, we are creating a function called **getAllBookmarks** taking **db** and **logger** objects as parameters which returns a function with the signature **func(c *gin.Context)**.
Then we are attaching the handler using **r.GET("/api/bookmarks", getAllBookmarks(db, logger))**.

Now, let's run the application and access the URL [http://localhost:8080/api/bookmarks](http://localhost:8080/api/bookmarks), 
and you should be able to the response with one bookmark.

It's working, but we are stuffing everything in **main.go** file.
No separation of concerns, and passing **db**, **logger** as inputs to all the functions doesn't look good.

Let's refactor the code to make it a little bit better.

## Refactoring the code
Before refactoring the code, let's understand a couple of things.

In Go, there is no concept of classes. Instead, it has **structs** which are used to define data structures.
We can define methods on structs as follows:

```go
type BookmarkRepository {
	db *pgx.Conn
    logger *config.Logger
}

func (b BookmarkRepository) GetAll(ctx context.Context) ([]Bookmarks, error) {
	b.logger.Infof("Fetching all bookmarks")
	b.db.Query(...)
}

var bookmarkRepo = BookmarkRepository{db: db, logger: logger}
bookmarks, err := bookmarkRepo.GetAll(ctx)
```

Here, we are defining a struct called **BookmarkRepository** with two fields **db** and **logger**.
Then we are defining a method called **GetAll** on the **BookmarkRepository** struct.
The **(b BookmarkRepository)** before the method name is called **receiver** through which you can access the fields of the struct.

Next, we may not want to directly expose our **BookmarkRepository** struct to the outside world.
So, we can create an interface and define the methods on the interface as follows:

```go
type BookmarkRepository interface {
	GetAll(ctx context.Context) ([]Bookmark, error)
}
```

Then you can create an unexported struct (with lowercase first letter) which implements the interface.
In Go, you don't explicitly declare this struct as implementing the interface.
If the struct has all the methods defined in the interface, then it is automatically considered as implementing the interface.

```go
type bookmarkRepo struct {
	db     *gorm.DB
	logger *config.Logger
}

func NewBookmarkRepository(db *gorm.DB, logger *config.Logger) BookmarkRepository {
	return bookmarkRepo{
		db:     db,
		logger: logger,
	}
}

func (r bookmarkRepo) GetAll(ctx context.Context) ([]Bookmark, error) {
	r.db.Query(...)
}

// --------- usage ------------
var db = ...
var logger = ...
var bookmarkRepo = NewBookmarkRepository(db, logger)
bookmarks, err := bookmarkRepo.GetAll(ctx)
```

Now, let's refactor the code to use this approach.

Create a file called **repository.go** in the **internal/domain** directory with the following content:

```go
package domain

import (
	"context"
	"github.com/jackc/pgx/v5"
	"github.com/sivaprasadreddy/bookmarks/internal/config"
	"time"
)

type Bookmark struct {
	ID        int
	Title     string
	Url       string
	CreatedAt time.Time
}

type BookmarkRepository interface {
	GetAll(ctx context.Context) ([]Bookmark, error)
	GetByID(ctx context.Context, id int) (*Bookmark, error)
	Create(ctx context.Context, b Bookmark) (*Bookmark, error)
	Update(ctx context.Context, b Bookmark) error
	Delete(ctx context.Context, id int) error
}

type bookmarkRepo struct {
	db     *pgx.Conn
	logger *config.Logger
}

func NewBookmarkRepository(db *pgx.Conn, logger *config.Logger) BookmarkRepository {
	return bookmarkRepo {
		db:     db,
		logger: logger,
	}
}

func (r bookmarkRepo) GetAll(ctx context.Context) ([]Bookmark, error) {
	query := `select id, title, url, created_at FROM bookmarks`
	rows, err := r.db.Query(ctx, query)
	if err != nil {
		return nil, err
	}
	defer rows.Close()

	var bookmarks []Bookmark
	for rows.Next() {
		var bookmark = Bookmark{}
		err = rows.Scan(&bookmark.ID, &bookmark.Title, &bookmark.Url, &bookmark.CreatedAt)
		if err != nil {
			return nil, err
		}
		bookmarks = append(bookmarks, bookmark)
	}
	if err := rows.Err(); err != nil {
		return nil, err
	}
	return bookmarks, nil
}

func (r bookmarkRepo) GetByID(ctx context.Context, id int) (*Bookmark, error) {
	panic("implement me")
}

func (r bookmarkRepo) Create(ctx context.Context, b Bookmark) (*Bookmark, error) {
	panic("implement me")
}

func (r bookmarkRepo) Update(ctx context.Context, b Bookmark) error {
	panic("implement me")
}

func (r bookmarkRepo) Delete(ctx context.Context, id int) error {
	panic("implement me")
}
```

You might be wondering why we are passing **context.Context** as input to all the methods.
In Go, you can use **context.Context** to pass request-scoped values, cancellation signals, and deadlines across API boundaries.
See [Context](https://golang.org/pkg/context/) for more details.

Now, let's refactor API handlers also.

Create a file called **handler.go** in the **internal/api** directory with the following content:

```go
package api

import (
  "github.com/gin-gonic/gin"
  "github.com/sivaprasadreddy/bookmarks/internal/config"
  "github.com/sivaprasadreddy/bookmarks/internal/domain"
  "net/http"
)

type BookmarkController struct {
  repo   domain.BookmarkRepository
  logger *config.Logger
}

func NewBookmarkController(repo domain.BookmarkRepository, logger *config.Logger) BookmarkController {
  return BookmarkController{
    repo:   repo,
    logger: logger,
  }
}

func (p BookmarkController) GetAll(c *gin.Context) {
  p.logger.Info("Finding all bookmarks")
  ctx := c.Request.Context()
  bookmarks, err := p.repo.GetAll(ctx)
  if err != nil {
    if err != nil {
      p.logger.Errorf("Error :%v", err)
    }
    c.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{
      "error": "Unable to fetch bookmarks",
    })
    return
  }
  c.JSON(http.StatusOK, bookmarks)
}
```

Finally, let's update the **main.go** file to use these changes as follows:

```go
package main

import (
	"fmt"
	"github.com/gin-gonic/gin"
	"github.com/sivaprasadreddy/bookmarks/internal/api"
	"github.com/sivaprasadreddy/bookmarks/internal/config"
	"github.com/sivaprasadreddy/bookmarks/internal/domain"
	"log"
)

func main() {
	cfg, err := config.GetConfig("config.json")
	if err != nil {
		log.Fatal(err)
	}
	logger := config.NewLogger(cfg)
	db := config.GetDb(cfg)
	repo := domain.NewBookmarkRepository(db, logger)
	handler := api.NewBookmarkController(repo, logger)

	logger.Infof("Application is running on %d", cfg.ServerPort)
	r := gin.Default()
	r.GET("/api/bookmarks", handler.GetAll)
	log.Fatal(r.Run(fmt.Sprintf(":%d", cfg.ServerPort)))
}
```

Now, this looks much better. However, coming from a Spring Boot background, 
you might be wondering where is my **Dependency Injection** and other cool **AOP** stuff?

In Go, there is no built-in support for Dependency Injection.
There are some 3rd party libraries available for Dependency Injection like [wire](https://github.com/google/wire).
But Go community prefers to keep things simple and create the structs and glue them together manually like we did above.

We are almost done with the refactoring, but I would like to go one step further.
I would like to keep the logic in **main.go** as minimal as possible and delegate the application initialization and starting the server to a separate package.

Let's create a file called **app.go** in the **cmd** directory with the following content:

```go
package cmd

import (
	"fmt"
	"github.com/gin-gonic/gin"
	"github.com/sivaprasadreddy/bookmarks/internal/api"
	"github.com/sivaprasadreddy/bookmarks/internal/config"
	"github.com/sivaprasadreddy/bookmarks/internal/domain"
	"log"
)

type App struct {
	Router *gin.Engine
	Cfg    config.AppConfig
}

func NewApp(cfg config.AppConfig) *App {
	logger := config.NewLogger(cfg)
	db := config.GetDb(cfg)

	repo := domain.NewBookmarkRepository(db, logger)
	handler := api.NewBookmarkController(repo, logger)

	router := gin.Default()

	router.GET("/api/bookmarks", handler.GetAll)

	return &App{
		Cfg:    cfg,
		Router: router,
	}
}

func (app App) Run() {
	log.Fatal(app.Router.Run(fmt.Sprintf(":%d", app.Cfg.ServerPort)))
}
```

* We created a struct called **App** which holds the key components of our application, i.e, **Gin Router** and **AppConfig**.
* We created a function called **NewApp()** which takes **AppConfig** as input, initialize the application and returns the **App** struct.
* We created a method called **Run** which starts the application.

Now, let's update the **main.go** file to use this as follows:

```go
package main

import (
	"github.com/sivaprasadreddy/bookmarks/cmd"
	"github.com/sivaprasadreddy/bookmarks/internal/config"
	"log"
)

func main() {
	cfg, err := config.GetConfig("config.json")
	if err != nil {
		log.Fatal(err)
	}
	app := cmd.NewApp(cfg)
	app.Run()
}
```

Now, we are talking. This looks much better.

We have come a long way and have the complete code structure in place.
Now, let's move on to the next topic of using database migrations to setup the database instead of manually making database changes.

## Database Migrations using golang-migrate
In Spring Boot, we can use [Flyway](https://flywaydb.org/) or [Liquibase](https://www.liquibase.org/) to manage database migrations.
All you have to do is place the migration scripts in the expected location, and the framework will take care of the rest.

In Go there are few libraries available for database migrations.
Among them, [golang-migrate](https://github.com/golang-migrate/migrate) is a popular one and has support for many databases.
Let's use it in our application.

Let's add the **golang-migrate** dependency to our project using the following command:

```shell
$ go get -u github.com/golang-migrate/migrate/v4
```

While using **golang-migrate**, we will create **up** and **down** migrations to support undoing the changes.

Let's create **db/migrations** directories in the project root directory.
Then create a file called **000001_init_schema.up.sql** in the **db/migrations** directory with the following content:

```sql
create table bookmarks
(
    id         bigserial primary key,
    title      varchar   not null,
    url        varchar   not null,
    created_at timestamp
);
```

Then create a file called **000001_init_schema.down.sql** in the **db/migrations** directory with the following content:

```sql
drop table bookmarks;
```

You can create more migration scripts to insert sample data, etc.

Before implementing the logic to apply db migrations, first we need to learn a little bit about including non-Go files in Go binaries.

### Embedding non-go files in the binary
In Java, when you build the jar/war file, by default, all the static resources that you put in **src/main/resources** will be bundled into the jar/war file.
But in Go, by default only compiled go code will be part of the binary.
Before Go 1.16, you need to use some 3rd party libraries to package non-go files into the binary.
Go 1.16 introduced a new feature called [Embedding](https://golang.org/pkg/embed/) which makes it easy to include non-go files in the binary.

We are going to use this feature to include the migration scripts in the binary.
Having everything related to the application in the binary makes it easy to deploy and run the application.

Create a file called **migrations.go** in the **db** directory with the following content:

```go
package db

import "embed"

//go:embed migrations/*.sql
var MigrationsFS embed.FS
```

Here, we are using the **//go:embed** directive to embed the SQL migration scripts in **MigrationsFS**.

Now, let's update the **internal/config/db.go** file to run the migrations as follows:

```go
package config

import (
  "context"
  "fmt"
  "github.com/golang-migrate/migrate/v4"
  _ "github.com/golang-migrate/migrate/v4/database/postgres"
  "github.com/golang-migrate/migrate/v4/source/iofs"
  "github.com/jackc/pgx/v5"
  "github.com/sivaprasadreddy/bookmarks/db"
)

func GetDb(config AppConfig, logger *Logger) *pgx.Conn {
  connStr := fmt.Sprintf("host=%s port=%d user=%s password=%s dbname=%s sslmode=disable",
    config.Db.Host, config.Db.Port, config.Db.UserName, config.Db.Password, config.Db.Database)
  conn, err := pgx.Connect(context.Background(), connStr)
  if err != nil {
    logger.Fatal(err)
  }
  applyDbMigrations(config, logger)
  return conn
}

func applyDbMigrations(config AppConfig, logger *Logger) {
  d, err := iofs.New(db.MigrationsFS, "migrations")
  if err != nil {
    logger.Fatalf("Error while loading db migrations from sources: %v", err)
  }
  databaseURL := fmt.Sprintf("postgres://%s:%s@%s:%d/%s?sslmode=disable",
    config.Db.UserName, config.Db.Password, config.Db.Host, config.Db.Port, config.Db.Database)
  m, err := migrate.NewWithSourceInstance("iofs", d, databaseURL)
  if err != nil {
    logger.Fatalf("Error while loading db migrations: %v", err)
  }
  err = m.Up()
  if err != nil && !errors.Is(err, migrate.ErrNoChange) {
    logger.Fatalf("Error while applying db migrations: %v", err)
  }
  logger.Infof("Database migrations applied successfully")
}
```

We have loaded the migration scripts from **migrations** directory of **MigrationsFS** and applied them to the database.
Note that we also need to import the **postgres** driver to use it with **golang-migrate**.
By default, Go doesn't allow declaring unused variables or imports.
So, we have to use **_** to import the package to avoid the error.

Also, notice that we are passing **config.Logger** to the **GetDb()** function.
So, we need to pass it from **app.go** file **NewApp(cfg config.AppConfig)** function as well.

Now, connect to the database and delete the **bookmarks** table and run the application.
You should see the **bookmarks** table got created and also there is **schema_migrations** table created by **golang-migrate** to keep track of the applied migrations.
This is similar to Flyway's **flyway_schema_history** table, but not quite the same though.

## Implement Create Bookmark API
We have already implemented the API to fetch all the bookmarks.
Now, let's implement the API to create a new bookmark.

Let's update the **internal/domain/repository.go** file to update the **Create()** method as follows:

```go
func (r bookmarkRepo) Create(ctx context.Context, b Bookmark) (*Bookmark, error) {
	query := "insert into bookmarks(title, url, created_at) values($1, $2, $3) RETURNING id"
	var lastInsertID int
	err := r.db.QueryRow(ctx, query, b.Title, b.Url, b.CreatedAt).Scan(&lastInsertID)
	if err != nil {
		r.logger.Errorf("Error while inserting bookmark: %v", err)
		return nil, err
	}
	b.ID = lastInsertID
	return &b, nil
}
```
Now, let's add a handler to create a new bookmark in **internal/api/handler.go** file as follows:

```go
type CreateBookmarkRequest struct {
	Title string `json:"title" binding:"required"`
	Url   string `json:"url" binding:"required,url"`
}

func (p BookmarkController) Create(c *gin.Context) {
	ctx := c.Request.Context()
	var model CreateBookmarkRequest
	if err := c.ShouldBindJSON(&model); err != nil {
        // you can extract error details as follows
        /*for _, err := range err.(validator.ValidationErrors) {
            fmt.Println(err.Field())
            fmt.Println(err.Tag())
            fmt.Println(err.Kind())
            fmt.Println(err.Type())
            fmt.Println(err.Value())
        }*/
		p.respondWithError(c, http.StatusBadRequest, err, "Invalid request payload")
		return
	}
	p.logger.Infof("Creating bookmark for URL: %s", model.Url)
	bookmark := domain.Bookmark{
		ID:        0,
		Title:     model.Title,
		Url:       model.Url,
		CreatedAt: time.Now(),
	}

	savedBookmark, err := p.repo.Create(ctx, bookmark)
	if err != nil {
		p.respondWithError(c, http.StatusInternalServerError, err, "Failed to create bookmark")
		return
	}
	c.JSON(http.StatusCreated, savedBookmark)
}

func (p BookmarkController) respondWithError(c *gin.Context, code int, err error, errMsg string) {
	if err != nil {
		p.logger.Errorf("Error :%v", err)
	}
	c.AbortWithStatusJSON(code, gin.H{
		"error": errMsg,
	})
}
```

We have created a struct called **CreateBookmarkRequest** to represent the request payload.
We have added **json** tags to map the request payload to the struct fields.
Also, we have added **binding** tags to validate the request payload.
Gin uses [validator](https://pkg.go.dev/gopkg.in/go-playground/validator.v9) package for validation.
You can extract the error details as shown in the commented code.

Then we have added a utility method called **respondWithError** to handle the error response to avoid the repetition.

Finally, we have to attach the handler to the router in **cmd/app.go** file as follows:

```go
router.POST("/api/bookmarks", handler.Create)
```

Now, let's run the application and create a new bookmark using the following curl command:

```shell
curl --location --request POST 'http://localhost:8080/api/bookmarks' \
--header 'Content-Type: application/json' \
--data-raw '{
    "title": "Google",
    "url": "https://google.com"
}'
```

You should see the following response:

```json
{
    "ID": 1,
    "Title": "Google",
    "Url": "https://google.com",
    "CreatedAt": "2021-09-18T12:11:51.091+05:30"
}
```

Notice that the keys are nothing but the Bookmark struct field names.
We can customize the response by using the **json** tags as follows:

```go
type Bookmark struct {
	ID        int       `json:"id"`
	Title     string    `json:"title"`
	Url       string    `json:"url"`
	CreatedAt time.Time `json:"createdAt"`
}
```

Now, you should see the following response:

```json
{
    "id": 1,
    "title": "Google",
    "url": "https://google.com",
    "createdAt": "2021-09-18T12:11:51.091+05:30"
}
```

## Implementing other API endpoints
We have implemented the API to fetch all the bookmarks and create a new bookmark.
The remaining API endpoints are pretty much similar to these API implementations.
So, I am going to leave it as an exercise for you to implement the remaining API endpoints.
You can find the complete code in the [GitHub repository](https://github.com/sivaprasadreddy/go-for-spring-boot-developers/tree/main/bookmarks-go).

## Dockerizing the Go application
Spring Boot has a built-in support for creating Docker images using **Buildpacks**.
You can also use **jib** or **Dockerfile** to create Docker images.

We can dockerize our Go application using the following **Dockerfile**:
    
```dockerfile 
FROM golang:1.21-buster as builder
# Create and change to the app directory.
WORKDIR /app
# Copy go.mod and if present go.sum.
COPY go.* ./
# Download all dependancies. Dependencies will be cached if the go.mod and go.sum files are not changed
RUN go mod download
# Copy local code to the container image.
COPY . ./
# Build the Go app
RUN GO111MODULE=on GOOS=linux CGO_ENABLED=0 go build -v -o server

######## Start a new stage from scratch #######
FROM gcr.io/distroless/base-debian10
WORKDIR /

# Copy the Pre-built binary file from the previous stage
COPY --from=builder /app/server ./server
COPY --from=builder /app/config.json ./config.json

# Run the templates service on container startup.
CMD ["/server"]
```

Notice that we are using a multi-stage build to create the Docker image.
In the first stage, we are using the official golang image to build the application and generate the binary.
In the second stage, we are using the distroless image to run the application.
We copied the binary and **config.json** file from the first stage to the second stage.
Finally, starting the application using the binary.

We can override the default configuration properties defined in **config.json** file using environment variables.
For example, if you want to override the server port, then you can pass the **SERVER_PORT** environment variable to the container.
You can pass database connection properties using **DB_HOST**, **DB_PORT**, **DB_USERNAME**, **DB_PASSWORD**, **DB_DATABASE** environment variables.


## Java/Spring Boot vs Go comparison
Each language and framework has its own pros and cons, and it's up to us to choose the right tool for the job.
There is no silver bullet, and there is no one size fits all solutions.

Sometimes performance is the most important factor, and sometimes developer productivity is the most important factor.
We need to evaluate the pros and cons of each technology and choose the right one for the problem at hand.

IMO, Java/Spring Boot and Go took very different approaches to improve the developer productivity.
I already discussed some of them in detail in my previous article [Code simplicity by Abstraction vs Verbosity](https://www.sivalabs.in/code-simplicity-by-abstraction-vs-verbosity/).

If I have to compare Java/Spring Boot and Go, I would say like this:

**Java/Spring Boot:**

* Java has a very mature ecosystem with a lot of libraries and tools available.
* Spring Boot is an opinionated framework, and it offers a lot of features out of the box.
* Spring Boot greatly improves developer productivity by providing many commonly needed features out of the box.
* Spring Boot has a steep learning curve, and it takes a lot of time to master it.
* Spring Boot consumes more resources (CPU, Memory) compared to Go.
  With GraalVM native image support, this is changing rapidly.
  However, there are many libraries that are not compatible with GraalVM native image yet and native compilation is currently taking a lot of time.

**Go:**

* Go is a very simple language with a small set of features.
* Go is a very opinionated language, and it forces you to do things in a certain way like formatting, unused variables, etc.
* Go has a rich standard library and toolchain (formatting, testing, benchmarking, cross-platform compilation, etc) support.
* Go is verbose, and it takes more lines of code to achieve the same thing compared to Java. IMO, this is mainly due to error handling approach in Go.
* Go consumes fewer resources (CPU, Memory) compared to Java/Spring Boot.
* In my opinion, the biggest advantage for Go is its simplicity. While the Go code looks more verbose, it's very easy to understand and maintain.

Go community prefers using only the necessary libraries and integrating them instead of using an all-in-one framework like **Spring Boot** or **Django**.

**Personally, I felt Go is more verbose and requires writing more lines of code compared to Java/Spring Boot.
But it also results in a less cognitive load while working with Go code.**

OTOH, once you understand the magic behind Spring Boot, it is super productive to build applications.
Spring Boot already solved a lot of common application needs like **configuration management**, **logging**, **monitoring**, etc.
You can also find Spring Boot integrations with almost everything under the sun, which greatly helps to build applications quickly.

## Conclusion

I am not trying to convince you one is better than the other.
If you are planning to build an application in Go coming from a Java/Spring Boot background, I hope this article might help you.

If you are coming from a Spring Boot background, then you might find it a little bit difficult to get used to the Go way of doing things.
Especially, Spring Boot has a lot of features and abstractions built-in to make the developer's life easy.
But in Go, you have to implement or integrate various libraries yourself.

However, once the skeleton is ready, you can focus on implementing the business logic with very less cognitive load.
As there is no annotations magic and ten layers of abstractions, it is very easy to understand the code.

Also, Go application consumes very less memory and starts very fast.
In a containerized environment, this is very important.

There are still many things I haven't covered in this article like **graceful shutdown**, **monitoring**, **testing**, etc.
But I hope this article will help you to get started with Go.

You can find the complete code in the [GitHub repository](https://github.com/sivaprasadreddy/go-for-spring-boot-developers).
In the repository, you can also find the implementation of the following:
* Remaining API endpoints 
* Graceful shutdown 
* Repository implementation using [GORM](https://gorm.io/) 
* Testing using [testcontainers-go](https://golang.testcontainers.org/)
# My First Year at AtomicJar as a Developer Advocate


I joined [AtomicJar](https://www.atomicjar.com/) as a **Developer Advocate** on 1st November 2022.
Time flies when you are enjoying your work, doesn't it?

Before joining AtomicJar, I worked as a TechLead/Architect at various companies for more than 15 years.
When I got the opportunity to join AtomicJar, I was very excited to work with my favourite technology [Testcontainers](https://www.testcontainers.org/).
I have been using Testcontainers for more than 4 years, and I am a big fan of it.

<!--more-->


But I was also a bit nervous about how I will perform as a Developer Advocate. 
In fact, I had little idea about what a Developer Advocate does.
So, I talked to the AtomicJar team and learned about the Developer Advocate role and responsibilities.
Then I realized that I have been doing some of the Developer Advocate's responsibilities like creating content, 
helping other developers to use Testcontainers, etc. for the past few years.
So, I decided to join AtomicJar as a Developer Advocate.

I would like to share my experience of working at AtomicJar for the past one year.

## Openness
We all come from different backgrounds, and we have different opinions and ideas.
I like the openness at AtomicJar, and everyone is free to share their opinions and ideas without the fear of being judged.

I do have strong opinions on some topics([cough](https://www.sivalabs.in/tomato-architecture-pragmatic-approach-to-software-design/), 
[cough](https://www.sivalabs.in/why-go-is-more-verbose-than-java/), ...), and they may not be aligned with others opinions.
We occasionally have healthy discussions on those topics, and I learned a lot from those discussions.

## Learning from others
AtomicJar has a very talented team, and I learned a lot from them.
Though there are specific roles (such as DevTeam, DevOps, UX), everyone is highly skilled in multiple areas.

* I can freely talk to [Oleg](https://twitter.com/shelajev) just about anything, and he always has some interesting insights to share. 
  Though he is my Manager, it always feels like I am working with a close buddy.
* Whenever [Sergei](https://twitter.com/bsideup) explains or demonstrates something, 
  I am like **"Maybe one day I will learn at least 10% of what he knows."**
  His quest for "perfection" is legendary.
* I read about "Leadership Qualities" but I have never seen anyone who has all those qualities like [Kevin](https://twitter.com/Kiview).
  He is a great leader, and every time I talked to him, I learn a lot.
* You know, I have a Love/Hate relationship with Go Language. 
  Occasionally when I get frustrated with Go, [Manu](https://twitter.com/mdelapenya) is always there to help me understand it better.
  Not just "getting things done" type of education, but properly explaining the concepts and making me understand it better.

I can go on and on about everyone, but I will stop here.
I am surrounded by people who are very knowledgeable, which inspires me to become better every day.

## Work-Life Balance
The last few years are challenging for many of us because of COVID, wars, etc.
I am not in my 20's anymore to work crazy hours, and I have a family to take care of.

I am very grateful to have a manager who often reminds me to take vacations and spend time with family :smile:

## Remote Work
I have been working remotely for the past 3 years, and I occasionally miss being in the office, having a casual talk with colleagues.
Fortunately, we have flexible working hours, and we can have a casual talk with our colleagues whenever we want.
It took me some time to get used to working remotely with async communication, but now I am very comfortable with it.

Overall, I am very happy that I can work from home and spend more time with my family.

## Few things I feel proud of
I feel proud of the following things I did in the past one year.
* I contributed to [Testcontainers Guides](https://testcontainers.com/guides/) that helps developers to quickly get started with Testcontainers.
* I have written a few blog posts on [AtomicJar Blog](https://www.atomicjar.com/author/siva/) that helps developers to use Testcontainers with various technologies.
* I have created a few videos on [AtomicJar YouTube Channel](https://www.youtube.com/@atomicjar/videos) explaining how to use Testcontainers effectively.
* I have created [testcontainers-jooq-codegen-maven-plugin](https://github.com/testcontainers/testcontainers-jooq-codegen-maven-plugin) that helps developers to generate jOOQ code from the database schema.

In addition to these, I have published many blog posts on my blog https://www.sivalabs.in and 
created a few videos on my [YouTube channel](https://www.youtube.com/@sivalabs/videos) that helps developers to use Testcontainers.

There is a lot more to do, and I am looking forward to doing more in the coming years.

## Conclusion
Often people ask me, **"Do I miss regular Software Development work?"**.
I would say **"No"** because I do a lot of coding every single day.
I do explore new technologies, learn new things, and experiment quite a bit while creating content.

As a passion, I have been sharing my knowledge by blogging and creating videos for the past 10 years in addition to doing my regular job.
Now doing the same thing as part of my full-time job, especially with my favourite technologies.

**I am very happy that I joined AtomicJar, and I am looking forward to more anniversaries.**

# Spring Boot + jOOQ Tutorial - 5 : Fetching Many-to-Many Relationships

In the [previous tutorial]({{< relref "2023-10-23-spring-boot-jooq-tutorial-part-4.md" >}}), 
we have learned how to fetch **One-to-Many** relationships using jOOQ.
In this tutorial, we will learn how to fetch **Many-to-Many** relationships using jOOQ.

<!--more-->


* [jOOQ Tutorial - 1 : Getting Started]({{< relref "2023-10-12-spring-boot-jooq-tutorial-part-1.md" >}})
* [jOOQ Tutorial - 2 : Implementing CRUD Operations]({{< relref "2023-10-16-spring-boot-jooq-tutorial-part-2.md" >}})
* [jOOQ Tutorial - 3 : Fetching One-to-One Relationships]({{< relref "2023-10-19-spring-boot-jooq-tutorial-part-3.md" >}})
* [jOOQ Tutorial - 4 : Fetching One-to-Many Relationships]({{< relref "2023-10-23-spring-boot-jooq-tutorial-part-4.md" >}})
* [jOOQ Tutorial - 5 : Fetching Many-to-Many Relationships]({{< relref "2023-10-26-spring-boot-jooq-tutorial-part-5.md" >}})

{{< box info >}}
**Source Code:**

You can find the complete source code of this project on GitHub:
https://github.com/sivaprasadreddy/spring-boot-jooq-demo
{{< /box >}}

In our sample database, we have **bookmarks** and **tags** tables.
Each bookmark can be associated with multiple bookmarks and vice-versa, 
so there is a **Many-to-Many** relationship between **bookmarks** and **tags** tables. 

Let's see how we can fetch a list of bookmarks along with the tags.

First, let's create **BookmarkWithTags** as a record.

```java 
package com.sivalabs.bookmarks.models;

import java.util.List;

public record BookmarkWithTags(Long id, String title, String url, List<TagInfo> tags) {
    public record TagInfo (Long id, String name){}
}
```

## Fetching Many-to-Many Relationships using MULTISET value constructor

We are going to use jOOQ's **MULTISET** value constructor to fetch the list of bookmarks along with tags.

Let's implement fetching bookmarks along with the tags as follows:

```java
package com.sivalabs.bookmarks.repositories;

import com.sivalabs.bookmarks.models.BookmarkWithTags;
import org.jooq.DSLContext;
import org.springframework.stereotype.Repository;

import java.util.List;

import static com.sivalabs.bookmarks.jooq.Tables.BOOKMARK_TAG;
import static com.sivalabs.bookmarks.jooq.tables.Bookmarks.BOOKMARKS;
import static com.sivalabs.bookmarks.jooq.tables.Tags.TAGS;
import static org.jooq.Records.mapping;
import static org.jooq.impl.DSL.multiset;
import static org.jooq.impl.DSL.select;

@Repository
public class BookmarkRepository {
    private final DSLContext dsl;

    public BookmarkRepository(DSLContext dsl) {
        this.dsl = dsl;
    }

    public List<BookmarkWithTags> getBookmarksWithTags() {
        return dsl
                .select(
                    BOOKMARKS.ID, BOOKMARKS.TITLE, BOOKMARKS.URL,
                    multiset(
                            select(TAGS.ID, TAGS.NAME)
                                .from(TAGS)
                                .join(BOOKMARK_TAG)
                                .on(BOOKMARK_TAG.TAG_ID.eq(TAGS.ID))
                                .where(BOOKMARK_TAG.BOOKMARK_ID.eq(BOOKMARKS.ID))
                    ).as("tags").convertFrom(r -> r.map(mapping(BookmarkWithTags.TagInfo::new)))
                )
                .from(BOOKMARKS)
                .fetch(mapping(BookmarkWithTags::new));
    }
}
```

## Sample Test Data Setup
We have the following **src/test/resources/test-data.sql** file to insert test data into the database.

```sql
# OMITTING OTHER INSERT STATEMENTS FOR BREVITY

INSERT INTO tags(id, name)
VALUES (1, 'java'),
       (2, 'spring-boot'),
       (3, 'spring-cloud'),
       (4, 'devops'),
       (5, 'security')
;

INSERT INTO bookmarks(id, title, url, created_by, created_at)
VALUES (1, 'SivaLabs', 'https://sivalabs.in', 1, CURRENT_TIMESTAMP),
       (2, 'Spring Initializr', 'https://start.spring.io', 2, CURRENT_TIMESTAMP),
       (3, 'Spring Blog', 'https://spring.io/blog', 2, CURRENT_TIMESTAMP)
;

insert into bookmark_tag(bookmark_id, tag_id)
VALUES (1, 1),
       (1, 2),
       (1, 3),
       (2, 2),
       (3, 2),
       (3, 3),
       (3, 4)
;
```

## Test Loading the Many-to-Many Relationships 
Now, let's write a test case to verify the above method.

```java
package com.sivalabs.bookmarks.repositories;

import com.sivalabs.bookmarks.models.BookmarkWithTags;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.jooq.JooqTest;
import org.springframework.boot.testcontainers.service.connection.ServiceConnection;
import org.springframework.context.annotation.Import;
import org.springframework.test.context.jdbc.Sql;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;

import java.util.List;

import static org.assertj.core.api.Assertions.assertThat;

@JooqTest
@Import({BookmarkRepository.class})
@Testcontainers
@Sql("classpath:/test-data.sql")
class BookmarkRepositoryTest {

    @Autowired
    BookmarkRepository bookmarkRepository;

    @Container
    @ServiceConnection
    static final PostgreSQLContainer<?> postgres =
            new PostgreSQLContainer<>("postgres:16-alpine");

    @Test
    void getBookmarksWithTags() {
        var bookmarksWithTags = bookmarkRepository.getBookmarksWithTags();
        assertThat(bookmarksWithTags).hasSize(3);

        var javaTag = new BookmarkWithTags.TagInfo(1L, "java");
        var springBootTag = new BookmarkWithTags.TagInfo(2L, "spring-boot");
        var springCloudTag = new BookmarkWithTags.TagInfo(3L, "spring-cloud");
        var devopsTag = new BookmarkWithTags.TagInfo(4L, "devops");

        var bookmark1 = new BookmarkWithTags(1L, "SivaLabs", "https://sivalabs.in",
                List.of(javaTag, springBootTag, springCloudTag));
        var bookmark2 = new BookmarkWithTags(2L, "Spring Initializr", "https://start.spring.io",
                List.of(springBootTag));
        var bookmark3 = new BookmarkWithTags(3L, "Spring Blog", "https://spring.io/blog",
                List.of(springBootTag, springCloudTag, devopsTag));

        assertThat(bookmarksWithTags).contains(bookmark1, bookmark2, bookmark3);
    }
}
```

If you run the test, it should pass.

## Conclusion
In this tutorial, we have learned how to fetch **Many-to-Many** relationships using jOOQ with the help of MULTISET value operator.

# Spring Boot + jOOQ Tutorial - 4 : Fetching One-to-Many Relationships

In the [previous tutorial]({{< relref "2023-10-19-spring-boot-jooq-tutorial-part-3.md" >}}), 
we have learned how to fetch **One-to-One** relationships using jOOQ.
In this tutorial, we will learn how to fetch **One-to-Many** relationships using jOOQ.

<!--more-->


* [jOOQ Tutorial - 1 : Getting Started]({{< relref "2023-10-12-spring-boot-jooq-tutorial-part-1.md" >}})
* [jOOQ Tutorial - 2 : Implementing CRUD Operations]({{< relref "2023-10-16-spring-boot-jooq-tutorial-part-2.md" >}})
* [jOOQ Tutorial - 3 : Fetching One-to-One Relationships]({{< relref "2023-10-19-spring-boot-jooq-tutorial-part-3.md" >}})
* [jOOQ Tutorial - 4 : Fetching One-to-Many Relationships]({{< relref "2023-10-23-spring-boot-jooq-tutorial-part-4.md" >}})
* [jOOQ Tutorial - 5 : Fetching Many-to-Many Relationships]({{< relref "2023-10-26-spring-boot-jooq-tutorial-part-5.md" >}})

{{< box info >}}
**Source Code:**

You can find the complete source code of this project on GitHub:
https://github.com/sivaprasadreddy/spring-boot-jooq-demo
{{< /box >}}

In our sample database, we have **users** and **bookmarks** tables.
Each user can create multiple bookmarks, so there is a **One-to-Many** relationship between **users** and **bookmarks** tables. 

Let's see how we can fetch a User details along with the bookmarks created by the user.

First, let's create **UserWithBookmarks** as a record.

```java 
package com.sivalabs.bookmarks.models;

import java.util.List;

public record UserWithBookmarks(Long id, String name, String email, List<BookmarkInfo> bookmarks) {
    public record BookmarkInfo (Long id, String title, String url){}
}
```

## Fetching One-to-Many Relationships using MULTISET value constructor

We are going to use jOOQ's **MULTISET** value constructor to fetch the list of bookmarks created by the user.
You can find more details about **MULTISET value constructor** here: https://www.jooq.org/doc/latest/manual/sql-building/column-expressions/multiset-value-constructor/.

Also, I highly recommend reading [How jOOQ 3.15's New Multiset Operator Will Change How You Think About SQL](https://blog.jooq.org/jooq-3-15s-new-multiset-operator-will-change-how-you-think-about-sql/) article.

Let's implement fetching a user details along with the bookmarks created by that user.

```java
@Repository
public class UserRepository {
    ...
    ...
    public Optional<UserWithBookmarks> getUserWithBookmarksById(Long userId) {
        return dsl
                .select(
                        USERS.ID, USERS.NAME, USERS.EMAIL,
                        multiset(
                                select(BOOKMARKS.ID, BOOKMARKS.TITLE, BOOKMARKS.URL)
                                        .from(BOOKMARKS)
                                        .where(BOOKMARKS.CREATED_BY.eq(USERS.ID))
                        ).as("bookmarks").convertFrom(r -> r.map(mapping(UserWithBookmarks.BookmarkInfo::new)))
                )
                .from(USERS)
                .where(USERS.ID.eq(userId))
                .fetchOptional()
                .map(mapping(UserWithBookmarks::new));
    }
}
```

## Sample Test Data Setup
We have the following **src/test/resources/test-data.sql** file to insert test data into the database.

```sql
# OMITTING OTHER INSERT STATEMENTS FOR BREVITY

INSERT INTO users (id, email, password, name, preferences_id)
VALUES (1, 'admin@gmail.com', 'admin', 'Admin', 2),
       (2, 'siva@gmail.com', 'siva', 'Siva', 1)
;
  
INSERT INTO bookmarks(id, title, url, created_by, created_at)
VALUES (1, 'SivaLabs', 'https://sivalabs.in', 1, CURRENT_TIMESTAMP),
       (2, 'Spring Initializr', 'https://start.spring.io', 2, CURRENT_TIMESTAMP),
       (3, 'Spring Blog', 'https://spring.io/blog', 2, CURRENT_TIMESTAMP)
;
```

## Test Loading the One-to-Many Relationships 
Now, let's write a test case to verify the above method.

```java
package com.sivalabs.bookmarks.repositories;

import com.sivalabs.bookmarks.models.User;
import com.sivalabs.bookmarks.models.UserWithBookmarks;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.jooq.JooqTest;
import org.springframework.boot.testcontainers.service.connection.ServiceConnection;
import org.springframework.context.annotation.Import;
import org.springframework.test.context.jdbc.Sql;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;

import java.util.List;
import java.util.Optional;
import java.util.UUID;

import static org.assertj.core.api.Assertions.assertThat;

@JooqTest
@Import({UserRepository.class})
@Testcontainers
@Sql("classpath:/test-data.sql")
class UserRepositoryTest {

    @Autowired
    UserRepository userRepository;

    @Container
    @ServiceConnection
    static final PostgreSQLContainer<?> postgres =
            new PostgreSQLContainer<>("postgres:16-alpine");

    @Test
    void getUserWithBookmarks() {
        Optional<UserWithBookmarks> userOptional = userRepository.getUserWithBookmarksById(2L);
        assertThat(userOptional).isPresent();
        UserWithBookmarks user = userOptional.get();
        assertThat(user.id()).isEqualTo(2L);
        assertThat(user.name()).isEqualTo("Siva");
        assertThat(user.email()).isEqualTo("siva@gmail.com");

        assertThat(user.bookmarks()).hasSize(2);

        var bookmark1 = new UserWithBookmarks.BookmarkInfo(2L, "Spring Initializr", "https://start.spring.io");
        var bookmark2 = new UserWithBookmarks.BookmarkInfo(3L, "Spring Blog", "https://spring.io/blog");

        assertThat(user.bookmarks()).contains(bookmark1, bookmark2);
    }
}
```

If you run the test, it should pass.

## Conclusion
In this tutorial, we have learned how to fetch **One-to-Many** relationships using jOOQ with the help of MULTISET value operator.
In the next tutorial, we will learn how to fetch **Many-to-Many** relationships using jOOQ.

# Spring Boot + jOOQ Tutorial - 3 : Fetching One-to-One Relationships

In the [previous tutorial]({{< relref "2023-10-16-spring-boot-jooq-tutorial-part-2.md" >}}), 
we have learned how to implement basic CRUD Operations using jOOQ.
In this tutorial, we will learn how to fetch **One-to-One** relationships using jOOQ.

<!--more-->


* [jOOQ Tutorial - 1 : Getting Started]({{< relref "2023-10-12-spring-boot-jooq-tutorial-part-1.md" >}})
* [jOOQ Tutorial - 2 : Implementing CRUD Operations]({{< relref "2023-10-16-spring-boot-jooq-tutorial-part-2.md" >}})
* [jOOQ Tutorial - 3 : Fetching One-to-One Relationships]({{< relref "2023-10-19-spring-boot-jooq-tutorial-part-3.md" >}})
* [jOOQ Tutorial - 4 : Fetching One-to-Many Relationships]({{< relref "2023-10-23-spring-boot-jooq-tutorial-part-4.md" >}})
* [jOOQ Tutorial - 5 : Fetching Many-to-Many Relationships]({{< relref "2023-10-26-spring-boot-jooq-tutorial-part-5.md" >}})

{{< box info >}}
**Source Code:**

You can find the complete source code of this project on GitHub:
https://github.com/sivaprasadreddy/spring-boot-jooq-demo
{{< /box >}}

Generally, when we are displaying a list of records, we will display minimal information about the record, and 
when the user clicks on the record, we will display the complete information about the record.

In our sample application, let's say we will display the list of users with minimal information like **id**, **name**, and **email**.
When the user clicks on the user, we will display the complete information about the user including the user preferences.

So, let's update our **findUserById()** method to fetch the user preferences as well.

First, let's create **UserPreferences** as a record.

```java 
public record UserPreferences(Long id, String theme, String language) {
}
```

Update the **User** class to include the **UserPreferences**.

```java
package com.sivalabs.bookmarks.models;

public record User (
     Long id,
     String name,
     String email,
     String password,
     UserPreferences preferences
) {
    public User(Long id, String name, String email, String password) {
        this(id, name, email, password, null);
    }

    public static User create(Long id, String name, String email, String password) {
        return new User(id, name, email, password, null);
    }
}
```

In SQL, we can fetch the user preferences using the **LEFT OUTER JOIN** query as shown below:

```sql
SELECT u.id, u.name, u.email, u.password, up.id as "preferences_id", up.theme, up.language
FROM users u LEFT OUTER JOIN user_preferences up ON u.preferences_id = up.id
WHERE u.id = 1;
```

Let's implement the above query using jOOQ.

```java
public Optional<User> findUserById(Long id) {
    return dsl
            .select(
                USERS.ID, USERS.NAME, USERS.EMAIL, USERS.PASSWORD,
                USER_PREFERENCES.ID, USER_PREFERENCES.THEME, USER_PREFERENCES.LANGUAGE)
            .from(USERS.leftOuterJoin(USER_PREFERENCES).on(USERS.PREFERENCES_ID.eq(USER_PREFERENCES.ID)))
            .where(USERS.ID.eq(id))
            .fetchOptional(record -> new User(
                    record.get(USERS.ID),
                    record.get(USERS.NAME),
                    record.get(USERS.EMAIL),
                    record.get(USERS.PASSWORD),
                    new UserPreferences(
                            record.get(USER_PREFERENCES.ID),
                            record.get(USER_PREFERENCES.THEME),
                            record.get(USER_PREFERENCES.LANGUAGE)
                    )
            ));
}
```

Now, let's update our test case to verify the above method.

```java
package com.sivalabs.bookmarks.repositories;

import com.sivalabs.bookmarks.models.User;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.jooq.JooqTest;
import org.springframework.boot.testcontainers.service.connection.ServiceConnection;
import org.springframework.context.annotation.Import;
import org.springframework.test.context.jdbc.Sql;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;

import java.util.List;
import java.util.Optional;
import java.util.UUID;

import static org.assertj.core.api.Assertions.assertThat;

@JooqTest
@Import({UserRepository.class})
@Testcontainers
@Sql("classpath:/test-data.sql")
class UserRepositoryTest {

    @Autowired
    UserRepository userRepository;

    @Container
    @ServiceConnection
    static final PostgreSQLContainer<?> postgres =
            new PostgreSQLContainer<>("postgres:16-alpine");

    @Test
    void findUserById() {
        Optional<User> userOptional = userRepository.findUserById(1L);
        assertThat(userOptional).isPresent();
        assertThat(userOptional.get().id()).isEqualTo(1L);
        assertThat(userOptional.get().name()).isEqualTo("Admin");
        assertThat(userOptional.get().email()).isEqualTo("admin@gmail.com");
        assertThat(userOptional.get().password()).isEqualTo("admin");
        assertThat(user.preferences().id()).isEqualTo(2L);
        assertThat(user.preferences().theme()).isEqualTo("Dark");
        assertThat(user.preferences().language()).isEqualTo("EN");
    }
}
```

If you run the test, it should pass.

## Using Implicit Join Path
The jOOQ generated code provides a better way to fetch the One-to-One relationships using
the implicit join path to the associated table like **USERS.userPreferences().ID**, **USERS.userPreferences().THEME**, etc.

Let's update our **findUserById()** method to use the implicit join path.

```java
public Optional<User> findUserById(Long id) {
    return dsl
            .select(
                USERS.ID, USERS.NAME, USERS.EMAIL, USERS.PASSWORD,
                USERS.userPreferences().ID, USERS.userPreferences().THEME, USERS.userPreferences().LANGUAGE)
            .from(USERS)
            .where(USERS.ID.eq(id))
            .fetchOptional(record -> new User(
                record.get(USERS.ID),
                record.get(USERS.NAME),
                record.get(USERS.EMAIL),
                record.get(USERS.PASSWORD),
                new UserPreferences(
                    record.get(USER_PREFERENCES.ID),
                    record.get(USER_PREFERENCES.THEME),
                    record.get(USER_PREFERENCES.LANGUAGE)
                )
            ));
}
```

## Using Row Value Expressions
We can also fetch the One-to-One relationships using jOOQ's row value expression as follows:

```java
public Optional<User> findUserById(Long id) {
    return dsl
            .select(
                USERS.ID, USERS.NAME, USERS.EMAIL, USERS.PASSWORD,
                row(
                    USERS.userPreferences().ID,
                    USERS.userPreferences().THEME,
                    USERS.userPreferences().LANGUAGE
                ).mapping(UserPreferences::new).as("preferences"))
            .from(USERS)
            .where(USERS.ID.eq(id))
            .fetchOptional()
            .map(mapping((userId, name, email, password, preferences) ->
                            new User(userId, name, email, password, preferences)));
}
```

You might want to fetch the same information for other methods as well such as **findUserByEmail()**.
In that case, you can extract the select clause and result mapping into separate methods as shown in 
the **Part-2 Implementing findUserById()** section.

## Conclusion
In this tutorial, we have learned how to fetch **One-to-One** relationships using jOOQ.
In the next tutorial, we will learn how to fetch **One-to-Many** relationships using jOOQ.

# Spring Boot + jOOQ Tutorial - 2 : Implementing CRUD Operations

In the [previous tutorial]({{< relref "2023-10-12-spring-boot-jooq-tutorial-part-1.md" >}}), we have seen how to generate jOOQ code using 
the [testcontainers-jooq-codegen-maven-plugin](https://github.com/testcontainers/testcontainers-jooq-codegen-maven-plugin) 
and use jOOQ Typesafe DSL to execute SQL queries.
In this tutorial, we will learn how to implement basic CRUD Operations using jOOQ.

<!--more-->


* [jOOQ Tutorial - 1 : Getting Started]({{< relref "2023-10-12-spring-boot-jooq-tutorial-part-1.md" >}})
* [jOOQ Tutorial - 2 : Implementing CRUD Operations]({{< relref "2023-10-16-spring-boot-jooq-tutorial-part-2.md" >}})
* [jOOQ Tutorial - 3 : Fetching One-to-One Relationships]({{< relref "2023-10-19-spring-boot-jooq-tutorial-part-3.md" >}})
* [jOOQ Tutorial - 4 : Fetching One-to-Many Relationships]({{< relref "2023-10-23-spring-boot-jooq-tutorial-part-4.md" >}})
* [jOOQ Tutorial - 5 : Fetching Many-to-Many Relationships]({{< relref "2023-10-26-spring-boot-jooq-tutorial-part-5.md" >}})


{{< box info >}}
**Source Code:**

You can find the complete source code of this project on GitHub:
https://github.com/sivaprasadreddy/spring-boot-jooq-demo
{{< /box >}}

In this tutorial, we will see how to perform basic CRUD (**C**reate, **R**ead, **U**pdate, **D**elete) operations on **USERS** table using jOOQ.

## Implementing findAllUsers()
First, let us start with fetching all the users from the **USERS** table.
Assume we are only interested in the **id**, **name**, **email** and **password** columns of the **USERS** table.
We will deal with fetching relations like **UserPreferences**, **Bookmarks** later.

Let's create **User** record as follows:

```java
package com.sivalabs.bookmarks.models;

public record User(Long id, String name, String email, String password) { }
```

Let's implement **findAllUsers()** method in **UserRepository** class as follows:

```java
package com.sivalabs.bookmarks.repositories;

import com.sivalabs.bookmarks.models.User;
import org.jooq.DSLContext;
import org.jooq.impl.SQLDataType;
import org.springframework.stereotype.Repository;

import java.util.List;

import static com.sivalabs.bookmarks.jooq.tables.Users.USERS;
import static org.jooq.impl.DSL.inline;

@Repository
class UserRepository {
    private final DSLContext dsl;

    UserRepository(DSLContext dsl) {
        this.dsl = dsl;
    }

    public List<User> findAllUsers() {
        return dsl
                .select(USERS.ID, USERS.NAME, USERS.EMAIL,USERS.PASSWORD)
                .from(USERS)
                .fetch(r -> new User(
                    r.get(USERS.ID),
                    r.get(USERS.NAME),
                    r.get(USERS.EMAIL),
                    r.get(USERS.PASSWORD))
                );
    }
}
```

* We are querying the **USERS** table to only return the **id**, **name**, **email** and **password** columns.
* The **fetch()** method returns a list of **UsersRecord** objects, and we are converting them to **User** objects.

If the **User** record has a constructor which takes all the selected column value types as arguments 
(in this case, Long, String, String, String), then we can use the following syntax:

```java
public List<User> findAllUsers() {
    return dsl
        .select(USERS.ID, USERS.NAME, USERS.EMAIL, USERS.PASSWORD)
        .from(USERS)
        .fetch(r -> r.into(User.class));
}
```

Here the **UsersRecord** fields are mapped into a **User** object using **Reflection**.

Instead, you can use **org.jooq.Records.mapping()** to map the **UsersRecord** fields to a **User** object as follows:

```java
import static org.jooq.Records.mapping;

public List<User> findAllUsers(){
    return dsl.select(USERS.ID,USERS.NAME,USERS.EMAIL,USERS.PASSWORD)
            .from(USERS)
            //using a lambda
            .fetch(mapping((id,name,email,password) -> new User(id,name,email,password)));
    
            //using a constructor reference
            //.fetch(mapping(User::new));
        
            //using a factory method reference
            //.fetch(mapping(User::create));
}

// factory method in User class
public record User (Long id, String name, String email, String password
) {
    public static User create(Long id, String name, String email, String password) {
        return new User(id, name, email, password);
    }
}
```

If you want to select all the columns from the **USERS** table, and we have a corresponding constructor, 
then we can use the following syntax:

```java
public record User (Long id, String name, String email, String password,
                    Long userPreferencesId, LocalDateTime createdAt, LocalDateTime updatedAt) { }

public List<User> findAllUsers(){
    return dsl.selectFrom(USERS).fetch(r->r.into(User.class));
}
```

**But the best practice is to select only the columns/data you need for the usecase and map the result to a DTO/record.**

## Implementing findUserById()
Let's implement **findUserById()** method in **UserRepository** class as follows:

```java
public Optional<User> findUserById(Long id) {
    return dsl
            .select(USERS.ID, USERS.NAME, USERS.EMAIL, USERS.PASSWORD)
            .from(USERS)
            .where(USERS.ID.eq(id))
        .fetchOptional()
        //using reflection
        //.map(r -> r.into(User.class));
        // using factory method
        .map(mapping(User::create));
}
```
The implementation is very similar to **findAllUsers()** method, 
except that we are using **fetchOptional()** method to return an **Optional** object.

If you want to load all the columns from the **USERS** table, and you have a corresponding **User** constructor, 
then you can use the following syntax:

```java
public Optional<User> findUserById(Long id){
    return dsl.fetchOptional(USERS,USERS.ID.eq(id))
            .map(r -> r.into(User.class));
}
```

If you have many **findByXXX()** methods which have the same select columns, and the same mapping logic,
then you can extract it into a method as follows:

```java
public Optional<User> findUserById(Long id) {
    return getSelectUserSpec()
            .where(USERS.ID.eq(id))
            .fetchOptional(new UserRecordMapper());
}
        
public Optional<User> findUserByEmail(String email) {
    return getSelectUserSpec()
            .where(USERS.EMAIL.equalIgnoreCase(email))
            .fetchOptional(new UserRecordMapper());
}

private SelectJoinStep<Record4<Long, String, String, String>> getSelectUserSpec() {
        return dsl.select(USERS.ID, USERS.NAME, USERS.EMAIL, USERS.PASSWORD)
        .from(USERS);
}

static class UserRecordMapper
        implements RecordMapper<Record4<Long, String, String, String>, User> {
    @Override
    public User map(Record4<Long, String, String, String> userRecord) {
        return new User(
                userRecord.get(USERS.ID),
                userRecord.get(USERS.NAME),
                userRecord.get(USERS.EMAIL),
                userRecord.get(USERS.PASSWORD)
        );
    }
}
```

## Implementing createUser()
Let's implement **createUser()** method in **UserRepository** class as follows:

```java
public User createUser(User user) {
    return dsl.insertInto(USERS)
            .set(USERS.NAME, user.name())
            .set(USERS.EMAIL, user.email())
            .set(USERS.PASSWORD, user.password())
            .returning()
            .fetchOne(record -> new User(
                    record.getId(),
                    record.getName(),
                    record.getEmail(),
                    record.getPassword()));
}
```

We are using **dsl.insertInto(USERS)** and then setting values for various columns using **set()** method.
This is one way of inserting a new record, and there are few other ways you can do it using jOOQ.
We are using **returning()** method to return the newly inserted row details and mapping it to the User object.

You can also use a slightly different approach/syntax as follows:

```java
public User createUser(User user){
    return dsl.insertInto(USERS,USERS.NAME,USERS.EMAIL,USERS.PASSWORD)
            .values(user.name(),user.email(),user.password())
            .returning()
            .fetchOne(record -> new User(
                        record.getId(),
                        record.getName(),
                        record.getEmail(),
                        record.getPassword()));
}
```

Another approach using **UsersRecord** (**UpdatableRecord**) object:

```java
public User createUser(User user){
    UsersRecord record=dsl.newRecord(USERS);
    record.setName(user.name());
    record.setEmail(user.email());
    record.setPassword(user.password());
    record.store();
    
    return new User(record.getId(),
                    record.getName(),
                    record.getEmail(),
                    record.getPassword());
}
```

Yet another approach using **UsersRecord** populating data from a **User** object:

```java
public User createUser(User user){
    UsersRecord record = dsl.newRecord(USERS,user);
    record.store();
    return new User(record.getId(),
                    record.getName(),
                    record.getEmail(),
                    record.getPassword());
}
```

Check **org.jooq.Record.from(Object source)**, which is internally called by **dsl.newRecord(USERS, user)**, 
API docs for knowing the mapping rules.

{{< box info >}}
**Persisting Associated Objects**

Unlike JPA, there is no support for automatic persisting associated objects in jOOQ.

For ex, if **User** has **UserPreferences**, then you need to persist **UserPreferences** separately.
{{< /box >}}

## Implementing updateUser()
Let's implement **updateUser()** method in **UserRepository** class as follows:

```java
public void updateUser(User user) {
    dsl.update(USERS)
        .set(USERS.NAME, user.name())
        .where(USERS.ID.eq(user.id()))
        .execute();
    
    //Another approach to check for record existence before updating
    /*
    dsl.fetchOptional(USERS, USERS.ID.eq(user.id()))
            .ifPresent(record -> {
                record.setName(user.name());
                record.store();
            });
    */
}
```

Here we are only updating the **name** column of the **USERS** table.
The execute() method returns the number of rows updated.

## Implementing deleteUser()
Let's implement **deleteUser()** method in **UserRepository** class as follows:

```java
public void deleteUser(Long id) {
    dsl.deleteFrom(USERS)
        .where(USERS.ID.eq(id))
        .execute();
}
```

## Implementing Tests using Testcontainers

Let's write UserRepositoryTest to test the above CRUD operations.

```java
package com.sivalabs.bookmarks.repositories;

import com.sivalabs.bookmarks.models.User;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.jooq.JooqTest;
import org.springframework.boot.testcontainers.service.connection.ServiceConnection;
import org.springframework.context.annotation.Import;
import org.springframework.test.context.jdbc.Sql;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;

import java.util.List;
import java.util.Optional;
import java.util.UUID;

import static org.assertj.core.api.Assertions.assertThat;

@JooqTest
@Import({UserRepository.class})
@Testcontainers
@Sql("classpath:/test-data.sql")
class UserRepositoryTest {

    @Autowired
    UserRepository userRepository;

    @Container
    @ServiceConnection
    static final PostgreSQLContainer<?> postgres =
            new PostgreSQLContainer<>("postgres:16-alpine");

    @Test
    void findAllUsers() {
        List<User> users = userRepository.findAllUsers();
        assertThat(users).hasSize(2);
        //more assertions
    }

    @Test
    void findUserById() {
        Optional<User> userOptional = userRepository.findUserById(1L);
        assertThat(userOptional).isPresent();
        assertThat(userOptional.get().id()).isEqualTo(1L);
        assertThat(userOptional.get().name()).isEqualTo("Admin");
        assertThat(userOptional.get().email()).isEqualTo("admin@gmail.com");
        assertThat(userOptional.get().password()).isEqualTo("admin");
    }

    @Test
    void createUser() {
        User user = new User(null, "SivaLabs", "sivalabs@gmail.com", "siva1234");

        User savedUser = userRepository.createUser(user);
        assertThat(savedUser.id()).isNotNull();
        assertThat(savedUser.name()).isEqualTo("SivaLabs");
        assertThat(savedUser.email()).isEqualTo("sivalabs@gmail.com");
        assertThat(savedUser.password()).isEqualTo("siva1234");
    }

    @Test
    void updateUser() {
        User user = createTestUser();
        User updateUser = new User(user.id(), "TestName1", user.email(), user.password());
        userRepository.updateUser(updateUser);

        User updatedUser = userRepository.findUserById(updateUser.id()).orElseThrow();

        assertThat(updatedUser.id()).isEqualTo(updateUser.id());
        assertThat(updatedUser.name()).isEqualTo("TestName1");
        assertThat(updatedUser.email()).isEqualTo(user.email());
        assertThat(updatedUser.password()).isEqualTo(user.password());
    }

    @Test
    void deleteUser() {
        User user = createTestUser();
        userRepository.deleteUser(user.id());

        Optional<User> optionalUser = userRepository.findUserById(user.id());
        assertThat(optionalUser).isEmpty();
    }

    private User createTestUser() {
        String uuid = UUID.randomUUID().toString();
        User user = new User(null, uuid, uuid+"@gmail.com", "Secret");
        return userRepository.createUser(user);
    }
}
```

## Conclusion
In this second part, we have learned how to implement basic CRUD operations using jOOQ.
In the next part, we will explore how to **fetch One-to-One associations** such as **User** and **UserPreferences**.

# Contact Me


You can reach out to us using any of the following channels:

* **Twitter**: [@sivalabs](https://twitter.com/sivalabs)
* **LinkedIn**: [K Siva Prasad Reddy](https://www.linkedin.com/in/ksivaprasadreddy/)
# Spring Boot + jOOQ Tutorial - 1 : Getting Started

[jOOQ](https://www.jooq.org/) is a Java persistence library that provides SQL DSL for writing typesafe SQL queries. 
It supports most of the popular databases like **MySQL**, **PostgreSQL**, **Oracle**, **SQL Server**, and many more. 
In this tutorial, we will learn how to get started with jOOQ for implementing persistence layer in a Spring Boot application.
You can also use jOOQ in other JVM based languages like **Kotlin**, **Scala**, etc.

<!--more-->


* [jOOQ Tutorial - 1 : Getting Started]({{< relref "2023-10-12-spring-boot-jooq-tutorial-part-1.md" >}})
* [jOOQ Tutorial - 2 : Implementing CRUD Operations]({{< relref "2023-10-16-spring-boot-jooq-tutorial-part-2.md" >}})
* [jOOQ Tutorial - 3 : Fetching One-to-One Relationships]({{< relref "2023-10-19-spring-boot-jooq-tutorial-part-3.md" >}})
* [jOOQ Tutorial - 4 : Fetching One-to-Many Relationships]({{< relref "2023-10-23-spring-boot-jooq-tutorial-part-4.md" >}})
* [jOOQ Tutorial - 5 : Fetching Many-to-Many Relationships]({{< relref "2023-10-26-spring-boot-jooq-tutorial-part-5.md" >}})

In this jOOQ tutorial series, you will learn how to use jOOQ in a Spring Boot application for implementing:
* **Basic CRUD Operations**
* **Loading One-to-One Relationships**
* **Loading One-to-Many Relationships**
* **Loading Many-to-Many Relationships**

{{< box info >}}
**Source Code:**

You can find the complete source code of this project on GitHub:
https://github.com/sivaprasadreddy/spring-boot-jooq-demo
{{< /box >}}

## Prerequisites
* Install JDK 17 or later
* Install any container runtime like **Docker Desktop**, **OrbStack**, etc.
 
**NOTE:** **jOOQ** doesn't require Docker. 
But, we will use [Testcontainers](https://testcontainers.com/) for jOOQ code generation and testing, which needs a container runtime.

## Sample Database
We will use the following sample database for this tutorial.

{{< figure src="/images/jooq-demo-db.webp" >}}

## Create a Spring Boot Project
Let's create a Spring Boot project using [Spring Initializr](https://start.spring.io/) by selecting the starters
**JOOQ Access Layer**, **Flyway Migration**, **PostgreSQL Driver** and **Testcontainers**.

You can click on this [link](https://start.spring.io/#!type=maven-project&language=java&packaging=jar&jvmVersion=17&groupId=com.sivalabs&artifactId=spring-boot-jooq-demo&name=spring-boot-jooq-demo&packageName=com.sivalabs.bookmarks&dependencies=flyway,postgresql,testcontainers,jooq) to create a project with the required dependencies.

### Add Flyway Migration Scripts
We will use [Flyway](https://flywaydb.org/) for database schema migration.
Let's add the following SQL scripts under the **src/main/resources/db/migration** directory.

**V1__create_tables.sql**
```sql
CREATE TABLE user_preferences
(
    id         bigserial primary key,
    theme      varchar(255),
    language   varchar(255),
    created_at timestamp with time zone default CURRENT_TIMESTAMP,
    updated_at timestamp with time zone
);

CREATE TABLE users
(
    id             bigserial primary key,
    name           varchar(255) not null,
    email          varchar(255) not null,
    password       varchar(255) not null,
    preferences_id bigint REFERENCES user_preferences (id),
    created_at     timestamp with time zone default CURRENT_TIMESTAMP,
    updated_at     timestamp with time zone,
    CONSTRAINT user_email_unique UNIQUE (email)
);

CREATE TABLE bookmarks
(
    id         bigserial primary key,
    url        varchar(1024) not null,
    title      varchar(1024),
    created_by bigint        not null REFERENCES users (id),
    created_at timestamp with time zone default CURRENT_TIMESTAMP,
    updated_at timestamp with time zone
);

CREATE TABLE tags
(
    id         bigserial primary key,
    name       varchar(100) not null,
    created_at timestamp with time zone default CURRENT_TIMESTAMP,
    updated_at timestamp with time zone,
    CONSTRAINT tag_name_unique UNIQUE (name)
);

CREATE TABLE bookmark_tag
(
    bookmark_id bigint not null REFERENCES bookmarks (id),
    tag_id      bigint not null REFERENCES tags (id)
);

ALTER SEQUENCE user_preferences_id_seq RESTART WITH 101;
ALTER SEQUENCE users_id_seq RESTART WITH 101;
ALTER SEQUENCE bookmarks_id_seq RESTART WITH 101;
ALTER SEQUENCE tags_id_seq RESTART WITH 101;
```

We are altering the sequences to start with 101, so that we can insert some test data with ids 1, 2, 3, etc.

## Using jOOQ to execute native SQL queries
When we add jOOQ starter, Spring Boot autoconfigures the jOOQ's **DSLContext** as a bean.
We can use the **DSLContext** bean to execute native SQL queries.

Let's create **UserRepository** class as follows:

```java
package com.sivalabs.bookmarks.repositories;

import org.jooq.DSLContext;
import org.jooq.Record;
import org.springframework.stereotype.Repository;

@Repository
class UserRepository {
    private final DSLContext dsl;

    UserRepository(DSLContext dsl) {
        this.dsl = dsl;
    }

    public String findUserNameById(Long id) {
        Record record =
                dsl.resultQuery("select * from users where id = ?", id)
                   .fetchOptional().orElseThrow();
        System.out.println(record);
        Object name = record.get("name");
        return (String) name;
    }
}
```

Before writing test, let's create an SQL script to add some test data to the database.

**src/test/resources/test-data.sql**

```sql
DELETE FROM bookmark_tag;
DELETE FROM bookmarks;
DELETE FROM tags;
DELETE FROM users;
DELETE FROM user_preferences;

INSERT INTO user_preferences (id, theme, language) VALUES
(1, 'Light', 'EN'),
(2, 'Dark', 'EN')
;

INSERT INTO users (id, email, password, name, preferences_id) VALUES
(1, 'admin@gmail.com', 'admin', 'Admin', 2),
(2, 'siva@gmail.com', 'siva', 'Siva', 1)
;

INSERT INTO tags(id, name)
VALUES (1, 'java'),
       (2, 'spring-boot'),
       (3, 'spring-cloud'),
       (4, 'devops'),
       (5, 'security')
;

INSERT INTO bookmarks(id, title, url, created_by, created_at)
VALUES (1, 'SivaLabs', 'https://sivalabs.in', 1, CURRENT_TIMESTAMP),
       (2, 'Spring Initializr', 'https://start.spring.io', 2, CURRENT_TIMESTAMP),
       (3, 'Spring Blog', 'https://spring.io/blog', 2, CURRENT_TIMESTAMP)
;

insert into bookmark_tag(bookmark_id, tag_id)
VALUES (1, 1),
       (1, 2),
       (1, 3),
       (2, 2)
;
```

Let's create a simple test case to verify the above code.

```java
package com.sivalabs.bookmarks.repositories;

import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.jooq.JooqTest;
import org.springframework.boot.testcontainers.service.connection.ServiceConnection;
import org.springframework.context.annotation.Import;
import org.springframework.test.context.jdbc.Sql;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;

import static org.assertj.core.api.Assertions.assertThat;

@JooqTest
@Import({UserRepository.class})
@Testcontainers
@Sql("classpath:/test-data.sql")
class UserRepositoryTest {

    @Container
    @ServiceConnection
    static final PostgreSQLContainer<?> postgres =
            new PostgreSQLContainer<>("postgres:16-alpine");

    @Autowired
    UserRepository userRepository;

    @Test
    void findUserNameById() {
        String username = userRepository.findUserNameById(1L);
        assertThat(username).isEqualTo("Admin");
    }
}
```

We are using **@JooqTest** slice test annotation to test our repository class.
We are using **@Testcontainers** and **@Container** annotations to start a PostgreSQL database and 
registering the DataSource properties using [ServiceConnection](https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.testing.testcontainers.service-connections) support.
Also, we are executing the **test-data.sql** script using **@Sql** annotation.

In our test, we are simply calling **userRepository.findUserNameById(1L)** and verifying the result.

{{< box tip >}}
**Logging SQL queries executed by jOOQ**

You can enable logging of SQL queries executed by jOOQ by adding the following property to **application.properties** file.

**logging.level.org.jooq.tools.LoggerListener=DEBUG**
{{< /box >}}

When we printed the jOOQ Record, we can see the following nicely formatted output in the console:

```shell
+----+-----+---------------+--------+--------------+--------------------------------+----------+
|  id|name |email          |password|preferences_id|created_at                      |updated_at|
+----+-----+---------------+--------+--------------+--------------------------------+----------+
|   1|Admin|admin@gmail.com|admin   |             1|2023-10-12T11:01:58.471277+05:30|{null}    |
+----+-----+---------------+--------+--------------+--------------------------------+----------+
```

So, we can use jOOQ to run native queries.

But, in the current implementation, you won't get any compilation error if you pass some string as **"id"** value. 
Also, you are getting the **"name"** value as **Object** type, and you need to convert it to desired type **String**.

This is where jOOQ's **Typesafe DSL** comes into the picture.
In order to use jOOQ Typesafe DSL, first we need to generate the jOOQ classes from our database schema.

## jOOQ Code Generation
We can use the [jOOQ Code Generation](https://www.jooq.org/doc/latest/manual/code-generation/) tool to generate the jOOQ classes from our database schema.
But to use the jOOQ Code Generation tool, we need to have an existing database up and running.

We can use the [Testcontainers](https://www.testcontainers.org/) to start a PostgreSQL database container and 
then we can use the jOOQ Code Generation tool to generate the jOOQ classes. 

There is [testcontainers-jooq-codegen-maven-plugin](https://github.com/testcontainers/testcontainers-jooq-codegen-maven-plugin) 
which can start a database container, run Flyway migrations and then generate jOOQ code.

Let's configure the **testcontainers-jooq-codegen-maven-plugin** in our **pom.xml** file as follows:

```xml
<properties>
    <testcontainers.version>1.19.1</testcontainers.version>
    <tc-jooq-codegen-plugin.version>0.0.3</tc-jooq-codegen-plugin.version>
</properties>

<build>
    <plugins>
        <plugin>
            <groupId>org.testcontainers</groupId>
            <artifactId>testcontainers-jooq-codegen-maven-plugin</artifactId>
            <version>${tc-jooq-codegen-plugin.version}</version>
            <dependencies>
                <dependency>
                    <groupId>org.testcontainers</groupId>
                    <artifactId>postgresql</artifactId>
                    <version>${testcontainers.version}</version>
                </dependency>
                <dependency>
                    <groupId>org.postgresql</groupId>
                    <artifactId>postgresql</artifactId>
                    <version>${postgresql.version}</version>
                </dependency>
            </dependencies>
            <executions>
                <execution>
                    <id>generate-jooq-sources</id>
                    <goals>
                        <goal>generate</goal>
                    </goals>
                    <phase>generate-sources</phase>
                    <configuration>
                        <database>
                            <type>POSTGRES</type>
                            <containerImage>postgres:16-alpine</containerImage>
                        </database>
                        <flyway>
                            <locations>
                                filesystem:${project.basedir}/src/main/resources/db/migration
                            </locations>
                        </flyway>
                        <jooq>
                            <generator>
                                <generate>
                                    <javaTimeTypes>true</javaTimeTypes>
                                </generate>
                                <database>
                                    <inputSchema>public</inputSchema>
                                    <includes>.*</includes>
                                    <excludes>
                                        flyway_schema_history
                                    </excludes>
                                </database>
                                <target>
                                    <clean>true</clean>
                                    <packageName>com.sivalabs.bookmarks.jooq</packageName>
                                    <directory>src/main/jooq</directory>
                                </target>
                            </generator>
                        </jooq>
                    </configuration>
                </execution>
            </executions>
        </plugin>

        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>build-helper-maven-plugin</artifactId>
            <executions>
                <execution>
                    <phase>generate-sources</phase>
                    <goals>
                        <goal>add-source</goal>
                    </goals>
                    <configuration>
                        <sources>
                            <source>src/main/jooq</source>
                        </sources>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>
```

We have configured the **testcontainers-jooq-codegen-maven-plugin** to generate jOOQ code in the **src/main/jooq** directory.
Then we have used the **build-helper-maven-plugin** to add the **src/main/jooq** directory as a source directory.

{{< box info >}}
**Should I check-in generated jOOQ code?**

You can also generate code in the **target/generated-sources/jooq** directory which will be automatically added as a source by Maven.
Then you don't need to use the **build-helper-maven-plugin**.

But, I personally prefer to generate code in the **src/main/jooq** directory and check-in the generated code into the source control.
{{< /box >}}

Now we can run the following command to generate the jOOQ code.

```shell
$ ./mvnw clean generate-sources
```

## Using jOOQ Typesafe DSL
Now let's rewrite our **UserRepository** class to use jOOQ Typesafe DSL.

```java
package com.sivalabs.bookmarks.repositories;

import com.sivalabs.bookmarks.jooq.tables.records.UsersRecord;
import org.jooq.DSLContext;
import org.springframework.stereotype.Repository;

import static com.sivalabs.bookmarks.jooq.tables.Users.USERS;

@Repository
class UserRepository {
    private final DSLContext dsl;

    UserRepository(DSLContext dsl) {
        this.dsl = dsl;
    }

    public String findUserNameById(Long id) {
        UsersRecord usersRecord = dsl.selectFrom(USERS)
                .where(USERS.ID.eq(id))
                .fetchOptional().orElseThrow();
        return usersRecord.getName();
    }
}
```

As you can see, we are using the **UsersRecord** class generated by the jOOQ Code Generation tool.
Now, we will get a compilation error if we pass any other type than **Long** for the **"id"** parameter.
Also, we are getting the **"name"** value as **String** type.

So, jOOQ typesafe DSL is very useful for writing typesafe SQL queries.

## Conclusion
In this first part, we have set the stage for exploring jOOQ.
In the next part, we will learn how to implement basic CRUD operations using jOOQ.

# Spring Security OAuth 2 Tutorial - 10 : Service to Service Communication using Client Credentials Flow


In this article, we will learn how to implement **Service to Service Communication** using **Client Credentials Flow**.
We will create the **archival-service** in which we will use a scheduler job to invoke the **messages-service** APIs to archive the messages.
For implementing this, we will use **Client Credentials Flow**.

<!--more-->


We will also implement **POST /api/messages/archive** API endpoint in **archival-service** which can only be called by users who have **ROLE_ADMIN** role.

Considering this, **archival-service** will act as a **Resource Server** and as a **Client** too.

* **Resource Server** - Exposes **POST /api/messages/archive** API endpoint which will be called from **messages-webapp**.
* **Client** - Invokes **messages-service** APIs to archive the messages.

{{< box info >}}
**Source Code:**

You can find the complete source code of this project on GitHub: 
https://github.com/sivaprasadreddy/spring-security-oauth2-microservices-demo
{{< /box >}}

## Create archival-service Client with Client Credentials Flow enabled in Keycloak

Let's create a new client named **archival-service**.

* **General Settings**:
    * **Client type**: OpenID Connect
    * **Client ID**: archival-service

* **Capability config**:
    * **Client authentication**: On
    * **Authorization**: Off
    * **Authentication flow**: Check **Service accounts roles** On and uncheck the rest of the checkboxes

* **Login settings**:
    * **Root URL**: http://localhost:8282
    * **Home URL**: http://localhost:8282

Once the client is created with the above configuration, you will be taken to the newly created Client's **Settings** page.

* Go to Service account roles tab and assign **ROLE_ADMIN** role.
* Click on the **Credentials** tab and copy the **Client secret** value.

In my case, the **Client secret** is **bL1a2V2kouKh4sBMX0UrSmc0d3qubD1a**.

## Create archival-service
You can generate **archival-service** using **Spring Initializr** by clicking on this [link](https://start.spring.io/#!type=maven-project&language=java&platformVersion=3.1.4&packaging=jar&jvmVersion=17&groupId=com.sivalabs&artifactId=archival-service&name=archival-service&packageName=com.sivalabs.archival&dependencies=web,validation,security,oauth2-resource-server,oauth2-client).
We have selected the starters **Web**, **Validation**, **Security**, **OAuth2 Client** and **OAuth2 Resource Server**.
Once the application is generated, open it in your favourite IDE.

Open **application.properties** and configure the following properties:

```properties
spring.application.name=archival-service
server.port=8282
OAUTH_SERVER=http://localhost:9191/realms/sivalabs

# Resource Server configuration
spring.security.oauth2.resourceserver.jwt.issuer-uri=${OAUTH_SERVER}

# Client configuration
spring.security.oauth2.client.registration.archival-service.provider=archival-service
spring.security.oauth2.client.registration.archival-service.client-id=archival-service
spring.security.oauth2.client.registration.archival-service.client-secret=bL1a2V2kouKh4sBMX0UrSmc0d3qubD1a
spring.security.oauth2.client.registration.archival-service.authorization-grant-type=client_credentials
spring.security.oauth2.client.registration.archival-service.scope=openid, profile
spring.security.oauth2.client.registration.archival-service.redirect-uri={baseUrl}/login/oauth2/code/archival-service

spring.security.oauth2.client.provider.archival-service.issuer-uri=${OAUTH_SERVER}
```

This configuration should be familiar to you if you have read the previous articles in this series.

* We have configured the Resource Server property **spring.security.oauth2.resourceserver.jwt.issuer-uri** to point to the Keycloak server.
* Next, we have configured the Client properties (**spring.security.oauth2.client.registration.archival-service**, 
**spring.security.oauth2.client.provider.archival-service.issuer-uri**) to point to the Keycloak server.

## Get Access Token using Client Credentials Flow
Now that we have configuration in place, let's see how to get the **access_token** using **Client Credentials Flow**.

Create **SecurityConfig** class with the following content:

```java
package com.sivalabs.archival.config;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.security.oauth2.client.AuthorizedClientServiceOAuth2AuthorizedClientManager;
import org.springframework.security.oauth2.client.OAuth2AuthorizedClientManager;
import org.springframework.security.oauth2.client.OAuth2AuthorizedClientService;
import org.springframework.security.oauth2.client.registration.ClientRegistrationRepository;

@Configuration
public class SecurityConfig {

  @Bean
  public OAuth2AuthorizedClientManager authorizedClientManager(
          ClientRegistrationRepository clientRegistrationRepository,
          OAuth2AuthorizedClientService authorizedClientService) {

    return new AuthorizedClientServiceOAuth2AuthorizedClientManager(
            clientRegistrationRepository, authorizedClientService);
  }
}
```

We are registering a bean of type **OAuth2AuthorizedClientManager** by injecting the auto-configured  
**ClientRegistrationRepository** and **OAuth2AuthorizedClientService** beans.
We will use **OAuth2AuthorizedClientManager** to get the **access_token**.

Create **SecurityHelper** class with the following content:

```java
package com.sivalabs.archival.domain;

import org.springframework.security.oauth2.client.OAuth2AuthorizeRequest;
import org.springframework.security.oauth2.client.OAuth2AuthorizedClient;
import org.springframework.security.oauth2.client.OAuth2AuthorizedClientManager;
import org.springframework.security.oauth2.core.OAuth2AccessToken;
import org.springframework.stereotype.Service;

@Service
public class SecurityHelper {
    private final OAuth2AuthorizedClientManager authorizedClientManager;

    public SecurityHelper(OAuth2AuthorizedClientManager authorizedClientManager) {
        this.authorizedClientManager = authorizedClientManager;
    }

    public OAuth2AccessToken getOAuth2AccessToken() {
        String clientRegistrationId = "archival-service";
        OAuth2AuthorizeRequest authorizeRequest =
                OAuth2AuthorizeRequest.withClientRegistrationId(clientRegistrationId)
                        // This principal value is unnecessary, but if you don't give it a value,
                        // it throws an exception.
                        .principal("dummy")
                        .build();
        OAuth2AuthorizedClient authorizedClient =
                this.authorizedClientManager.authorize(authorizeRequest);
        return authorizedClient.getAccessToken();
    }
}
```

We are using the client registration id **archival-service** to create the **OAuth2AuthorizeRequest**.
Then we are invoking **authorize()** method on **OAuth2AuthorizedClientManager** to get the **OAuth2AuthorizedClient**, 
which internally performs the authentication.
Finally, we are returning the **OAuth2AccessToken** from the **OAuth2AuthorizedClient**.

Now, we can use this token to invoke the **messages-service** APIs.

## Create MessageServiceClient

Create **MessageServiceClient** class with the following content:

```java
package com.sivalabs.archival.domain;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.core.ParameterizedTypeReference;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.ResponseEntity;
import org.springframework.security.oauth2.core.OAuth2AccessToken;
import org.springframework.stereotype.Service;
import org.springframework.web.client.RestTemplate;

@Service
public class MessageServiceClient {
    private static final Logger log = LoggerFactory.getLogger(MessageServiceClient.class);
    private static final String MESSAGES_SVC_URL = "http://localhost:8181";

    private final SecurityHelper securityHelper;
    private final RestTemplate restTemplate;

    public MessageServiceClient(SecurityHelper securityHelper, RestTemplate restTemplate) {
        this.securityHelper = securityHelper;
        this.restTemplate = restTemplate;
    }

    public void archiveMessages() {
        try {
            String url = MESSAGES_SVC_URL + "/api/messages/archive";
            OAuth2AccessToken oAuth2AccessToken = securityHelper.getOAuth2AccessToken();
            String accessToken = oAuth2AccessToken.getTokenValue();

            HttpHeaders headers = new HttpHeaders();
            headers.add("Authorization", "Bearer " + accessToken);
            HttpEntity<?> httpEntity = new HttpEntity<>(headers);
            ResponseEntity<Void> response = restTemplate.exchange(
                    url, HttpMethod.POST, httpEntity,
                    new ParameterizedTypeReference<>() {});
            log.info("Archive messages response code: {}", response.getStatusCode());
        } catch (Exception e) {
            log.error("Error while invoking Archive messages API", e);
        }
    }
}
```

We are getting the **accessToken** from **SecurityHelper** and adding it to the **Authorization** header.
Then we are invoking the **POST /api/messages/archive** API endpoint using **RestTemplate**.

## Implement Scheduler Job to Archive Messages
Spring Boot provides **@Scheduled** annotation to implement scheduled jobs.
First, we need to enable scheduling by adding **@EnableScheduling** annotation to the **ArchivalServiceApplication** class.

Then we can create a scheduled job to archive the messages by adding **@Scheduled** annotation to a method as follows:

```java
package com.sivalabs.archival.jobs;

import com.sivalabs.archival.domain.MessageServiceClient;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;

import java.time.Instant;

@Service
public class MessageArchivalJob {
    private static final Logger log = LoggerFactory.getLogger(MessageArchivalJob.class);

    private final MessageServiceClient messageServiceClient;

    public MessageArchivalJob(MessageServiceClient messageServiceClient) {
        this.messageServiceClient = messageServiceClient;
    }

    @Scheduled(fixedDelay = 30000)
    public void run() {
        log.info("Running MessageArchivalJob at {}", Instant.now());
        messageServiceClient.archiveMessages();
    }
}
```

We have configured the job to run every 30 seconds.

Now if you start the **archival-service** and **messages-service** , you should see the following logs:

```shell
Running MessageArchivalJob at 2023-09-29T14:48:11.606017Z
Archive messages response code: 200 OK
```

In the **messages-service** logs, you should see the following logs:

```shell
Archiving all messages
```

We haven't actually implemented the logic of archiving the messages, because that is not the focus of the article.
The important thing is we are able to get AccessToken using Client Credentials Flow and invoke the **messages-service** APIs.

{{< box info >}}
**Is it good using ROLE_ADMIN for Client Credentials Flow?**

In this article, we have assigned **ROLE_ADMIN** role to **archival-service** client and 
be able to invoke the **POST /api/messages/archive** API endpoint of **messages-service** 
which can only be allowed by users having **ROLE_ADMIN** role.

While technically, this works, it is not a good idea to use **ROLE_ADMIN** for Client Credentials Flow.
Instead, we should create a new role like **ROLE_ADMIN_JOB**, assign it to **archival-service** client 
and configure **messages-service** **POST /api/messages/archive** API endpoint to be accessible by users having 
either **ROLE_ADMIN** or **ROLE_ADMIN_JOB**.
{{< /box >}}

## Implement Archive Messages API Endpoint in archival-service
The last thing we want to implement is the **POST /api/messages/archive** API endpoint in **archival-service**.

Create **MessageArchivalController** class with the following content:

```java
package com.sivalabs.archival.api;

import com.sivalabs.archival.domain.MessageServiceClient;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.Map;

@RestController
class MessageArchivalController {
    private final MessageServiceClient messageServiceClient;

    MessageArchivalController(MessageServiceClient messageServiceClient) {
        this.messageServiceClient = messageServiceClient;
    }

    @PostMapping("/api/messages/archive")
    Map<String, String> archiveMessages() {
        messageServiceClient.archiveMessages();
        return Map.of("status", "success");
    }
}
```

Nothing special about this controller, we are just invoking **messageServiceClient.archiveMessages()** method.
But, we need to protect this API endpoint to be accessible only by users having **ROLE_ADMIN** role.

Update **SecurityConfig** class as follows:

```java
package com.sivalabs.archival.config;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.HttpMethod;
import org.springframework.security.config.Customizer;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.configurers.CorsConfigurer;
import org.springframework.security.config.annotation.web.configurers.CsrfConfigurer;
import org.springframework.security.config.http.SessionCreationPolicy;
import org.springframework.security.oauth2.client.AuthorizedClientServiceOAuth2AuthorizedClientManager;
import org.springframework.security.oauth2.client.OAuth2AuthorizedClientManager;
import org.springframework.security.oauth2.client.OAuth2AuthorizedClientService;
import org.springframework.security.oauth2.client.registration.ClientRegistrationRepository;
import org.springframework.security.web.SecurityFilterChain;

@Configuration
public class SecurityConfig {
    @Bean
    SecurityFilterChain securityFilterChain(HttpSecurity http) throws Exception {
        http
            .authorizeHttpRequests(c ->
                c
                    .requestMatchers(HttpMethod.POST, "/api/messages/archive").hasRole("ADMIN")
                    .anyRequest().authenticated()
            )
            .sessionManagement(c -> c.sessionCreationPolicy(SessionCreationPolicy.STATELESS))
            .cors(CorsConfigurer::disable)
            .csrf(CsrfConfigurer::disable)
            .oauth2ResourceServer(oauth2 -> oauth2
                .jwt(jwt -> jwt.jwtAuthenticationConverter(new KeycloakJwtAuthenticationConverter()))
            );

        return http.build();
    }

    @Bean
    public OAuth2AuthorizedClientManager authorizedClientManager(
            ClientRegistrationRepository clientRegistrationRepository,
            OAuth2AuthorizedClientService authorizedClientService) {

        return new AuthorizedClientServiceOAuth2AuthorizedClientManager(
                        clientRegistrationRepository, authorizedClientService);
    }
}
```

This configuration should be familiar to you if you have read the previous articles in this series.
It is similar to the **messages-service** configuration, except we have configured the **/api/messages/archive** API endpoint to be accessible only by users having **ROLE_ADMIN** role.
We are also using **KeycloakJwtAuthenticationConverter** to convert the **realm_access.roles** to **GrantedAuthority**.
You can copy the same class from **messages-service** to **archival-service**.

## Invoke Archive Messages API Endpoint from messages-webapp
Now that we have implemented the **POST /api/messages/archive** API endpoint in **archival-service**,
we can invoke this API endpoint from **messages-webapp**.

Add **archiveMessages()** method in **MessageServiceClient** of **messages-webapp** as follows:

```java
@Service
public class MessageServiceClient {
    //...
  
    public void archiveMessages() {
        try {
          HttpHeaders headers = new HttpHeaders();
          headers.add("Authorization", "Bearer " + securityHelper.getAccessToken());
          HttpEntity<?> httpEntity = new HttpEntity<>(headers);
          ResponseEntity<Message> response = restTemplate.exchange(
                  "http://localhost:8282/api/messages/archive", HttpMethod.POST, httpEntity,
                  new ParameterizedTypeReference<>() {
                  });
          log.info("Archive messages response code: {}", response.getStatusCode());
        } catch (Exception e) {
          log.error("Error while invoking Archive messages", e);
        }
    }
}
```

Add **archiveMessages()** handler method in **HomeController** of **messages-webapp** as follows:

```java
@Controller
public class HomeController {
    //...
    
    @PostMapping("/messages/archive")
    String archiveMessages() {
        messageServiceClient.archiveMessages();
        return "redirect:/";
    }
}
```

Finally, add **Archive Messages** button in **home.html** as follows:

```html
<div sec:authorize="hasRole('ADMIN')">
    <form method="post" action="/messages/archive">
      <input type="submit" value="Archive Messages">
    </form>
</div>
```

Now, if you run all the services and login to **messages-webapp** with a user having **ROLE_ADMIN** assigned, you should see the **Archive Messages** button.
Upon clicking on this button, it should invoke the **POST /api/messages/archive** API endpoint of **archival-service**,
which internally should invoke the **POST /api/messages/archive** API endpoint of **messages-service**.


## Conclusion
In this Spring Security OAuth2 Tutorial series, we have learned:
* Various OAuth2 / OpenID Connect Flows
* How to implement OAuth2 / OpenID Connect Flows using Keycloak and Spring Boot

I hope this series is helpful to understand how OAuth 2.0 works and how to implement it using Spring Boot and Keycloak.
# Spring Security OAuth 2 Tutorial - 9 : Invoking Secured Resource Server APIs from Client Application


In the previous articles, we have created **messages-webapp** and **messages-service** and invoked API endpoints using Postman.
In this article, we will learn how to invoke the secured **messages-service** API endpoints from the Client application **messages-webapp**.

<!--more-->


{{< box info >}}
**Source Code:**

You can find the complete source code of this project on GitHub: 
https://github.com/sivaprasadreddy/spring-security-oauth2-microservices-demo
{{< /box >}}

## Show List of Messages
As **GET /api/messages** API endpoint in **messages-service** is publicly accessible, we can invoke it from **messages-webapp** without any authentication.

{{< box tip >}}
**RestTemplate vs RestClient**

I am going to use the good old **RestTemplate** to invoke the API endpoints in **messages-service**.
But once Spring Boot 3.2.0 is released, I would recommend to use **RestClient** instead.
{{< /box >}}

In **messages-webapp**, create **AppConfig** class with the following content:

```java
package com.sivalabs.messages.config;

import org.springframework.boot.web.client.RestTemplateBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.client.RestTemplate;

@Configuration
public class AppConfig {

    @Bean
    public RestTemplate restTemplate(RestTemplateBuilder builder) {
        return builder.build();
    }
}
```
We have registered a **RestTemplate** bean so that we can inject it into other components.

Create **SecurityHelper** class with the following content:

```java
package com.sivalabs.messages.domain;

import org.springframework.stereotype.Service;

@Service
public class SecurityHelper {

    public String getAccessToken() {
        String accessToken = null;
        // logic to get Access token
        return accessToken;
    }
}
```

We have just created a placeholder method **getAccessToken()** to return the **accessToken**, which we are going to implement later.

Create **Message** class with the following content:

```java
package com.sivalabs.messages.domain;

import jakarta.validation.constraints.NotEmpty;

import java.time.Instant;

public class Message {
    private Long id;
    @NotEmpty
    private String content;
    @NotEmpty
    private String createdBy;
    private Instant createdAt;

    // constructors
    // setters and getters
}
```

Create **MessageServiceClient** class with the following content:

```java
package com.sivalabs.messages.domain;

import com.sivalabs.messages.domain.Message;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.core.ParameterizedTypeReference;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Service;
import org.springframework.web.client.RestTemplate;

import java.util.List;

@Service
public class MessageServiceClient {
    private static final Logger log = LoggerFactory.getLogger(MessageServiceClient.class);
    private static final String MESSAGE_SVC_BASE_URL = "http://localhost:8181";

    private final SecurityHelper securityHelper;
    private final RestTemplate restTemplate;

    public MessageServiceClient(SecurityHelper securityHelper, RestTemplate restTemplate) {
        this.securityHelper = securityHelper;
        this.restTemplate = restTemplate;
    }

    public List<Message> getMessages() {
        try {
            String url = MESSAGE_SVC_BASE_URL + "/api/messages";
            ResponseEntity<List<Message>> response = restTemplate.exchange(
                    url, HttpMethod.GET, null,
                    new ParameterizedTypeReference<>() {});
            return response.getBody();
        } catch (Exception e) {
            log.error("Error while fetching messages", e);
            return List.of();
        }
    }

    public void createMessage(Message message) {
        try {
            String url = MESSAGE_SVC_BASE_URL + "/api/messages";
            HttpHeaders headers = new HttpHeaders();
            headers.add("Authorization", "Bearer " + securityHelper.getAccessToken());
            HttpEntity<?> httpEntity = new HttpEntity<>(message, headers);
            ResponseEntity<Message> response = restTemplate.exchange(
                    url, HttpMethod.POST, httpEntity,
                    new ParameterizedTypeReference<>() {});
            log.info("Create message response code: {}", response.getStatusCode());
        } catch (Exception e) {
            log.error("Error while creating message", e);
        }
    }
}
```

Now, update **HomeController** to fetch the list of messages from **messages-service** and display them on the home page.

```java
package com.sivalabs.messages.web;

import com.sivalabs.messages.domain.MessageServiceClient;
import com.sivalabs.messages.domain.Message;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PostMapping;

import java.util.List;

@Controller
public class HomeController {
    private static final Logger log = LoggerFactory.getLogger(HomeController.class);

    private final MessageServiceClient messageServiceClient;

    public HomeController(MessageServiceClient messageServiceClient) {
        this.messageServiceClient = messageServiceClient;
    }

    @GetMapping("/")
    public String home(Model model, @AuthenticationPrincipal OAuth2User principal) {
        if(principal != null) {
          model.addAttribute("username", principal.getAttribute("name"));
        } else {
          model.addAttribute("username", "Guest");
        }
        List<Message> messages = messageServiceClient.getMessages();
        log.info("Message count: {}", messages.size());
        model.addAttribute("messages", messages);
        return "home";
    }
}
```

Update **home.html** to display the list of messages as follows:

```html
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml"
      xmlns:th="http://www.thymeleaf.org"
      xmlns:sec="http://www.thymeleaf.org/thymeleaf-extras-springsecurity">
<head>
    <title>Home</title>
</head>
<body>
<div>
    <p sec:authorize="!isAuthenticated()">
      <a href="/oauth2/authorization/messages-webapp">Login</a>
    </p>
    <h1>Welcome <span th:text="${username}">username</span></h1>

    <div id="messages" class="pt-2">
        <div class="message" th:each="message: ${messages}">
            <div class="alert alert-light" role="alert">
                <p th:text="${message.content}">content</p>
                <p>Posted By: <span th:text="${message.createdBy}">CreatedBy</span></p>
            </div>
        </div>
    </div>
</div>
</body>
</html>
```

Now, if you start the **messages-service** and **messages-webapp** and access http://localhost:8080, you should be able to see the list of messages.

Before moving to implementing the creating a new message functionality, let's see how to get the **access_token** to invoke **POST /api/messages** API endpoint of **messages-service**.

## Getting Access Token
We can get the current user details from **SecurityContextHolder** and extract various interesting information from it.

Let's implement the **getAccessToken()** method in **SecurityHelper** class as follows:

```java
package com.sivalabs.messages.domain;

import org.springframework.security.core.Authentication;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.security.oauth2.client.OAuth2AuthorizedClient;
import org.springframework.security.oauth2.client.OAuth2AuthorizedClientService;
import org.springframework.security.oauth2.client.authentication.OAuth2AuthenticationToken;
import org.springframework.stereotype.Service;

@Service
public class SecurityHelper {
    private final OAuth2AuthorizedClientService authorizedClientService;
  
    public SecurityHelper(OAuth2AuthorizedClientService authorizedClientService) {
      this.authorizedClientService = authorizedClientService;
    }
    
    public String getAccessToken() {
        Authentication authentication = SecurityContextHolder.getContext().getAuthentication();
        if(!(authentication instanceof OAuth2AuthenticationToken oauthToken)) {
          return null;
        }
        OAuth2AuthorizedClient client = authorizedClientService.loadAuthorizedClient(
                oauthToken.getAuthorizedClientRegistrationId(), oauthToken.getName());
  
        return client.getAccessToken().getTokenValue();
    }
}
```

We have injected the auto-configured **OAuth2AuthorizedClientService** bean, and we are loading the current AuthorizedClient, 
and then acquiring the accessToken.

## Create a New Message
Now that we have implemented the **getAccessToken()** method, **MessageServiceClient** already invokes the **POST /api/messages** API endpoint with the **access_token**,
let's add Create a New Message form in **home.html** and implement the handler in **HomeController**.

```html
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml"
      xmlns:th="http://www.thymeleaf.org"
      xmlns:sec="http://www.thymeleaf.org/thymeleaf-extras-springsecurity">
<head>
    <title>Home</title>
</head>
<body>
<div>
    <p sec:authorize="!isAuthenticated()">
      <a href="/oauth2/authorization/messages-webapp">Login</a>
    </p>
    <h1>Welcome <span th:text="${username}">username</span></h1>

    <div sec:authorize="isAuthenticated()">
      <div class="card">
        <div class="card-body">
          <form method="post" action="/messages">
            <div class="mb-3">
              <label for="content" class="form-label">Message</label>
              <textarea class="form-control" id="content" name="content"></textarea>
            </div>
            <button type="submit" class="btn btn-primary">Submit</button>
          </form>
        </div>
      </div>
    </div>
  
    <div id="messages" class="pt-2">
        <!-- Display messages -->
    </div>
    
</div>
</body>
</html>
```

We are showing the **Create New Message** form only if the user is authenticated.

In order to create a new message, we need the current user details like **username**.
Let's add a utility method in **SecurityHelper** to get the current user details.

```java
package com.sivalabs.messages.domain;

import org.springframework.security.core.Authentication;
import org.springframework.security.core.GrantedAuthority;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.security.oauth2.client.OAuth2AuthorizedClient;
import org.springframework.security.oauth2.client.OAuth2AuthorizedClientService;
import org.springframework.security.oauth2.client.authentication.OAuth2AuthenticationToken;
import org.springframework.security.oauth2.core.oidc.OidcUserInfo;
import org.springframework.security.oauth2.core.oidc.user.DefaultOidcUser;
import org.springframework.stereotype.Service;

import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

@Service
public class SecurityHelper {

    private final OAuth2AuthorizedClientService authorizedClientService;

    public SecurityHelper(OAuth2AuthorizedClientService authorizedClientService) {
        this.authorizedClientService = authorizedClientService;
    }

    public static Map<String, Object> getLoginUserDetails() {
        Map<String, Object> map = new HashMap<>();
        Authentication authentication = SecurityContextHolder.getContext().getAuthentication();
        if(!(authentication instanceof OAuth2AuthenticationToken)) {
            return null;
        }
        DefaultOidcUser principal = (DefaultOidcUser) authentication.getPrincipal();
        Collection<? extends GrantedAuthority> authorities = authentication.getAuthorities();
        List<String> roles = authorities.stream().map(GrantedAuthority::getAuthority).toList();
        OidcUserInfo userInfo = principal.getUserInfo();
        
        map.put("id", userInfo.getSubject());
        map.put("fullName", userInfo.getFullName());
        map.put("email", userInfo.getEmail());
        map.put("username", userInfo.getPreferredUsername());
        map.put("roles", roles);
        
        return map;
    }

    // other code omitted for brevity
}
```

As we are using OpenID Connect, the authenticated user principal is of type **DefaultOidcUser**.
We are extracting the user details from **DefaultOidcUser** and returning them as a **Map**.
To keep it simple, we are not using any DTO class to represent the user details, instead we are using a **Map**.

Add the following handler method in **HomeController** to create a new message:

```java
@Controller
public class HomeController {
    private final MessageServiceClient messageServiceClient;
    private final SecurityHelper securityHelper;
    
    // other code omitted for brevity
  
    @PostMapping("/messages")
    String createMessage(Message message) {
        Map<String, Object> loginUserDetails = SecurityHelper.getLoginUserDetails();
        message.setCreatedBy(loginUserDetails.get("username").toString());
        messageServiceClient.createMessage(message);
        return "redirect:/";
    }
}
```

Now, if you restart the **messages-webapp** and login into the application, you should be able to create a new message.

## Role Based Access Control
In the previous section, if we had printed the **loginUserDetails** map, we would have seen the following output:

```shell
{ 
  id = "ca1a2f34-1614-45dd-86c1-5eafff085d8a", 
  fullName = "Siva Katamreddy", 
  email = "siva@gmail.com", 
  username = "siva",
  roles = [
    OIDC_USER, 
    SCOPE_email, 
    SCOPE_openid, 
    SCOPE_profile
  ]
}
```
The **ROLE_ADMIN** and **ROLE_USER** are not listed under **roles**, just like we have noticed for the **messages-service** in the previous article.

In order to get the assigned roles as part of the claims, we need to update a setting in Keycloak.

* Go to Keycloak Admin Console -> Select **sivalabs** realm 
* **Client scopes** -> **roles** -> **Mappers** -> **realm_roles** -> **Add to ID token** -> **ON**.

Now the assigned roles will be placed inside the claims with the key **real_access**.

Similar to how we implemented a custom **JwtTokenConverter** in **messages-service**, 
we can implement a custom **GrantedAuthoritiesMapper** in **messages-webapp** to extract the roles from **real_access** claim.

Create **KeycloakAuthoritiesMapper** class with the following content:

```java
package com.sivalabs.messages.config;

import org.springframework.security.core.GrantedAuthority;
import org.springframework.security.core.authority.SimpleGrantedAuthority;
import org.springframework.security.core.authority.mapping.GrantedAuthoritiesMapper;
import org.springframework.security.oauth2.core.oidc.OidcIdToken;
import org.springframework.security.oauth2.core.oidc.user.OidcUserAuthority;
import org.springframework.security.oauth2.core.user.OAuth2UserAuthority;

import java.util.Collection;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;

class KeycloakAuthoritiesMapper implements GrantedAuthoritiesMapper {

    @Override
    public Collection<? extends GrantedAuthority> mapAuthorities(
            Collection<? extends GrantedAuthority> authorities) {
        Set<GrantedAuthority> mappedAuthorities = new HashSet<>();

        authorities.forEach(authority -> {
            if (authority instanceof SimpleGrantedAuthority) {
                mappedAuthorities.add(authority);
            }
            else if (authority instanceof OidcUserAuthority oidcUserAuthority) {
                OidcIdToken idToken = oidcUserAuthority.getIdToken();
                Map<String, Object> claims = idToken.getClaims();
                Map<String,Object> realm_access = (Map<String, Object>) claims.get("realm_access");
                if(realm_access != null && !realm_access.isEmpty()) {
                    List<String> roles = (List<String>) realm_access.get("roles");
                    var list = roles.stream()
                            .filter(role -> role.startsWith("ROLE_"))
                            .map(SimpleGrantedAuthority::new).toList();
                    mappedAuthorities.addAll(list);
                }
            } else if (authority instanceof OAuth2UserAuthority oauth2UserAuthority) {
                Map<String, Object> userAttributes = oauth2UserAuthority.getAttributes();
                // Map the attributes found in userAttributes
                // to one or more GrantedAuthority's and add it to mappedAuthorities
            }
        });
        return mappedAuthorities;
    }
}
```

We are checking the authority type and extracting the roles from **realm_access** claim.
As we are using OpenID Connect flow, authority is of type **OidcUserAuthority**.
If we were using OAuth 2.0 flow, authority would be of type **OAuth2UserAuthority**.

We have extracted the **roles** from **realm_access** claim and mapped them to **SimpleGrantedAuthority**.

Now, update **SecurityConfig** in **messages-webapp** to use this custom **GrantedAuthoritiesMapper** as follows:

```java
package com.sivalabs.messages.config;


@Configuration
public class SecurityConfig {
    private final ClientRegistrationRepository clientRegistrationRepository;

    public SecurityConfig(ClientRegistrationRepository clientRegistrationRepository) {
        this.clientRegistrationRepository = clientRegistrationRepository;
    }

    @Bean
    SecurityFilterChain securityFilterChain(HttpSecurity http) throws Exception {
        http
            .authorizeHttpRequests(c ->
                c.requestMatchers("/").permitAll()
                 .anyRequest().authenticated()
            )
            .cors(CorsConfigurer::disable)
            .csrf(CsrfConfigurer::disable)
                //.oauth2Login(Customizer.withDefaults())
                .oauth2Login(oauth2 ->
                    oauth2.userInfoEndpoint(userInfo -> userInfo
                            .userAuthoritiesMapper(new KeycloakAuthoritiesMapper())))
            .logout(logout -> logout
                .logoutSuccessHandler(oidcLogoutSuccessHandler())
            );
        return http.build();
    }

    // other code omitted for brevity
}
```

Now if you restart the **messages-webapp** and **messages-service** and print the **loginUserDetails** map, 
you should see the following output:

```shell
{ 
  id = "ca1a2f34-1614-45dd-86c1-5eafff085d8a", 
  fullName = "Siva Katamreddy", 
  email = "siva@gmail.com", 
  username = "siva",
  roles = [
    SCOPE_email, 
    SCOPE_openid, 
    SCOPE_profile,
    ROLE_USER,
    ROLE_ADMIN
  ]
}
```

Now, we can use the **roles** to implement Role Based Access Control.

Let's update **home.html** to show the message **You are an ADMIN** only if the user has **ROLE_ADMIN** role.

```html
<h1>Welcome <span th:text="${username}">username</span></h1>
<div sec:authorize="isAuthenticated()">
    <div sec:authorize="hasRole('ADMIN')">
        <p>You are an ADMIN</p>
    </div>
    <div sec:authorize="!hasRole('ADMIN')">
        <p>You are NOT an ADMIN</p>
    </div>
</div>
<!-- other code omitted for brevity -->
```

Now if you login with a user who has **ROLE_ADMIN** assigned, you should see the message **You are an ADMIN**.
Otherwise, you should see the message **You are NOT an ADMIN**.

## Conclusion
In this article, we have learned how to invoke the **messages-service** Resource Server APIs from the **messages-webapp** Client application. 
We also learned how to customize the **GrantedAuthoritiesMapper** to convert roles as Authorities and implement Role Based Access Control. 

In the next article, we will create **archival-service** and learn how to invoke **messages-service** APIs using **Client Credentials Flow**.

# Spring Security OAuth 2 Tutorial - 8 : Securing Resource Server


In the previous article, we have created **messages-webapp** and secured it with **Spring Security OAuth 2.0** using Authorization Code Flow.
In this article, we will create **messages-service**, which is a Spring Boot Resource Server, and secure it with Spring Security OAuth 2.0.

<!--more-->


{{< box info >}}
**Source Code:**

You can find the complete source code of this project on GitHub: 
https://github.com/sivaprasadreddy/spring-security-oauth2-microservices-demo
{{< /box >}}

## Create messages-service
You can generate **messages-service** using **Spring Initializr** by clicking on this [link](https://start.spring.io/#!type=maven-project&language=java&platformVersion=3.1.4&packaging=jar&jvmVersion=17&groupId=com.sivalabs&artifactId=messages-service&name=messages-service&packageName=com.sivalabs.messages&dependencies=web,validation,security,oauth2-resource-server).
We have selected the starters **Web**, **Validation**, **Security** and **OAuth2 Resource Server**.
Once the application is generated, open it in your favourite IDE.

### Configure OAuth 2.0 Resource Server Properties
The **messages-service** is Resource Server of **bearer-only** type.
That means if anybody invokes a secured API endpoint with valid **access_token** sent as **Authorization** header, 
then the service will give the response. Otherwise, it will not initiate the OAuth 2.0 Authorization flow, 
but simply return 401 or 403 HTTP status code.

The **bearer-only** type Resource Server need not be registered with the Authorization Server (Keycloak).
We just need to configure the **issuer-uri** in **application.properties** file as follows:

```properties
spring.application.name=messages-service
server.port=8181
spring.security.oauth2.resourceserver.jwt.issuer-uri=http://localhost:9191/realms/sivalabs
```

### Implement API Endpoints
Let's implement our first API endpoint **/api/messages** which returns a list of messages.

Create **Message** class with the following content:

```java
package com.sivalabs.messages.domain;

import jakarta.validation.constraints.NotEmpty;

import java.time.Instant;

public class Message {
    private Long id;
    @NotEmpty
    private String content;
    @NotEmpty
    private String createdBy;
    private Instant createdAt;

    // constructors
    // setters and getters
}
```

Create **MessageRepository** class with the following content:

```java
package com.sivalabs.messages.repository;

import com.sivalabs.messages.model.Message;

import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.atomic.AtomicLong;

import jakarta.annotation.PostConstruct;
import org.springframework.stereotype.Repository;

@Repository
public class MessageRepository {
    private static final AtomicLong ID = new AtomicLong(0L);
    private static final List<Message> MESSAGES = new ArrayList<>();

    @PostConstruct
    void init() {
        getDefaultMessages().forEach( p -> {
            p.setId(ID.incrementAndGet());
            MESSAGES.add(p);
        });
    }

    public List<Message> getMessages() {
        return MESSAGES;
    }

    public Message createMessage(Message message) {
        message.setId(ID.incrementAndGet());
        message.setCreatedAt(Instant.now());
        MESSAGES.add(message);
        return message;
    }

    private List<Message> getDefaultMessages() {
        List<Message> messages = new ArrayList<>();
        messages.add(new Message(null, "Test Message 1", "admin", Instant.now()));
        messages.add(new Message(null, "Test Message 2", "admin", Instant.now()));
        return messages;
    }
}
```
Our focus is on securing the API endpoints using OAuth 2, so we are not using any database to store the messages.
Instead, we are using a simple in-memory list to store the messages.

Create **MessageController** class with the following content:

```java
package com.sivalabs.messages.api;

import com.sivalabs.messages.domain.Message;
import com.sivalabs.messages.domain.MessageRepository;
import jakarta.validation.Valid;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.List;

@RestController
@RequestMapping("/api/messages")
class MessageController {
    private final MessageRepository messageRepository;

    MessageController(MessageRepository messageRepository) {
        this.messageRepository = messageRepository;
    }

    @GetMapping
    List<Message> getMessages() {
        return messageRepository.getMessages();
    }

    @PostMapping
    Message createMessage(@RequestBody @Valid Message message) {
        return messageRepository.createMessage(message);
    }
}
```

We have implemented two API endpoints **GET /api/messages** to fetch all messages and **POST /api/messages** to create a new message.

Now if we try to access http://localhost:8181/api/messages we will get 401 HTTP status code as we have not sent any **access_token**.
By default, Spring Security secured all the endpoints.

## Accessing Secured API Endpoints using Postman
Remember, we have seen how to get **access_token** using Postman in the previous [article](https://www.sivalabs.in/spring-security-oauth2-tutorial-authorization-code-flow/#authorization-code-flow-using-postman).

Let's get the **access_token** using Postman and invoke the **GET /api/messages** API endpoint.

* Open a New Request tab in Postman
* Select **GET** as **HTTP Method** and enter the URL **http://localhost:8181/api/messages**
* Go to the **Authorization** tab and select **OAuth 2.0** as **Type**.
* Under **Configure New Token** section:
    * **Grant Type**: Authorization Code
    * **Callback URL**: http://localhost:8080/login/oauth2/code/messages-webapp
    * **Auth URL**: http://localhost:9191/realms/sivalabs/protocol/openid-connect/auth
    * **Access Token URL**: http://localhost:9191/realms/sivalabs/protocol/openid-connect/token
    * **Client ID**: messages-webapp
    * **Client Secret**: qVcg0foCUNyYbgF0Sg52zeIhLYyOwXpQ
    * **Scope**: openid profile
    * **State**: randomstring
    * **Client Authentication**: Send as Basic Auth header
* Click on **Get New Access Token** button
* Postman will open a popup showing Keycloak Login page
* Login with the user credentials **siva/siva1234**
* Now you should be able to see the response with **Token Details**
* Click on **Use Token** button
* Now, click on **Send** button to invoke the API endpoint

You should be able to see the response with the list of messages.

So, using Postman, we first got the **access_token** and then invoked the API endpoint by sending the **access_token** 
as **Authorization** header.

## Customizing Security Configuration
By default, Spring Security OAuth 2.0 Resource Server implementation will secure all the endpoints.
But, we want to allow access to the **GET /api/messages** API endpoint without authentication.

So, let's customize the security configuration using the following **SecurityConfig** class:

```java
package com.sivalabs.messages.config;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.HttpMethod;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;
import org.springframework.security.config.annotation.web.configurers.CorsConfigurer;
import org.springframework.security.config.annotation.web.configurers.CsrfConfigurer;
import org.springframework.security.config.http.SessionCreationPolicy;
import org.springframework.security.web.SecurityFilterChain;

@Configuration
@EnableWebSecurity
public class SecurityConfig {

    @Bean
    SecurityFilterChain securityFilterChain(HttpSecurity http) throws Exception {
        http
            .authorizeHttpRequests(c ->
                c.requestMatchers(HttpMethod.GET, "/api/messages").permitAll()
                 .anyRequest().authenticated()
            )
            .sessionManagement(c -> c.sessionCreationPolicy(SessionCreationPolicy.STATELESS))
            .cors(CorsConfigurer::disable)
            .csrf(CsrfConfigurer::disable)
            .oauth2ResourceServer(oauth2 ->
                oauth2.jwt(Customizer.withDefaults())
            );
        return http.build();
    }
}
```

As you can see, we have allowed access to the **GET /api/messages** API endpoint without authentication and 
secured all the rest of the endpoints. Also, we have configured the OAuth2 Resource Server to use **JWT** token based security 
with default configuration.

Now if you restart the application and access http://localhost:8181/api/messages you will be able to see the response without any authentication.
But, while invoking the **POST /api/messages** API endpoint, you need to configure **Authentication** as mentioned in the previous section.

{{< box info >}}
**How it works?**

We have configured the **issuer-uri**,http://localhost:9191/realms/sivalabs, in **application.properties** file.
When you start the application, Spring Security OAuth 2.0 uses the discovery endpoint (http://localhost:9191/realms/sivalabs/.well-known/openid-configuration) 
to fetch the **jwks_uri** and uses it to download the public key which is used to validate the JWT token.
{{< /box >}}

So, we are able to configure certail API endpoints publicly accessible without requiring authentication 
and secured the rest of the endpoints. But, what about the Role Based Access?

Remember, we want to allow only the users with **ROLE_ADMIN** to invoke the **POST /api/messages/archive** API endpoint.

But, before looking into that, let's see how to get the current user details.

## Getting Current User Details
We can get the current user details from **SecurityContextHolder** and extract various interesting information from it.
Let's implement an API endpoint **GET /api/me** to get the current user details as follows.

```java
package com.sivalabs.messages.api;

import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.security.oauth2.jwt.Jwt;
import org.springframework.security.oauth2.server.resource.authentication.JwtAuthenticationToken;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.HashMap;
import java.util.List;
import java.util.Map;

@RestController
class UserInfoController {

    @GetMapping("/api/me")
    Map<String, Object> currentUserDetails() {
        return getLoginUserDetails();
    }

    Map<String, Object> getLoginUserDetails() {
        Map<String, Object> map = new HashMap<>();
        JwtAuthenticationToken authentication =
                (JwtAuthenticationToken) SecurityContextHolder.getContext().getAuthentication();
        Jwt jwt = (Jwt) authentication.getPrincipal();

        map.put("username", jwt.getClaimAsString("preferred_username"));
        map.put("email", jwt.getClaimAsString("email"));
        map.put("name", jwt.getClaimAsString("name"));
        map.put("token", jwt.getTokenValue());
        map.put("authorities", authentication.getAuthorities());
        map.put("roles", getRoles(jwt));

        return map;
    }

    List<String> getRoles(Jwt jwt) {
        Map<String,Object> realm_access = (Map<String, Object>) jwt.getClaims().get("realm_access");
        if(realm_access != null && !realm_access.isEmpty()) {
            return  (List<String>) realm_access.get("roles");
        }
        return List.of();
    }
}
```

Now if we invoke the **GET /api/me** API endpoint from Postman by configuring the **Authorization** as we did earlier, 
we will get the current user details similar to the following:

```json
{
  
  "name": "Siva Katamreddy",
  "email": "siva@gmail.com",
  "username": "siva",
  "token": "eyJhbGciOiJSUzI1NiIsInR5c.....qgGIu8iF86azw",
  "roles": [
    "default-roles-sivalabs",
    "offline_access",
    "uma_authorization"
  ],
  "authorities": [
    {
      "authority": "SCOPE_openid"
    },
    {
      "authority": "SCOPE_email"
    },
    {
      "authority": "SCOPE_profile"
    }
  ]
}
```

If we go to https://jwt.io/ and paste the token value, we can see the decoded response as follows:

```json
{
  "exp": 1695919675,
  "iat": 1695919375,
  "auth_time": 1695914182,
  "jti": "64128bc7-8f4d-48ff-978f-93b0764f39cd",
  "iss": "http://localhost:9191/realms/sivalabs",
  "aud": "account",
  "sub": "ca1a2f34-1614-45dd-86c1-5eafff085d8a",
  "typ": "Bearer",
  "azp": "messages-webapp",
  "session_state": "3e5865f1-0f0e-4ada-b2a1-97e7b118af4d",
  "acr": "0",
  "allowed-origins": [
    "http://localhost:8080"
  ],
  "realm_access": {
    "roles": [
      "default-roles-sivalabs",
      "offline_access",
      "uma_authorization"
    ]
  },
  "resource_access": {
    "account": {
      "roles": [
        "manage-account",
        "manage-account-links",
        "view-profile"
      ]
    }
  },
  "scope": "openid email profile",
  "sid": "3e5865f1-0f0e-4ada-b2a1-97e7b118af4d",
  "email_verified": true,
  "name": "Siva Katamreddy",
  "preferred_username": "siva",
  "given_name": "Siva",
  "family_name": "Katamreddy",
  "email": "siva@gmail.com"
}
```

If we compare the **/api/me** API response with the decoded token data, the interesting things to observe are:
* The **roles** are derived from the **realm_access** claim in the JWT token.
* The **authorities** are derived from the **scope** claim in the JWT token.

The default **JwtAuthenticationConverter** implementation converts the **scope** claim into **authorities** adding **SCOPE_** prefix.

Keycloak sends the **roles** in the **realm_access** claim in the JWT token.
We have extracted the **roles** from the **realm_access** claim and added it to the response.

Now, let's create **ROLE_USER** and **ROLE_ADMIN** roles in Keycloak and assign them to the user **siva**.

* Go to Keycloak Admin Console and select the **sivalabs** realm
* Click on **Realm roles** and create **ROLE_USER** and **ROLE_ADMIN** roles
* Click on **Users** and select the user **siva**
* Click on **Role Mappings** and assign **ROLE_USER** and **ROLE_ADMIN** roles to the user **siva**

Now if we invoke the **GET /api/me** API endpoint from Postman, we will get the updated response as follows:

```json
{

  "name": "Siva Katamreddy",
  "email": "siva@gmail.com",
  "username": "siva",
  "token": "eyJhbGciOiJSUzI1NiIsInR5c.....qgGIu8iF86azw",
  "roles": [
    "default-roles-sivalabs",
    "offline_access",
    "uma_authorization",
    "ROLE_USER",
    "ROLE_ADMIN"
  ],
  "authorities": [
    {
      "authority": "SCOPE_openid"
    },
    {
      "authority": "SCOPE_email"
    },
    {
      "authority": "SCOPE_profile"
    }
  ]
}
```

What we just did is, figuring out the current user details and extracting the roles from the JWT token.
But, for Spring Security to treat roles as authorities, somehow we need to convert the roles into authorities.

To do that, we can create a custom **JwtAuthenticationConverter** by implementing 
**Converter<Jwt, AbstractAuthenticationToken>** as follows:

```java
package com.sivalabs.messages.config;

import org.springframework.core.convert.converter.Converter;
import org.springframework.security.authentication.AbstractAuthenticationToken;
import org.springframework.security.core.GrantedAuthority;
import org.springframework.security.core.authority.SimpleGrantedAuthority;
import org.springframework.security.oauth2.jwt.Jwt;
import org.springframework.security.oauth2.server.resource.authentication.JwtAuthenticationToken;
import org.springframework.security.oauth2.server.resource.authentication.JwtGrantedAuthoritiesConverter;

import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;

class KeycloakJwtAuthenticationConverter implements Converter<Jwt, AbstractAuthenticationToken> {
    private final Converter<Jwt, Collection<GrantedAuthority>> delegate = new JwtGrantedAuthoritiesConverter();

    @Override
    public AbstractAuthenticationToken convert(Jwt jwt) {
        List<GrantedAuthority> authorityList = extractRoles(jwt);
        Collection<GrantedAuthority> authorities = delegate.convert(jwt);
        if (authorities != null) {
            authorityList.addAll(authorities);
        }
        return new JwtAuthenticationToken(jwt, authorityList);
    }

    private List<GrantedAuthority> extractRoles(Jwt jwt) {
        Map<String,Object> realm_access = (Map<String, Object>) jwt.getClaims().get("realm_access");
        if(realm_access == null || realm_access.isEmpty()) {
            return List.of();
        }
        List<String> roles = (List<String>) realm_access.get("roles");
        if (roles == null || roles.isEmpty()) {
            roles = List.of("ROLE_USER");
        }
        return roles.stream()
                        .filter(role -> role.startsWith("ROLE_"))
                        .map(SimpleGrantedAuthority::new).collect(Collectors.toList());
    }
}
```

What we did here:

* We are extracting the **roles** from the **realm_access** claim and converting them into **authorities**. 
  Note that we are considering the roles having **ROLE_** prefix only.
* We are converting the **scope** claim into **authorities** using Spring Security's **JwtGrantedAuthoritiesConverter**.
* We are combining the **authorities** from **scope** claim and **roles** from **realm_access** claim and creating a new **JwtAuthenticationToken**.

Now, let's register this custom **JwtAuthenticationConverter** in **SecurityConfig** as follows:

```java
@Configuration
@EnableWebSecurity
public class SecurityConfig {
    @Bean
    SecurityFilterChain securityFilterChain(HttpSecurity http) throws Exception {
        http
        ...
        ...
        .oauth2ResourceServer(oauth2 ->
                oauth2.jwt(jwt -> jwt.jwtAuthenticationConverter(new KeycloakJwtAuthenticationConverter()))
        );
        return http.build();
    }
}
```

Now if we invoke the **GET /api/me** API endpoint from Postman, we will get the updated response as follows:

```json
{

  "name": "Siva Katamreddy",
  "email": "siva@gmail.com",
  "username": "siva",
  "token": "eyJhbGciOiJSUzI1NiIsInR5c.....qgGIu8iF86azw",
  "roles": [
    "default-roles-sivalabs",
    "offline_access",
    "uma_authorization",
    "ROLE_USER",
    "ROLE_ADMIN"
  ],
  "authorities": [
    {
      "authority": "ROLE_USER"
    },
    {
      "authority": "ROLE_ADMIN"
    },
    {
      "authority": "SCOPE_openid"
    },
    {
      "authority": "SCOPE_email"
    },
    {
      "authority": "SCOPE_profile"
    }
  ]
}
```

We don't actually need the **GET /api/me** API endpoint, but we just implemented it to understand how to get the current user details
and how to use a custom **JwtAuthenticationConverter** to map Keycloak roles as authorities.

## Verify the Role Based Access Control
Let's verify the Role Based Access Control by implementing the **POST /api/messages/archive** API endpoint.

Add the following endpoint to the **MessageController** class with the following content:

```java
@RestController
@RequestMapping("/api/messages")
class MessageController {
    private static final Logger log = LoggerFactory.getLogger(MessageController.class);
    ...
    ...

   @PostMapping("/archive")
   Map<String,String> archiveMessages() {
      log.info("Archiving all messages");
      return Map.of("status", "success");
   }
}
```

Update the **SecurityConfig** class as follows:

```java
@Configuration
@EnableWebSecurity
public class SecurityConfig {
    @Bean
    SecurityFilterChain securityFilterChain(HttpSecurity http) throws Exception {
        http
          .authorizeHttpRequests(c ->
            c.requestMatchers(HttpMethod.GET, "/api/messages").permitAll()
             .requestMatchers(HttpMethod.POST, "/api/messages/archive").hasAnyRole("ADMIN")
             .anyRequest().authenticated()
          )
        ...
        ...
      
        return http.build();
    }
}
```

Now if we invoke the **POST /api/messages/archive** API endpoint from Postman by configuring the **Authorization** as we did earlier,
we will get the following response with HTTP status code 200.

```json
{
  "status": "success"
}
```

Now, go to Keycloak Admin Console and remove the **ROLE_ADMIN** role from the user **siva**.
Now try getting a new **access_token** and invoke the **POST /api/messages/archive** API endpoint from Postman.
You will get Forbidden 403 HTTP status code as the user **siva** doesn't have the **ROLE_ADMIN** role.

## Conclusion
In this article, we have created the **messages-service** Resource Server and secured it using 
Spring Security OAuth 2.0. We also learned how to implement Role Based Access Control. 

In the next article, we will integrate **messages-webapp** with **messages-service** Resource Server.

# Spring Security OAuth 2 Tutorial - 7 : Securing Spring MVC Client Application


In this article, we will create **messages-webapp** which is a **Spring MVC + Thymeleaf** web application 
and secure it with **Spring Security OAuth 2.0** using **Keycloak**.

<!--more-->


{{< box info >}}
**Source Code:**

You can find the complete source code of this project on GitHub: 
https://github.com/sivaprasadreddy/spring-security-oauth2-microservices-demo
{{< /box >}}


## Setup Keycloak using Docker Compose
In the previous article, we have already seen how to setup Keycloak using Docker Compose.

Create **docker-compose.yml** file with the following content:

```yaml
version: '3.8'
name: spring-security-oauth2-microservices-demo
services:
  keycloak:
    image: quay.io/keycloak/keycloak:22.0.3
    command: ['start-dev']
    container_name: keycloak
    hostname: keycloak
    environment:
      - KEYCLOAK_ADMIN=admin
      - KEYCLOAK_ADMIN_PASSWORD=admin1234
    ports:
      - "9191:8080"
```

Run the following command to start the Keycloak instance:

```shell
$ docker compose up -d
```

Now you can access the Keycloak admin console at http://localhost:9191/ and login using the credentials **admin/admin1234**.

## Create Keycloak Realm, Client and Users
In the previous articles, we have learned how to create a realm, client and users.
Please follow the steps mentioned in https://www.sivalabs.in/spring-security-oauth2-tutorial-introduction/#create-new-realm 
to create a new realm, client and user with one change for **Valid redirect URIs**.

Set **Valid redirect URIs** value to **http://localhost:8080/login/oauth2/code/messages-webapp** instead of **http://localhost:8080/callback**.

Now you should have the following details:

* **Keycloak Realm:** sivalabs
* **Client Configuration:**
  * **Client ID:** messages-webapp
  * **Client Secret:** O3SVuBs0Z25kpYoRtL5C0FhLwAnIx1CW (you might have different value)
  * **Root URL**: http://localhost:8080
  * **Home URL**: http://localhost:8080
  * **Valid redirect URIs:** http://localhost:8080/login/oauth2/code/messages-webapp
  * **Valid post logout redirect URIs**: http://localhost:8080/
  * **Web origins**: http://localhost:8080
* **User:** siva/siva1234

{{< box info >}}
**NOTE:**

Did you observe the **Valid redirect URIs** value? It is different from what we have configured (http://localhost:8080/callback) in the previous articles.
Spring Security implemented authentication filter to handle the OAuth 2.0 Authorization Code Grant flow.

In Spring Security OAuth 2.0 implementation, the default value of **redirect-uri** is **{baseUrl}/login/oauth2/code/{registrationId}**.
We are going to use **messages-webapp** as the **registrationId** for our client application.
So, in Keycloak we need to configure the **Valid redirect URIs** as **http://localhost:8080/login/oauth2/code/messages-webapp**.
{{< /box >}}

{{< box warning >}}
**IMPORTANT:**

Make sure you have configured the **Root URL**, **Home URL**, **Valid redirect URIs**, 
**Valid post logout redirect URIs** and **Web origins** exactly as mentioned above.
Having an extra **"/"** at the end or not having **"/"** at the end may result in errors like **invalid_redirect_uri**.
{{< /box >}}

## Create messages-webapp
You can generate **messages-webapp** using **Spring Initializr** by clicking on this [link](https://start.spring.io/#!type=maven-project&language=java&platformVersion=3.1.4&packaging=jar&jvmVersion=17&groupId=com.sivalabs&artifactId=messages-webapp&name=messages-webapp&packageName=com.sivalabs.messages&dependencies=web,validation,oauth2-client,security,thymeleaf).
We have selected the starters **Web**, **Validation**, **OAuth2 Client**, **Security** and **Thymeleaf**.
Once the application is generated, open it in your favourite IDE.

### Configure OAuth 2.0 Client Registration Properties
An OAuth 2.0 Client application can use multiple Authentication Providers such as **Google**, **Facebook**, **GitHub**, **Okta**, **Keycloak**, etc.
In our case, we are going to use only one Authentication Provider which is **Keycloak**.

We need to configure the client application details using **spring.security.oauth2.client.registration.{registrationId}.\*** 
and **spring.security.oauth2.client.provider.{registrationId}.\*** properties in **application.properties** file. 
We will use **messages-webapp** as **registrationId** and configure the properties as follows:

```properties
spring.security.oauth2.client.registration.messages-webapp.client-id=messages-webapp
spring.security.oauth2.client.registration.messages-webapp.client-secret=O3SVuBs0Z25kpYoRtL5C0FhLwAnIx1CW
spring.security.oauth2.client.registration.messages-webapp.authorization-grant-type=authorization_code
spring.security.oauth2.client.registration.messages-webapp.scope=openid, profile
spring.security.oauth2.client.registration.messages-webapp.redirect-uri={baseUrl}/login/oauth2/code/messages-webapp

spring.security.oauth2.client.provider.messages-webapp.issuer-uri=http://localhost:9191/realms/sivalabs
#spring.security.oauth2.client.provider.messages-webapp.authorization-uri=http://localhost:9191/realms/sivalabs/protocol/openid-connect/auth
#spring.security.oauth2.client.provider.messages-webapp.token-uri=http://localhost:9191/realms/sivalabs/protocol/openid-connect/token
#spring.security.oauth2.client.provider.messages-webapp.jwk-set-uri=http://localhost:9191/realms/sivalabs/protocol/openid-connect/certs
#spring.security.oauth2.client.provider.messages-webapp.user-info-uri=http://localhost:9191/realms/sivalabs/protocol/openid-connect/userinfo
```

If you observe the above configuration, we have commented out the properties **authorization-uri**, **token-uri**, **jwk-set-uri** and **user-info-uri**.
Spring Security OAuth 2.0 Client implementation will automatically discover these endpoints using the **issuer-uri** by invoking 
**{issuer-uri}/.well-known/openid-configuration** endpoint.

If you go to [http://localhost:9191/realms/sivalabs/.well-known/openid-configuration](http://localhost:9191/realms/sivalabs/.well-known/openid-configuration) you can see the following response
containing all the endpoints information:

```json
{
    "issuer": "http://localhost:9191/realms/sivalabs",
    "authorization_endpoint": "http://localhost:9191/realms/sivalabs/protocol/openid-connect/auth",
    "token_endpoint": "http://localhost:9191/realms/sivalabs/protocol/openid-connect/token",
    "introspection_endpoint": "http://localhost:9191/realms/sivalabs/protocol/openid-connect/token/introspect",
    "userinfo_endpoint": "http://localhost:9191/realms/sivalabs/protocol/openid-connect/userinfo",
    "end_session_endpoint": "http://localhost:9191/realms/sivalabs/protocol/openid-connect/logout",
    "frontchannel_logout_session_supported": true,
    "frontchannel_logout_supported": true,
    "jwks_uri": "http://localhost:9191/realms/sivalabs/protocol/openid-connect/certs",
    "check_session_iframe": "http://localhost:9191/realms/sivalabs/protocol/openid-connect/login-status-iframe.html",
    "grant_types_supported": [
        "authorization_code",
        "implicit",
        "refresh_token",
        "password",
        "client_credentials",
        "urn:ietf:params:oauth:grant-type:device_code",
        "urn:openid:params:grant-type:ciba"
    ],
    ...,
  "response_types_supported": [
        "code",
        "none",
        "id_token",
        "token",
        "id_token token",
        "code id_token",
        "code token",
        "code id_token token"
    ],
    ...
    ...
}
```

### Implement Home Page
When we add **spring-boot-starter-security** dependency, Spring Security will automatically secure all the endpoints.
And we also added **spring-boot-starter-oauth2-client** dependency which will automatically configure OAuth 2.0 Client 
using the properties configured in **application.properties** .

Create **HomeController** class with the following content:

```java
@Controller
public class HomeController {

    @GetMapping("/")
    public String home(Model model, @AuthenticationPrincipal OAuth2User principal) {
        model.addAttribute("username", principal.getAttribute("name"));
        return "home";
    }
}
```

We are injecting the authenticated user principal object using **@AuthenticationPrincipal** annotation.
The **OAuth2User** interface represents the authenticated user principal.

Create **home.html** file under **src/main/resources/templates** folder with the following content:

```html
<!DOCTYPE html>
<html>
<head>
    <title>Home</title>
</head>
<body>
<div>
    <h1>Welcome <span th:text="${username}">username</span></h1>
</div>
</body>
</html>
```

Now if you run the application and access http://localhost:8080/ you will be redirected to the Keycloak login page.
Once you login successfully using **siva/siva1234** credentials, you will be redirected to the home page, 
and you can see the username in the home page.

### Customize Security Configuration
As we discussed earlier, Spring Security will automatically secure all the endpoints.
But, we want to allow access to the home page without authentication.
So, we need to customize the security configuration to allow access to the home page.

Create **SecurityConfig** class with the following content:

```java
package com.sivalabs.messages.config;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.security.config.Customizer;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;
import org.springframework.security.config.annotation.web.configurers.CorsConfigurer;
import org.springframework.security.config.annotation.web.configurers.CsrfConfigurer;
import org.springframework.security.web.SecurityFilterChain;

@Configuration
@EnableWebSecurity
public class SecurityConfig {

    @Bean
    SecurityFilterChain securityFilterChain(HttpSecurity http) throws Exception {
        http
            .authorizeHttpRequests(c ->
                    c.requestMatchers("/").permitAll()
                    .anyRequest().authenticated()
            )
            .cors(CorsConfigurer::disable)
            .csrf(CsrfConfigurer::disable)
            .oauth2Login(Customizer.withDefaults());
        return http.build();
    }
}
```

As we made Home page accessible to all, the **@AuthenticationPrincipal** could be **null**.
Let's update the **HomeController** to handle this scenario.

```java
@Controller
public class HomeController {

    @GetMapping("/")
    public String home(Model model, @AuthenticationPrincipal OAuth2User principal) {
        if(principal != null) {
            model.addAttribute("username", principal.getAttribute("name"));
        } else {
            model.addAttribute("username", "Guest");
        }
        return "home";
    }
}
```

Now, if we restart the application and access http://localhost:8080/ we will see the home page without any authentication.

Now we need a way to login into the application. We can add a **Login** link in the home page as follows:

```html
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml"
      xmlns:th="http://www.thymeleaf.org"
      xmlns:sec="http://www.thymeleaf.org/thymeleaf-extras-springsecurity">
<head>
  <title>Home</title>
</head>
<body>
<div >
  <p sec:authorize="!isAuthenticated()">
    <a href="/oauth2/authorization/messages-webapp">Login</a>
  </p>

  <h1>Welcome <span th:text="${username}">username</span></h1>

</div>
</body>
</html>
```

We are checking whether user is already loggged in or not and conditionally showing the Login link. 
And, we are using the Spring Security OAuth 2.0 default login URL **/oauth2/authorization/{registrationId}** to initiate the OAuth 2.0 Authorization Code Grant flow.

Now if you access http://localhost:8080/ you will see the Login link. 
Click on the Login link and you will be redirected to the Keycloak login page.
Once you login successfully, you will be redirected to the home page and you can see the username in the home page.

### Implement Logout
By default, Spring Security OAuth 2.0 Client implementation configures the logout functionality
such that you can initiate logout by calling the URL **/logout**. Then the HTTP Session will be invalidated, 
clears the **SecurityContextHolder**, and then redirect to the configured **Valid post logout redirect URIs**.

If you want to customize the logout functionality, then you can update **SecurityConfig** as follows:

```java
package com.sivalabs.messages.config;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.security.config.Customizer;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;
import org.springframework.security.config.annotation.web.configurers.CorsConfigurer;
import org.springframework.security.config.annotation.web.configurers.CsrfConfigurer;
import org.springframework.security.oauth2.client.oidc.web.logout.OidcClientInitiatedLogoutSuccessHandler;
import org.springframework.security.oauth2.client.registration.ClientRegistrationRepository;
import org.springframework.security.web.SecurityFilterChain;
import org.springframework.security.web.authentication.logout.LogoutSuccessHandler;

@Configuration
@EnableWebSecurity
public class SecurityConfig {
  private final ClientRegistrationRepository clientRegistrationRepository;

  public SecurityConfig(ClientRegistrationRepository clientRegistrationRepository) {
    this.clientRegistrationRepository = clientRegistrationRepository;
  }

  @Bean
  SecurityFilterChain securityFilterChain(HttpSecurity http) throws Exception {
    http
            .authorizeHttpRequests(c ->
                    c.requestMatchers("/").permitAll()
                            .anyRequest().authenticated()
            )
            .cors(CorsConfigurer::disable)
            .csrf(CsrfConfigurer::disable)
            .oauth2Login(Customizer.withDefaults())
            .logout(logout -> logout
                    .clearAuthentication(true)
                    .invalidateHttpSession(true)
                    .logoutSuccessHandler(oidcLogoutSuccessHandler())
            );
    return http.build();
  }

  private LogoutSuccessHandler oidcLogoutSuccessHandler() {
    OidcClientInitiatedLogoutSuccessHandler oidcLogoutSuccessHandler =
            new OidcClientInitiatedLogoutSuccessHandler(this.clientRegistrationRepository);
    oidcLogoutSuccessHandler.setPostLogoutRedirectUri("{baseUrl}/");
    return oidcLogoutSuccessHandler;
  }
}
```

## Conclusion
In this article, we have created the **messages-webapp** Client application and secured it using 
Spring Security OAuth 2.0 Authorization Code Flow. 

In the next article, we will create the **messages-service** Resource Server and secure it using Spring Security OAuth 2.0, 
and call its APIs from **messages-webapp**.
# Spring Security OAuth 2 Tutorial - 6 : Microservices Sample Project Setup

In the previous articles, we have learned about various OAuth 2.0 / OpenID Connect flows using web browser, cURL and Postman.
Now it's time to put what we have learned into practice. What better way to do that than to build a sample project?

<!--more-->


While implementing OAuth 2.0 / OpenID Connect based security using a Security framework like [Spring Security](https://spring.io/projects/spring-security),
many activities are performed by the framework under the hood. 
It is important to understand what is happening under the hood so that we can use the framework effectively.

In this article, I will briefly introduce you to the sample microservices based project that we will be working on for this 
Spring Security OAuth 2 Tutorial series.

## Sample Spring Boot Microservices Project
We are going to build a sample microservices project that has the following components:

* **Authorization Server** - This is the OAuth 2.0 Authorization Server that will issue access tokens to the clients. 
    We will use **Keycloak** as the **Authorization Server**.
* **messages-webapp** - A Spring MVC based **Client** application that the end user will be using.
* **messages-service** - A Spring Boot REST API (**Resource Server**) that manages the user data (messages) 
    and will be used by the messages-webapp application.
* **archival-service** - A Spring Boot application that handles archival of the messages at regular intervals 
    using a scheduled job. It also exposes an API endpoint to trigger the archival process. 
    This service will play the roles of both a **Resource Server** and a **Client**.

{{< figure src="/images/oauth2-microservices.webp" alt="OAuth2 Microservices" >}}

Let's get into what functionality each of these components provides.

## messages-webapp (Client)
This is a Spring Boot web application that uses Spring MVC and Thymeleaf to render the UI.
This application will be protected by the Authorization Server (Keycloak) and will be using the **Authorization Code Flow** 
to authenticate the user and obtain the access token.

* Anyone (unauthorized users also) can view the list of messages.
* Any authenticated user can post a new message.
* Any authenticated user with **ADMIN** role can trigger the message archival process.

## messages-service (Resource Server)
This is a Spring Boot REST API that manages the user data (messages) and will be used by the **messages-webapp** application.
This is a Resource Server and exposes the following API endpoints:

* **GET /api/messages** - Returns the list of messages. Publicly accessible, no authentication required.
* **POST /api/messages** - Creates a new message. Any authenticated user can call this API endpoint. 
* **POST /api/messages/archive** - Archives the messages older than N days. Only authenticated users with either **ADMIN** or **ADMIN_JOB** roles can call this API endpoint.

## archival-service (Resource Server and Client)
This is a Spring Boot service that handles archival of the messages at regular intervals using a scheduled job. 
This service exposes an API endpoint to trigger the archival process only by **ADMIN** users.
This application will be protected by the Authorization Server (Keycloak) and will be using the **Client Credentials Flow** 
to obtain the access token to invoke **POST /api/messages/archive** API endpoint of **messages-service**.

This service exposes the following API endpoint:

* **POST /api/messages/archive** - Internally delegates the archival process to **messages-service**. Only authenticated users with **ADMIN** role can call this API.

{{< box info >}}
**Why not messages-webapp directly call messages-service POST /api/messages/archive endpoint?:**

The reason is, I would like to demonstrate a scenario where a service acts as both a **Resource Server and a Client**.
The archival-service is a **Resource Server** exposing the **POST /api/messages/archive** API endpoint and also a **Client** for the **messages-service**.
{{< /box >}}

## Keycloak Setup
In the previous articles, we have learned how to setup Keycloak and create a realm, client and users.
We will be using the similar Keycloak setup for the service configurations.

### Exporting and Importing Keycloak Realm Configuration
If, for any reason, the keycloak docker container is deleted, then we need to again create the realm, client and users manually.
We can use Keycloak's **export** and **import** feature to export the realm configuration and import it back when needed.

Once the keycloak instance is up and running, we can create the realm, clients and users and then use the following steps to export the realm configuration.

```shell
$ docker ps
# copy the keycloak container id

# ssh into keycloak container
$ docker exec -it <container-id> bash

# export the realm configuration along with users info
$ /opt/keycloak/bin/kc.sh export --dir /opt/keycloak/data/import --realm sivalabs --users realm_file

# exit from the container
$ exit 

# copy the exported realm configuration to local machine
$ docker cp <container-id>:/opt/keycloak/data/import/sivalabs-realm.json ~/Downloads/sivalabs-realm.json
```

Once you have the exported realm configuration file, copy it into the **realm-config** folder of the sample project.
Then you can use it to automatically import it using docker-compose as follows:

```yaml
version: '3.8'
name: spring-security-oauth2-microservices-demo
services:
  keycloak:
    image: quay.io/keycloak/keycloak:22.0.3
    command: ['start-dev --import-realm']
    volumes:
      - ./realm-config:/opt/keycloak/data/import
    container_name: keycloak
    hostname: keycloak
    environment:
      - KEYCLOAK_ADMIN=admin
      - KEYCLOAK_ADMIN_PASSWORD=admin1234
    ports:
      - "9191:8080"
```

Don't worry if anything is not clear at this point, we will be exploring the sample project in detail in the upcoming articles.

## Conclusion
In this article, we have explored the sample microservices project that we will be working on for this Spring Security OAuth 2 Tutorial series.
In the next article, we will work on setting up Keycloak, creating the **messages-webapp** application and securing it in detail.

# Spring Security OAuth 2 Tutorial - 5 : Implicit & Resource Owner Password Credentials Flows


In the [Part 4: OAuth 2.0 Authorization Code Flow with PKCE]({{< relref "2023-09-27-spring-security-oauth2-tutorial-part4.md" >}}), 
we learned how to acquire **access_token** using **Authorization Code Flow with PKCE**. 

In this article, we will explore how to use **Implicit Flow** and **Resource Owner Password Credentials Flow**.

<!--more-->


## Implicit Flow
The **Implicit Flow** is a kind of shorter version of **Authorization Code Flow** where you will be directly getting **access_token**
using **authorization_endpoint** itself.

{{< box warning >}}
**IMPORTANT**

The **Implicit Flow** and **Resource Owner Password Credentials Flow** are **DEPRECATED**.

Unless you have a good reason, you shouldn't be using them.
{{< /box >}}

If you have been following this series, you know how to create a Client with certain **Authentication flows** enabled.
Instead of creating a new client, let's enable **Implicit flow** for **messages-webapp** client.

To learn how to create a new **Client**, please see [Create Client](https://www.sivalabs.in/spring-security-oauth2-tutorial-introduction/#create-client)

Open the following URL in the browser window:

```shell
http://localhost:9191/realms/sivalabs/protocol/openid-connect/auth?
  response_type=id_token%20token
  &client_id=messages-webapp
  &redirect_uri=http://localhost:8080/callback
  &scope=openid%20profile
  &state=randomstring
  &nonce=another_randomstring
```

* Then you will be redirected to Keycloak's Login page.
* Login with the user credentials **siva/siva1234**.
* Then you will be redirected the **Redirect URI** which has **access_token** and **id_token**.

```shell
http://localhost:8080/callback#
  state=randomstring
  &session_state=51692fdb-8b72-45b7-a341-fa73e97b5139
  &id_token=eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJMeVVPTDg4LVBGM3BYQzFpN3BIeGdFZTJwaWZJY3RyTXJiNklHOElmRTlVIn0.eyJleHAiOjE2OTU2MTUzOTQsImlhdCI6MTY5NTYxNDQ5NCwiYXV0aF90aW1lIjoxNjk1NjE0NDk0LCJqdGkiOiI2MjFmNTJmMC0wMDBmLTQ1ZmUtYWYzOC1iY2YzZWM2ZDk1MTEiLCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjkxOTEvcmVhbG1zL3NpdmFsYWJzIiwiYXVkIjoibWVzc2FnZXMtd2ViYXBwIiwic3ViIjoiY2ExYTJmMzQtMTYxNC00NWRkLTg2YzEtNWVhZmZmMDg1ZDhhIiwidHlwIjoiSUQiLCJhenAiOiJtZXNzYWdlcy13ZWJhcHAiLCJub25jZSI6ImFub3RoZXJfcmFuZG9tc3RyaW5nIiwic2Vzc2lvbl9zdGF0ZSI6IjUxNjkyZmRiLThiNzItNDViNy1hMzQxLWZhNzNlOTdiNTEzOSIsImF0X2hhc2giOiJiamR6MC1NeWltQ0xrSzdqaWRRbHp3IiwiYWNyIjoiMSIsInNfaGFzaCI6IlJtVE5Ld0lYaTNXRFhzRFlObTQtUHciLCJzaWQiOiI1MTY5MmZkYi04YjcyLTQ1YjctYTM0MS1mYTczZTk3YjUxMzkiLCJlbWFpbF92ZXJpZmllZCI6dHJ1ZSwibmFtZSI6IlNpdmEgS2F0YW1yZWRkeSIsInByZWZlcnJlZF91c2VybmFtZSI6InNpdmEiLCJnaXZlbl9uYW1lIjoiU2l2YSIsImZhbWlseV9uYW1lIjoiS2F0YW1yZWRkeSIsImVtYWlsIjoic2l2YUBnbWFpbC5jb20ifQ.TIcmVBti96HuZvrYe_14mVJlfopXI2PhdMdWBtPPASpJc-DKrL9argy08sYZKqJTTcmWwnIwKK2o1vddVxA4zUP2tnqqg6ymz1trN3J8r4h-WSvIp907vnS0R7iHei56L6MQX2DZLJ8pOdSmti8wg_9fu4gQJBE2sHRTlrlOP39dh8yohMGidM-Z5iFbLCIzOQXA6B6ewMZll5iwL3ssJ716Ve9cO4qHGCneRGpb3mO7jclY87YSGM-wqr6ur00ylQ_BCGyCdl-f-xskSeDX09iQKFSTX_acMxB7FNi21BL7dMx8_22XPFOwNkX8ha8Vb7eTRMYEMyB776i33FLu0A
  &access_token=eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJMeVVPTDg4LVBGM3BYQzFpN3BIeGdFZTJwaWZJY3RyTXJiNklHOElmRTlVIn0.eyJleHAiOjE2OTU2MTUzOTQsImlhdCI6MTY5NTYxNDQ5NCwiYXV0aF90aW1lIjoxNjk1NjE0NDk0LCJqdGkiOiI1N2NiYmRkMC0wNGFmLTRlMDctYWZlNC02ZmQ5MmY0YjA2MzAiLCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjkxOTEvcmVhbG1zL3NpdmFsYWJzIiwiYXVkIjoiYWNjb3VudCIsInN1YiI6ImNhMWEyZjM0LTE2MTQtNDVkZC04NmMxLTVlYWZmZjA4NWQ4YSIsInR5cCI6IkJlYXJlciIsImF6cCI6Im1lc3NhZ2VzLXdlYmFwcCIsIm5vbmNlIjoiYW5vdGhlcl9yYW5kb21zdHJpbmciLCJzZXNzaW9uX3N0YXRlIjoiNTE2OTJmZGItOGI3Mi00NWI3LWEzNDEtZmE3M2U5N2I1MTM5IiwiYWNyIjoiMSIsImFsbG93ZWQtb3JpZ2lucyI6WyJodHRwOi8vbG9jYWxob3N0OjgwODAiXSwicmVhbG1fYWNjZXNzIjp7InJvbGVzIjpbImRlZmF1bHQtcm9sZXMtc2l2YWxhYnMiLCJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9uIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiYWNjb3VudCI6eyJyb2xlcyI6WyJtYW5hZ2UtYWNjb3VudCIsIm1hbmFnZS1hY2NvdW50LWxpbmtzIiwidmlldy1wcm9maWxlIl19fSwic2NvcGUiOiJvcGVuaWQgZW1haWwgcHJvZmlsZSIsInNpZCI6IjUxNjkyZmRiLThiNzItNDViNy1hMzQxLWZhNzNlOTdiNTEzOSIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJuYW1lIjoiU2l2YSBLYXRhbXJlZGR5IiwicHJlZmVycmVkX3VzZXJuYW1lIjoic2l2YSIsImdpdmVuX25hbWUiOiJTaXZhIiwiZmFtaWx5X25hbWUiOiJLYXRhbXJlZGR5IiwiZW1haWwiOiJzaXZhQGdtYWlsLmNvbSJ9.XsYc69HnM9VaJZFF568nRiZhh8RYEw6Hq2WGnJ4jr3tmZvgMF0QK2RtlBT9BuX4A11XHjyNZqGYNf55x0k4bPXjhPzWI-lC0shhsKrXGYrnhVcComXxMbO_38ypRY_EMeBWRTXu0bvcKInYMjVoItfoLheH-kbcziK6O16yFGftOG-YYw0uVzs_DrOkjQjs1BS2L56yXcRgN72EBXMT-Cv6OLMTSj6WXjfg1nmRl0NRJdeZv0iafSolmqSpJeqXwPzM2hgZ2hPzaq90qipndQrZ05xesMtzXMNlev0ozYPN7xKSa7arHMYky8y4OMpCQDJzkSwekjUEQUSU9Sqg_VA
  &token_type=Bearer
  &expires_in=900
```

As you can see, the **access_token** and **id_token** are returned via the front-channel thru browser URL which not secure.

## Resource Owner Password Credentials Flow
In **Resource Owner Password Credentials Flow**, we will get **access_token** and **id_token** by using the resource owner (end user) credentials.

Instead of creating a new client, let's enable **Direct access grants**, which enables **Resource Owner Password Credentials Grant**, 
for **messages-webapp** client.

Now we can get the **access_token** and **id_token** using the token_endpoint with the following cURL command:

```shell
curl --location 'http://localhost:9191/realms/sivalabs/protocol/openid-connect/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'grant_type=password' \
--data-urlencode 'client_id=messages-webapp' \
--data-urlencode 'username=siva' \
--data-urlencode 'password=siva1234' \
--data-urlencode 'client_secret=qVcg0foCUNyYbgF0Sg52zeIhLYyOwXpQ' \
--data-urlencode 'scope=openid'
```

This should return the JSON response something like this:

```json
{
  "access_token": "eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJMeVVPTDg4LVBGM3BYQzFpN3BIeGdFZTJwaWZJY3RyTXJiNklHOElmRTlVIn0.eyJleHAiOjE2OTU2MTU4NTYsImlhdCI6MTY5NTYxNTU1NiwianRpIjoiZmE4ZDRjZGUtMTk3NS00Mzc1LWJhMTAtMWVjYmJmZjIzZGFlIiwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo5MTkxL3JlYWxtcy9zaXZhbGFicyIsImF1ZCI6ImFjY291bnQiLCJzdWIiOiJjYTFhMmYzNC0xNjE0LTQ1ZGQtODZjMS01ZWFmZmYwODVkOGEiLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJtZXNzYWdlcy13ZWJhcHAiLCJzZXNzaW9uX3N0YXRlIjoiYjJiM2YzYTAtYTQwMi00MTdjLWJjOTQtMmMxNzcwZjY0YzE0IiwiYWNyIjoiMSIsImFsbG93ZWQtb3JpZ2lucyI6WyJodHRwOi8vbG9jYWxob3N0OjgwODAiXSwicmVhbG1fYWNjZXNzIjp7InJvbGVzIjpbImRlZmF1bHQtcm9sZXMtc2l2YWxhYnMiLCJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9uIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiYWNjb3VudCI6eyJyb2xlcyI6WyJtYW5hZ2UtYWNjb3VudCIsIm1hbmFnZS1hY2NvdW50LWxpbmtzIiwidmlldy1wcm9maWxlIl19fSwic2NvcGUiOiJvcGVuaWQgZW1haWwgcHJvZmlsZSIsInNpZCI6ImIyYjNmM2EwLWE0MDItNDE3Yy1iYzk0LTJjMTc3MGY2NGMxNCIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJuYW1lIjoiU2l2YSBLYXRhbXJlZGR5IiwicHJlZmVycmVkX3VzZXJuYW1lIjoic2l2YSIsImdpdmVuX25hbWUiOiJTaXZhIiwiZmFtaWx5X25hbWUiOiJLYXRhbXJlZGR5IiwiZW1haWwiOiJzaXZhQGdtYWlsLmNvbSJ9.J4UfeHwDT906wPa58pr9a_AIM8UqrBVB8mO1Dvt_FXNLO5DUHUPWv9X25htrHIkl6L4M_SeElOep8MfFBQWeLq-yjrkd7PsXZFh1bjqSscu0bdeIqMOgBdDKZz97XY6UYS5sWjzaws4KvYcbWvLp3athhMts0Jx1Tag57FPWF1SBXwe9ROLQIDmWbm7-jCqiU9GLmBYv9S03SqvXubAXbINDX63-8i7Uyj1NdrXSGARpELdaoc9WZiX28D1eclaPzyI4r3gJ7wxmpRQ-sBTLcC5fanm__Yfuwja0UgjhK6jtCUS_SuxL2uJv4fu3BiQYGyLGHPKmJ9e1ev9PWFU7CQ",
  "expires_in": 300,
  "refresh_expires_in": 1800,
  "refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICI5N2E1NTU3Ni01MThlLTQ1MDItOWQyNi1jNzVmYjZhNGRhZWEifQ.eyJleHAiOjE2OTU2MTczNTYsImlhdCI6MTY5NTYxNTU1NiwianRpIjoiOTI0YmVhM2MtYWRmZC00NDUwLWFjZWYtM2U5M2Q3MDI3NzQxIiwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo5MTkxL3JlYWxtcy9zaXZhbGFicyIsImF1ZCI6Imh0dHA6Ly9sb2NhbGhvc3Q6OTE5MS9yZWFsbXMvc2l2YWxhYnMiLCJzdWIiOiJjYTFhMmYzNC0xNjE0LTQ1ZGQtODZjMS01ZWFmZmYwODVkOGEiLCJ0eXAiOiJSZWZyZXNoIiwiYXpwIjoibWVzc2FnZXMtd2ViYXBwIiwic2Vzc2lvbl9zdGF0ZSI6ImIyYjNmM2EwLWE0MDItNDE3Yy1iYzk0LTJjMTc3MGY2NGMxNCIsInNjb3BlIjoib3BlbmlkIGVtYWlsIHByb2ZpbGUiLCJzaWQiOiJiMmIzZjNhMC1hNDAyLTQxN2MtYmM5NC0yYzE3NzBmNjRjMTQifQ._1_ODyFLuiOt6kquAQM9MAJpVCBlD6XphSsdaI0d9YM",
  "token_type": "Bearer",
  "id_token": "eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJMeVVPTDg4LVBGM3BYQzFpN3BIeGdFZTJwaWZJY3RyTXJiNklHOElmRTlVIn0.eyJleHAiOjE2OTU2MTU4NTYsImlhdCI6MTY5NTYxNTU1NiwiYXV0aF90aW1lIjowLCJqdGkiOiJkMWRiMzZkMC1kYmMyLTRmYmUtODE4ZS1mMmNjMWUwNzU5NjMiLCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjkxOTEvcmVhbG1zL3NpdmFsYWJzIiwiYXVkIjoibWVzc2FnZXMtd2ViYXBwIiwic3ViIjoiY2ExYTJmMzQtMTYxNC00NWRkLTg2YzEtNWVhZmZmMDg1ZDhhIiwidHlwIjoiSUQiLCJhenAiOiJtZXNzYWdlcy13ZWJhcHAiLCJzZXNzaW9uX3N0YXRlIjoiYjJiM2YzYTAtYTQwMi00MTdjLWJjOTQtMmMxNzcwZjY0YzE0IiwiYXRfaGFzaCI6IjZ4Rk1lTjR6a1Q1UkIzdE9lWFRteEEiLCJhY3IiOiIxIiwic2lkIjoiYjJiM2YzYTAtYTQwMi00MTdjLWJjOTQtMmMxNzcwZjY0YzE0IiwiZW1haWxfdmVyaWZpZWQiOnRydWUsIm5hbWUiOiJTaXZhIEthdGFtcmVkZHkiLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJzaXZhIiwiZ2l2ZW5fbmFtZSI6IlNpdmEiLCJmYW1pbHlfbmFtZSI6IkthdGFtcmVkZHkiLCJlbWFpbCI6InNpdmFAZ21haWwuY29tIn0.UPR78Ps7colWhBAHcGr6hvx4bQdEnvt6j9oDuFtRLTYRMCSb-dlG73ehWDov7AEczcPtfnoEamBgJ_5Hr9XQvA0oQ_9ye8JT6bPrUuUxB-CK6BPsdtyHNyV6jnxolkz26QYsfsB-fHBObnlwVZEKouiGwdGl08GR8xPlqZgK-f1Lo6kKnqLE4i6hdEzJP_z1N5-TE2pSQ-eqc792Mg5z2WB04ug5m2B_NSODVSnXZ_KzYwI7aEKgHBnDac9uisDJrSfqxr2V4lysyRMjUN4odK10hFBn0MdsI-Z1a7d3DKjssyBGZ9k0ZXNWfOBAtzZH7W5s7fLcOViPLYBxgTNUkw",
  "not-before-policy": 0,
  "session_state": "b2b3f3a0-a402-417c-bc94-2c1770f64c14",
  "scope": "openid email profile"
}
```

As you can see, with Resource Owner Password Credentials Flow, the end user needs to share their credentials 
with the client application, which is not recommended.

## Summary
In this part, we have learned how to get **access_token** using **Implicit Flow** and **Resource Owner Password Credentials Flow**.
Remember that **Implicit Flow** and **Resource Owner Password Credentials Flow** are DEPRECATED and not recommended to use anymore.

So far, we have learned how to use various OAuth 2 and OpenID Connect flows.
**Next, we will start building Spring Boot application(s) and protect them using Spring Security OAuth 2 and Keycloak.**

# Spring Security OAuth 2 Tutorial - 4 : Authorization Code Flow with PKCE

In the [Part 3: OAuth 2.0 Client Credentials Flow]({{< relref "2023-09-25-spring-security-oauth2-tutorial-part3.md" >}}), 
we learned how to acquire **access_token** using **Client Credentials Flow**. 
In this article, we will explore how to use **Authorization Code Flow with PKCE**.

<!--more-->


## Authorization Code Flow with PKCE
The **Authorization Code Flow with PKCE** is an OpenId Connect flow primarily designed to secure native, 
mobile applications and Single Page Applications (SPA).
The PKCE, typically pronounced as "pixy", is an acronym for **Proof Key for Code Exchange**.

{{< box info >}}
**IMPORTANT**

The **Authorization Code Flow with PKCE** can be used to secure web apps running on server as well.
In this scenario, PKCE acts as an additional layer of protection.
{{< /box >}}

## Create a "public" Client
Let's create a new client named **messages-spa**.

* **General Settings**:
  * **Client type**: OpenID Connect
  * **Client ID**: messages-spa

* **Capability config**:
  * **Client authentication**: Off
  * **Authorization**: Off
  * **Authentication flow**: Check **Standard flow** On and uncheck the rest of the checkboxes

* **Login settings**:
  * **Root URL**: http://localhost:3000
  * **Home URL**: http://localhost:3000
  * **Valid redirect URIs**: http://localhost:3000/callback
  * **Valid post logout redirect URIs**: http://localhost:3000
  * **Web origins**: http://localhost:3000
    
Once the client is created with the above configuration, you will be taken to the newly created Client's **Settings** page.
Click on the **Advanced** tab, go to **Advanced Settings** section and update **Proof Key for Code Exchange Code Challenge Method** value to **S256**.

## Getting Access Token using Authorization Code Flow with PKCE
In **Authorization Code Flow with PKCE**, first we get an **authorization_code** via **Redirect URL** on front-channel (browser url).

First, we need to generate a **code verifier** which is a cryptographically random string using the characters A-Z, a-z, 0-9, 
and the punctuation characters -._~ (hyphen, period, underscore, and tilde), between 43 and 128 characters long.

Once the **code verifier** is generated, generate the **code challenge** which is a BASE64-URL-encoded string of the SHA256 hash of the code verifier.

You can use https://www.oauth.com/playground/authorization-code-with-pkce.html to generate code verifier and code challenge.

* **code verifier:** M7Q3C-V-_BafRd251gvQLhHw5lYRUYuAlbtT7BCF8cnDiHSV
* **code challenge:** gD2zOSHT__2UcU_BkXqqMld3b7TQ764LaPUOXMSDCMw

Now open the following URL in a browser window:

```shell
http://localhost:9191/realms/sivalabs/protocol/openid-connect/auth?
  response_type=code
  &client_id=messages-spa
  &redirect_uri=http://localhost:3000/callback
  &state=randomstring
  &response_mode=query
  &scope=openid
  &code_challenge=gD2zOSHT__2UcU_BkXqqMld3b7TQ764LaPUOXMSDCMw
  &code_challenge_method=S256
```

* Then you will be redirected to Keycloak's Login page.
* Login with the user credentials **siva/siva1234**.
* Then you will be redirected the **Redirect URI** which has **code** as a query parameter.

```shell

http://localhost:3000/callback?
  state=randomstring
  &session_state=d0da5382-4408-4548-af53-fe4fb948c18c
  &code=052aa7cf-a868-4b95-aa59-6a75eaff26d8.d0da5382-4408-4548-af53-fe4fb948c18c.63de5a66-e986-46ca-9b0f-3944529f3ad9
```

The **authorization_code** is **052aa7cf-a868-4b95-aa59-6a75eaff26d8.d0da5382-4408-4548-af53-fe4fb948c18c.63de5a66-e986-46ca-9b0f-3944529f3ad9**.

Now that we have **authorization_code**, we can get **access_token** using **token_endpoint** as follows:

```shell
curl --location 'http://localhost:9191/realms/sivalabs/protocol/openid-connect/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'grant_type=authorization_code' \
--data-urlencode 'client_id=messages-spa' \
--data-urlencode 'redirect_uri=http://localhost:3000/callback' \
--data-urlencode 'code_verifier=M7Q3C-V-_BafRd251gvQLhHw5lYRUYuAlbtT7BCF8cnDiHSV' \
--data-urlencode 'code=052aa7cf-a868-4b95-aa59-6a75eaff26d8.d0da5382-4408-4548-af53-fe4fb948c18c.63de5a66-e986-46ca-9b0f-3944529f3ad9'
```

Here we are sending the original **code_verifier** which should match with the **code_verifier** sent in authorization_code request.

This should return the JSON response something like this:

```json
{
  "access_token": "eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJMeVVPTDg4LVBGM3BYQzFpN3BIeGdFZTJwaWZJY3RyTXJiNklHOElmRTlVIn0.eyJleHAiOjE2OTU1NzAyNjcsImlhdCI6MTY5NTU2OTk2NywiYXV0aF90aW1lIjoxNjk1NTY5OTQxLCJqdGkiOiJkNzNjOTk4MS1jYzhmLTQ2NzItODcwYy03YzI2OTQ3ZDY3ZTkiLCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjkxOTEvcmVhbG1zL3NpdmFsYWJzIiwiYXVkIjoiYWNjb3VudCIsInN1YiI6ImNhMWEyZjM0LTE2MTQtNDVkZC04NmMxLTVlYWZmZjA4NWQ4YSIsInR5cCI6IkJlYXJlciIsImF6cCI6Im1lc3NhZ2VzLXNwYSIsInNlc3Npb25fc3RhdGUiOiJmZDU1YTdmZi0xNTI2LTQ4MjYtODM1OS00MTFkZTU0ZmJhMWYiLCJhY3IiOiIxIiwiYWxsb3dlZC1vcmlnaW5zIjpbImh0dHA6Ly9sb2NhbGhvc3Q6MzAwMCJdLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiZGVmYXVsdC1yb2xlcy1zaXZhbGFicyIsIm9mZmxpbmVfYWNjZXNzIiwidW1hX2F1dGhvcml6YXRpb24iXX0sInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6Im9wZW5pZCBlbWFpbCBwcm9maWxlIiwic2lkIjoiZmQ1NWE3ZmYtMTUyNi00ODI2LTgzNTktNDExZGU1NGZiYTFmIiwiZW1haWxfdmVyaWZpZWQiOnRydWUsIm5hbWUiOiJTaXZhIEthdGFtcmVkZHkiLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJzaXZhIiwiZ2l2ZW5fbmFtZSI6IlNpdmEiLCJmYW1pbHlfbmFtZSI6IkthdGFtcmVkZHkiLCJlbWFpbCI6InNpdmFAZ21haWwuY29tIn0.f4Ic6ggLBteou71NHzBWaCdcxKaaXnCSM38Glg3Xs1ruQz7uMtMJKewJOpZmCwrvodg0xCF0TZEl-wKWz3CnWEFFE92nIGGp3BIL42Coc4f8_aB_M0YvH3hUQznswYgZLcqvpN3k3e4yA70NfU9LWbNJudBkaLBYCDgPv62_t5620TPyg4cYxjcf2HHoCG4pU3Ms5DX-Zu6tc-aa3RT0uXp7CNzgQF3yqP0kmyu9SY8hkhLV05hdtsc5Szj0mH0e8T55IajM8Z_eYCi20VFaBehohM_Zr6FsP_S69numxhHLBy06JgQd9zUV8DpMPhOC0R4oTla4RpmPZo0B5ApFmw",
  "expires_in": 300,
  "refresh_expires_in": 1774,
  "refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICI5N2E1NTU3Ni01MThlLTQ1MDItOWQyNi1jNzVmYjZhNGRhZWEifQ.eyJleHAiOjE2OTU1NzE3NDEsImlhdCI6MTY5NTU2OTk2NywianRpIjoiNDYwMzlmNGQtMjBjZi00OWUxLTk2ZWEtNjI4NmI0MjdiOWQ2IiwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo5MTkxL3JlYWxtcy9zaXZhbGFicyIsImF1ZCI6Imh0dHA6Ly9sb2NhbGhvc3Q6OTE5MS9yZWFsbXMvc2l2YWxhYnMiLCJzdWIiOiJjYTFhMmYzNC0xNjE0LTQ1ZGQtODZjMS01ZWFmZmYwODVkOGEiLCJ0eXAiOiJSZWZyZXNoIiwiYXpwIjoibWVzc2FnZXMtc3BhIiwic2Vzc2lvbl9zdGF0ZSI6ImZkNTVhN2ZmLTE1MjYtNDgyNi04MzU5LTQxMWRlNTRmYmExZiIsInNjb3BlIjoib3BlbmlkIGVtYWlsIHByb2ZpbGUiLCJzaWQiOiJmZDU1YTdmZi0xNTI2LTQ4MjYtODM1OS00MTFkZTU0ZmJhMWYifQ.QGgw00X_biM0kMLFDEG14UhjCaDBSuk1K8euqTxQpWk",
  "token_type": "Bearer",
  "id_token": "eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJMeVVPTDg4LVBGM3BYQzFpN3BIeGdFZTJwaWZJY3RyTXJiNklHOElmRTlVIn0.eyJleHAiOjE2OTU1NzAyNjcsImlhdCI6MTY5NTU2OTk2NywiYXV0aF90aW1lIjoxNjk1NTY5OTQxLCJqdGkiOiJjYjViMTQ0Ni0zOGQxLTQwN2ItYjIwYy03ZjQ0Mjg1MmZhYmYiLCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjkxOTEvcmVhbG1zL3NpdmFsYWJzIiwiYXVkIjoibWVzc2FnZXMtc3BhIiwic3ViIjoiY2ExYTJmMzQtMTYxNC00NWRkLTg2YzEtNWVhZmZmMDg1ZDhhIiwidHlwIjoiSUQiLCJhenAiOiJtZXNzYWdlcy1zcGEiLCJzZXNzaW9uX3N0YXRlIjoiZmQ1NWE3ZmYtMTUyNi00ODI2LTgzNTktNDExZGU1NGZiYTFmIiwiYXRfaGFzaCI6ImRETGgycU1RdU5FT3F4MkRSYVR3cGciLCJhY3IiOiIxIiwic2lkIjoiZmQ1NWE3ZmYtMTUyNi00ODI2LTgzNTktNDExZGU1NGZiYTFmIiwiZW1haWxfdmVyaWZpZWQiOnRydWUsIm5hbWUiOiJTaXZhIEthdGFtcmVkZHkiLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJzaXZhIiwiZ2l2ZW5fbmFtZSI6IlNpdmEiLCJmYW1pbHlfbmFtZSI6IkthdGFtcmVkZHkiLCJlbWFpbCI6InNpdmFAZ21haWwuY29tIn0.BTuFqcyrjB99R9kBfeLbVRSL5EMtgwS-BSgue77eGmFZxoBMV6UXM2qjWCoF9Wdozy2xmOQcqehvNGpnvR34YHfdrCxymuIZ7jIp7LvNW8sf5ZzsqnCq2OLCKPH4pyWFPqAGfvBYXqzhammNjPITKTKHH11nIcGnDGt7bhfDf-iDj3saDf0CZNkyOmh1Udi3Bkl3O4OBERWbg9xpsNGVks0SDONkDtLSL0IO1ODfgJRq1ZASRdn2i67EMDh5kpLnpKHPj7a1rMZyIUTdhbF4_sODWj3D6STFZxS_FjhcXdjYHLvtLdQBIOmHDjTqEXpwNp5OrVBtEvjgMj4ERNyoMw",
  "not-before-policy": 0,
  "session_state": "fd55a7ff-1526-4826-8359-411de54fba1f",
  "scope": "openid email profile"
}
```

With this **access_token** a Resource Server can invoke another Resource Server secured API endpoints.

{{< box info >}}
**How is PKCE secure?**

When I first looked at the PKCE flow, I was wondering how exactly PKCE is secure?
Anybody can create code verifier and code challenge, and there is no need to know client_secret.

This [How is PKCE secure?](https://security.stackexchange.com/questions/239946/how-is-pkce-secure)
question is exactly what I had in my mind. Please read the answers to understand how PKCE Flow is secure.

Another interesting question worth reading [What is PKCE actually protecting?](https://security.stackexchange.com/questions/175465/what-is-pkce-actually-protecting). 
{{< /box >}}

## Authorization Code Flow with PKCE using Postman

We can use [Postman](https://www.postman.com/) to get **access_token** using **Authorization Code Flow with PKCE** as follows:

* Open a New Request tab in Postman
* Go to the **Authorization** tab and select **OAuth 2.0** as **Type**.
* Under **Configure New Token** section:
  * **Grant Type**: Authorization Code (With PKCE)
  * **Callback URL**: http://localhost:3000/callback
  * **Auth URL**: http://localhost:9191/realms/sivalabs/protocol/openid-connect/auth
  * **Access Token URL**: http://localhost:9191/realms/sivalabs/protocol/openid-connect/token
  * **Client ID**: messages-spa
  * **Client Secret**: "" (leave it as blank)
  * **Code Challenge Method**: S256
  * **Code Verifier**: M7Q3C-V-_BafRd251gvQLhHw5lYRUYuAlbtT7BCF8cnDiHSV
  * **Scope**: profile
  * **State**: randomstring
  * **Client Authentication**: Send as Basic Auth header
* Click on **Get New Access Token** button
* Postman will show Keycloak's Login page in a popup
* Login with the credentials **siva/siva1234**
* Now you should be able to see the response with **Token Details**

{{< figure src="/images/oauth2-code-flow-pkce-postman.webp" alt="OAuth2 Authorization Code Flow with PKCE using Postman" >}}

If you specify **"openid profile"** as **scope** then you will also get **id_token**.

## Summary
In this part, we have learned how to get **access_token** using **Authorization Code Flow with PKCE**.
We also learned how to use Postman to do the same.

In the next part, we will explore how **OAuth 2.0 Implicit Flow** and **Resource Owner Password Flow** works.

# Spring Security OAuth 2 Tutorial - 3 : Client Credentials Flow

In the [Part 2: OAuth 2.0 Authorization Code Flow]({{< relref "2023-09-24-spring-security-oauth2-tutorial-part2.md" >}}), 
we learned how to authenticate a user using **Authorization Code Flow**. 

In this article, we will explore how to use **Client Credentials Flow** that is typically used for **Service-to-Service** 
communication without any user (Resource Owner) context.

<!--more-->


## Client Credentials Flow
Sometimes a Resource Server needs to interact with another Resource Server without any user context.
For instance, **Resource Server A** may run a scheduled job that will invoke a secured REST API endpoint on **Resource Server B**.
In these kinds of scenarios, we can use **Client Credentials Flow** to get an **access_token** from **Authorization Server**.

## Create a Client with Client Credentials Flow enabled
In order to use **Client Credentials Flow** for a Client, **Client Credentials Grant** support should be enabled.
In Keycloak **Client Credentials Grant** can be enabled by enabling **Service accounts roles** Authentication flow.

{{< box info >}}
**OAuth2 Client multiple Grants**

An OAuth 2.0 client can have multiple grant types like Authorization Code, Client Credentials, Implicit, etc enabled.
{{< /box >}}

Let's create a new client named **archival-service**.

* **General Settings**:
  * **Client type**: OpenID Connect
  * **Client ID**: archival-service

* **Capability config**:
  * **Client authentication**: On
  * **Authorization**: Off
  * **Authentication flow**: Check **Service accounts roles** On and uncheck the rest of the checkboxes

* **Login settings**:
  * **Root URL**: http://localhost:8282
  * **Home URL**: http://localhost:8282
    
Once the client is created with the above configuration, you will be taken to the newly created Client's **Settings** page.
Click on the **Credentials** tab and copy the **Client secret** value.

In my case, the **Client secret** is **bL1a2V2kouKh4sBMX0UrSmc0d3qubD1a**.

## Getting Access Token using Client Credentials Flow

We can get **access_token** using **Client Credentials Flow** using the following cURL command:

```shell
curl --location 'http://localhost:9191/realms/sivalabs/protocol/openid-connect/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'grant_type=client_credentials' \
--data-urlencode 'client_id=archival-service' \
--data-urlencode 'client_secret=bL1a2V2kouKh4sBMX0UrSmc0d3qubD1a'
```

This should return a JSON response something like this:

```json
{
  "access_token": "eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJMeVVPTDg4LVBGM3BYQzFpN3BIeGdFZTJwaWZJY3RyTXJiNklHOElmRTlVIn0.eyJleHAiOjE2OTU1NTU3NjgsImlhdCI6MTY5NTU1NTQ2OCwianRpIjoiZDQwZjEwZjYtMGVjNi00YzAyLWI2ZTktOGE5NzMyYmIwODUzIiwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo5MTkxL3JlYWxtcy9zaXZhbGFicyIsImF1ZCI6ImFjY291bnQiLCJzdWIiOiIwZjQyYjZkYi0yZWRjLTQyNDUtODAwNC04OWM0Mjg1NDY2MjQiLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJhcmNoaXZhbC1zZXJ2aWNlIiwiYWNyIjoiMSIsImFsbG93ZWQtb3JpZ2lucyI6WyJodHRwOi8vbG9jYWxob3N0OjgyODIiXSwicmVhbG1fYWNjZXNzIjp7InJvbGVzIjpbImRlZmF1bHQtcm9sZXMtc2l2YWxhYnMiLCJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9uIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiYWNjb3VudCI6eyJyb2xlcyI6WyJtYW5hZ2UtYWNjb3VudCIsIm1hbmFnZS1hY2NvdW50LWxpbmtzIiwidmlldy1wcm9maWxlIl19fSwic2NvcGUiOiJlbWFpbCBwcm9maWxlIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJjbGllbnRIb3N0IjoiMTkyLjE2OC4xMTcuMSIsInByZWZlcnJlZF91c2VybmFtZSI6InNlcnZpY2UtYWNjb3VudC1hcmNoaXZhbC1zZXJ2aWNlIiwiY2xpZW50QWRkcmVzcyI6IjE5Mi4xNjguMTE3LjEiLCJjbGllbnRfaWQiOiJhcmNoaXZhbC1zZXJ2aWNlIn0.HUjqMP8lxHJfNhxlwtZm8pui_LafuEGf9LxXh5K5tWIQHO6DkfZBWqAhqYggv3mv1QsbRO4akEnldTzrstoFN-4xO_kxKFv10VZLdcUcKf_hhjoySFcKlp0eh1cBRn-C-YyDun5kNOg8kx38G_T7T6F1GMXPAAbMabngmhX2Ke_V4SJZsZvphNu15tmKXnqk8ZUyPRGWSfB9yb19JU79FmiE7RJPMAKSFyV5nHt1HxhSsE7NHeMaqUAn7M5bsQa6em-YxNp3oB2ROA_s0RsACHk41U3d3gARBpZcR2ad2kO0DL_jCV22I2uS4XJCpkvE1MMB5nD6CPwt3Re-8-gGjw",
  "expires_in": 300,
  "refresh_expires_in": 0,
  "token_type": "Bearer",
  "not-before-policy": 0,
  "scope": "email profile"
}
```

With this **access_token** a Resource Server can invoke another Resource Server secured API endpoints.


{{< box info >}}
**OAuth2 Roles and Permissions**

Just having access_token doesn't necessarily mean you can access any resource on a Resource Server.
The access_token should have required roles/privileges to be able to successfully invoke a REST API endpoints on another Resource Server.

We will talk about roles and permission in the upcoming articles in the series.
{{< /box >}}

## Client Credentials Flow using Postman

We can use [Postman](https://www.postman.com/) to get **access_token** using **Client Credentials Flow** as follows:

* Open a New Request tab in Postman
* Go to the **Authorization** tab and select **OAuth 2.0** as **Type**.
* Under **Configure New Token** section:
  * **Grant Type**: Client Credentials
  * **Access Token URL**: http://localhost:9191/realms/sivalabs/protocol/openid-connect/token
  * **Client ID**: archival-service
  * **Client Secret**: bL1a2V2kouKh4sBMX0UrSmc0d3qubD1a
  * **Scope**: openid profile
  * **Client Authentication**: Send as Basic Auth header
* Click on **Get New Access Token** button
* Now you should be able to see the response with **Token Details**

{{< figure src="/images/oauth2-client-credentials-flow-postman.webp" alt="OAuth2 Client Credentials Flow using Postman" >}}


## Summary
In this part, we have learned how to get **access_token** using **Client Credentials Flow**.
We also learned how to use Postman to do the same.

In the next part, we will explore how **OAuth 2.0 Authorization Code Flow with PKCE** works.

# Spring Security OAuth 2 Tutorial - 2 : Authorization Code Flow

In the [Part 1: Getting familiar with OAuth 2 concepts]({{< relref "2023-09-22-spring-security-oauth2-tutorial-part1.md" >}}), we learned how to set up Keycloak, created a realm, a client with Standard flow enabled and a user.
In this Part 2, you will learn how to authenticate a user using **Authorization Code Flow**.

<!--more-->


First of all, let's clear up the confusion between **Authorization Code Grant Type** vs **Authorization Code Flow**.

As I mentioned earlier, **OAuth 2.0** spec concern about only **Authorization** and 
**OpenID Connect** spec is added as a layer on top of OAuth 2.0 to handle **Authentication**.

The **Authorization Code Grant Type** is OAuth 2.0 terminology, whereas **Authorization Code Flow** is OpenID Connect terminology.
They both work in the same way and the difference is in the **scope**. 
The difference will become clear while we explore Authorization Code Flow with examples later in this article.

To quickly recap, we have the following details of **Client** and **User** that we have created in the previous part.

* **Client id**: messages-webapp
* **Client secret**: **qVcg0foCUNyYbgF0Sg52zeIhLYyOwXpQ**
* **Username**: siva
* **Password**: siva1234

## OAuth 2.0 High level Architecture 
Here is a high-level Architecture diagram of an OAuth 2.0 based system.

{{< figure src="/images/oauth2-roles.webp" alt="OAuth2 Roles" >}}

* **The Resource Owner** (end user) wants to his/her data stored on the **Resource Server** using the **Client** application.
* The **Resource Server** data is protected, and you need **access_token** to be able to access the data.
* The **Client** application offloaded the responsibility of managing the users, issuing the **access_tokens**, validating them to the **Authorization Server**.
* When you (Resource Owner) try to access a protected resource on the **Client** application, you will be redirected to **Authorization Server** where you need to authenticate yourself by providing the user credentials.
* If the authentication is successful, the **Authorization Server** issues an **access_token** to the **Client**.
* Then the **Client** can access the protected user's data using the **access_token**.

This is a very high-level flow of how an end-user can access their protected resources using OAuth 2.0 based security.

Now if we map these various components in our current Keycloak setup, it will look like this:

* **Authorization Server**: Keycloak
* **Client**: messages-webapp (which is not yet created)
* **Resource Server**: messages-service (which is not yet created)
* **Resource Owner**: user(siva)

As I mentioned in the previous part, there are different ways such as **Authorization Code Flow**, 
**Client Credentials Flow**, **Implicit Flow**, **Resource Owner Password Flow** to get **access_token**.

In this part, we are going to focus on acquiring **access_token** using **Authorization Code Flow**.

Before getting into it, first let's see how to get some important URLs that we will use going forward.

Login into Keycloak Admin Console, select **sivalabs** realm, click on **Realm settings**.
You can see the **Endpoints** section and a link to **OpenID Endpoint Configuration**. 
If you click on that link, you will see the following JSON response:

```json
{
    "issuer": "http://localhost:9191/realms/sivalabs",
    "authorization_endpoint": "http://localhost:9191/realms/sivalabs/protocol/openid-connect/auth",
    "token_endpoint": "http://localhost:9191/realms/sivalabs/protocol/openid-connect/token",
    "introspection_endpoint": "http://localhost:9191/realms/sivalabs/protocol/openid-connect/token/introspect",
    "userinfo_endpoint": "http://localhost:9191/realms/sivalabs/protocol/openid-connect/userinfo",
    "end_session_endpoint": "http://localhost:9191/realms/sivalabs/protocol/openid-connect/logout",
    "frontchannel_logout_session_supported": true,
    "frontchannel_logout_supported": true,
    "jwks_uri": "http://localhost:9191/realms/sivalabs/protocol/openid-connect/certs",
    "check_session_iframe": "http://localhost:9191/realms/sivalabs/protocol/openid-connect/login-status-iframe.html",
    "grant_types_supported": [
        "authorization_code",
        "implicit",
        "refresh_token",
        "password",
        "client_credentials",
        "urn:ietf:params:oauth:grant-type:device_code",
        "urn:openid:params:grant-type:ciba"
    ],
    ...,
  "response_types_supported": [
        "code",
        "none",
        "id_token",
        "token",
        "id_token token",
        "code id_token",
        "code token",
        "code id_token token"
    ],
    ...
    ...
}
```

We will use **authorization_endpoint**, **token_endpoint** during the **Authorization Code Flow**.

## Authentication using Authorization Code Flow

In **Authorization Code Flow**, first we get an **authorization_code** via **Redirect URL** on front-channel (browser url)
and then use that **authorization_code** along with **client_id** and **client_secret** to get **access_token** via the back-channel (code on server).

### Get Authorization Code

First, we are going to request for authorization_code using the **authorization_endpoint** as follows:

* Open the following URL in a browser window.

```shell
http://localhost:9191/realms/sivalabs/protocol/openid-connect/auth?
  response_type=code
  &client_id=messages-webapp
  &scope=profile
  &state=randomstring
  &redirect_uri=http://localhost:8080/callback
```

* Then you will be redirected to Keycloak's Login page. 
* Login with the user credentials **siva/siva1234**.
* Then you will be redirected the **Redirect URI** which has **code** as a query parameter.

```shell
http://localhost:8080/callback?
  state=randomstring
  &session_state=07aecbd0-dc7e-487e-a1e5-ee36b3fdd5d7
  &code=6e4fb072-df61-4d13-998d-469abf60492b.07aecbd0-dc7e-487e-a1e5-ee36b3fdd5d7.27a26554-da99-417c-a775-bf6559c98f02
```

The **authorization_code** is **6e4fb072-df61-4d13-998d-469abf60492b.07aecbd0-dc7e-487e-a1e5-ee36b3fdd5d7.27a26554-da99-417c-a775-bf6559c98f02**.

**NOTE:** The **redirect_uri** you specified in the request URL should match one of the Client's **Valid redirect URIs** 
configured in the **Login settings**.

### Get AccessToken
Now that we have **authorization_code**, we can get **access_token** using **token_endpoint** as follows:

```shell
curl --location 'http://localhost:9191/realms/sivalabs/protocol/openid-connect/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'grant_type=authorization_code' \
--data-urlencode 'client_id=messages-webapp' \
--data-urlencode 'client_secret=qVcg0foCUNyYbgF0Sg52zeIhLYyOwXpQ' \
--data-urlencode 'code=6e4fb072-df61-4d13-998d-469abf60492b.07aecbd0-dc7e-487e-a1e5-ee36b3fdd5d7.27a26554-da99-417c-a775-bf6559c98f02' \
--data-urlencode 'redirect_uri=http://localhost:8080/callback' \
--data-urlencode 'scope=profile'
```

This should return a JSON response something like this:

```json
{
    "access_token": "eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJMeVVPTDg4LVBGM3BYQzFpN3BIeGdFZTJwaWZJY3RyTXJiNklHOElmRTlVIn0.eyJleHAiOjE2OTU1NDAxODEsImlhdCI6MTY5NTUzOTg4MSwiYXV0aF90aW1lIjoxNjk1NTM5ODQ3LCJqdGkiOiIyYjRmNjNlMC0xNzg2LTRmYzctOTIxNy01ZDEzMjg5MDhhN2QiLCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjkxOTEvcmVhbG1zL3NpdmFsYWJzIiwiYXVkIjoiYWNjb3VudCIsInN1YiI6ImNhMWEyZjM0LTE2MTQtNDVkZC04NmMxLTVlYWZmZjA4NWQ4YSIsInR5cCI6IkJlYXJlciIsImF6cCI6Im1lc3NhZ2VzLXdlYmFwcCIsInNlc3Npb25fc3RhdGUiOiIwN2FlY2JkMC1kYzdlLTQ4N2UtYTFlNS1lZTM2YjNmZGQ1ZDciLCJhY3IiOiIxIiwiYWxsb3dlZC1vcmlnaW5zIjpbImh0dHA6Ly9sb2NhbGhvc3Q6ODA4MCJdLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiZGVmYXVsdC1yb2xlcy1zaXZhbGFicyIsIm9mZmxpbmVfYWNjZXNzIiwidW1hX2F1dGhvcml6YXRpb24iXX0sInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6ImVtYWlsIHByb2ZpbGUiLCJzaWQiOiIwN2FlY2JkMC1kYzdlLTQ4N2UtYTFlNS1lZTM2YjNmZGQ1ZDciLCJlbWFpbF92ZXJpZmllZCI6dHJ1ZSwibmFtZSI6IlNpdmEgS2F0YW1yZWRkeSIsInByZWZlcnJlZF91c2VybmFtZSI6InNpdmEiLCJnaXZlbl9uYW1lIjoiU2l2YSIsImZhbWlseV9uYW1lIjoiS2F0YW1yZWRkeSIsImVtYWlsIjoic2l2YUBnbWFpbC5jb20ifQ.W23gLsCkaWMi85BvvgzHhAdAEKMBPP6pzLhHnkKamzfG0Sw6XILPlXagrLSZBgzkaNSgt3ttZ6JLizh_oKuKmZBucAWtsBCo6ZIiPZDUDrEk95sDybhIEJZH-b1LSWMVLa2NXLE_FkEtTE44zCi2B2MxFQzfCuf7RJ5WnnW8XLAD6qbZVF64HT2wYNRiZiuq9MPwKPU-pOAGjGa53ko37EOgc0RjVE14D1--RFlGfLmGc4aGrl6OSElf1X8ya4wyDT8vP9pbWK4Fe-XLI9zOlYdY506TdOtAoe28Pu4GPhL5PrBQ1-2pY30_fN6BVgFk0qyff76yFyNNMFNNm6aA5w",
    "expires_in": 299,
    "refresh_expires_in": 1765,
    "refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICI5N2E1NTU3Ni01MThlLTQ1MDItOWQyNi1jNzVmYjZhNGRhZWEifQ.eyJleHAiOjE2OTU1NDE2NDcsImlhdCI6MTY5NTUzOTg4MiwianRpIjoiOTZkYzFhYzYtMWI1ZC00MjA2LWFiM2EtOWQ4MDg2Y2ZmMjc1IiwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo5MTkxL3JlYWxtcy9zaXZhbGFicyIsImF1ZCI6Imh0dHA6Ly9sb2NhbGhvc3Q6OTE5MS9yZWFsbXMvc2l2YWxhYnMiLCJzdWIiOiJjYTFhMmYzNC0xNjE0LTQ1ZGQtODZjMS01ZWFmZmYwODVkOGEiLCJ0eXAiOiJSZWZyZXNoIiwiYXpwIjoibWVzc2FnZXMtd2ViYXBwIiwic2Vzc2lvbl9zdGF0ZSI6IjA3YWVjYmQwLWRjN2UtNDg3ZS1hMWU1LWVlMzZiM2ZkZDVkNyIsInNjb3BlIjoiZW1haWwgcHJvZmlsZSIsInNpZCI6IjA3YWVjYmQwLWRjN2UtNDg3ZS1hMWU1LWVlMzZiM2ZkZDVkNyJ9.G5shFoEF1ptXUHM0FommjZ40fF1r61rd0fHU33R1CAc",
    "token_type": "Bearer",
    "not-before-policy": 0,
    "session_state": "07aecbd0-dc7e-487e-a1e5-ee36b3fdd5d7",
    "scope": "email profile"
}
```

**NOTE:** The **authorization_code** is valid only for a short period of time. 
So, once you get **authorization_code** then you should quickly invoke the token endpoint.
Otherwise, you will get response like this:

```json
{
    "error": "invalid_grant",
    "error_description": "Code not valid"
}
```

Finally, we got **access_token** which we can use while accessing a protected resource on a Resource Server.

## Authorization Code Flow using Postman
As you can see, getting **access_token** is a 2-step process and **authorization_code** might become invalid 
by the time we invoke **token_endpoint**.

We can use [Postman](https://www.postman.com/) to make it simple as follows:

* Open a New Request tab in Postman
* Go to the **Authorization** tab and select **OAuth 2.0** as **Type**.
* Under **Configure New Token** section:
  * **Grant Type**: Authorization Code
  * **Callback URL**: http://localhost:8080/callback
  * **Auth URL**: http://localhost:9191/realms/sivalabs/protocol/openid-connect/auth
  * **Access Token URL**: http://localhost:9191/realms/sivalabs/protocol/openid-connect/token
  * **Client ID**: messages-webapp
  * **Client Secret**: qVcg0foCUNyYbgF0Sg52zeIhLYyOwXpQ
  * **Scope**: profile
  * **State**: randomstring
  * **Client Authentication**: Send as Basic Auth header
* Click on **Get New Access Token** button
* Postman will open a popup showing Keycloak Login page
* Login with the user credentials **siva/siva1234**
* Now you should be able to see the response with **Token Details**

{{< figure src="/images/oauth2-code-flow-postman.webp" alt="OAuth2 Auth Code Flow using Postman" >}}

Simple, isn't it :-)

## Getting ID Token
If you have noticed, in the previous request we specified only **"profile"** as **scope**, 
and we got **access_token** and **refresh_token** in the response. This is OAuth 2.0 **Authorization Code Grant Type**.

Now, if you specify **"openid profile"** as **scope** then you will also get **id_token**, 
which contains more information regarding the authenticated user which is the Resource Owner, in the response.
This is **OpenID Connect** based **Authorization Code Flow**.

```json
{
    "access_token": "eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJMeVVPTDg4LVBGM3BYQzFpN3BIeGdFZTJwaWZJY3RyTXJiNklHOElmRTlVIn0.eyJleHAiOjE2OTU1NDIyMTMsImlhdCI6MTY5NTU0MTkxMywiYXV0aF90aW1lIjoxNjk1NTQxMTg0LCJqdGkiOiIyZDM1NWFjNC0zOTRmLTQ3ZjktOTFiMC05Yjc3MjU0NDBjZDQiLCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjkxOTEvcmVhbG1zL3NpdmFsYWJzIiwiYXVkIjoiYWNjb3VudCIsInN1YiI6ImNhMWEyZjM0LTE2MTQtNDVkZC04NmMxLTVlYWZmZjA4NWQ4YSIsInR5cCI6IkJlYXJlciIsImF6cCI6Im1lc3NhZ2VzLXdlYmFwcCIsInNlc3Npb25fc3RhdGUiOiJmNDJmMWU5NC0zMmFiLTRjYzktYTRiOS1lMGRmNTIwNWY1NTQiLCJhY3IiOiIwIiwiYWxsb3dlZC1vcmlnaW5zIjpbImh0dHA6Ly9sb2NhbGhvc3Q6ODA4MCJdLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiZGVmYXVsdC1yb2xlcy1zaXZhbGFicyIsIm9mZmxpbmVfYWNjZXNzIiwidW1hX2F1dGhvcml6YXRpb24iXX0sInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6Im9wZW5pZCBlbWFpbCBwcm9maWxlIiwic2lkIjoiZjQyZjFlOTQtMzJhYi00Y2M5LWE0YjktZTBkZjUyMDVmNTU0IiwiZW1haWxfdmVyaWZpZWQiOnRydWUsIm5hbWUiOiJTaXZhIEthdGFtcmVkZHkiLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJzaXZhIiwiZ2l2ZW5fbmFtZSI6IlNpdmEiLCJmYW1pbHlfbmFtZSI6IkthdGFtcmVkZHkiLCJlbWFpbCI6InNpdmFAZ21haWwuY29tIn0.IQ52KDoyaGztq1hYqMjZici6eWxPOnLa3RSz7E23Cp1607wVwrCtWwYLAfwewzedrdkjYO9w58qEww-LvsWuzOr85i85JcHL7gQ7kYXS4kRvsKNUs-41wnJUEI6V4-9BduQdSarE43VMUNVS9ZUlNWCsAa7yVKOLdlLEbtM7bWWyGekojye6AQrB9enpKnRGcZaQWRhUZ6k07-d91QeyT2P7VRMit4gjaZmnaSlUKUvKzSYdmRhpB8vcNfggFTeNPgedYQOOwA-vbMWW0LQVElgtvi6eSUTksyFMwaP9iGdI0HyTW7IFi9ddFn3hq43GNahfrckoDeKXtkKQEPOgng",
    "expires_in": 299,
    "refresh_expires_in": 1765,
    "refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICI5N2E1NTU3Ni01MThlLTQ1MDItOWQyNi1jNzVmYjZhNGRhZWEifQ.eyJleHAiOjE2OTU1NDM3MTIsImlhdCI6MTY5NTU0MTkxMywianRpIjoiYWQyNWQwN2UtZWY0OS00M2YxLWI1ZWUtOThlYzU2ZmZjYWY0IiwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo5MTkxL3JlYWxtcy9zaXZhbGFicyIsImF1ZCI6Imh0dHA6Ly9sb2NhbGhvc3Q6OTE5MS9yZWFsbXMvc2l2YWxhYnMiLCJzdWIiOiJjYTFhMmYzNC0xNjE0LTQ1ZGQtODZjMS01ZWFmZmYwODVkOGEiLCJ0eXAiOiJSZWZyZXNoIiwiYXpwIjoibWVzc2FnZXMtd2ViYXBwIiwic2Vzc2lvbl9zdGF0ZSI6ImY0MmYxZTk0LTMyYWItNGNjOS1hNGI5LWUwZGY1MjA1ZjU1NCIsInNjb3BlIjoib3BlbmlkIGVtYWlsIHByb2ZpbGUiLCJzaWQiOiJmNDJmMWU5NC0zMmFiLTRjYzktYTRiOS1lMGRmNTIwNWY1NTQifQ.xClg0y5senUjiPIa_AzqDlSQ6M5isUBthzgYprfbNN8",
    "id_token": "eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJMeVVPTDg4LVBGM3BYQzFpN3BIeGdFZTJwaWZJY3RyTXJiNklHOElmRTlVIn0.eyJleHAiOjE2OTU1NDIyMTMsImlhdCI6MTY5NTU0MTkxMywiYXV0aF90aW1lIjoxNjk1NTQxMTg0LCJqdGkiOiJjYzkyOGVkOC1kMGEzLTQ0YjUtOTEyMS1mOTYwOTc0YzM3ODciLCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjkxOTEvcmVhbG1zL3NpdmFsYWJzIiwiYXVkIjoibWVzc2FnZXMtd2ViYXBwIiwic3ViIjoiY2ExYTJmMzQtMTYxNC00NWRkLTg2YzEtNWVhZmZmMDg1ZDhhIiwidHlwIjoiSUQiLCJhenAiOiJtZXNzYWdlcy13ZWJhcHAiLCJzZXNzaW9uX3N0YXRlIjoiZjQyZjFlOTQtMzJhYi00Y2M5LWE0YjktZTBkZjUyMDVmNTU0IiwiYXRfaGFzaCI6IkF0R1B6RXdnR3lfejB5TDhWcXc0QmciLCJhY3IiOiIwIiwic2lkIjoiZjQyZjFlOTQtMzJhYi00Y2M5LWE0YjktZTBkZjUyMDVmNTU0IiwiZW1haWxfdmVyaWZpZWQiOnRydWUsIm5hbWUiOiJTaXZhIEthdGFtcmVkZHkiLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJzaXZhIiwiZ2l2ZW5fbmFtZSI6IlNpdmEiLCJmYW1pbHlfbmFtZSI6IkthdGFtcmVkZHkiLCJlbWFpbCI6InNpdmFAZ21haWwuY29tIn0.f6kIyFDu9kaMBufQWgtta-ZnEDGUNWDa9YVmrsuZDVCWy4roV-dd4Bj41Ncg1bYquHmCTgjk4u2FT7tYxOu4aj5aZS9xGvBDn3zUMOehxY4VxLa-3YSZ-DLsEburxGWFZmrDXbGGP59o7fgLYxsDEsXxuq7LVwzOPYgtNsnleihdADSPNJP9wQ-4_ozWUx1rXEkcJo93S9w6BIiGeKBZBXHMCD8wjxGJPLJh078UumqgypZiEBsmlJRMDQE23k8s2K1Txaru7zDbw1mAu43-lki-sTJgqXnOG8kg3tF5lFK0fLTlG9cB5i8oXZ2zJ3mUbTDx6gZCjfA85Oc_q_vUdw",
    "token_type": "Bearer",
    "not-before-policy": 0,
    "session_state": "f42f1e94-32ab-4cc9-a4b9-e0df5205f554",
    "scope": "openid email profile"
}
```

## Summary
In this part, we have learned how to get **access_token** and **id_token** using **Authorization Code Flow**.
We also learned how to use Postman to perform the flow easily.

In the next part, we will explore how **OAuth 2.0 Client Credentials Flow** works.

# Spring Security OAuth 2 Tutorial - 1 : Getting familiar with OAuth 2 concepts

Security is a complex topic to understand in-depth. In addition to that, implementing security for complex microservices based systems 
using [OAuth 2.0](https://datatracker.ietf.org/doc/html/rfc6749) and 
[OpenID Connect](https://openid.net/specs/openid-connect-core-1_0.html) specs is even harder. 
Frameworks and libraries, like [Spring Security](https://spring.io/projects/spring-security), help to reduce the complexity,
but still there is a steep learning curve to understand how to properly implement security.

<!--more-->


In this **Spring Security OAuth2 Tutorial series**, I would like to share my learning on implementing security 
for a simple microservices based application using [Spring Security OAuth2](https://docs.spring.io/spring-security/reference/servlet/oauth2/index.html). 

There are many Identity Provider solutions like **Keycloak**, **Okta**, **Auth0**, etc.
In this series, we are going to use [Keycloak](https://www.keycloak.org/), which is an Open Source Identity and Access Management solution.

{{< box warning >}}
**I am NOT a Security/OAuth2 Expert**

I am certainly not an expert on Security, OAuth2, Keycloak.
I am sharing my learning based on my understanding of the concepts.
If you think any of the concepts or explanations is incorrect, please feel free to let me know.
{{< /box >}}


* [Spring Security OAuth 2 Tutorial - 1 : Getting familiar with OAuth 2 concepts]({{< relref "2023-09-22-spring-security-oauth2-tutorial-part1.md" >}})
* [Spring Security OAuth 2 Tutorial - 2 : Authorization Code Flow]({{< relref "2023-09-24-spring-security-oauth2-tutorial-part2.md" >}})
* [Spring Security OAuth 2 Tutorial - 3 : Client Credentials Flow]({{< relref "2023-09-25-spring-security-oauth2-tutorial-part3.md" >}})
* [Spring Security OAuth 2 Tutorial - 4 : Authorization Code Flow with PKCE]({{< relref "2023-09-27-spring-security-oauth2-tutorial-part4.md" >}})
* [Spring Security OAuth 2 Tutorial - 5 : Implicit & Resource Owner Password Credentials Flows]({{< relref "2023-09-27-spring-security-oauth2-tutorial-part5.md" >}})
* [Spring Security OAuth 2 Tutorial - 6 : Microservices Sample Project Setup]({{< relref "2023-09-28-spring-security-oauth2-tutorial-part6.md" >}})
* [Spring Security OAuth 2 Tutorial - 7 : Securing Spring MVC Client Application]({{< relref "2023-09-28-spring-security-oauth2-tutorial-part7.md" >}})
* [Spring Security OAuth 2 Tutorial - 8 : Securing Resource Server]({{< relref "2023-10-02-spring-security-oauth2-tutorial-part8.md" >}})
* [Spring Security OAuth 2 Tutorial - 9 : Invoking Secured Resource Server APIs from Client Application]({{< relref "2023-10-06-spring-security-oauth2-tutorial-part9.md" >}})
* [Spring Security OAuth 2 Tutorial - 10 : Service to Service Communication using Client Credentials Flow]({{< relref "2023-10-09-spring-security-oauth2-tutorial-part10.md" >}})

We are not going to jump right into the implementation of Spring Security OAuth2.
Instead, we will learn one thing at a time in slow pace so that it won't be overwhelming.

## Understand the basics of OAuth 2.0 and OpenID Connect
The first step in the journey of learning OAuth 2.0 and OpenID Connect is to understand some core concepts
such as what are the various roles in OAuth2, what are the various Grant Types and which one to use when.

Instead of me repeating what I learned from this great video, I highly recommend watching this video
to get a clear understanding of OAuth 2.0 and OpenID Connect concepts.

**OAuth 2.0 and OpenID Connect (in plain English)**

{{< youtube 996OiexHze0 >}}

To quickly summarize, in an OAuth2 system there are various components that play different roles and there are different approaches to authenticate a user.

### OAuth2.0 Roles

* **Resource Owner:** The resource owner is typically the end user, who authorizes an application (Client) to access his/her account.
* **Resource Server:** Server hosting the protected resources. This is the API you want to access.
* **Client:** An application (which Resource Owner is using) requesting access to a protected resource on behalf of the Resource Owner.
* **Authorization Server:** Server that authenticates the Resource Owner and issues access tokens after successful authorization.

### OAuth2.0 Grant Types
* Authorization Code Flow
* Authorization Code Flow with PKCE
* Client Credentials Flow
* Implicit Flow (not recommended)
* Resource Owner Password Flow (not recommended)

OAuth 2.0 only concerns about **Authorization**, but not **Authentication**.
**OpenID Connect** specification is created to address **Authentication** which is built as a layer on top of OAuth 2.0.

We will discuss more on which grant type to use when in the upcoming parts of this series.

## Installing Keycloak
As mentioned earlier, we are going to use Keycloak for implementing OAuth2.0/OpenID Connect based security.
Let's get started with the installation of Keycloak using docker compose. 

```yaml
version: '3.8'

services:
  keycloak:
    image: quay.io/keycloak/keycloak:22.0.3
    command: ['start-dev']
    container_name: keycloak
    hostname: keycloak
    environment:
      - KEYCLOAK_ADMIN=admin
      - KEYCLOAK_ADMIN_PASSWORD=admin1234
    ports:
      - "9191:8080"
```

Now start the Keycloak container as follows:

```shell
$ docker compose up -d
```

We started the keycloak container in dev mode and configured Admin User credentials using environment variables.
The container's port 8080 is mapped to host's port 9191.

Now you can access the Keycloak Admin Console at [http://localhost:9191](http://localhost:9191).

## Create new Realm
From the Keycloak's documentation:
> A realm manages a set of users, credentials, roles, and groups. 
> A user belongs to and logs into a realm. Realms are isolated from one another and can only manage 
> and authenticate the users that they control.

You can create a new realm to isolate a set of clients, users, roles, etc from others.

Once you login into Admin Console using the credentials **admin/admin1234**, 
there is a dropdown in the top left corner that provides an option to create a new realm. 

Enter **Realm name** as **sivalabs**, keep **Enabled** **On** and click on **Create**.

Upon successful creation, the new realm should be automatically selected. 
If not, select the **sivalabs** realm from the dropdown.

## Create Client
To create a new client, click on **Clients** on the left side nav menu and click on **Create client** button. 

* **General Settings**:
  * **Client type**: OpenID Connect
  * **Client ID**: messages-webapp

* **Capability config**:
  * **Client authentication**: On
  * **Authorization**: Off
  * **Authentication flow**: Check **Standard flow** On and uncheck the rest of the checkboxes

* **Login settings**:
  * **Root URL**: http://localhost:8080
  * **Home URL**: http://localhost:8080
  * **Valid redirect URIs**: http://localhost:8080/callback
  * **Valid post logout redirect URIs**: http://localhost:8080
  * **Web origins**: http://localhost:8080

Once the client is created with the above configuration, you will be taken to the newly created Client's **Settings** page.
Click on the **Credentials** tab and copy the **Client secret** value.

In my case, the **Client secret** is **qVcg0foCUNyYbgF0Sg52zeIhLYyOwXpQ**.

{{< box info >}}
**Keycloak Client AccessType**

If you have used Keycloak's earlier versions, you might know that there was an option called **AccessType** 
with possible values of **public**, **confidential**, **bearer-only**. In the new version, there is no explicit **AccessType** field.  

* If **Client authentication** is **Off** then it is **public** client.
* If **Client authentication** is **On**, then it is **confidential** client.
* If **Client authentication** is **On**, but none of the **Authentication flow** options is enabled, then it is **bearer-only** client.

Now you might wonder, what are **public**, **confidential**, **bearer-only** clients?

* A **public** client is a user-facing application which doesn't have a server backend, like Single Page Applications. They can't securely store any sensitive data such as **Client secret**.
* A **confidential** client is a web application which runs on a backend server, like a Spring MVC application. They can securely store sensitive data such as **Client secret**.
* A **bearer-only** client is typically a backend only application (API) which expects **access_token** from the caller. If they don't provide a valid **access_token** then it will simply return **Unauthorized** response and does not initiate authentication flow.
{{< /box >}}

## Create User
Now, let's create a User. Click on **Users** on the left side nav menu, and click on **Add user** button.

Create a user with the following details:

* **Username**: siva
* **Email**: siva@gmail.com
* **Email verified**: Yes
* **First name**: Siva
* **Last name**: Katamreddy

Once the user is created, go to the **Credentials** tab and set the **Password** to **siva1234** with **Temporary: Off**.

Now you should be able to login into **sivalabs** realm using the **siva** user's credentials.

Go to [http://localhost:9191/realms/sivalabs/account/](http://localhost:9191/realms/sivalabs/account/) and SignIn with the credentials **siva/siva1234**.

## Summary
Okay, now we have the initial setup ready to explore OAuth 2.0 and OpenID Connect.

In the next [Spring Security OAuth 2 Tutorial - 2 : Authorization Code Flow]({{< relref "2023-09-24-spring-security-oauth2-tutorial-part2.md" >}}) article, we are going to explore **how to get Access Token using Authorization Code Flow**.

# The new JdbcClient Introduced in Spring Framework 6.1

Spring framework 6.1 introduced a new **JdbcClient** API, which is a wrapper on top of **JdbcTemplate**,
for performing database operations using a fluent API.

Spring Boot 3.2 includes Spring framework 6.1, so let's take a quick look at 
how we can use **JdbcClient** to implement various database operations in a simplified manner.

<!--more-->


First, let's go to https://start.spring.io/ and create a Spring Boot application by selecting 
**Spring JDBC**, **PostgreSQL Driver**, **Flyway Migration**, and **Testcontainers** starters.

## Create Bookmark domain class
Let's start with creating a Java record representing a **Bookmark** as follows:

```java
import java.time.Instant;

public record Bookmark(Long id, String title, String url, Instant createdAt) {}
```

## Create Flyway Migration Script
Let's add the following migration script under **src/main/resources/db/migration** directory. 

**V1__create_tables.sql**
```sql
create table bookmarks
(
    id         bigserial primary key,
    title      varchar   not null,
    url        varchar   not null,
    created_at timestamp
);
```

## Implementing CRUD operations using JdbcClient
Let's implement CRUD operations on **Bookmark** domain class using **JdbcClient** API.

```java
@Repository
@Transactional(readOnly = true)
public class BookmarkRepository {
    private final JdbcClient jdbcClient;

    public BookmarkRepository(JdbcClient jdbcClient) {
        this.jdbcClient = jdbcClient;
    }
    ...
    ...
    ...
}
```

### Fetch all bookmarks
We can fetch all bookmarks using **JdbcClient** as follows:

```java
public List<Bookmark> findAll() {
    String sql = "select id, title, url, created_at from bookmarks";
    return jdbcClient.sql(sql).query(Bookmark.class).list();
}
```

The **JdbcClient** API will take care of dynamically creating a **RowMapper** by using **SimplePropertyRowMapper**.
It will perform the mapping between bean property names to table column names by converting camelCase to underscore notation. 

If you need more control over the mapping, you can create a **RowMapper** yourself and use it as follows:

```java
public List<Bookmark> findAll() {
    String sql = "select id, title, url, created_at from bookmarks";
    return jdbcClient.sql(sql).query(new BookmarkRowMapper()).list();
}

static class BookmarkRowMapper implements RowMapper<Bookmark> {
    @Override
    public Bookmark mapRow(ResultSet rs, int rowNum) throws SQLException {
        return new Bookmark(
                rs.getLong("id"),
                rs.getString("title"),
                rs.getString("url"),
                rs.getTimestamp("created_at").toInstant()
        );
    }
}
```

### Find bookmark By ID

We can fetch a bookmark by **id** using **JdbcClient** as follows:

```java
public Optional<Bookmark> findById(Long id) {
    String sql = "select id, title, url, created_at from bookmarks where id = :id";
    return jdbcClient.sql(sql).param("id", id).query(Bookmark.class).optional();
    
    // If you want to use your own RowMapper
    //return jdbcClient.sql(sql).param("id", id).query(new BookmarkRowMapper()).optional();
}
```

### Create a new bookmark
We can use PostgreSQL **INSERT INTO ... RETURNING COL1, COL2** syntax and then use **KeyHolder** to get the generated primary key value.

So, we can insert a new row into the **bookmarks** table and get the generated primary key value as follows:

```java
@Transactional
public Long save(Bookmark bookmark) {
    String sql = "insert into bookmarks(title, url, created_at) values(:title,:url,:createdAt) returning id";
    KeyHolder keyHolder = new GeneratedKeyHolder();
    jdbcClient.sql(sql)
                .param("title", bookmark.title())
                .param("url", bookmark.url())
                .param("createdAt", Timestamp.from(bookmark.createdAt()))
                .update(keyHolder);
    return keyHolder.getKeyAs(Long.class);
}
```

### Update a bookmark
We can update a bookmark as follows:

```java
@Transactional
public void update(Bookmark bookmark) {
    String sql = "update bookmarks set title = ?, url = ? where id = ?";
    int count = jdbcClient.sql(sql)
            .param(1, bookmark.title())
            .param(2, bookmark.url())
            .param(3, bookmark.id())
            .update();
    if (count == 0) {
        throw new RuntimeException("Bookmark not found");
    }
}
```

In the **update(...)** method, I have used positional parameters **(?)** instead of using named parameters **(:title)** 
for the demonstration purpose. I highly recommend using named parameters over positional parameters.

### Delete a bookmark
We can delete a bookmark as follows:

```java
@Transactional
public void delete(Long id) {
    String sql = "delete from bookmarks where id = ?";
    int count = jdbcClient.sql(sql).param(1, id).update();
    if (count == 0) {
        throw new RuntimeException("Bookmark not found");
    }
}
```

## Test Repository using Testcontainers
We should always make sure that the database is in a known state so that we can write predictable assertions.
So, let's create **src/test/resources/test-data.sql** file with the following content:

```sql
TRUNCATE TABLE bookmarks;
ALTER SEQUENCE bookmarks_id_seq RESTART WITH 1;

INSERT INTO bookmarks(title, url, created_at) VALUES
('How (not) to ask for Technical Help?','https://sivalabs.in/how-to-not-to-ask-for-technical-help', CURRENT_TIMESTAMP),
('Getting Started with Kubernetes','https://sivalabs.in/getting-started-with-kubernetes', CURRENT_TIMESTAMP),
('Few Things I learned in the HardWay in 15 years of my career','https://sivalabs.in/few-things-i-learned-the-hardway-in-15-years-of-my-career', CURRENT_TIMESTAMP),
('All the resources you ever need as a Java & Spring application developer','https://sivalabs.in/all-the-resources-you-ever-need-as-a-java-spring-application-developer', CURRENT_TIMESTAMP),
('Spring Boot Integration Testing using Testcontainers Starter','https://sivalabs.in/spring-boot-integration-testing-using-testcontainers-starter', CURRENT_TIMESTAMP),
('Testing Spring Boot Applications','https://sivalabs.in/spring-boot-testing', CURRENT_TIMESTAMP)
;
```

Now, we can add the annotation **@Sql("/test-data.sql")** to our test class
so that before running each test, the specified SQL script will be executed.

Let's test our **BookmarkRepository**, of course, using [Testcontainers](https://testcontainers.com) as follows:

```java
package com.sivalabs.bookmarks.domain;

import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.jdbc.JdbcTest;
import org.springframework.jdbc.core.simple.JdbcClient;
import org.springframework.test.context.jdbc.Sql;

import java.time.Instant;
import java.util.List;
import java.util.Optional;

import static org.assertj.core.api.Assertions.assertThat;

@JdbcTest(properties = {
   "spring.test.database.replace=none",
   "spring.datasource.url=jdbc:tc:postgresql:16-alpine:///db"
})
@Sql("/test-data.sql")
class BookmarkRepositoryTest {

    @Autowired
    JdbcClient jdbcClient;

    BookmarkRepository bookmarkRepository;

    @BeforeEach
    void setUp() {
        bookmarkRepository = new BookmarkRepository(jdbcClient);
    }

    @Test
    void shouldFindAllBookmarks() {
        List<Bookmark> bookmarks = bookmarkRepository.findAll();
        assertThat(bookmarks).isNotEmpty();
        assertThat(bookmarks).hasSize(6);
    }

    @Test
    void shouldCreateBookmark() {
        Bookmark bookmark = new Bookmark(null, "My Title", "https://sivalabs.in", Instant.now());
        Long id = bookmarkRepository.save(bookmark);
        assertThat(id).isNotNull();
    }

    @Test
    void shouldGetBookmarkById() {
        Bookmark bookmark = new Bookmark(null, "My Title", "https://sivalabs.in", Instant.now());
        Long id = bookmarkRepository.save(bookmark);

        Optional<Bookmark> bookmarkOptional = bookmarkRepository.findById(id);
        assertThat(bookmarkOptional).isPresent();
        assertThat(bookmarkOptional.get().id()).isEqualTo(id);
        assertThat(bookmarkOptional.get().title()).isEqualTo(bookmark.title());
        assertThat(bookmarkOptional.get().url()).isEqualTo(bookmark.url());
    }

    @Test
    void shouldEmptyWhenBookmarkNotFound() {
        Optional<Bookmark> bookmarkOptional = bookmarkRepository.findById(9999L);
        assertThat(bookmarkOptional).isEmpty();
    }

    @Test
    void shouldUpdateBookmark() {
        Bookmark bookmark = new Bookmark(null, "My Title", "https://sivalabs.in", Instant.now());
        Long id = bookmarkRepository.save(bookmark);

        Bookmark changedBookmark = new Bookmark(id, "My Updated Title", "https://www.sivalabs.in", bookmark.createdAt());
        bookmarkRepository.update(changedBookmark);

        Bookmark updatedBookmark = bookmarkRepository.findById(id).orElseThrow();
        assertThat(updatedBookmark.id()).isEqualTo(changedBookmark.id());
        assertThat(updatedBookmark.title()).isEqualTo(changedBookmark.title());
        assertThat(updatedBookmark.url()).isEqualTo(changedBookmark.url());
    }

    @Test
    void shouldDeleteBookmark() {
        Bookmark bookmark = new Bookmark(null, "My Title", "https://sivalabs.in", Instant.now());
        Long id = bookmarkRepository.save(bookmark);

        bookmarkRepository.delete(id);

        Optional<Bookmark> optionalBookmark = bookmarkRepository.findById(id);
        assertThat(optionalBookmark).isEmpty();
    }
}
```

We have used the Testcontainers special JDBC URL to start PostgreSQL database and run the tests using it. 

{{< box info >}}
**Spring Boot Tutorials**

You can find more Spring Boot tutorials on [Spring Boot Tutorials]({{% relref "/pages/spring-boot-tutorials" %}}) page. 
{{< /box >}}

## Summary
The new **JdbcClient** API provides a nice fluent API to implement data access layer using JDBC.
While you can still use good old JdbcTemplate, I would highly recommend using JdbcClient over JdbcTemplate going forward.

You can find the sample code for this tutorial in this 
[GitHub](https://github.com/sivaprasadreddy/spring-boot-tutorials-blog-series/tree/main/spring-boot-jdbcclient-tutorial) repository.

# Spring Boot REST API Best Practices - Part 4


In this **Spring Boot REST API Best Practices** series, we have learned how to implement CRUD operations so far.
In this **Part-4**, we will explore how to **implement exception handling** for our APIs.

<!--more-->


* [Spring Boot REST API Best Practices - Part 1 : Implementing Get Collection API]({{< relref "09-spring-boot-rest-api-tutorial-part-1.md" >}})
* [Spring Boot REST API Best Practices - Part 2 : Implementing Create and Update APIs]({{< relref "10-spring-boot-rest-api-tutorial-part-2.md" >}})
* [Spring Boot REST API Best Practices - Part 3 : Implementing FindById and DeleteById APIs]({{< relref "11-spring-boot-rest-api-tutorial-part-3.md" >}})
* [Spring Boot REST API Best Practices - Part 4 : Exception Handling in REST APIs]({{< relref "12-spring-boot-rest-api-tutorial-part-4.md" >}}) (This article)

You can find the sample code for this tutorial in this
[GitHub](https://github.com/sivaprasadreddy/spring-boot-tutorials-blog-series/tree/main/spring-boot-rest-api-tutorial) repository.

As mentioned in the Part-3, if a request handling method in a controller throws an Exception, 
then Spring Boot will handle it and return the response using its default Exception Handling mechanism.

If all you care about is returning a proper HTTP Status code when an Exception is thrown, you can simply use 
**@ResponseStatus** annotation to specify which HTTP Status code should be used instead of default **INTERNAL_SERVER_ERROR - 500**.

```java
import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.ResponseStatus;

@ResponseStatus(HttpStatus.NOT_FOUND)
public class BookmarkNotFoundException extends RuntimeException {
    
}

// --------------------------------------

import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.ResponseStatus;

@ResponseStatus(HttpStatus.UNPROCESSABLE_ENTITY)
public class InvalidBookmarkUrlException extends RuntimeException {

}
```

But most likely, you would like to return a customized error response body with an appropriate HTTP Status Code as the response.
So, let's see what are the different approaches to handling Exceptions and returning error responses.

## Different approaches to handling exceptions
We can handle exceptions in different ways, and depending on your use-case, you can choose one of the approaches that fit best for you.

* Handling exceptions in the controller handler method
* Using Controller level **@ExceptionHandler**
* GlobalExceptionHandler using **@RestControllerAdvice**
  * Spring's **ProblemDetails** for HTTP APIs ([RFC 7807](https://www.rfc-editor.org/rfc/rfc7807.html)).
  * Using 3rd party libraries
    * [Error Handling Spring Boot Starter](https://github.com/wimdeblauwe/error-handling-spring-boot-starter) by [Wim Deblauwe](https://twitter.com/wimdeblauwe)
    * [problem-spring-web](https://github.com/zalando/problem-spring-web) by Zalando

## Handling exceptions in the controller handler method
This is the best approach if you want at most control over the exception handling logic for a particular API endpoint.

For example, in **POST /api/bookmarks** API endpoint implementation, if the bookmark URL already exists 
then, **BookmarkService** may throw **DuplicateBookmarkException**. 
If the bookmark title contains certain blocked words, then **BookmarkService** may throw **BookmarkTitleNotAllowedException**.
So, if you want to handle all those different exceptions in the controller handler method itself then you can follow this approach.

**BookmarkController.java**

```java
@RestController
@RequestMapping("/api/bookmarks")
class BookmarkController {
    private final BookmarkService bookmarkService;
    //...
    //...

    @PostMapping
    ResponseEntity<BookmarkDTO> create(@RequestBody @Validated CreateBookmarkRequest request) {
        CreateBookmarkCommand cmd = new CreateBookmarkCommand(
                request.title(),
                request.url()
        );
        try {
            BookmarkDTO bookmark = bookmarkService.create(cmd);
            URI location = ServletUriComponentsBuilder
                    .fromCurrentRequest()
                    .path("/api/bookmarks/{id}")
                    .buildAndExpand(bookmark.id()).toUri();
            return ResponseEntity.created(location).body(bookmark);
        } catch(DuplicateBookmarkException e) {
            return ResponseEntity.status(HttpStatus.BAD_REQUEST).build();
        } catch(BookmarkTitleNotAllowedException e) {
            return ResponseEntity.status(HttpStatus.UNPROCESSABLE_ENTITY).build();
        }
    }
}
```

## Using Controller level @ExceptionHandler
Sometimes we may end up handling the same type of Exceptions in the same manner from multiple API handler methods in a controller.
For example, the duplicate **url** check and **title** validation logic apply to both Create and Update API endpoints.
In such cases, instead of duplicating the try-catch logic in multiple places, we can use Controller level **@ExceptionHandler** as follows:

```java
@RestController
@RequestMapping("/api/bookmarks")
class BookmarkController {
    private final BookmarkService bookmarkService;
    //...
    //...

    @PostMapping
    ResponseEntity<BookmarkDTO> create(@RequestBody @Validated CreateBookmarkRequest request) {
        CreateBookmarkCommand cmd = new CreateBookmarkCommand(request.title(), request.url());
        BookmarkDTO bookmark = bookmarkService.create(cmd);
        URI location = ServletUriComponentsBuilder
                .fromCurrentRequest()
                .path("/api/bookmarks/{id}")
                .buildAndExpand(bookmark.id()).toUri();
        return ResponseEntity.created(location).body(bookmark);
    }

    @PutMapping("/{id}")
    void update(@PathVariable(name = "id") Long id,
                @RequestBody @Validated UpdateBookmarkRequest request) {
        UpdateBookmarkCommand cmd = new UpdateBookmarkCommand(id, request.title(), request.url());
        bookmarkService.update(cmd);
    }

    @ExceptionHandler(DuplicateBookmarkException.class)
    public ResponseEntity<ApiError> handleDuplicateBookmarkException(DuplicateBookmarkException e) {
        ApiError error = new ApiError(e.getMessage());
        return ResponseEntity.status(HttpStatus.BAD_REQUEST).body(error);
    }

    @ExceptionHandler(BookmarkTitleNotAllowedException.class)
    public ResponseEntity<ApiError> handleBookmarkTitleNotAllowedException(BookmarkTitleNotAllowedException e) {
        ApiError error = new ApiError(e.getMessage());
        return ResponseEntity.status(HttpStatus.UNPROCESSABLE_ENTITY).body(error);
    }
}
```

In this approach, you don't have to duplicate the exception handling logic in multiple handler methods in the controller.
If **BookmarkTitleNotAllowedException** or **DuplicateBookmarkException** is thrown from **create(...)** or **update(...)** methods, 
they will be handled by the respective **@ExceptionHandler** methods. 

You can also handle multiple types of Exceptions in the same ExceptionHandler method using
**@ExceptionHandler({DuplicateBookmarkException.class, BookmarkTitleNotAllowedException.class})**.
In this case, the ExceptionHandler method should use the common base Exception class of 
**DuplicateBookmarkException** and **BookmarkTitleNotAllowedException** as a method parameter.

## GlobalExceptionHandler using @RestControllerAdvice
In the previous section, we have seen how to use **@ExceptionHandler** at the Controller level.
What if the same type of exceptions may occur in different Controllers, and we want to handle those Exceptions in the same way?
In such cases, we can use the Global Exception Handling approach by using **@RestControllerAdvice**.

Create **GlobalExceptionHandler** as follows:

```java
package com.sivalabs.bookmarks.api;

@RestControllerAdvice
public class GlobalExceptionHandler {
    
    @ExceptionHandler(DuplicateBookmarkException.class)
    public ResponseEntity<ApiError> handleDuplicateBookmarkException(DuplicateBookmarkException e) {
      ApiError error = new ApiError(e.getMessage());
      return ResponseEntity.status(HttpStatus.BAD_REQUEST).body(error);
    }
  
    @ExceptionHandler(BookmarkTitleNotAllowedException.class)
    public ResponseEntity<ApiError> handleBookmarkTitleNotAllowedException(BookmarkTitleNotAllowedException e) {
      ApiError error = new ApiError(e.getMessage());
      return ResponseEntity.status(HttpStatus.UNPROCESSABLE_ENTITY).body(error);
    }
}
```

By using **ControllerAdvice** approach, we don't have to duplicate the same **@ExceptionHandler** logic in multiple Controllers.

{{< box tip >}}
**IMPORTANT** 

If you have an **@ExceptionHandler** handling the same Exception in both Controller and **GlobalExceptionHandler** then
Controller level **@ExceptionHandler** method takes priority.
{{< /box >}}

### Spring Boot Error Responses using Problem Details for HTTP APIs
Spring Framework 6 implemented the Problem Details for HTTP APIs specification, ([RFC 7807](https://www.rfc-editor.org/rfc/rfc7807.html)).

{{< box info >}}
**Spring Boot 3: Error Responses using Problem Details for HTTP APIs**

You can read the [Spring Boot 3 : Error Responses using Problem Details for HTTP APIs](https://www.sivalabs.in/spring-boot-3-error-reporting-using-problem-details/)
post to learn how to use ProblemDetails API for handling Exceptions.
{{< /box >}}

We can enable RFC 7807 responses either by adding the property **spring.mvc.problemdetails.enabled=true** 
or create a global exception handler using **@ControllerAdvice** by extending **ResponseEntityExceptionHandler**.

To quickly demonstrate, here is how you can use **ProblemDetails** API to return error responses in **RFC 7807** format.

```java
@RestControllerAdvice
public class GlobalExceptionHandler extends ResponseEntityExceptionHandler {

    @ExceptionHandler(BookmarkNotFoundException.class)
    ProblemDetail handleBookmarkNotFoundException(BookmarkNotFoundException e) {
        ProblemDetail problemDetail = ProblemDetail.forStatusAndDetail(HttpStatus.NOT_FOUND, e.getMessage());
        problemDetail.setTitle("Bookmark Not Found");
        problemDetail.setType(URI.create("https://api.bookmarks.com/errors/not-found"));
        problemDetail.setProperty("errorCategory", "Generic");
        problemDetail.setProperty("timestamp", Instant.now());
        return problemDetail;
    }
}
```

Now when an unhandled **BookmarkNotFoundException** is thrown, the following response will be returned:

```json
{
  "type": "https://api.bookmarks.com/errors/not-found",
  "title": "Bookmark Not Found",
  "status": 404,
  "detail": "Bookmark with id=111 not found",
  "instance": "/api/bookmarks/111",
  "errorCategory": "Generic",
  "timestamp": "2023-08-30T05:21:59.828411Z"
}
```

By extending **ResponseEntityExceptionHandler**, you can leverage the Spring's default Exception handling for various common exceptions 
such as **MethodArgumentNotValidException**, **BindException**, **MissingServletRequestParameterException**, etc.
If you want to customize the exception handling for any of those Exceptions,
then you can override those respective methods and implement your own logic.

### Using Error Handling Spring Boot Starter
We can use [Error Handling Spring Boot Starter](https://github.com/wimdeblauwe/error-handling-spring-boot-starter) 
that can handle Exceptions and return meaningful error responses without having to write custom code.

Add the following library dependency to your **pom.xml**:

```xml
<properties>
  <error-handling-spring-boot-starter.version>4.2.0</error-handling-spring-boot-starter.version>
</properties>

<dependency>
    <groupId>io.github.wimdeblauwe</groupId>
    <artifactId>error-handling-spring-boot-starter</artifactId>
    <version>${error-handling-spring-boot-starter.version}</version>
</dependency>
```

Now if you try to invoke **POST /api/bookmarks** API endpoint without providing **title** and **url** in the request payload:

```shell
curl --location 'http://localhost:8080/api/bookmarks' \
--header 'Content-Type: application/json' \
--data '{}'
```

Then you will get the following error response:

```json
{
    "code": "VALIDATION_FAILED",
    "message": "Validation failed for object='createBookmarkRequest'. Error count: 2",
    "fieldErrors": [
        {
            "code": "REQUIRED_NOT_EMPTY",
            "message": "URL is required",
            "property": "url",
            "rejectedValue": null,
            "path": "url"
        },
        {
            "code": "REQUIRED_NOT_EMPTY",
            "message": "Title is required",
            "property": "title",
            "rejectedValue": null,
            "path": "title"
        }
    ]
}
```

You can read the [documentation](https://github.com/wimdeblauwe/error-handling-spring-boot-starter#documentation) 
to learn more about **Error Handling Spring Boot Starter**.  

### Using Zalando's problem-spring-web library
Another popular library that can handle Exceptions and return error responses in RFC 7807 format is 
[problem-spring-web](https://github.com/zalando/problem-spring-web) created by Zalando.

You can learn how to use **problem-spring-web** library by watching my 
[Spring Boot Tips : Part 7 - Exception Handling in Spring Boot REST APIs using problem-spring-web](https://www.youtube.com/watch?v=hVfajuyEJMQ) video.

{{< youtube hVfajuyEJMQ >}}

You can find the sample code for this tutorial in this
[GitHub](https://github.com/sivaprasadreddy/spring-boot-tutorials-blog-series/tree/main/spring-boot-rest-api-tutorial) repository.

{{< box info >}}
**Spring Boot Tutorials**

You can find more Spring Boot tutorials on [Spring Boot Tutorials]({{% relref "/pages/spring-boot-tutorials" %}}) page. 
{{< /box >}}

## Summary
In this final part of **Spring Boot REST API Best Practices** series, we have explored how to implement exception handling using different approaches.

I hope this series is helpful in understanding how to implement Spring Boot REST APIs following some best practices.


# Spring Boot REST API Best Practices - Part 3


In this **Spring Boot REST API Best Practices - Part-3**, we will see how to implement **FindById and DeleteById API endpoints**.

<!--more-->


You can find the sample code for this tutorial in this
[GitHub](https://github.com/sivaprasadreddy/spring-boot-tutorials-blog-series/tree/main/spring-boot-rest-api-tutorial) repository.

* [Spring Boot REST API Best Practices - Part 1 : Implementing Get Collection API]({{< relref "09-spring-boot-rest-api-tutorial-part-1.md" >}})
* [Spring Boot REST API Best Practices - Part 2 : Implementing Create and Update APIs]({{< relref "10-spring-boot-rest-api-tutorial-part-2.md" >}})
* [Spring Boot REST API Best Practices - Part 3 : Implementing FindById and DeleteById APIs]({{< relref "11-spring-boot-rest-api-tutorial-part-3.md" >}}) (This article)
* [Spring Boot REST API Best Practices - Part 4 : Exception Handling in REST APIs]({{< relref "12-spring-boot-rest-api-tutorial-part-4.md" >}})



## Implementing GET /api/bookmarks/{id} API endpoint
We would like to implement the API endpoint to get a single resource for the given id with HTTP Status Code 200.
If the resource is not found for the given id then return HTTP Status Code 404, optionally with response body including the error message.

We can implement the **GET /api/bookmarks/{id}** API endpoint as follows:

**BookmarkRepository.java**

```java
interface BookmarkRepository extends JpaRepository<Bookmark, Long> {
    @Query("""
           SELECT
            new com.sivalabs.bookmarks.domain.BookmarkDTO(b.id, b.title, b.url, b.createdAt)
           FROM Bookmark b
           WHERE b.id = ?1
        """)
    Optional<BookmarkDTO> findBookmarkById(Long id);
}
```

**BookmarkService.java**

```java
@Service
@Transactional(readOnly = true)
public class BookmarkService {
      private final BookmarkRepository repo;
      //...
      //...

    public Optional<BookmarkDTO> findById(Long id) {
        return repo.findBookmarkById(id);
    }
}
```

**BookmarkController.java**

```java
@RestController
@RequestMapping("/api/bookmarks")
class BookmarkController {
    private final BookmarkService bookmarkService;
    //...
    //...

    @GetMapping("/{id}")
    ResponseEntity<BookmarkDTO> findById(@PathVariable(name = "id") Long id) {
        return bookmarkService.findById(id)
                .map(ResponseEntity::ok)
                .orElseGet(() -> ResponseEntity.notFound().build());
    }
}
```
Now you can start the application and invoke the API endpoint using CURL as follows:

```shell
$ curl --location 'http://localhost:8080/api/bookmarks/1'
  
  // response
  {"id":1,"title":"SivaLabs blog","url":"https://sivalabs.in","createdAt":"2023-08-23T04:24:17.975268Z"}
```

## Implementing DELETE /api/bookmarks/{id} API endpoint
We will follow the similar pattern and implement the **DELETE /api/bookmarks/{id}** API endpoint for deleting a bookmark.

**BookmarkService.java**

```java
@Service
@Transactional(readOnly = true)
public class BookmarkService {
      private final BookmarkRepository repo;
      //...
      //...

    @Transactional
    public void delete(Long postId) {
        Bookmark entity = repo.findById(postId)
                .orElseThrow(()-> BookmarkNotFoundException.of(postId));
        repo.delete(entity);
    }
}
```

**BookmarkController.java**

```java
@RestController
@RequestMapping("/api/bookmarks")
class BookmarkController {
    private final BookmarkService bookmarkService;
    //...
    //...

    @DeleteMapping("/{id}")
    void delete(@PathVariable(name = "id") Long id) {
        bookmarkService.delete(id);
    }
}
```

You can invoke the API endpoint using CURL as follows:

```shell
$ curl --location --request DELETE 'http://localhost:8080/api/bookmarks/17'
```

## Testing the API endpoint using RestAssured and Testcontainers

Let's write tests for our API endpoints as follows:

```java
package com.sivalabs.bookmarks.api.controllers;

@SpringBootTest(webEnvironment = RANDOM_PORT)
@Testcontainers
class BookmarkControllerTests {

    @Container
    @ServiceConnection
    static PostgreSQLContainer<?> postgres = 
            new PostgreSQLContainer<>(DockerImageName.parse("postgres:15.4-alpine"));

    @LocalServerPort
    private Integer port;

    @Autowired
    private BookmarkService bookmarkService;
   
    @BeforeEach
    void setUp() {
        RestAssured.port = port;
    }
    
    //...
    //...
    
    @Test
    void shouldGetBookmarkByIdSuccessfully() {
        CreateBookmarkCommand cmd = new CreateBookmarkCommand("SivaLabs blog", "https://sivalabs.in");
        BookmarkDTO bookmark = bookmarkService.create(cmd);

        given().contentType(ContentType.JSON)
                .when()
                .get("/api/bookmarks/{id}", bookmark.id())
                .then()
                .statusCode(200)
                .body("id", equalTo(bookmark.id()))
                .body("title", equalTo("SivaLabs blog"))
                .body("url", equalTo("https://sivalabs.in"))
                .body("createdAt", notNullValue())
                .body("updatedAt", nullValue());
    }

    @Test
    void shouldGet404WhenBookmarkNotExists() {
        Long nonExistingId = 99999L;
        given().contentType(ContentType.JSON)
                .when()
                .get("/api/bookmarks/{id}", nonExistingId)
                .then()
                .statusCode(404);
    }

    @Test
    void shouldDeleteBookmarkByIdSuccessfully() {
        CreateBookmarkCommand cmd = new CreateBookmarkCommand("SivaLabs blog", "https://sivalabs.in");
        BookmarkDTO bookmark = bookmarkService.create(cmd);

        given().contentType(ContentType.JSON)
                .when()
                .delete("/api/bookmarks/{id}", bookmark.id())
                .then()
                .statusCode(200);

        Optional<BookmarkDTO> optionalBookmark = bookmarkService.findById(bookmark.id());
        assertThat(optionalBookmark).isEmpty();
    }
}
```

Now you can run the tests using **./mvnw test** from the command-line.

## Exception handling - Pending
If you have noticed, we haven't handled the failure scenarios so far.
For example, if you try to create a new bookmark without passing the mandatory fields(**title**, **url**) 
or try to delete a bookmark with a non-existing id then you will get a response similar to the following:

```shell
$ curl --location --request DELETE 'http://localhost:8080/api/bookmarks/99999'
```

```json
{
    "timestamp": "2023-08-23T12:37:37.772+00:00",
    "status": 500,
    "error": "Internal Server Error",
    "trace": "com.sivalabs.bookmarks.domain.BookmarkNotFoundException: Bookmark with id=17 not found 
              at com.sivalabs.bookmarks.domain.BookmarkNotFoundException.of(BookmarkNotFoundException.java:9)
              at com.sivalabs.bookmarks.domain.BookmarkService.lambda$delete$1(BookmarkService.java:65)
              at java.base/java.util.Optional.orElseThrow(Optional.java:403)
              at com.sivalabs.bookmarks.domain.BookmarkService.delete(BookmarkService.java:65)
              at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              ...
              ...
              at java.base/java.lang.Thread.run(Thread.java:833)",
    "message": "Bookmark with id=17 not found",
    "path": "/api/bookmarks/17"
}
```

This is a default error response returned by Spring Boot. 
But, most likely we might want to customize the error response format.

**We will explore how to handle errors in the upcoming **Part-4** in this series.**

{{< box info >}}
**Spring Boot Tutorials**

You can find more Spring Boot tutorials on [Spring Boot Tutorials]({{% relref "/pages/spring-boot-tutorials" %}}) page. 
{{< /box >}}

## Summary
In this third part of **Spring Boot REST API Best Practices** series, we have learned how to implement
an API endpoints to find and delete a resource by given id.

You can find the sample code for this tutorial in this 
[GitHub](https://github.com/sivaprasadreddy/spring-boot-tutorials-blog-series/tree/main/spring-boot-rest-api-tutorial) repository.

# Spring Boot REST API Best Practices - Part 2


In this **Spring Boot REST API Best Practices - Part-2**, I will explain some of the best practices we should follow 
while implementing **Create and Update API endpoints**.

<!--more-->


You can find the sample code for this tutorial in this
[GitHub](https://github.com/sivaprasadreddy/spring-boot-tutorials-blog-series/tree/main/spring-boot-rest-api-tutorial) repository.

* [Spring Boot REST API Best Practices - Part 1 : Implementing Get Collection API]({{< relref "09-spring-boot-rest-api-tutorial-part-1.md" >}})
* [Spring Boot REST API Best Practices - Part 2 : Implementing Create and Update APIs]({{< relref "10-spring-boot-rest-api-tutorial-part-2.md" >}}) (This article)
* [Spring Boot REST API Best Practices - Part 3 : Implementing FindById and DeleteById APIs]({{< relref "11-spring-boot-rest-api-tutorial-part-3.md" >}})
* [Spring Boot REST API Best Practices - Part 4 : Exception Handling in REST APIs]({{< relref "12-spring-boot-rest-api-tutorial-part-4.md" >}})

This article is a continuation of [Spring Boot REST API Best Practices - Part 1]({{< relref "09-spring-boot-rest-api-tutorial-part-1.md" >}}).
So, if you haven't already, please read **Part-1** first. We are going to build the APIs on top of the code we have implemented in Part-1.

## Implementing POST /api/bookmarks API endpoint
We may think of implementing the **POST /api/bookmarks** API endpoint as follows:

```java
package com.sivalabs.bookmarks.api.controllers;

import com.sivalabs.bookmarks.domain.BookmarkService;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.List;

@RestController
@RequestMapping("/api/bookmarks")
class BookmarkController {
    private final BookmarkService bookmarkService;
    //...
    //...
   
    @PostMapping
    @ResponseStatus(HttpStatus.CREATED)
    BookmarkDTO create(@RequestBody @Validated BookmarkDTO bookmark) {
      return bookmarkService.create(bookmark);
    }
}
```

At a glance this may looks fine.

* We are not using JPA entities to bind request payload or returning as response.
* We are returning the proper response code 201 when the bookmark resource is successfully created.

If we generate Open API documentation from this code using [springdoc-openapi](https://springdoc.org/) 
then the expected request payload will be shown as follows:

```json
{
   "id": 0,
   "title": "",
   "url": "",
   "createdAt": ""
}
```

When I look at the request payload, I have a bunch of questions:

* Should I generate the **id** from client side and send it in the payload, or it will be generated on the server-side?
* If I include the **id** in the request payload, does it override the bookmark details if an entity wth same **id** value exist or ignore the **id** and creates a new bookmark?
* Should I include **createdAt** or server will use the timestamp of the record insertion into DB?
* What if I set a future date for **createdAt**?

All these questions came up because we are not explicit in our contract.

The actual API behaviour we want is, the client should only send the **title** and **url**.
Then we will automatically generate the **id** and use the current timestamp as **createdAt** value.

To avoid the confusion and bring more clarity to what is the expected payload, 
it is better to create a request class for this specific API endpoint as follows:

```java
package com.sivalabs.bookmarks.api.models;

import jakarta.validation.constraints.NotEmpty;

public record CreateBookmarkRequest(
        @NotEmpty(message = "Title is required")
        String title,
        @NotEmpty(message = "URL is required")
        String url) {
}
```

The next question that comes to our mind is, should we return **BookmarkDTO** or **ResponseEntity&lt;BookmarkDTO&gt;**?

I would prefer to use **ResponseEntity** as return type if:
* I need to send different HTTP Status Codes for different kinds of failures or validation errors.
* I need to add headers.

Basically, if I want more fine-grained control over the response I would choose **ResponseEntity**, otherwise I simply return the response object. 

Now we have some clarity on how our Controller method implementation would look like. What about Service layer implementation?

Should we send **CreateBookmarkRequest** as input to **BookmarkService.create(...)** method? 
Or, create **BookmarkDTO** object from **CreateBookmarkRequest** and then send it to **BookmarkService.create(...)** method?

My preference is to create a new **CreateBookmarkCommand** class with **title** and **url** properties 
and send it to **BookmarkService.create(...)** method.
This may seem unnecessary because **CreateBookmarkRequest** and **CreateBookmarkCommand** are exactly same in this scenario.

But imagine this API endpoint can only be invoked by an authenticated user.
Then we may need to include **createdBy** property in the input to **BookmarkService.create(...)** method 
which is not available in **CreateBookmarkRequest**.
So, to keep each layer's responsibilities separate, I would use a separate command object.

```java
package com.sivalabs.bookmarks.domain;

public record CreateBookmarkCommand(String title, String url) {}
```

Here is the final implementation for **POST /api/bookmarks** API endpoint.

**BookmarkService.java**

```java
@Service
@Transactional(readOnly = true)
public class BookmarkService {
      private final BookmarkRepository repo;
      //...
      //...

      @Transactional
      public BookmarkDTO create(CreateBookmarkCommand cmd) {
         Bookmark bookmark = new Bookmark();
         bookmark.setTitle(cmd.title());
         bookmark.setUrl(cmd.url());
         bookmark.setCreatedAt(Instant.now());
         return BookmarkDTO.from(repo.save(bookmark));
      }
}
```

To create BookmarkDTO instance from Bookmark entity we will create a static method as follows:

```java
package com.sivalabs.bookmarks.domain;

import java.time.Instant;

public record BookmarkDTO(
        Long id,
        String title,
        String url,
        Instant createdAt
) {
    static BookmarkDTO from(Bookmark bookmark) {
        return new BookmarkDTO(bookmark.getId(),
                bookmark.getTitle(),
                bookmark.getUrl(),
                bookmark.getCreatedAt()
        );
    }
}
```

**BookmarkController.java**

```java
package com.sivalabs.bookmarks.api.controllers;

@RestController
@RequestMapping("/api/bookmarks")
class BookmarkController {
    private final BookmarkService bookmarkService;
    //...
    //...
   
    @PostMapping
    ResponseEntity<BookmarkDTO> create(@RequestBody @Validated CreateBookmarkRequest request) {
         CreateBookmarkCommand cmd = new CreateBookmarkCommand(request.title(), request.url());
         BookmarkDTO bookmark = bookmarkService.create(cmd);
         URI location = ServletUriComponentsBuilder
                          .fromCurrentRequest()
                          .path("/api/bookmarks/{id}")
                          .buildAndExpand(bookmark.id()).toUri();
         return ResponseEntity.created(location).body(bookmark);
   }
}
```
Now you can start the application and invoke the API endpoint using CURL as follows:

```shell
$ curl --location 'http://localhost:8080/api/bookmarks' \
   --header 'Content-Type: application/json' \
   --data '{
       "title": "SivaLabs blog",
       "url": "https://sivalabs.in"
  }'
  
  // response
  {"id":17,"title":"SivaLabs blog","url":"https://sivalabs.in","createdAt":"2023-08-23T04:24:17.975268Z"}
```

## Implementing PUT /api/bookmarks/{id} API endpoint
We will follow the similar pattern and implement the **PUT /api/bookmarks/{id}** API endpoint for updating a bookmark.

**UpdateBookmarkCommand.java**

```java
package com.sivalabs.bookmarks.domain;

public record UpdateBookmarkCommand(
        Long id,
        String title,
        String url) {
}
```

**BookmarkNotFoundException.java**

```java
package com.sivalabs.bookmarks.domain;

public class BookmarkNotFoundException extends RuntimeException {
   public BookmarkNotFoundException(Long id) {
      super(String.format("Bookmark with id=%d not found", id));
   }

   public static BookmarkNotFoundException of(Long id) {
      return new BookmarkNotFoundException(id);
   }
}
```

**BookmarkService.java**

```java
package com.sivalabs.bookmarks.domain;

@Service
@Transactional(readOnly = true)
public class BookmarkService {
    private final BookmarkRepository repo;

    //...
    //...
    @Transactional
    public void update(UpdateBookmarkCommand cmd) {
        Bookmark bookmark = repo.findById(cmd.id())
                .orElseThrow(() -> BookmarkNotFoundException.of(cmd.id()));
        bookmark.setTitle(cmd.title());
        bookmark.setUrl(cmd.url());
        bookmark.setUpdatedAt(Instant.now());
        repo.save(bookmark);
    }
}
```

**UpdateBookmarkRequest.java**

```java
package com.sivalabs.bookmarks.api.models;

import jakarta.validation.constraints.NotEmpty;

public record UpdateBookmarkRequest(
        @NotEmpty(message = "Title is required")
        String title,
        @NotEmpty(message = "URL is required")
        String url) {
}
```

**BookmarkController.java**

```java
package com.sivalabs.bookmarks.api.controllers;

@RestController
@RequestMapping("/api/bookmarks")
class BookmarkController {
    private final BookmarkService bookmarkService;

    @PutMapping("/{id}")
    void update(@PathVariable(name = "id") Long id,
                @RequestBody @Validated UpdateBookmarkRequest request) {
        UpdateBookmarkCommand cmd = new UpdateBookmarkCommand(id, request.title(), request.url());
        bookmarkService.update(cmd);
    }
}
```

You can try invoking this endpoint using CURL as follows:

```shell
$ curl -v --location --request PUT 'http://localhost:8080/api/bookmarks/17' \
   --header 'Content-Type: application/json' \
   --data '{
       "title": "SivaLabs - TechBlog",
       "url": "https://www.sivalabs.in"
   }'
```

There are a couple of things that can be improved in the current implementation:
* **Exception Handling** - This we will tackle in the upcoming part in this series.
* Inserting and updating the **createdAt** and **updatedAt** column values.

In the current implementation, we are manually setting the **createdAt** and **updatedAt** values as follows:

```java
bookmark.setCreatedAt(Instant.now());

bookmark.setUpdatedAt(Instant.now());
```

But instead of manually setting these values, we can leverage some of the JPA and Spring Data JPA features 
to automatically set these values while inserting or updating the entities.

### Using @PrePersist and @PreUpdate
We can use JPA's **@PrePersist** and **@PreUpdate** annotations to automatically set the **createdAt** and **updatedAt** values as follows:

```java
@Entity
class Bookmark {
   ... 
   ... 
   @Column(name = "created_at", nullable = false, updatable = false)
   private Instant createdAt;
   @Column(name = "updated_at", insertable = false)
   private Instant updatedAt;

   @PreUpdate
   @PrePersist
   public void updateTimeStamps() {
      updatedAt = Instant.now();
      if (createdAt == null) {
         createdAt = Instant.now();
      }
   }
}
```

### Using Spring Data JPA's @CreatedDate and @LastModifiedDate
You can also use Spring Data JPA's **@CreatedDate** and **@LastModifiedDate** annotations 
to automatically set the **createdAt** and **updatedAt** values as follows:

```java
@Entity
@EntityListeners(AuditingEntityListener.class)
class Bookmark {
   ... 
   ... 
   @Column(name = "created_at", nullable = false, updatable = false)
   @CreatedDate
   private Instant createdAt;
   
   @Column(name = "updated_at", insertable = false)
   @LastModifiedDate
   private Instant updatedAt;
}
```

You also need to enable auditing in your Spring Boot application by adding **@EnableJpaAuditing** annotation on a configuration class.

**Hibernate** also provides **@CreationTimestamp** and **@UpdateTimestamp** annotations for similar purpose.
But I prefer to use one of the above-mentioned 2 approaches to keep the code independent of underlying persistence implementation framework.

{{< box tip >}}
**Which Java DataType to use for storing Date or Timestamp in Database?**

Please read this excellent [StackOverflow Answer](https://stackoverflow.com/a/32443004/755932) to find 
which Java datatype is more suitable for storing **Date** or **Timestamp** values in the database.
{{< /box >}}

## Testing the API endpoint using RestAssured and Testcontainers

Let's write tests for our API endpoints as follows:

```java
package com.sivalabs.bookmarks.api.controllers;

import com.sivalabs.bookmarks.domain.BookmarkDTO;
import com.sivalabs.bookmarks.domain.BookmarkService;
import com.sivalabs.bookmarks.domain.CreateBookmarkCommand;
import io.restassured.RestAssured;
import io.restassured.http.ContentType;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.test.web.server.LocalServerPort;
import org.springframework.boot.testcontainers.service.connection.ServiceConnection;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import org.testcontainers.utility.DockerImageName;

import static io.restassured.RestAssured.given;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.matchesRegex;
import static org.hamcrest.Matchers.notNullValue;
import static org.hamcrest.Matchers.nullValue;
import static org.springframework.boot.test.context.SpringBootTest.WebEnvironment.RANDOM_PORT;

@SpringBootTest(webEnvironment = RANDOM_PORT)
@Testcontainers
class BookmarkControllerTests {

    @Container
    @ServiceConnection
    static PostgreSQLContainer<?> postgres = 
            new PostgreSQLContainer<>(DockerImageName.parse("postgres:15.4-alpine"));

    @LocalServerPort
    private Integer port;

    @Autowired
    private BookmarkService bookmarkService;
   
    @BeforeEach
    void setUp() {
        RestAssured.port = port;
    }

    @Test
    void shouldCreateBookmarkSuccessfully() {
       given().contentType(ContentType.JSON)
              .body(
                """
                 {
                     "title": "SivaLabs blog",
                     "url": "https://sivalabs.in"
                 }
               """)
              .when()
              .post("/api/bookmarks")
              .then()
              .statusCode(201)
              .header("Location", matchesRegex(".*/api/bookmarks/[0-9]+$"))
              .body("id", notNullValue())
              .body("title", equalTo("SivaLabs blog"))
              .body("url", equalTo("https://sivalabs.in"))
              .body("createdAt", notNullValue())
              .body("updatedAt", nullValue());
    }

    @Test
    void shouldUpdateBookmarkSuccessfully() {
       CreateBookmarkCommand cmd = new CreateBookmarkCommand("SivaLabs blog", "https://sivalabs.in");
       BookmarkDTO bookmark = bookmarkService.create(cmd);

       given().contentType(ContentType.JSON)
              .body(
                """
                 {
                     "title": "SivaLabs - Tech Blog",
                     "url": "https://www.sivalabs.in"
                 }
               """)
              .when()
              .put("/api/bookmarks/{id}", bookmark.id())
              .then()
              .statusCode(200);
    }
}
```

Now you can run the tests using **./mvnw test** from the command-line.

We will explore how to implement **FindById and DeleteById API endpoints** in the upcoming **Part-3** in this series.

{{< box info >}}
**Spring Boot Tutorials**

You can find more Spring Boot tutorials on [Spring Boot Tutorials]({{% relref "/pages/spring-boot-tutorials" %}}) page. 
{{< /box >}}

## Summary
In this second part of **Spring Boot REST API Best Practices** series, we have learned how to implement
an API endpoints to create and update a resource by following some best practices.

You can find the sample code for this tutorial in this 
[GitHub](https://github.com/sivaprasadreddy/spring-boot-tutorials-blog-series/tree/main/spring-boot-rest-api-tutorial) repository.

# Spring Boot REST API Best Practices - Part 1


In this **Spring Boot REST API Best Practices Series**, I will explain some of the best practices we should follow while implementing REST APIs.
Also, I will explain some of the common mistakes developers do and how to avoid them.

<!--more-->


* [Spring Boot REST API Best Practices - Part 1 : Implementing Get Collection API]({{< relref "09-spring-boot-rest-api-tutorial-part-1.md" >}}) (This article)
* [Spring Boot REST API Best Practices - Part 2 : Implementing Create and Update APIs]({{< relref "10-spring-boot-rest-api-tutorial-part-2.md" >}})
* [Spring Boot REST API Best Practices - Part 3 : Implementing FindById and DeleteById APIs]({{< relref "11-spring-boot-rest-api-tutorial-part-3.md" >}})
* [Spring Boot REST API Best Practices - Part 4 : Exception Handling in REST APIs]({{< relref "12-spring-boot-rest-api-tutorial-part-4.md" >}})

In this **Part-1**, we are going to implement our first API endpoint which is to fetch a list of resources.
We are going to explore some of the common mistakes developers do and how to avoid them.

## Create a Spring Boot application
First, let's go to https://start.spring.io/ and create a Spring Boot application by selecting **Spring Web**,
**Validation**, **Spring Data JPA**, **PostgreSQL Driver**, **Flyway Migration**, 
and **Testcontainers** starters.

Our sample application is going to be a very simple one, but I am going to follow the same practices 
that I would follow in a real-world application.

You can find the sample code for this tutorial in this
[GitHub](https://github.com/sivaprasadreddy/spring-boot-tutorials-blog-series/tree/main/spring-boot-rest-api-tutorial) repository.


The REST API we are going to build is to manage bookmarks.
A bookmark contains **id**, **title**, **url**, **createdAt** and **updatedAt** properties.

## Create Bookmark entity
Create the JPA entity **Bookmark** as follows:

```java
package com.sivalabs.bookmarks.domain;

import jakarta.persistence.*;
import java.time.Instant;

@Entity
@Table(name = "bookmarks")
class Bookmark {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;
    @Column(nullable = false)
    private String title;
    @Column(nullable = false)
    private String url;
    @Column(name = "created_at", nullable = false, updatable = false)
    private Instant createdAt;
    @Column(name = "updated_at", insertable = false)
    private Instant updatedAt;

    //constructors, setters & getters
}
```

Note that the entity class is not **public**, so it's visibility is limited to **com.sivalabs.bookmarks.domain** package.

## Create Flyway Migration Scripts
We are going to use Flyway for database migrations.
To learn more about Flyway checkout [Spring Boot Flyway Database Migration Tutorial]({{< relref "08-spring-boot-flyway-database-migration-tutorial.md" >}})

Let's create the following migration script under **src/main/resources/db/migration** directory. 

**V1__init.sql**
```sql
create table bookmarks
(
  id         bigserial primary key,
  title      varchar not null,
  url        varchar not null,
  created_at timestamp,
  updated_at timestamp
);

INSERT INTO bookmarks(title, url, created_at) VALUES
('How (not) to ask for Technical Help?','https://sivalabs.in/how-to-not-to-ask-for-technical-help', CURRENT_TIMESTAMP),
('Announcing My Spring Boot Tips Video Series on YouTube','https://sivalabs.in/announcing-my-springboot-tips-video-series', CURRENT_TIMESTAMP),
('Kubernetes - Exposing Services to outside of Cluster using Ingress','https://sivalabs.in/kubernetes-ingress', CURRENT_TIMESTAMP),
('Kubernetes - Blue/Green Deployments','https://sivalabs.in/kubernetes-blue-green-deployments', CURRENT_TIMESTAMP),
('Kubernetes - Releasing a new version of the application using Deployment Rolling Updates','https://sivalabs.in/kubernetes-deployment-rolling-updates', CURRENT_TIMESTAMP),
('Getting Started with Kubernetes','https://sivalabs.in/getting-started-with-kubernetes', CURRENT_TIMESTAMP),
('Get Super Productive with Intellij File Templates','https://sivalabs.in/get-super-productive-with-intellij-file-templates', CURRENT_TIMESTAMP),
('Few Things I learned in the HardWay in 15 years of my career','https://sivalabs.in/few-things-i-learned-the-hardway-in-15-years-of-my-career', CURRENT_TIMESTAMP),
('All the resources you ever need as a Java & Spring application developer','https://sivalabs.in/all-the-resources-you-ever-need-as-a-java-spring-application-developer', CURRENT_TIMESTAMP),
('GoLang from a Java developer perspective','https://sivalabs.in/golang-from-a-java-developer-perspective', CURRENT_TIMESTAMP),
('Imposing Code Structure Guidelines using ArchUnit','https://sivalabs.in/impose-architecture-guidelines-using-archunit', CURRENT_TIMESTAMP),
('Spring Boot Integration Testing using TestContainers Starter','https://sivalabs.in/spring-boot-integration-testing-using-testcontainers-starter', CURRENT_TIMESTAMP),
('Creating Yeoman based Spring Boot Generator','https://sivalabs.in/creating-yeoman-based-springboot-generator', CURRENT_TIMESTAMP),
('Testing REST APIs using Postman and Newman','https://sivalabs.in/testing-rest-apis-with-postman-newman', CURRENT_TIMESTAMP),
('Testing Spring Boot Applications','https://sivalabs.in/spring-boot-testing', CURRENT_TIMESTAMP)
;
```

## Create Spring Data JPA Repository for Bookmark entity
Create the Spring Data JPA repository BookmarkRepository interface as follows:

```java
package com.sivalabs.bookmarks.domain;

import org.springframework.data.jpa.repository.JpaRepository;

interface BookmarkRepository extends JpaRepository<Bookmark, Long> {
}
```

## Create BookmarkService
Let's create **BookmarkService** which is a transactional business service which will be exposed to outside the "domain" package.

```java
package com.sivalabs.bookmarks.domain;

import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

@Service
@Transactional(readOnly = true)
public class BookmarkService {
    private final BookmarkRepository repo;

    BookmarkService(BookmarkRepository repo) {
        this.repo = repo;
    }

}
```


{{< box tip >}}
**IMPORTANT CONSIDERATIONS:**

The following things are important to design our component as a cohesive component hiding the internal implementation details:

1. Notice that **Bookmark** entity and **BookmarkRepository** are not **public**. 
   They are **package-private** scoped classes/interfaces.
   They are supposed to be used by **BookmarkService** only and is hidden from outside 
   the **com.sivalabs.bookmarks.domain** package.

2. The **BookmarkService** is a transactional service layer component which will be used by **web layer** or **other service layer** components.
   The **BookmarkService** class is annotated with **@Transactional(readOnly = true)** which means 
   all the **public** methods are transactional and allows only **read-only** operations on the database.  
   We can override this read-only behaviour for the methods which needs to perform insert/update/delete database operations
   by adding **@Transactional** annotation.
{{< /box >}}

## Running application locally using Testcontainers
Spring Boot 3.1.0 introduced support for [Testcontainers](https://testcontainers.com/) which we can use for writing integration tests 
and for local development also.

While generating the application, we have selected **PostgreSQL Driver**, and **Testcontainers** starters.
So, the generated application will have a **TestApplication.java** under **src/test/java/** similar to the following:

```java
import org.springframework.boot.SpringApplication;
import org.springframework.boot.test.context.TestConfiguration;
import org.springframework.boot.testcontainers.service.connection.ServiceConnection;
import org.springframework.context.annotation.Bean;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.utility.DockerImageName;

@TestConfiguration(proxyBeanMethods = false)
public class TestApplication {

  @Bean
  @ServiceConnection
  PostgreSQLContainer<?> postgresContainer() {
    return new PostgreSQLContainer<>(DockerImageName.parse("postgres:15.4-alpine"));
  }

  public static void main(String[] args) {
    SpringApplication
            .from(Application::main)
            .with(TestApplication.class)
            .run(args);
  }
}
```

We can start the application locally by running **TestApplication.java** from the IDE or 
by running **./mvnw spring-boot:test-run** from the command-line.

Now we have all the basic code setup ready to start implementing our API endpoints.
Let's start with implementing get all bookmarks API endpoint.

## Implementing GET /api/bookmarks API endpoint
We may think of implementing the **GET /api/bookmarks** API endpoint as follows:

**BookmarkService.java**
```java
@Service
@Transactional(readOnly = true)
public class BookmarkService {
    private final BookmarkRepository repo;

    BookmarkService(BookmarkRepository repo) {
        this.repo = repo;
    }

    public List<Bookmark> findAll() {
        return repo.findAll();
    }
}
```

And, create the **BookmarkController**, and implement the API endpoint as follows:

```java
package com.sivalabs.bookmarks.api.controllers;

import com.sivalabs.bookmarks.domain.BookmarkService;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.List;

@RestController
@RequestMapping("/api/bookmarks")
class BookmarkController {
    private final BookmarkService bookmarkService;

    BookmarkController(BookmarkService bookmarkService) {
        this.bookmarkService = bookmarkService;
    }

    @GetMapping
    List<Bookmark> findAll() {
        return bookmarkService.findAll();
    }
}
```

You may see this kind of implementation in many tutorials and examples, but this is a bad implementation.

**Problems with this implementation:**

1. We are exposing database entity as REST API response directly and in most of the cases it is a bad practice.
   If we have to make any changes to the entity then API response format will be changed too which might not be desirable. 
   So, we should create a DTO and expose only the necessary fields for that API.
   
2. If we are fetching the data only to return to the client, then it is better to use [DTO projections](https://docs.spring.io/spring-data/jpa/docs/current/reference/html/#projections) instead of loading entities.

3. The **findAll()** method will load all the records in the table and this may lead to **OutOfMemoryExceptions** if there are millions of records.
   If the table is ever-growing with new data, it is always advised to use **Pagination**. 

So, let's re-implement this API with **pagination** support and using **DTO projections**.

Let's create a **PagedResult** class that represents a generic paginated query result as follows:

```java
package com.sivalabs.bookmarks.domain;

import com.fasterxml.jackson.annotation.JsonProperty;
import java.util.List;

public record PagedResult<T>(
        List<T> data,
        long totalElements,
        int pageNumber,
        int totalPages,
        @JsonProperty("isFirst") boolean isFirst,
        @JsonProperty("isLast") boolean isLast,
        @JsonProperty("hasNext") boolean hasNext,
        @JsonProperty("hasPrevious") boolean hasPrevious) {}
```

Create **BookmarkDTO** as a record as follows:

```java
package com.sivalabs.bookmarks.domain;

import java.time.Instant;

public record BookmarkDTO(
        Long id,
        String title,
        String url,
        Instant createdAt) {}
```

Now, let's add a method to **BookmarkRepository** to fetch bookmarks with pagination and using DTO projection as follows:

```java
interface BookmarkRepository extends JpaRepository<Bookmark, Long> {
    @Query("""
               SELECT
                new com.sivalabs.bookmarks.domain.BookmarkDTO(b.id, b.title, b.url, b.createdAt)
               FROM Bookmark b
            """)
    Page<BookmarkDTO> findBookmarks(Pageable pageable);
}
```

Create a class to wrap all the query parameters as follows:

```java
public record FindBookmarksQuery(int pageNo, int pageSize) {}
```

This wrapper class **FindBookmarksQuery** will be convenient if you want to enhance the API 
with some filtering and sorting capabilities in the future.

Now, let's update the **BookmarkService** as follows:

```java
@Service
@Transactional(readOnly = true)
public class BookmarkService {
    private final BookmarkRepository repo;
  
    BookmarkService(BookmarkRepository repo) {
      this.repo = repo;
    }
  
    public PagedResult<BookmarkDTO> findBookmarks(FindBookmarksQuery query) {
        Sort sort = Sort.by(Sort.Direction.DESC, "createdAt");
        //from user POV, page number starts from 1, but for Spring Data JPA page number starts from 0.
        int pageNo = query.pageNo() > 0 ? query.pageNo() - 1 : 0;
        Pageable pageable = PageRequest.of(pageNo, query.pageSize(), sort);
        Page<BookmarkDTO> page = repo.findBookmarks(pageable);
        return new PagedResult<>(
                page.getContent(),
                page.getTotalElements(),
                page.getNumber() + 1, // for user page number starts from 1
                page.getTotalPages(),
                page.isFirst(),
                page.isLast(),
                page.hasNext(),
                page.hasPrevious()
        );
    }
}
```

Finally, let's update the **BookmarkController** as follows:

```java
@RestController
@RequestMapping("/api/bookmarks")
class BookmarkController {
    private final BookmarkService bookmarkService;

    BookmarkController(BookmarkService bookmarkService) {
        this.bookmarkService = bookmarkService;
    }

    @GetMapping
    PagedResult<BookmarkDTO> findBookmarks(
            @RequestParam(name = "page", defaultValue = "1") Integer pageNo,
            @RequestParam(name = "size", defaultValue = "10") Integer pageSize) {
        FindBookmarksQuery query = new FindBookmarksQuery(pageNo, pageSize);
        return bookmarkService.findBookmarks(query);
    }
}
```

Now if you run the application and access the [http://localhost:8080/api/bookmarks](http://localhost:8080/api/bookmarks) API endpoint 
then you will get the response similar to the following:

```json
{
    "isFirst": true,
    "isLast": false,
    "hasNext": true,
    "hasPrevious": false,
    "totalElements": 15,
    "pageNumber": 1,
    "totalPages": 2,
    "data": [
      {
        "id": 1,
        "title": "SivaLabs blog",
        "url": "https://wwww.sivalabs.in",
        "createdAt": "2023-08-22T10:24:58.956786"
      },
      ...
      ...
    ]
}
```

## Testing the API endpoint using RestAssured and Testcontainers
Let's be a good citizen by writing an automated test for our API endpoint.
We are going to use **RestAssured** for invoking the API endpoint and [Testcontainers](https://testcontainers.com) for provisioning the PostgreSQL database.

We should always make sure that database is in a known state so that we can write predictable assertions.
So, let's create **src/test/resources/test_data.sql** file with the following content:

```sql
TRUNCATE TABLE bookmarks;
ALTER SEQUENCE bookmarks_id_seq RESTART WITH 1;

INSERT INTO bookmarks(title, url, created_at) VALUES
('How (not) to ask for Technical Help?','https://sivalabs.in/how-to-not-to-ask-for-technical-help', CURRENT_TIMESTAMP),
('Announcing My Spring Boot Tips Video Series on YouTube','https://sivalabs.in/announcing-my-springboot-tips-video-series', CURRENT_TIMESTAMP),
('Kubernetes - Exposing Services to outside of Cluster using Ingress','https://sivalabs.in/kubernetes-ingress', CURRENT_TIMESTAMP),
('Kubernetes - Blue/Green Deployments','https://sivalabs.in/kubernetes-blue-green-deployments', CURRENT_TIMESTAMP),
('Kubernetes - Releasing a new version of the application using Deployment Rolling Updates','https://sivalabs.in/kubernetes-deployment-rolling-updates', CURRENT_TIMESTAMP),
('Getting Started with Kubernetes','https://sivalabs.in/getting-started-with-kubernetes', CURRENT_TIMESTAMP),
('Get Super Productive with Intellij File Templates','https://sivalabs.in/get-super-productive-with-intellij-file-templates', CURRENT_TIMESTAMP),
('Few Things I learned in the HardWay in 15 years of my career','https://sivalabs.in/few-things-i-learned-the-hardway-in-15-years-of-my-career', CURRENT_TIMESTAMP),
('All the resources you ever need as a Java & Spring application developer','https://sivalabs.in/all-the-resources-you-ever-need-as-a-java-spring-application-developer', CURRENT_TIMESTAMP),
('GoLang from a Java developer perspective','https://sivalabs.in/golang-from-a-java-developer-perspective', CURRENT_TIMESTAMP),
('Imposing Code Structure Guidelines using ArchUnit','https://sivalabs.in/impose-architecture-guidelines-using-archunit', CURRENT_TIMESTAMP),
('Spring Boot Integration Testing using TestContainers Starter','https://sivalabs.in/spring-boot-integration-testing-using-testcontainers-starter', CURRENT_TIMESTAMP),
('Creating Yeoman based Spring Boot Generator','https://sivalabs.in/creating-yeoman-based-springboot-generator', CURRENT_TIMESTAMP),
('Testing REST APIs using Postman and Newman','https://sivalabs.in/testing-rest-apis-with-postman-newman', CURRENT_TIMESTAMP),
('Testing Spring Boot Applications','https://sivalabs.in/spring-boot-testing', CURRENT_TIMESTAMP)
;
```

Now, we can add the annotation **@Sql("/test-data.sql")** to our test method 
so that before running the test the specified SQL script will be executed.

Let's write our API test as follows:

```java
package com.sivalabs.bookmarks.api.controllers;

import io.restassured.RestAssured;
import io.restassured.http.ContentType;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.test.web.server.LocalServerPort;
import org.springframework.boot.testcontainers.service.connection.ServiceConnection;
import org.springframework.test.context.jdbc.Sql;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import org.testcontainers.utility.DockerImageName;

import static io.restassured.RestAssured.given;
import static org.hamcrest.Matchers.equalTo;
import static org.springframework.boot.test.context.SpringBootTest.WebEnvironment.RANDOM_PORT;

@SpringBootTest(webEnvironment = RANDOM_PORT)
@Testcontainers
class BookmarkControllerTests {

    @Container
    @ServiceConnection
    static PostgreSQLContainer<?> postgres = 
            new PostgreSQLContainer<>(DockerImageName.parse("postgres:15.4-alpine"));

    @LocalServerPort
    private Integer port;

    @BeforeEach
    void setUp() {
        RestAssured.port = port;
    }

    @Test
    @Sql("/test-data.sql")
    void shouldGetBookmarksByPage() {
        given().contentType(ContentType.JSON)
                .when()
                .get("/api/bookmarks?page=1&size=10")
                .then()
                .statusCode(200)
                .body("data.size()", equalTo(10))
                .body("totalElements", equalTo(15))
                .body("pageNumber", equalTo(1))
                .body("totalPages", equalTo(2))
                .body("isFirst", equalTo(true))
                .body("isLast", equalTo(false))
                .body("hasNext", equalTo(true))
                .body("hasPrevious", equalTo(false));
    }
}
```

Now you can run the test and see Testcontainers spinning up a PostgreSQL database and Spring Boot automatically configured 
to use that database while running the test.

We will explore how to implement **Create and Update Bookmark API endpoints** in the upcoming
[Spring Boot REST API Best Practices - Part 2]({{< relref "10-spring-boot-rest-api-tutorial-part-2.md" >}}) in this series.

{{< box info >}}
**Spring Boot Tutorials**

You can find more Spring Boot tutorials on [Spring Boot Tutorials]({{% relref "/pages/spring-boot-tutorials" %}}) page. 
{{< /box >}}

## Summary
In this first part of **Spring Boot REST API Best Practices** series, we have learned how to implement
an API endpoint that will return a collection of resources by following some best practices such as pagination and DTO projections.

You can find the sample code for this tutorial in this 
[GitHub](https://github.com/sivaprasadreddy/spring-boot-tutorials-blog-series/tree/main/spring-boot-rest-api-tutorial) repository.

# Spring Boot Flyway Database Migration Tutorial


In the previous [Spring Boot JdbcTemplate Tutorial]({{< relref "06-spring-boot-jdbctemplate-tutorial.md" >}}) 
we have seen how to initialize database using **schema.sql** and **data.sql** scripts.
This may be useful for demos and quick prototypes, but for real world applications we should use a database migration tool.

<!--more-->


[Flyway](https://flywaydb.org/) is one of the most popular Java-based database migration libraries.
Database migrations can be performed using Flyway as a standalone library, 
using [flyway-maven-plugin](https://documentation.red-gate.com/fd/maven-goal-184127408.html) 
or using [Flyway Gradle plugin](https://documentation.red-gate.com/fd/gradle-task-184127407.html).

Spring Boot provides out-of-the-box support for Flyway database migrations.
Let us see how we can create a Spring Boot application using Spring Data JPA to interact with PostgreSQL database and 
Flyway for implementing database migrations.

First, go to https://start.spring.io/ and create a Spring Boot application by selecting **Spring Web**, 
**Spring Data JPA**, **PostgreSQL Driver**, **Flyway Migration**, and **Testcontainers** starters.

## Create Flyway Migration Scripts
Flyway follows the **V&lt;VERSION&gt;__&lt;DESCRIPTION&gt;.sql** naming convention for its versioned migration scripts.
Let's add the following two migration scripts under **src/main/resources/db/migration** folder. 

**V1__create_tables.sql**
```sql
create table bookmarks
(
    id         bigserial not null,
    title      varchar   not null,
    url        varchar   not null,
    created_at timestamp,
    primary key (id)
);
```


**V2__create_bookmarks_indexes.sql**
```sql
CREATE INDEX idx_bookmarks_title ON bookmarks(title);
```

## Setup Postgres Database using Testcontainers
Spring Boot 3.1.0 introduced support for [Testcontainers](https://testcontainers.com/) which we can use for writing integration tests 
and for local development also.

While generating the application, we have selected **PostgreSQL Driver**, and **Testcontainers** starters.
So, the generated application will have a **TestApplication.java** under **src/test/java/** similar to the following:

```java
import org.springframework.boot.SpringApplication;
import org.springframework.boot.test.context.TestConfiguration;
import org.springframework.boot.testcontainers.service.connection.ServiceConnection;
import org.springframework.context.annotation.Bean;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.utility.DockerImageName;

@TestConfiguration(proxyBeanMethods = false)
public class TestApplication {

  @Bean
  @ServiceConnection
  PostgreSQLContainer<?> postgresContainer() {
    return new PostgreSQLContainer<>(DockerImageName.parse("postgres:15-alpine"));
  }

  public static void main(String[] args) {
    SpringApplication
            .from(Application::main)
            .with(TestApplication.class)
            .run(args);
  }
}
```

## Executing Flyway Migrations

Now you can run the **TestApplication** class from your IDE or run **./mvnw spring-boot:test-run** from the command line to start the application.
Then you should notice the following Flyway execution related logs as follows:

```shell
INFO 4009 --- [           main] tc.postgres:15-alpine                    : Container is started (JDBC URL: jdbc:postgresql://127.0.0.1:33331/test?loggerLevel=OFF)
INFO 4009 --- [           main] o.f.c.internal.license.VersionPrinter    : Flyway Community Edition 9.16.3 by Redgate
INFO 4009 --- [           main] o.f.c.internal.license.VersionPrinter    : See release notes here: https://rd.gt/416ObMi
INFO 4009 --- [           main] o.f.c.internal.license.VersionPrinter    : 
INFO 4009 --- [           main] o.f.c.i.database.base.BaseDatabaseType   : Database: jdbc:postgresql://127.0.0.1:33331/test (PostgreSQL 15.3)
INFO 4009 --- [           main] o.f.c.i.s.JdbcTableSchemaHistory         : Schema history table "public"."flyway_schema_history" does not exist yet
INFO 4009 --- [           main] o.f.core.internal.command.DbValidate     : Successfully validated 2 migrations (execution time 00:00.010s)
INFO 4009 --- [           main] o.f.c.i.s.JdbcTableSchemaHistory         : Creating Schema History table "public"."flyway_schema_history" ...
INFO 4009 --- [           main] o.f.core.internal.command.DbMigrate      : Current version of schema "public": << Empty Schema >>
INFO 4009 --- [           main] o.f.core.internal.command.DbMigrate      : Migrating schema "public" to version "1 - create tables"
INFO 4009 --- [           main] o.f.core.internal.command.DbMigrate      : Migrating schema "public" to version "2 - create bookmarks indexes"
INFO 4009 --- [           main] o.f.core.internal.command.DbMigrate      : Successfully applied 2 migrations to schema "public", now at version v2 (execution time 00:00.041s)
```

Flyway keeps track of all the applied migrations history in **flyway_schema_history** table by default.
If you check the data in **flyway_schema_history** table now, you can see the following rows:

```shell
| installed_rank | version | description              | type | script                           | checksum   | installed_by | installed_on               | execution_time | success |
|:---------------|:--------|:-------------------------|:-----|:---------------------------------|:-----------|:-------------|:---------------------------|:---------------|:--------|
| 1              | 1       | create tables            | SQL  | V1__create_tables.sql            | 1020037327 | test         | 2023-08-09 09:13:04.439012 | 6              | true    |
| 2              | 2       | create bookmarks indexes | SQL  | V2__create_bookmarks_indexes.sql | 732086927  | test         | 2023-08-09 09:13:04.456876 | 4              | true    |
```

If you keep the same database instance running and restart the application, then Flyway doesn't re-run the already applied migrations.
If you have added any new migrations, then only those migration scripts will be executed.

{{< box warning >}}
**Flyway Rules**

The following rules must be followed, otherwise Flyway will throw error while applying migrations:

* There should be no duplicate version numbers in flyway migration script file names. 
   
   Ex: **V1__init.sql**, **V1__indexes.sql**. Here version number 1 is used multiple times.
* Once a migration is applied, you should not change its content.
{{< /box >}}

## Customizing Flyway Configuration
Spring Boot provides sensible defaults for Flyway migrations, 
but you can configure various Flyway configuration properties 
using **spring.flyway.{property-name}** properties in **application.properties** file.

### Flyway migrations for different databases
If you are building a product which can be used with different databases, then you can configure flyway migration locations as follows:

```properties
spring.flyway.locations=classpath:db/migration/{vendor}
```

Then you can place **mysql** specific scripts under **src/main/resources/db/migration/mysql** directory,
**postgresql** specific scripts under **src/main/resources/db/migration/postgresql**, etc.
You can check the vendor names in **org.springframework.boot.jdbc.DatabaseDriver** class.

In addition to this, some interesting flyway configuration properties are:

```properties
# disable flyway execution
spring.flyway.enabled=false

# If you have an existing database and start using Flyway for new database changes.
spring.flyway.baseline-on-migrate=true

# To customize flyway migrations tracking table name
spring.flyway.table=db_migrations

# In case of any flyway execution errors, clean up the database and re-run all the migrations
## NEVER USE THIS IN PRODUCTION. ONLY SUITABLE FOR PROTOTYPING
spring.flyway.clean-disabled=false
spring.flyway.clean-on-validation-error=true
```

## Java based Flyway Migrations
In addition to SQL scripts, you can also write database migrations using Java classes.
You can create **V3__InsertSampleData.java** in **db.migration** package as follows:

```java
package db.migration;

import org.flywaydb.core.api.migration.BaseJavaMigration;
import org.flywaydb.core.api.migration.Context;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.SingleConnectionDataSource;

public class V3__InsertSampleData extends BaseJavaMigration {

  public void migrate(Context context) {
      JdbcTemplate jdbcTemplate = new JdbcTemplate(
        new SingleConnectionDataSource(context.getConnection(), true));
      
      Long userId = jdbcTemplate.query("...");
      // insert roles for userId
      jdbcTemplate.update("...");
  }
  
}
```

If the database changes involve complex logic which is difficult to write using plain SQL,
then Java based migrations come very handy.

You can use Flyway Database Migrations with different persistence technologies 
such as **JdbcTemplate**, **Spring Data Jdbc**, **Spring Data JPA**, **jOOQ**, etc.

{{< box info >}}
**Spring Boot Tutorials**

You can find more Spring Boot tutorials on [Spring Boot Tutorials]({{% relref "/pages/spring-boot-tutorials" %}}) page. 
{{< /box >}}

## Summary
Spring Boot's out of the box support for Flyway Database Migrations made it really easy to implement 
a production-grade database migration process.

You can find the sample code for this tutorial in this 
[GitHub](https://github.com/sivaprasadreddy/spring-boot-tutorials-blog-series/tree/main/spring-boot-flyway-tutorial) repository.

# Spring Boot Database Transaction Management Tutorial


Spring provides a high-level abstraction on top of **JDBC** with 
[JdbcTemplate]({{< relref "06-spring-boot-jdbctemplate-tutorial.md" >}}) 
to make it easier to perform database operations. 
Spring also provides a high-level abstraction on top of **JPA** 
with **Spring Data JPA** to make it easy to implement CRUD operations, sorting, pagination, etc.

<!--more-->


Whether you use **JdbcTemplate** or **JPA/Hibernate** or **Spring Data JPA**, you need to take care of 
handling database transactions.

A database transaction is a single unit of work, which either completes fully or does not complete at all, 
and leaves the database in a consistent state. While implementing database transactions you need to take 
**ACID (Atomicity, Consistency, Isolation, Durability)** properties into consideration.

Let's understand how we can handle database transactions in our Spring Boot applications.

## Transaction Handling using JDBC
First, let's take a quick look at how we usually handle database transactions in plain JDBC.

```java
class UserService {
    
    void register(User user) {
        String sql = "...";
        Connection conn = dataSource.getConnection(); // <1>
        try(conn) {  // <6>
            conn.setAutoCommit(false);  // <2>
            PreparedStatement pstmt = conn.prepareStatement(sql);
            pstmt.setString(1, user.getEmail());
            pstmt.setString(2, user.getPassword());
            pstmt.executeUpdate();  // <3>
            conn.commit();  // <4>
        } catch(SQLException e) {
            conn.rollback();  // <5>
        }
    }
}
```

In the above code snippet:

* **(1)** We obtained a database connection from DataSource connection pool
* **(2)** By default, every database operation will be running in its own transaction automatically.
  To take control of defining our transaction scope, we set AutoCommit to false.
* **(3)** Then we performed the necessary database operations.
* **(4)** If everything works fine then we are committing the transaction.
* **(5)** If any exception occurred then we rollback the transaction. 
* **(6)** As we are using **try-with-resources** syntax, the connection will be automatically closed.

In this implementation there is a good amount of **boilerplate code to handle database transaction**.

Spring simplifies database transaction handling mechanism by using **Aspect Oriented Programming (AOP)**
behind the scenes. We can implement database transaction handling in a declarative approach using **@Transactional** annotation 
or programmatically using **TransactionTemplate**. 

## Spring's declarative transaction handling using @Transactional annotation
Typically, in a 3-tier layered architecture, we will have a **Controller** in web layer, 
**Service** in business logic layer and **Repository** in data access layer. 

Generally, a **"Unit Of Work"** is modelled as a **Service** method and 
is considered to be a **"transactional boundary"**.

We can add Spring's **@Transactional** annotation on a Service layer method to define the transaction scope.
All the database operations in that method will be running in a single transaction.
If the method is executed successfully then the transaction will be committed.
If any **RuntimeException** occurs during the method execution then the transaction will be rolled back.

```java
@Service
class UserService {
    private final JdbcTemplate jdbcTemplate;
    
    @Transactional
    void register(User user) {
        String sql = "...";
        jdbcTemplate.update(sql);
    }
}
```

You can add **@Transactional** annotation on a method level to make that particular method transactional, 
or at class level to make all the public methods in that class as transactional.

When you add **@Transactional** annotation on a class or on any of its methods, **Spring creates a Proxy** for that class 
and apply the transaction handling logic using Spring AOP behind the scenes. 

{{< figure src="/images/spring-txn-proxy.webp" width="60%" height="40%" >}}

When you add **@Transactional** annotation, by default:

* The propagation level is set to **REQUIRED** which participates in the current transaction if exists, or starts a new transaction.
* Rollbacks the transaction if any **RuntimeException** is thrown by the method.
* Does NOT rollbacks the transaction if any **Checked Exception** is thrown by the method.

You can customize this behaviour by specifying the **@Transactional** attributes as follows:

```java
@Service
class UserService {
    private final JdbcTemplate jdbcTemplate;
    
    @Transactional(
            propagation = Propagation.REQUIRES_NEW,
            rollbackFor = DuplicateUserException.class,
            noRollbackFor = IllegalArgumentException.class)
    void register(User user) throws DuplicateUserException {
        String sql = "...";
        jdbcTemplate.update(sql);
    }
}

class DuplicateUserException extends Exception {}
```

In the above code snippet, we specified propagation level to be **REQUIRES_NEW** instead of the default **REQUIRED**.
The **REQUIRES_NEW** propagation level, suspends the current transaction if one exists, 
starts a new transaction, executes the database operations, commits the transaction 
and, then resumes the previously suspended transaction.

Also, we specified to rollback the transaction if **DuplicateUserException** occurs even though it is a **Checked Exception**.
And, we specified not to rollback the transaction if **IllegalArgumentException** occurs even though it is a **RuntimeException**.

{{< youtube jviq49ukATo >}}

## Spring's programmatic transaction handling using TransactionTemplate
Spring also provides a programmatic approach to handle database transactions using **TransactionTemplate** as follows:

```java
@Service
class UserService {
    private final TransactionTemplate transactionTemplate;

    void register(User user) {
        transactionTemplate.execute(status -> {
            String sql = "...";
            jdbcTemplate.update(sql);
            
            // If anything goes wrong, rollback
            //status.setRollbackOnly();

            return result;
        });
    }
}
```

You can use **TransactionTemplate** if you want to have more control over transaction management.

## A common mistake while using @Transactional annotation
We need to understand how proxies work to avoid mis-configuring the transaction handling logic.

Let's take an example scenario where a Controller calls a transactional method which in turn calls 
other transactional method.

```shell
UserController  --> UserService --> AccountService
```

Let's assume we have the user registration process implemented as follows:

```java
@Controller
class UserController {
    private final UserService userService;
    
    @PostMapping("/register")
    String register() {
        User user = ...;
        userService.register(user);
        return "...";
    }
}

//--------------------------------------------------------------

@Service
class UserService {
    private final UserRepository userRepository;
    private final AccountService accountService;
    
    @Transactional
    public void register(User user) {
        userRepository.save(user);
        
        Account account = ...;
        accountService.create(account);

        UserPreferences preferences = ...;
        userRepository.savePreferences(preferences);
    }
}

//--------------------------------------------------------------

@Service
class AccountService {
    
    @Transactional(propagation = Propagation.REQUIRES_NEW)
    public void create(Account account) {
        ...
    }
}
```
In the above example scenario, from the **UserController** we are calling **UserService.register(..)** method
which is annotated with **@Transactional**. As there is no new transaction in progress, a new transaction will be created.
The user details will be saved into the database within the current transaction.
After that we are calling **AccountService.create(account)** which is annotated with **@Transactional(propagation = Propagation.REQUIRES_NEW)**.
Now the current transaction which is in progress will be temporarily suspended and a new transaction will be started.
The account creation will be executed in that new transaction and if no exception occurred that transaction will be committed immediately.
Then the previous transaction will be resumed and saves the user preferences. 

If a **RuntimeException** occurs while saving preferences then only user record insertion will be rolled back, 
but not account creation as account creation happened in a separate transaction which is already committed successfully.

**This will work as expected.**

But, imagine the account creation method is also in **UserService** as follows:

```java
@Controller
class UserController {
    private final UserService userService;
    
    @PostMapping("/register")
    String register() {
        User user = ...;
        userService.register(user);
        return "...";
    }
}

//--------------------------------------------------------------

@Service
class UserService {
    private final UserRepository userRepository;
    
    @Transactional
    public void register(User user) {
        userRepository.save(user);
        
        Account account = ...;
        this.create(account);

        UserPreferences preferences = ...;
        userRepository.savePreferences(preferences);
    }

    @Transactional(propagation = Propagation.REQUIRES_NEW)
    public void create(Account account) {
        ...
    }
}
```

We may assume the account creation logic will be executed in a new transaction 
because we have configured **REQUIRES_NEW** propagation level. **But this does NOT work.**

When you added **@Transactional** annotation on **UserService**, a proxy is created, and that proxy is injected into the **UserController**.
When **userService.register(..)** method is called from **UserController**, you are calling the **register()** method on the proxy.
The proxy will apply the transaction handling logic according to the **@Transactional** semantics and then 
delegates the execution of actual logic to the actual **UserService** instance.

Now, from within the **UserService.register()** method when the **create(account)** method is called, 
it will be a local method call on the actual **UserService** object which is unaware of transaction handling.

{{< figure src="/images/spring-txn-gotcha.webp" width="80%" height="60%" >}}

In the above scenario, the **create(account)** logic will also be executed in the same 
parent transaction that is created when you called **UserService.register()** method.

So, if you want to run a method with different transaction semantics the better way is to move that method into a separate class.
There are alternative ways such as injecting the same class dependency into itself lazily, but it is not a recommended approach.

{{< box info >}}
**Spring Boot Tutorials**

You can find more Spring Boot tutorials on [Spring Boot Tutorials]({{% relref "/pages/spring-boot-tutorials" %}}) page. 
{{< /box >}}

## Summary
Spring's transaction handling support greatly helps in implementing persistence layer logic 
without polluting with low-level transaction handling boilerplate code. 

We have looked at a common mistake often beginners make when one transactional method calls 
another method with different transactional semantics in the same class and talked about a possible solution.

# Spring Boot JdbcTemplate Tutorial


## Introducing Spring Boot JDBC Support
Spring's **JdbcTemplate** provides high-level abstraction on top of **DataSource** to perform database operations.
In addition to that Spring's declarative Transaction Management capabilities helps to manage database transactions 
in a simplified way without having to write boilerplate code.

<!--more-->


Spring Boot simplifies the configuration of **DataSource**, **TransactionManager**, etc. by using 
it's AutoConfiguration mechanism.

{{< box info >}}
**How Spring Boot AutoConfiguration magic works?**

If you want to learn more about Spring Boot AutoConfiguration, 
see [How Spring Boot AutoConfiguration magic works?]({{< relref "2016-03-13-how-springboot-autoconfiguration-magic.md" >}}).
{{< /box >}}

Let's see how we can perform CRUD operations using **JdbcTemplate** with PostgreSQL database.

First, go to https://start.spring.io/ and create a Spring Boot application by selecting **JDBC API**, **PostgreSQL Driver** 
and **Testcontainers** starters.

Imagine we are building a simple application to manage bookmarks.
So, we are going to create **bookmarks** table with **id, title, url and created_at** columns.

## Initializing the Database
Spring Boot provides a convenient mechanism to initialize a database.
We can create **schema.sql** and **data.sql** files under **src/main/resources** which will be automatically executed 
upon starting the application.
However, this automatic script execution is enabled by default only when using in-memory databases like HSQL, H2, etc. but disabled otherwise.

We can enable the script initialization by adding the following property in **src/main/resources/application.properties** file.

```properties
spring.sql.init.mode=always
```

Now, let's create **src/main/resources/schema.sql** file as follows:

```sql
create table if not exists bookmarks
(
    id         bigserial not null,
    title      varchar   not null,
    url        varchar   not null,
    created_at timestamp,
    primary key (id)
);
```

To insert some sample data, create **src/main/resources/data.sql** file as follows:

```sql
truncate table bookmarks;
ALTER SEQUENCE bookmarks_id_seq RESTART WITH 1;

insert into bookmarks(title, url, created_at) values
('How (not) to ask for Technical Help?','https://sivalabs.in/how-to-not-to-ask-for-technical-help', CURRENT_TIMESTAMP),
('Announcing My Spring Boot Tips Video Series on YouTube','https://sivalabs.in/announcing-my-springboot-tips-video-series', CURRENT_TIMESTAMP),
('Kubernetes - Exposing Services to outside of Cluster using Ingress','https://sivalabs.in/kubernetes-ingress', CURRENT_TIMESTAMP),
('Kubernetes - Blue/Green Deployments','https://sivalabs.in/kubernetes-blue-green-deployments', CURRENT_TIMESTAMP),
('Kubernetes - Releasing a new version of the application using Deployment Rolling Updates','https://sivalabs.in/kubernetes-deployment-rolling-updates', CURRENT_TIMESTAMP),
('Getting Started with Kubernetes','https://sivalabs.in/getting-started-with-kubernetes', CURRENT_TIMESTAMP),
('Get Super Productive with Intellij File Templates','https://sivalabs.in/get-super-productive-with-intellij-file-templates', CURRENT_TIMESTAMP),
('Few Things I learned in the HardWay in 15 years of my career','https://sivalabs.in/few-things-i-learned-the-hardway-in-15-years-of-my-career', CURRENT_TIMESTAMP),
('All the resources you ever need as a Java & Spring application developer','https://sivalabs.in/all-the-resources-you-ever-need-as-a-java-spring-application-developer', CURRENT_TIMESTAMP),
('GoLang from a Java developer perspective','https://sivalabs.in/golang-from-a-java-developer-perspective', CURRENT_TIMESTAMP),
('Imposing Code Structure Guidelines using ArchUnit','https://sivalabs.in/impose-architecture-guidelines-using-archunit', CURRENT_TIMESTAMP),
('Spring Boot Integration Testing using TestContainers Starter','https://sivalabs.in/spring-boot-integration-testing-using-testcontainers-starter', CURRENT_TIMESTAMP),
('Creating Yeoman based Spring Boot Generator','https://sivalabs.in/creating-yeoman-based-springboot-generator', CURRENT_TIMESTAMP),
('Testing REST APIs using Postman and Newman','https://sivalabs.in/testing-rest-apis-with-postman-newman', CURRENT_TIMESTAMP),
('Testing Spring Boot Applications','https://sivalabs.in/spring-boot-testing', CURRENT_TIMESTAMP)
;
```

{{< box tip >}}
**Use a DB Migration Tool**

Though Spring Boot provides easy way to initialize database using **schema.sql** and **data.sql**, 
prefer using a proper database migration tools like **Liquibase** or **Flyway**.
{{< /box >}}

## Implementing CRUD operations using JdbcTemplate
Let's start with creating a class representing a **Bookmark**.

```java
import java.time.Instant;

public record Bookmark(
        Long id,
        String title,
        String url,
        Instant createdAt) {}
```

We have used Java records to model our **Bookmark** domain object.

Now, let's create **BookmarkRepository** class injecting **JdbcTemplate** as follows:

```java
@Repository
public class BookmarkRepository {
    private final JdbcTemplate jdbcTemplate;

    public BookmarkRepository(JdbcTemplate jdbcTemplate) {
        this.jdbcTemplate = jdbcTemplate;
    }
    
}
```

Let's start with implementing **findAll()** method to fetch all the records from the **bookmarks** table.

### Implementing findAll() method
When we query the database, it will return a **ResultSet**.
We can provide a **RowMapper** implementation to map the **ResultSet** data into our **Bookmark** domain object as follows:

```java
@Repository
public class BookmarkRepository {
    private final JdbcTemplate jdbcTemplate;

    public BookmarkRepository(JdbcTemplate jdbcTemplate) {
        this.jdbcTemplate = jdbcTemplate;
    }

    public List<Bookmark> findAll() {
        String sql = "select id, title, url, created_at from bookmarks";
        return jdbcTemplate.query(sql, INSTANCE);
    }

    static class BookmarkRowMapper implements RowMapper<Bookmark> {
        public static final BookmarkRowMapper INSTANCE = new BookmarkRowMapper();
        
        private BookmarkRowMapper(){}
        
        @Override
        public Bookmark mapRow(ResultSet rs, int rowNum) throws SQLException {
            return new Bookmark(
                    rs.getLong("id"),
                    rs.getString("title"),
                    rs.getString("url"),
                    rs.getTimestamp("created_at").toInstant()
            );
        }
    }
}
```
We have created **BookmarkRowMapper** implementing **RowMapper** interface as a **Singleton**.
Then we have implemented **findAll()** method to fetch all rows from **bookmarks** table 
and mapped them into **Bookmark** objects using **BookmarkRowMapper**.

{{< box warning >}}
**IMPORTANT**

Fetching all rows from a table might lead to **OutOfMemoryException** if there are huge number of records.
Always prefer to use **pagination** to fetch only a subset of records and process them.
{{< /box >}}

### Implement findById() method
Let's implement **findById(Long id)** method to fetch a bookmark by **id** as follows:

```java
public Optional<Bookmark> findById(Long id) {
    String sql = "select id, title, url, created_at from bookmarks where id = ?";
    try {
        Bookmark bookmark = jdbcTemplate.queryForObject(sql, INSTANCE, id);
        return Optional.of(bookmark);
    } catch (EmptyResultDataAccessException e) {
        return Optional.empty();
    }
}
```

The **findById()** method is returning **Optional&lt;Bookmark&gt;** because a bookmark with given **id** may or may not exist.
We are using **jdbcTemplate.queryForObject(...)** method which throws **EmptyResultDataAccessException** if no records found.
So, we are handling the exception and returning **Optional.empty()**.

### Implement create() method
In **create()** method, we are going to insert a record into bookmarks table and return the auto_generated primary key value.

```java
public Long create(Bookmark bookmark) {
    KeyHolder keyHolder = new GeneratedKeyHolder();

    jdbcTemplate.update(connection -> {
        String sql = "insert into bookmarks(title, url, created_at) values(?,?,?)";
        PreparedStatement ps = connection.prepareStatement(sql, new String[] { "id" });
        ps.setString(1, bookmark.title());
        ps.setString(2, bookmark.url());
        ps.setTimestamp(3, Timestamp.from(bookmark.createdAt()));
        return ps;
    }, keyHolder);

    return (long) keyHolder.getKey();
}
```

Note that, in newer versions of PostgreSQL we are specifying the auto_generated keys 
using **connection.prepareStatement(sql, new String[] { "id" })**.
In previous versions, we can specify the same using **connection.prepareStatement(sql, Statement.RETURN_GENERATED_KEYS)**.

### Implement update() method
Let's implement **update()** method in such a way that if a bookmark exists by the given **id** then 
we need to update **title** and **url** columns. Otherwise, throw an Exception.

```java
public void update(Bookmark bookmark) {
    String sql = "update bookmarks set title = ?, url = ? where id = ?";
    int count = jdbcTemplate.update(sql, bookmark.title(), bookmark.url(), bookmark.id());
    if (count == 0) {
        throw new RuntimeException("Bookmark not found");
    }
}
```

The **jdbcTemplate.update(...)** method return the number of rows affected by the executed query.
If the count is 0 means there is no bookmark exists with the given id and hence we are throwing an Exception.

### Implement delete() method
Let's implement **delete()** method in such a way that if a bookmark exists by the given **id** then
delete that row, otherwise throw an Exception.

```java
public void delete(Long id) {
    String sql = "delete from bookmarks where id = ?";
    int count = jdbcTemplate.update(sql, id);
    if (count == 0) {
        throw new RuntimeException("Bookmark not found");
    }
}
```

## Testing Repository using Testcontainers
We should test our repositories interacting with PostgreSQL database using same type of database.
[Testcontainers](https://testcontainers.com) can help us to test our application with real dependencies instead of using mocks or in-memory variations such as H2.

{{< box tip >}}
**Tip:**

You can also watch my [Testing Database Repositories using Testcontainers](https://www.youtube.com/watch?v=sWWwVAJ2cCw) video.
{{< /box >}}

Let's write **BookmarkRepositoryTest** using Testcontainers as follows:

```java
package com.sivalabs.bookmarks.domain;

import static org.assertj.core.api.Assertions.assertThat;

import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.jdbc.JdbcTest;
import org.springframework.jdbc.core.JdbcTemplate;

import java.time.Instant;
import java.util.List;
import java.util.Optional;

@JdbcTest(properties = {
   "spring.test.database.replace=none",
   "spring.datasource.url=jdbc:tc:postgresql:15.2-alpine:///db"
})
class BookmarkRepositoryTest {

    @Autowired
    JdbcTemplate jdbcTemplate;

    BookmarkRepository bookmarkRepository;

    @BeforeEach
    void setUp() {
        bookmarkRepository = new BookmarkRepository(jdbcTemplate);
    }

    @Test
    void shouldFindAllBookmarks() {
        List<Bookmark> bookmarks = bookmarkRepository.findAll();
        assertThat(bookmarks).isNotEmpty();
        assertThat(bookmarks).hasSize(15);
    }

    @Test
    void shouldCreateBookmark() {
        Bookmark bookmark = new Bookmark(null, "My Title", "https://sivalabs.in", Instant.now());
        Long id = bookmarkRepository.save(bookmark);
        assertThat(id).isNotNull();
    }

    @Test
    void shouldGetBookmarkById() {
        Bookmark bookmark = new Bookmark(null, "My Title", "https://sivalabs.in", Instant.now());
        Long id = bookmarkRepository.save(bookmark);

        Optional<Bookmark> bookmarkOptional = bookmarkRepository.findById(id);
        assertThat(bookmarkOptional).isPresent();
    }

    @Test
    void shouldEmptyWhenBookmarkNotFound() {
        Optional<Bookmark> bookmarkOptional = bookmarkRepository.findById(9999L);
        assertThat(bookmarkOptional).isEmpty();
    }

    @Test
    void shouldUpdateBookmark() {
        Bookmark bookmark = new Bookmark(null, "My Title", "https://sivalabs.in", Instant.now());
        Long id = bookmarkRepository.save(bookmark);

        Bookmark updatedBookmark = new Bookmark(id, "My Updated Title", "https://www.sivalabs.in", bookmark.createdAt());
        bookmarkRepository.update(updatedBookmark);

        updatedBookmark = bookmarkRepository.findById(id).orElseThrow();
        assertThat(updatedBookmark.id()).isEqualTo(id);
        assertThat(updatedBookmark.title()).isEqualTo("My Updated Title");
        assertThat(updatedBookmark.url()).isEqualTo("https://www.sivalabs.in");
    }
    
    @Test
    void shouldDeleteBookmark() {
        Bookmark bookmark = new Bookmark(null, "My Title", "https://sivalabs.in", Instant.now());
        Long id = bookmarkRepository.save(bookmark);

        bookmarkRepository.delete(id);
    }
}
```

We are using Spring Boot Test Slice annotation **@JdbcTest** to test only Repository instead of loading the entire application.
Then we are using Testcontainers special JDBC URL support to spin up a PostgreSQL database using **postgres:15.2-alpine** image.
Then we have written various tests to test our CRUD operations.

{{< box info >}}
**Spring Boot Tutorials**

You can find more Spring Boot tutorials on [Spring Boot Tutorials]({{% relref "/pages/spring-boot-tutorials" %}}) page. 
{{< /box >}}

## Summary
Spring's **JdbcTemplate** provides high-level abstraction to perform database operations without having to write boilerplate code.
We have learned how to implement CRUD operations using **JdbcTemplate** and also written tests using **Testcontainers**.


# Spring Boot Profiles Tutorial


## Introducing Spring Profiles
Typically, software applications run in different environments.
During development, it will be **local**, and then we may deploy it on **QA**, **Staging**, **Performance** 
and finally in **Production** environments.
You may have to configure your application with different configuration properties while running the application 
in different environments. 

<!--more-->


For example, if you are using a database then you may configure the database connection properties to a locally running database 
during development. While deploying it on QA, Staging, Performance and Production environments, 
you will need to configure the database properties to that environment specific database.

Spring provides great support for externalizing the application properties so that we can configure different properties 
while running in different environments.

{{< box tip >}}
**Spring Boot Application Configuration Tutorial**

To learn more about how to configure application properties,
please refer [Spring Boot Application Configuration Tutorial]({{< relref "03-spring-boot-application-configuration-tutorial.md" >}}) tutorial.
{{< /box >}}

To make it even simpler, we can use Spring Profiles where we can configure our application properties using different profiles 
and then enable the desired profiles based on our needs.

## Profile specific configuration using properties
Let's assume we are building a Spring Boot application which uses PostgreSQL database.
We want to run our application locally pointing to PostgreSQL database running on our laptop.
Also, we want to deploy our application in QA and Production environments.

We can configure the default properties in **default** profile configuration file **src/main/resources/application.properties** as follows:

```properties
spring.datasource.url=jdbc:postgresql://localhost:5432/postgres
spring.datasource.username=postgres
spring.datasource.password=postgres
```

We can configure any profile specific properties in **src/main/resources/application-{profile}.{properties/yml}** file
and Spring Boot automatically use profile specific properties when you activate that profile.
So, we can configure QA profile configuration properties in **src/main/resources/application-qa.properties** as follows:

```properties
spring.datasource.url=jdbc:postgresql://postgres_qa:5432/postgres
spring.datasource.username=${DB_USERNAME}
spring.datasource.password=${DB_PASSWORD}
```

In **qa** profile configuration file we have configured the JDBC URL pointing to a remote database **postgres_qa**.
And, username and password are configured to use environment variables **DB_USERNAME**, **DB_PASSWORD** respectively.

Instead of creating separate properties file for different profiles, you can put all profile properties 
in **application.properties** file and separate profile-specific configuration by using `"#---"` or `"!---"` separator as follows:

**application.properties**

```properties
spring.datasource.url=jdbc:postgresql://localhost:5432/postgres
spring.datasource.username=postgres
spring.datasource.password=postgres

#---
spring.config.activate.on-profile=qa
spring.datasource.url=jdbc:postgresql://postgres_qa:5432/postgres
spring.datasource.username=${DB_USERNAME}
spring.datasource.password=${DB_PASSWORD}
```

## Profile specific configuration using Configuration classes
Profiles support is not only limited to configuring properties.
We can also use Profiles to register beans in Spring Configuration classes as follows:

```java
@Configuration
class RepositoriesConfiguration {
    
    @Bean
    @Profile("jdbc")
    UserRepository jdbcUserRepository() {
        return new JdbcUserRepository();
    }

    @Bean
    @Profile("mongo")
    UserRepository mongoUserRepository() {
        return new MongoUserRepository();
    }
}
```

In this Spring configuration class, we have registered 2 different **UserRepository** beans but with different profiles.
Now, when the application is started using **"jdbc"** profile then **JdbcUserRepository** instance will be registered. 
If **"mongo"** profile is enabled then **MongoUserRepository** instance will be registered.

In addition to applying **@Profile** at the **@Bean** level, we can also use **@Profile** at the class level on **@Configuration** classes.
In that case, all the beans registered in that class will be associated with that profile and will be registered in the ApplicationContext 
only if that profile is active.

## Using Profile Expressions
You can associate more than one profile using **@Profile** annotation as follows:

```java
@Configuration
class RepositoriesConfiguration {
    @Bean
    @Profile({"mongo", "cloud"})
    UserRepository mongoUserRepository() {
        return new MongoUserRepository();
    }
}
```
In this example, the **MongoUserRepository** bean is registered with **"mongo"** and **"cloud"** profile.
If either **"mongo"** or **"cloud"** profile is activate then **MongoUserRepository** bean will be registered.

In addition to associating multiple profiles, we can also use **Profile Expressions** to implement complex bean registration needs.

```java
@Configuration
class RepositoriesConfiguration {
    @Bean
    @Profile({"mongo & cloud", "docker", "!jdbc"})
    UserRepository mongoUserRepository() {
        return new MongoUserRepository();
    }
}
```
As you can see, we can use the logical operators **&, | and !** operators also.
In the above example, the **MongoUserRepository** bean will be registered if at least one of the following condition is true:

* Enabled both **mongo** and **cloud** profiles
* Enabled **docker** profile
* Not enabled **jdbc** profile

{{< box info >}}
**Important:**

If you define a bean without associating to any profile then that bean is associated with **"default"** profile which is active by default.
{{< /box >}}

While Profile Expressions gives more power to configure profiles, **DON'T OVER USE PROFILES**.
Otherwise, soon you will find yourself wondering which bean is going to be registered in which scenario. 

{{< figure src="https://media.tenor.com/GXgi36aVH3gAAAAd/confused-meme-confused.gif" alt="confused" >}}


## How to activate Spring Profiles
Once we configured profile specific properties and beans, we need to activate the profile(s) in order to 
enable profile-specific configurations.

A common way to enable profile(s) is using **SPRING_PROFILES_ACTIVE** environment variable.

```shell
$ export SPRING_PROFILES_ACTIVE=mongo
$ java -jar app.jar

$ SPRING_PROFILES_ACTIVE=docker,jdbc java -jar app.jar
```

You can also enable profiles using **spring.profiles.active** System Property as follows:

```shell
$ java -Dspring.profiles.active=docker,jdbc -jar app.jar
```

In tests, you can enable profiles using **@ActiveProfiles** annotation as follows:

```java
@SpringBootTest
@ActiveProfiles({"test", "integration-test"})
class CustomerControllerTest {

  @Test
  void testSomething() {
      ...
  }
}
```

If you don't activate any profiles, then the **"default"** profile will be active, and you can see it in the application logs as follows: 

```shell
2023-07-29T09:55:53.709+05:30  INFO 4338 --- [  restartedMain] c.s.h.Application    : Starting Application using Java 17.0.4.1 with PID 4338...
2023-07-29T09:55:53.710+05:30  INFO 4338 --- [  restartedMain] c.s.h.Application    : No active profile set, falling back to 1 default profile: "default"
...
...
```

If you enable **"docker", "jdbc"** profiles and start the application, you can see the activated profiles in the application logs as follows:

```shell
2023-07-29T09:55:53.709+05:30  INFO 4338 --- [  restartedMain] c.s.h.Application    : Starting Application using Java 17.0.4.1 with PID 4338...
2023-07-29T09:55:53.710+05:30  INFO 4338 --- [  restartedMain] c.s.h.Application    : The following 2 profiles are active: "docker", "jdbc"
...
...
```

{{< box info >}}
**Spring Boot Tutorials**

You can find more Spring Boot tutorials on [Spring Boot Tutorials]({{% relref "/pages/spring-boot-tutorials" %}}) page. 
{{< /box >}}

## Summary
Spring Profiles are typically used for configuring environment specific properties,
but you can use profiles for any arbitrary criteria. 
While Spring Profiles comes handy, overusing them will make understanding the application 
configuration confusing in the long run. 

In this tutorial, we have learned how to use profiles to configure profile-specific properties,
register beans and activating the desired profiles.

# Spring Boot Logging Tutorial

Logging is a common and important requirement for running applications in production.
Spring Boot provides great support for application logging out of the box and offers various customization options.
In this tutorial, you will learn how to implement logging in your Spring Boot application using **Logback** and **Log4j2**. 

<!--more-->


{{< box info >}}
**Getting Started with Spring Boot**

If you are new to Spring Boot, refer [Getting Started with Spring Boot]({{< relref "01-getting-started-with-spring-boot.md" >}}) tutorial 
to learn how to create a Spring Boot project.
{{< /box >}}

## Spring Boot Default Logging Support
When you create a Spring Boot application with any starter added, they have a dependency on 
**spring-boot-starter** which in turn depends on **spring-boot-starter-logging** 
which transitively add **logback** dependencies for logging.

Spring Boot default logging configuration is already configured in **spring-boot-starter** with **CONSOLE** appender.
You can view the default configuration files (**defaults.xml**, **base.xml**, **console-appender.xml**, **file-appender.xml**) 
packaged in **spring-boot.jar** under the package **org/springframework/boot/logging/logback**.

So, by default Spring Boot is configured with Slf4j and Logback for logging and you can start logging in your application as follows:

```java
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

@Service
class CustomerService {
    private static final Logger log = LoggerFactory.getLogger(CustomerService.class);
    
    public Customer findById(Long id) {
        log.info("Fetching customer by id: {}", id);
        ...
    }
}
```

This will produce a **INFO** level log entry in your **CONSOLE** similar to the following:

```shell
2023-07-26T19:44:04.492+05:30  INFO 69881 --- [nio-8080-exec-1] c.s.demo.CustomerService      : Fetching customer by id: : 1
```

## Customizing Log levels
By default, Spring Boot is configured with **root** log level set to **INFO**.
You can configure the log level for **root** or for any specific package in **src/main/resources/application.properties** as follows:

```properties
logging.level.root=WARN
logging.level.com.sivalabs=DEBUG
logging.level.org.springframework=INFO
logging.level.org.hibernate.validator=DEBUG
```

## Enabling File Appender
Spring Boot already configured **RollingFileAppender**, but is disabled by default.
You can enable File Appender by simply configuring either **logging.file.name** or **logging.file.path** property in **application.properties** as follows:

```properties
# relative to the current directory
logging.file.name=product-service.log

# absolute path
logging.file.name=/var/log/product-service.log

# write logs to /var/log/product-service/spring.log file
logging.file.path=/var/log/product-service
```

Note that when you configure **logging.file.name** or **logging.file.path** properties, both **Console** and **File** Appenders are enabled.
If you want to disable Console appender and enable only File appender, or you want to have more control over logback configuration, 
you can create **src/main/resources/logback.xml** and configure the logging as per your needs. 

## Custom Logback configuration
If the Spring Boot logging customization options via **logging.*** properties are not sufficient, 
you can create your own **src/main/resources/logback.xml** file and Spring Boot will configure logging according to this configuration.

However, Spring Boot provides some logback extensions which helps you to use advanced configuration options.
In order to use these extensions, create the configuration file with name **logback-spring.xml** instead of **logback.xml**.

A sample **logback-spring.xml** file looks as follows:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml" />
    <include resource="org/springframework/boot/logging/logback/console-appender.xml" />

    <root level="INFO">
        <appender-ref ref="FILE" />
    </root>
    <logger name="org.springframework" level="WARN"/>
    <logger name="com.sivalabs" level="DEBUG"/>

</configuration>
```

Spring Boot Logback extensions comes handy if you want to use any Spring Boot property or customize logging based on Spring profiles.

```xml
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <springProperty scope="context" 
                    name="logstashHost" 
                    source="logstash.host"
                    defaultValue="localhost"/>
    <include resource="org/springframework/boot/logging/logback/defaults.xml" />

    <springProfile name="docker">
        <include resource="org/springframework/boot/logging/logback/file-appender.xml" />
        <property name="LOG_FILE" value="/var/logs/myapp.log"/>

        <appender name="logstash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
            <destination>${logstashHost}</destination>
            ...
            ...
        </appender>
        
        <root level="INFO">
            <appender-ref ref="FILE" />
            <appender-ref ref="logstash" />
        </root>
    </springProfile>
    
    <springProfile name="!docker">
        <include resource="org/springframework/boot/logging/logback/console-appender.xml" />
        <root level="INFO">
            <appender-ref ref="CONSOLE" />
        </root>
    </springProfile>
    
    <logger name="com.sivalabs" level="DEBUG"/>
    <logger name="org.springframework" level="INFO"/>

</configuration>
```

In the above configuration, we have enabled appenders based on Spring profile using `<springProfile name="...">`.
If **docker** profile is enabled then we are using **FILE**, **Logstash** appenders.
If the **docker** profile is not activated then we are using **CONSOLE** appender only.
Also, notice that we are using **<springProperty .../>** to use the **logstash.host** property value to specify the Logstash host. 

{{< box tip >}}
**Tip:**

You can also watch my [How to implement Logging in Spring Boot applications](https://www.youtube.com/watch?v=tmj6QphzAPo) video.
{{< /box >}}

## Using Log4j2 instead of Logback
If you prefer to use **Log4j2** instead of **Logback**, you can exclude **spring-boot-starter-logging** from **spring-boot-starter**
and include **spring-boot-starter-log4j2** in **pom.xml** as follows.

```xml
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter</artifactId>
    <exclusions>
        <exclusion>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-logging</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-log4j2</artifactId>
</dependency>
```

In case you are using **Gradle**, you can configure to use **log4j2** instead of **logback** as follows:

```groovy
configurations.all {
	exclude group: 'org.springframework.boot', module: 'spring-boot-starter-logging'
}

dependencies {
    implementation('org.springframework.boot:spring-boot-starter-log4j2')
    ...
    ...
}
```

Spring Boot by default includes **log4j2.xml** configuration file in **org/springframework/boot/logging/log4j2/log4j2.xml**.
But if you want to provide a customized configuration, then you can create **src/main/resources/log4j2-spring.xml** as follows:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
    <Properties>
        <Property name="LOG_PATTERN">
            %d{yyyy-MM-dd HH:mm:ss.SSS} %-5p %c{1}:%L - %m%n
        </Property>
    </Properties>
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT" follow="true">
            <PatternLayout pattern="${LOG_PATTERN}"/>
        </Console>
    </Appenders>
    <Loggers>
        <Logger name="com.sivalabs" level="debug" additivity="false">
            <AppenderRef ref="Console" />
        </Logger>

        <Root level="info">
            <AppenderRef ref="Console" />
        </Root>
    </Loggers>
</Configuration>
```

If you use slf4j API to create logger objects, then you don't have to make any code changes.
The slf4j API will delegate the actual logging to log4j2.

{{< box info >}}
**Spring Boot Tutorials**

You can find more Spring Boot tutorials on [Spring Boot Tutorials]({{% relref "/pages/spring-boot-tutorials" %}}) page. 
{{< /box >}}

## Summary
Spring Boot makes it easy to implement logging by providing the default configuration with customization options.
In this tutorial, we have learned about Spring Boot default logging support using Slf4J and Logback and how to customize it.
Then we have explored how to use Log4j2 instead of default Logback implementation.

# Spring Boot Application Configuration Tutorial


In the previous [Spring Boot Testing Tutorial]({{< relref "02-spring-boot-testing-tutorial.md" >}}),
we have learned how to write unit, slice and integration tests for a Spring Boot application.

In this tutorial, you will learn how to configure your Spring Boot application using **properties** and **YAML** files 
for running the application in different environments.

<!--more-->


## Externalizing Spring Boot Application Configuration
Any non-trivial application will have some configuration that you would like to make it configurable 
instead of hard-coding those values in the application code. 

Spring Boot provides [many ways to configure your application properties](https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.external-config),
among which we most likely use only the following approaches:

* Default properties in **src/main/resources/application.{properties/yml}**
* Profile-specific properties in **src/main/resources/application-{profile}.{properties/yml}**
* Overriding the default configuration using **Environment Variables** or **System Properties**

Typically, in our applications we will have 2 types of configuration properties:

* Spring Boot defined properties to configure services like DataSource, Kafka etc. 
  Ex: **spring.datasource.url**, **spring.datasource.username**, **spring.datasource.password**, etc

* Your application specific configuration properties.

## Configuring Default Properties
Spring Boot by default loads properties from **src/main/resources/application.{properties/yml}**.
For example, we can configure our application properties using **application.properties** file as follows:

```properties
spring.datasource.url=jdbc:postgresql://localhost:5432/postgres
spring.datasource.username=postgres
spring.datasource.password=secret123

ftp.host=ftpsrv001
ftp.port=21
ftp.username=appuser1
ftp.password=secret321
```

The same configuration values can be configured using **application.yml** file as follows:

```yaml
spring:
  datasource:
    url: jdbc:postgresql://localhost:5432/postgres
    username: postgres
    password: secret123

ftp:
  host: ftpsrv001
  port: 21
  username: appuser1
  password: secret321
```

## Configuring Profile-specific Properties
Our application may run in different environments such as **local**, **dev**, **qa**, **staging** and **production**.
We may want to configure different values for different environments. 
In such cases, we can use Spring's profile concept to configure different values for different environments.

Let's say we want to configure different database connection properties based on running environment.
We can configure local database parameters in default **application.properties** file, 
**qa** and **staging** properties using qa and staging profiles specific properties files as follows:

**application.properties**

```properties
spring.datasource.url=jdbc:postgresql://localhost:5432/postgres
spring.datasource.username=postgres
spring.datasource.password=secret123
```

**application-qa.properties**

```properties
spring.datasource.url=jdbc:postgresql://qadb:5432/postgres
spring.datasource.username=postgres
spring.datasource.password=secret123
```

**application-staging.properties**

```properties
spring.datasource.url=jdbc:postgresql://stagingdb:5432/postgres
spring.datasource.username=postgres
spring.datasource.password=secret123
```

When you run your application with a specific profile, then first properties from default profile will be loaded 
and then profile-specific properties will override them.

Instead of creating separate properties file for different profiles, you can put all profile properties in 
**application.properties** file and separate profile-specific configuration by using `"#---"` or `"!---"` separator as follows: 

**application.properties**

```properties
spring.datasource.url=jdbc:postgresql://localhost:5432/postgres
spring.datasource.username=postgres
spring.datasource.password=secret123

#---
spring.config.activate.on-profile=qa
spring.datasource.url=jdbc:postgresql://qadb:5432/postgres
spring.datasource.username=postgresqa
spring.datasource.password=secret1235

#---
spring.config.activate.on-profile=staging
spring.datasource.url=jdbc:postgresql://stagingdb:5432/postgres
spring.datasource.username=postgresstaging
spring.datasource.password=secret1236
```

If you are using YAML files, then you can configure different profile properties in the same **application.yml** file 
using `"---"` separator as follows:

```yaml
spring:
  datasource:
    url: jdbc:postgresql://localhost:5432/postgres
    username: postgres
    password: secret123
---
spring:
  config:
    activate:
      on-profile: "qa"
  datasource:
    url: jdbc:postgresql://qadb:5432/postgres
    username: postgres
    password: secret1235
```

## Using Environment Variables
A common approach is defining the default and profile-specific properties in the configuration files which will be packaged inside the jar file 
and then override those values using Environment Variables if required.

Spring Boot support [Relaxed Binding](https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.external-config.typesafe-configuration-properties.relaxed-binding)
which allows us to use environment variables with capitalized variable names separated with underscores follows:

* The **spring.datasource.url** property value can be overridden by environment variable **SPRING_DATASOURCE_URL**.
* The **app.jwt.expiryDuration** property value can be overridden by environment variable **APP_JWT_EXPIRY_DURATION**.

## How to use the configured properties from application?
There are multiple ways we can use the configured properties values.
In the above examples, we have configured DataSource properties which will be used by Spring Boot to create **DataSource** bean.

Let us see how can use the configured FTP server details.

### Using Environment
You can access all your application properties using Spring's **Environment** abstraction.
You can inject **Environment** bean and access the properties as follows:

```java
import org.springframework.core.env.Environment;

@Service
class MyService {
    private final Environment environment;
    
    public MyService(Environment environment) {
        this.environment = environment;
    }
    
    void upload(File file) {
      String host = environment.getProperty("ftp.host", "localhost");
      Integer port = environment.getProperty("ftp.port", Integer.class, 21);
      String username = environment.getRequiredProperty("ftp.username");
      String password = environment.getRequiredProperty("ftp.password");
      ...
      ...
    }
}
```

As you can see, we can obtain the configured values using **environment.getProperty("...")**.
You can also use other overloaded methods to specify default value, dataType of the property etc.
When you use **environment.getRequiredProperty("...")** and if the property value is **null** then it will throw **IllegalStateException**.

Instead of using Spring's **Environment** abstraction, which ties your code to the Spring framework, 
you can inject the property values directly using **@Value** annotation.

### Using @Value("${property.name}")
You can inject the property values using **@Value** annotation as follows:

```java
@Service
class MyService {
    private final String ftpHost;
    private final Integer ftpPort;
    private final String ftpUsername;
    private final String ftpPassword;
    
    public MyService(@Value("${ftp.host:localhost}") String ftpHost,
                     @Value("${ftp.port:21}") Integer ftpPort,
                     @Value("${ftp.username}") String ftpUsername,
                     @Value("${ftp.password}") String ftpPassword
                     ) {
        this.ftpHost = ftpHost;
        this.ftpPort = ftpPort;
        this.ftpUsername = ftpUsername;
        this.ftpPassword = ftpPassword;
    }
    
    void upload(File file) {
      ...
      ...
    }
}
```

We have used Spring property reference syntax **@Value("${property.name}")** to inject the configured value.
Optionally, we can also specify a default value using **@Value("${property.name:defaultValue}")** syntax.

You can also inject the property value using Field Injection, but it is highly discouraged.

```java
@Service
class MyService {
    @Value("${ftp.host:localhost}")
    private String ftpHost;
    
    @Value("${ftp.port:21}")
    private Integer ftpPort;
    
    @Value("${ftp.username}")
    private String ftpUsername;
    
    @Value("${ftp.password}")
    private String ftpPassword;
    
}
```

While using the **@Value("${property.name}")** approach works, it is very verbose.
There is an alternative way to use the configured properties using Typesafe Configuration Properties binding.

### Using Typesafe Configuration Properties Binding
Instead of binding each property value separately using **@Value**, we can create a class with the properties and 
let the Spring framework bind the configured values into an instance of that configuration properties class 
by using **@ConfigurationProperties** as follows:

```java
@ConfigurationProperties(prefix = "ftp")
public class FtpProperties {
    private String host;
    private Integer port;
    private String username;
    private String password;
}
```

Once we defined the configuration properties class, we need to enable the binding using **@EnableConfigurationProperties(FtpProperties.class)**.
This is typically done on the application main entrypoint class as follows:

```java
@SpringBootApplication
@EnableConfigurationProperties(FtpProperties.class)
public class MyApplication {

  public static void main(String[] args) {
    SpringApplication.run(MyApplication.class, args);
  }
}
```

We can have as many configuration properties class as we want and register them using
**@EnableConfigurationProperties({FtpProperties.class, SomeProperties.class, OtherProperties.class})**.
Instead of specifying each configuration properties class explicitly, we can use **@ConfigurationPropertiesScan**
which will scan for all the classes that are annotated with **@ConfigurationProperties** and automatically register them.

```java
@SpringBootApplication
@ConfigurationPropertiesScan
public class MyApplication {

  public static void main(String[] args) {
    SpringApplication.run(MyApplication.class, args);
  }
}
```

However, a common approach is having one configuration properties class 
and using nested classes for a different group of properties as follows:

```java
@ConfigurationProperties(prefix = "app")
public class ApplicationProperties {
    private final FtpProperties ftp = new FtpProperties();
    private final Jwt jwt = new Jwt();
    
    public static class FtpProperties {
      private String host;
      private Integer port;
      private String username;
      private String password;
      //setters & getters
    }

  public static class Jwt {
    private String secret;
    private Long expiryDuration;
    //setters & getters
  }
  
}
```

Now you can configure your application properties with **app.** prefix as follows:

```properties
app.ftp.host=ftpsrv001
app.ftp.port=21
app.ftp.username=appuser1
app.ftp.password=secret321

app.jwt.secret=supersecret
app.jwt.expiryDuration=3600000
```

### Using Java records for Configuration Properties Binding
Usually, once the application is started and configuration properties class is initialized by Spring,
we don't change the property values in that object. In that case, Java record is a better fit than a regular class.

Let's see how we can use Java records as configuration properties class.

```java
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.context.properties.bind.DefaultValue;

@ConfigurationProperties(prefix = "app")
public record ApplicationProperties(
    FtpProperties ftp,
    Jwt jwt){

    public record FtpProperties(
        @DefaultValue("localhost")
        String host,
        @DefaultValue("21")
        Integer port,
        String username,
        String password){
    }
    
    public record Jwt(
        String secret, 
        Long expiryDuration){
    }
}
```

Using records for binding configuration properties will help you 
to prevent accidental modification of the values as you can't change the values of a record.

## Validating Configuration Binding Properties

We can use Java Bean Validation API to validate the configuration property values so that we can fail-fast in case of invalid configuration 
during the application startup.

In order to use Java Bean Validation API we need to add the following dependency:

```xml
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-validation</artifactId>
</dependency>
```

Now we can use Bean Validation annotations like **@NotNull**, **@NotEmpty**, **@Min**, **@Max** as follows:

```java
import jakarta.validation.Valid;
import jakarta.validation.constraints.Min;
import jakarta.validation.constraints.NotEmpty;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.validation.annotation.Validated;

@ConfigurationProperties(prefix = "app")
@Validated
public class ApplicationProperties {
    @Valid
    private final FtpProperties ftp = new FtpProperties();
    
    @Valid
    private final Jwt jwt = new Jwt();
    
    public static class FtpProperties {
      @NotEmpty  
      private String host;
      private Integer port;
      @NotEmpty
      private String username;
      @NotEmpty
      private String password;
      //setters & getters
    }

  public static class Jwt {
    @NotEmpty
    private String secret;
    @NotNull
    @Min(30000)
    private Long expiryDuration;
    //setters & getters
  }
  
}
```

Note that, in order to trigger the validation, we need to add **@Validated** annotation on top-level class, 
and to validate properties of the nested class properties, we need to add **@Valid** on the property declaration.

{{< box tip >}}
**Tip:**

You can also watch my [Spring Boot Tips : Part 2 - Managing Application Configuration Properties In The Right Way](https://www.youtube.com/watch?v=4OX7hbAGuRE) video.
{{< /box >}}

## Advanced Configuration Options
In addition to the approaches described above, Spring Boot supports more advanced options to configure the application properties.

* [Spring Cloud Config Server](https://spring.io/projects/spring-cloud-config)
* [Spring Cloud Vault](https://spring.io/projects/spring-cloud-vault)
* [AWS Secrets Manager and Parameter Store](https://docs.awspring.io/spring-cloud-aws/docs/3.0.1/reference/html/index.html#spring-cloud-aws-secrets-manager)
* [Kubernetes ConfigMaps and Secrets](https://spring.io/projects/spring-cloud-kubernetes)

These configuration options will be explained in separate tutorials in the future.

{{< box info >}}
**Spring Boot Tutorials**

You can find more Spring Boot tutorials on [Spring Boot Tutorials]({{% relref "/pages/spring-boot-tutorials" %}}) page.
{{< /box >}}

## Summary
We have learned how to externalize our application configuration using default properties, profile-specific properties and also environment variables.
We also looked at how to consume those configured properties from our application.
Finally, we learned how to validate our application configuration using Java Bean Validation API annotations.


# Spring Boot Testing Tutorial


In the previous [Getting Started with Spring Boot]({{< relref "01-getting-started-with-spring-boot.md" >}}) tutorial,
we have learned how to create a Spring Boot application and built a simple REST API.

In this tutorial, you will learn how to write **unit**, **slice** and **integration tests** for your Spring Boot application.

<!--more-->


## Testing Spring Boot applications
We should write Unit Tests to verify the business logic of a particular unit(a class, a method, or a set of classes),
and they shouldn't be talking to any external services like database, queues or other webservices etc.
If the unit we are testing depends on any of those external services, then we can provide mock implementations of
those dependencies and verify the unit's behaviour.

In addition to Unit Tests, we should write Integration Tests which exercise the behaviour of a sub-system or a component
by talking to the real services and dependent collaborators.

When we generate our Spring Boot application **spring-boot-starter-test** dependency is automatically added which transitively
adds the most commonly used testing libraries such as **SpringTest**, **JUnit5**, **Mockito**, **Assertj**, **JsonPath**,
**JsonAssert** to our application as test dependencies.

## Types of Tests

**Unit Tests:** These are the tests to verify the behaviour of a single unit, and ideally
they shouldn't depend on frameworks like Spring or Hibernate, etc.

**Slice Tests:** These are the tests to verify a slice of the application such as Web layer or
Persistence layer etc. Spring Boot provides support for testing slices of the application
using **@WebMvcTest**, **@DataJpaTest**, **@DataMongoTest** etc.

**Integration Tests:** These are the tests, which tests the application in a blackbox manner.
We pass in some input, and we expect specific output, we don't know or care how it works internally.
Spring Boot provides support for writing Integration Tests using **@SpringBootTest**.

{{< box info >}}
**Spring Boot Tutorial Series GitHub Code Repository:**
https://github.com/sivaprasadreddy/spring-boot-tutorials-blog-series
{{< /box >}}

## Unit Testing with JUnit 5 and Mockito
We are going to write tests for the REST API we have implemented in the 
previous [Getting Started with Spring Boot](https://www.sivalabs.in/getting-started-with-spring-boot/#simple-rest-api-using-spring-boot) tutorial.

Let's start with writing Unit Tests for **GreetingService**.
We will use JUnit 5 and Mockito for writing unit tests.

**src/test/java/com/sivalabs/helloworld/GreetingServiceTest.java**
```java
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;

import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.BDDMockito.given;

@ExtendWith(MockitoExtension.class) // (1)
class GreetingServiceTest {

    @Mock // (2)
    private ApplicationProperties properties;

    @InjectMocks // (3)
    private GreetingService greetingService;

    @BeforeEach // (4)
    void setUp() {
        given(properties.getGreeting()).willReturn("Hello");
    }

    @Test
    void shouldGreetWithDefaultNameWhenNameIsNotProvided() {
        given(properties.getDefaultName()).willReturn("World");

        String greeting = greetingService.sayHello(null);

        Assertions.assertEquals("Hello World", greeting); //JUnit 5 based assertion (5)
        assertThat(greeting).isEqualTo("Hello World"); //Assertj based assertion (6)
    }

    @Test
    void shouldGreetWithGivenName() {
        String greeting = greetingService.sayHello("John");

        assertThat(greeting).isEqualTo("Hello John");
    }
}
```

* **(1)** We are using **MockitoExtension** so that we can create mock objects and inject them using annotations.
* **(2)** The **@Mock** annotation will initialize properties with a mock implementation of **ApplicationProperties** class.
* **(3)** The **@InjectMocks** annotation will create an instance of **GreetingService** by injecting the dependencies (properties) using the defined mock objects.
* **(4)** The **@BeforeEach** annotated method will be executed before each test execution so that we can put any common test setup in there.
* **(5)** We are asserting the expected output with actual output using **JUnit 5** based assertions.
* **(6)** We are asserting the expected output with actual output using **Assertj** based assertions.

{{< box tip >}}
**TIP:**

While JUnit5 assertions such as **Assertions.assertEquals()** does the job, **Assertj** assertions are fluent, and you get various convenient assertion methods depending on the type of the object.

To learn more about using AssertJ, you can watch [Why using Assertj assertions much better?](https://www.youtube.com/watch?v=Ri_fiEr3wMg) video. 
{{< /box >}}

## Testing application slices using Spring's Test Slice annotations

Now let's write a test for **HelloWorldController** using **@WebMvcTest**.

With **@WebMvcTest** only the web layer components such as Controllers, Interceptors, etc will be loaded into ApplicationContext.
So we need to add dependent beans by some means such as using **@MockBean** or configuring dependent beans using **@TestConfiguration**.

In our unit test we used Mockito's **@Mock** annotation to create a mock bean, but here we are using Spring's **@MockBean** instead of **@Mock**.
While using **@WebMvcTest** the test instance and **ApplicationContext** creation is taken care by Spring,
which is unaware of plain mock objects created by **@Mock**. When you use Spring's **@MockBean** that mock bean will become part of
ApplicationContext and will be injected into the Controller.

**src/test/java/com/sivalabs/helloworld/HelloWorldControllerTest.java**
```java
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTest;
import org.springframework.boot.test.mock.mockito.MockBean;
import org.springframework.test.web.servlet.MockMvc;

import static org.hamcrest.CoreMatchers.is;
import static org.mockito.BDDMockito.given;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;

@WebMvcTest(controllers = HelloWorldController.class) // (1)
class HelloWorldControllerTest {
    @Autowired
    private MockMvc mockMvc; // (2)

    @MockBean
    private GreetingService greetingService; // (3)

    @Test
    void shouldReturnGreetingSuccessfully() throws Exception {
        given(greetingService.sayHello("Siva")).willReturn("Hello Siva"); // (4)

        mockMvc.perform(get("/api/hello?name={name}", "Siva"))
            .andExpect(status().isOk())
            .andExpect(jsonPath("$.greeting", is("Hello Siva"))); // (5)
    }
}
```

* **(1)** Using **@WebMvcTest(controllers = HelloWorldController.class)** we are only testing the web layer controller and loading only **HelloWorldController**.
* **(2)** While using **@WebMvcTest** annotation **MockMvc** will be autoconfigured, and we can autowire and use it to invoke API endpoints.
* **(3)** Using **@MockBean** we are injecting a mock **GreetingService** bean into **HelloWorldController**.
* **(4)** Setting up the mock behaviour on mock **GreetingService** bean.
* **(5)** Invoking **GET /api/hello** API and asserting the Http Response Code and body using **jsonPath** assertions.


{{< box info >}}
**NOTE:**

As Slice tests only load a small subset of Spring components, they will be faster than integration tests written using **@SpringBootTest**.
{{< /box >}}

## Integration Tests using @SpringBootTest

Finally, let's write an integration test which loads the entire application then make an API call and test the results.

**src/test/java/com/sivalabs/helloworld/SpringBootHelloWorldApplicationTests.java**
```java
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.web.servlet.MockMvc;

import static org.hamcrest.CoreMatchers.is;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.*;

@SpringBootTest(webEnvironment= SpringBootTest.WebEnvironment.RANDOM_PORT) // (1)
@AutoConfigureMockMvc  // (2)
class SpringBootHelloWorldApplicationTests {

    @Autowired
    private MockMvc mockMvc;

    @Test
    void shouldReturnGreetingSuccessfully() throws Exception {
        mockMvc.perform(get("/api/hello?name={name}", "Siva"))
            .andExpect(status().isOk())
            .andExpect(jsonPath("$.greeting", is("Hello Siva")));
    }
}
```

* **(1)** We are using **@SpringBootTest** to load the entire application and also specified **webEnvironment= SpringBootTest.WebEnvironment.RANDOM_PORT** to start the application on a random available port so that there won't be any port conflicts with any running applications. This is especially very useful while running tests on build servers like Jenkins where multiple application builds run in parallel.
* **(2)** While using **@SpringBootTest** the **MockMvc** bean won't be autoconfigured, so we are using **@AutoConfigureMockMvc** to configure the **MockMvc** bean.

{{< box tip >}}
**NOTE:**

We have just scratched the surface of testing a Spring Boot application.
We will explore more testing techniques in upcoming posts as we go on.

You can also watch my [Java Testing Made Easy: Learn to write Unit, Integration, E2E & Performance Tests](https://www.youtube.com/watch?v=FGoLvCc6BeI&list=PLuNxlOYbv61jtHHFHBOc9N7Dg5jn013ix) video series.
{{< /box >}}

## Run tests
We can run tests using the build tool as follows:

Maven:

```shell
./mvnw verify
```

Gradle:

```shell
./gradlew test
```

{{< box info >}}
**Spring Boot Tutorials**

You can find more Spring Boot tutorials on [Spring Boot Tutorials]({{% relref "/pages/spring-boot-tutorials" %}}) page.
{{< /box >}}

## Summary
We have learned how to write unit tests using JUnit 5 and Mockito.
Then we learned about testing a slice of the application using Spring Boot Test Slice support.
Finally, we have learned how to write an integration test by bootstrapping the entire application.

# Privacy Policy


# Privacy Policy

Last updated: July 25, 2023

This Privacy Policy describes Our policies and procedures on the collection, use and disclosure of Your information when You use the Service and tells You about Your privacy rights and how the law protects You.

We use Your Personal data to provide and improve the Service. By using the Service, You agree to the collection and use of information in accordance with this Privacy Policy. This Privacy Policy has been created with the help of the [Free Privacy Policy Generator](https://www.freeprivacypolicy.com/free-privacy-policy-generator/).

# Interpretation and Definitions

## Interpretation

The words of which the initial letter is capitalized have meanings defined under the following conditions. The following definitions shall have the same meaning regardless of whether they appear in singular or in plural.

## Definitions

For the purposes of this Privacy Policy:

- __Account__ means a unique account created for You to access our Service or parts of our Service.
- __Affiliate__ means an entity that controls, is controlled by or is under common control with a party, where "control" means ownership of 50% or more of the shares, equity interest or other securities entitled to vote for election of directors or other managing authority.


- __Company__ (referred to as either "the Company", "We", "Us" or "Our" in this Agreement) refers to SivaLabs.


- __Cookies__ are small files that are placed on Your computer, mobile device or any other device by a website, containing the details of Your browsing history on that website among its many uses.

- __Country__ refers to: Telangana,  India

- __Device__ means any device that can access the Service such as a computer, a cellphone or a digital tablet.


- __Personal Data__ is any information that relates to an identified or identifiable individual.


- __Service__ refers to the Website.

- __Service Provider__ means any natural or legal person who processes the data on behalf of the Company. It refers to third-party companies or individuals employed by the Company to facilitate the Service, to provide the Service on behalf of the Company, to perform services related to the Service or to assist the Company in analyzing how the Service is used.
	

- __Usage Data__ refers to data collected automatically, either generated by the use of the Service or from the Service infrastructure itself (for example, the duration of a page visit).

- __Website__ refers to SivaLabs, accessible from [https://www.sivalabs.in/](https://www.sivalabs.in/)

- __You__ means the individual accessing or using the Service, or the company, or other legal entity on behalf of which such individual is accessing or using the Service, as applicable.


# Collecting and Using Your Personal Data

## Types of Data Collected

### Personal Data

While using Our Service, We may ask You to provide Us with certain personally identifiable information that can be used to contact or identify You. Personally identifiable information may include, but is not limited to:

- Email address
- First name and last name
- Usage Data

### Usage Data

Usage Data is collected automatically when using the Service.

Usage Data may include information such as Your Device's Internet Protocol address (e.g. IP address), browser type, browser version, the pages of our Service that You visit, the time and date of Your visit, the time spent on those pages, unique device identifiers and other diagnostic data.

When You access the Service by or through a mobile device, We may collect certain information automatically, including, but not limited to, the type of mobile device You use, Your mobile device unique ID, the IP address of Your mobile device, Your mobile operating system, the type of mobile Internet browser You use, unique device identifiers and other diagnostic data.

We may also collect information that Your browser sends whenever You visit our Service or when You access the Service by or through a mobile device.


### Tracking Technologies and Cookies

We use Cookies and similar tracking technologies to track the activity on Our Service and store certain information. Tracking technologies used are beacons, tags, and scripts to collect and track information and to improve and analyze Our Service. The technologies We use may include:

- __Cookies or Browser Cookies.__ A cookie is a small file placed on Your Device. You can instruct Your browser to refuse all Cookies or to indicate when a Cookie is being sent. However, if You do not accept Cookies, You may not be able to use some parts of our Service. Unless you have adjusted Your browser setting so that it will refuse Cookies, our Service may use Cookies.
- __Web Beacons.__ Certain sections of our Service and our emails may contain small electronic files known as web beacons (also referred to as clear gifs, pixel tags, and single-pixel gifs) that permit the Company, for example, to count users who have visited those pages or opened an email and for other related website statistics (for example, recording the popularity of a certain section and verifying system and server integrity).

Cookies can be "Persistent" or "Session" Cookies. Persistent Cookies remain on Your personal computer or mobile device when You go offline, while Session Cookies are deleted as soon as You close Your web browser. Learn more about cookies on the [Free Privacy Policy website](https://www.freeprivacypolicy.com/blog/sample-privacy-policy-template/#Use_Of_Cookies_And_Tracking) article.

We use both Session and Persistent Cookies for the purposes set out below:

- __Necessary / Essential Cookies__

	Type: Session Cookies

	Administered by: Us

	Purpose: These Cookies are essential to provide You with services available through the Website and to enable You to use some of its features. They help to authenticate users and prevent fraudulent use of user accounts. Without these Cookies, the services that You have asked for cannot be provided, and We only use these Cookies to provide You with those services.
- __Cookies Policy / Notice Acceptance Cookies__

	Type: Persistent Cookies

	Administered by: Us

	Purpose: These Cookies identify if users have accepted the use of cookies on the Website.
- __Functionality Cookies__

	Type: Persistent Cookies

	Administered by: Us

	Purpose: These Cookies allow us to remember choices You make when You use the Website, such as remembering your login details or language preference. The purpose of these Cookies is to provide You with a more personal experience and to avoid You having to re-enter your preferences every time You use the Website.



For more information about the cookies we use and your choices regarding cookies, please visit our Cookies Policy or the Cookies section of our Privacy Policy.


## Use of Your Personal Data

The Company may use Personal Data for the following purposes:

- __To provide and maintain our Service__, including to monitor the usage of our Service.
- __To manage Your Account:__ to manage Your registration as a user of the Service. The Personal Data You provide can give You access to different functionalities of the Service that are available to You as a registered user.
- __For the performance of a contract:__ the development, compliance and undertaking of the purchase contract for the products, items or services You have purchased or of any other contract with Us through the Service.
- __To contact You:__ To contact You by email, telephone calls, SMS, or other equivalent forms of electronic communication, such as a mobile application's push notifications regarding updates or informative communications related to the functionalities, products or contracted services, including the security updates, when necessary or reasonable for their implementation.
- __To provide You__ with news, special offers and general information about other goods, services and events which we offer that are similar to those that you have already purchased or enquired about unless You have opted not to receive such information.
- __To manage Your requests:__ To attend and manage Your requests to Us.

- __For business transfers:__ We may use Your information to evaluate or conduct a merger, divestiture, restructuring, reorganization, dissolution, or other sale or transfer of some or all of Our assets, whether as a going concern or as part of bankruptcy, liquidation, or similar proceeding, in which Personal Data held by Us about our Service users is among the assets transferred.
- __For other purposes__: We may use Your information for other purposes, such as data analysis, identifying usage trends, determining the effectiveness of our promotional campaigns and to evaluate and improve our Service, products, services, marketing and your experience. 

We may share Your personal information in the following situations:

- __With Service Providers:__ We may share Your personal information with Service Providers to monitor and analyze the use of our Service,  to contact You.
- __For business transfers:__ We may share or transfer Your personal information in connection with, or during negotiations of, any merger, sale of Company assets, financing, or acquisition of all or a portion of Our business to another company.
- __With Affiliates:__ We may share Your information with Our affiliates, in which case we will require those affiliates to honor this Privacy Policy. Affiliates include Our parent company and any other subsidiaries, joint venture partners or other companies that We control or that are under common control with Us.
- __With business partners:__ We may share Your information with Our business partners to offer You certain products, services or promotions.
- __With other users:__ when You share personal information or otherwise interact in the public areas with other users, such information may be viewed by all users and may be publicly distributed outside. 
- __With Your consent__: We may disclose Your personal information for any other purpose with Your consent.

## Retention of Your Personal Data

The Company will retain Your Personal Data only for as long as is necessary for the purposes set out in this Privacy Policy. We will retain and use Your Personal Data to the extent necessary to comply with our legal obligations (for example, if we are required to retain your data to comply with applicable laws), resolve disputes, and enforce our legal agreements and policies.

The Company will also retain Usage Data for internal analysis purposes. Usage Data is generally retained for a shorter period of time, except when this data is used to strengthen the security or to improve the functionality of Our Service, or We are legally obligated to retain this data for longer time periods.

## Transfer of Your Personal Data

Your information, including Personal Data, is processed at the Company's operating offices and in any other places where the parties involved in the processing are located. It means that this information may be transferred to — and maintained on — computers located outside of Your state, province, country or other governmental jurisdiction where the data protection laws may differ than those from Your jurisdiction.

Your consent to this Privacy Policy followed by Your submission of such information represents Your agreement to that transfer.

The Company will take all steps reasonably necessary to ensure that Your data is treated securely and in accordance with this Privacy Policy and no transfer of Your Personal Data will take place to an organization or a country unless there are adequate controls in place including the security of Your data and other personal information.

## Delete Your Personal Data

You have the right to delete or request that We assist in deleting the Personal Data that We have collected about You.

Our Service may give You the ability to delete certain information about You from within the Service.

You may update, amend, or delete Your information at any time by signing in to Your Account, if you have one, and visiting the account settings section that allows you to manage Your personal information. You may also contact Us to request access to, correct, or delete any personal information that You have provided to Us.

Please note, however, that We may need to retain certain information when we have a legal obligation or lawful basis to do so.

## Disclosure of Your Personal Data

### Business Transactions

If the Company is involved in a merger, acquisition or asset sale, Your Personal Data may be transferred. We will provide notice before Your Personal Data is transferred and becomes subject to a different Privacy Policy.

### Law enforcement

Under certain circumstances, the Company may be required to disclose Your Personal Data if required to do so by law or in response to valid requests by public authorities (e.g. a court or a government agency).

### Other legal requirements

The Company may disclose Your Personal Data in the good faith belief that such action is necessary to:

- Comply with a legal obligation
- Protect and defend the rights or property of the Company
- Prevent or investigate possible wrongdoing in connection with the Service
- Protect the personal safety of Users of the Service or the public
- Protect against legal liability

## Security of Your Personal Data

The security of Your Personal Data is important to Us, but remember that no method of transmission over the Internet, or method of electronic storage is 100% secure. While We strive to use commercially acceptable means to protect Your Personal Data, We cannot guarantee its absolute security.


# Children's Privacy

Our Service does not address anyone under the age of 13. We do not knowingly collect personally identifiable information from anyone under the age of 13. If You are a parent or guardian and You are aware that Your child has provided Us with Personal Data, please contact Us. If We become aware that We have collected Personal Data from anyone under the age of 13 without verification of parental consent, We take steps to remove that information from Our servers.

If We need to rely on consent as a legal basis for processing Your information and Your country requires consent from a parent, We may require Your parent's consent before We collect and use that information.


# Links to Other Websites

Our Service may contain links to other websites that are not operated by Us. If You click on a third party link, You will be directed to that third party's site. We strongly advise You to review the Privacy Policy of every site You visit.

We have no control over and assume no responsibility for the content, privacy policies or practices of any third party sites or services.

# Changes to this Privacy Policy

We may update Our Privacy Policy from time to time. We will notify You of any changes by posting the new Privacy Policy on this page.

We will let You know via email and/or a prominent notice on Our Service, prior to the change becoming effective and update the "Last updated" date at the top of this Privacy Policy.

You are advised to review this Privacy Policy periodically for any changes. Changes to this Privacy Policy are effective when they are posted on this page.

# Contact Us

If you have any questions about this Privacy Policy, You can contact us:


- By email: sivalabs.in@gmail.com
# SivaLabs Weekly Newsletter - 2023, Week 29


## What I Learned This Week

* If you are thinking of what is a better way to structure your code, then I highly recommend watching
  [Facts and Myths about CQRS • Oskar Dudycz • Devoxx Poland 2023](https://www.youtube.com/watch?v=9COWKz1E32w) talk.
* I have updated my blog to make it easy to read posts and optimized CSS/JS to make it a bit faster.
  Along the way, I learned some new features of **Hugo** such as including **Table of Contents**, **opening external links in a new tab**, etc.
* **Spring Boot Tutorials for the Impatient:** I have planned to write concise and to the point tutorials on the latest Spring Boot version.
  I have published [Getting Started with Spring Boot](https://www.sivalabs.in/getting-started-with-spring-boot/) article, 
  which is the first tutorial in the series. So, stay tuned!!
* Published [Testing REST API integrations using WireMock and Testcontainers](https://testcontainers.com/guides/testing-rest-api-integrations-using-wiremock/) guide.
* I have published [Creating a Blog using Hugo, Asciidoc, and Netlify](https://www.sivalabs.in/creating-blog-using-hugo-asciidoc-netlify/) article
  which you can follow to create your blog with **ABSOLUTELY ZERO COST**.
* **AtomicJar** team announced [Testcontainers desktop app](https://testcontainers.com/desktop/) to make local development simpler using [Testcontainers](https://testcontainers.com/). Check it out.

## A happy moment
One of my [YouTube channel](https://www.youtube.com/c/sivalabs) subscribers pinged me and said he was looking for Job change, and my [Top 5 Tips to succeed in Technical Interviews](https://www.youtube.com/watch?v=xNf512h5bnU) helped him a lot.
That made me feel the time and effort of making videos is worth it :smile:

## Tweet of the week

{{< x user="asaikali" id="1682210450820218882" >}}

Keep learning. Cheers.

# Getting Started with Spring Boot


## Introducing Spring Boot
Spring Boot is the most popular framework for building applications in the Java world. 
Spring Boot is an Opinionated and Convention Over Configuration based approach to 
building Spring framework based applications.

Using Spring Boot, you can build different kinds of applications such as 
monolithic applications, microservices, serverless applications, Batch applications, etc.

<!--more-->


Let us take a quick look at what are the key features of Spring Boot that make it so popular.

## Spring Boot Key Features

### AutoConfiguration
Spring Boot takes an opinionated view of the application and auto-configures the components(a.k.a beans)
based on the default conventions without requiring you to configure everything explicitly.
However, you can customize or override the bean configurations in various ways if required.

For example, if you add **spring-boot-starter-data-jpa** dependency it will add **Hibernate** as the JPA implementation
as it is the most commonly used JPA provider. Also, Spring Boot tries to auto-configure the components required
for using Spring Data Jpa such as **DataSource, EntityManagerFactory, PlatformTransactionManager** etc.
If there is an in-memory JDBC driver in the classpath like **H2** or **HSQL** then Spring Boot will auto-configure
the **DataSource** with in-memory settings.

If you want to use any non in-memory databases like **MySQL**, **Postgresql** etc then you can add the respective JDBC driver jar
and configure the JDBC connection parameters in **application.properties** file. Then Spring Boot will use those properties
to configure DataSource bean as opposed to using default in-memory database.
You can even configure a DataSource bean by yourself using **@Bean** annotation then Spring Boot will backoff
and use the DataSource bean configured by you instead of auto-configuring it.

With Spring Boot's auto-configuration, the need for Spring application configuration is greatly reduced.

### Convention Over Configuration
Spring Boot relies upon various default conventions so that it can auto-configure the application.

For example, in most of the Spring based (not Spring Boot based) applications we use configuration properties
and register the **PropertySourcesPlaceholderConfigurer** bean. In Spring Boot you can put your configuration
properties in **src/main/resources/application.properties** file and Spring Boot will automatically register
**PropertySourcesPlaceholderConfigurer** bean loading the properties from that file. You don't have to
explicitly specify the properties file name.

Similarly, you might want to configure different values for properties based on environment (**dev, qa, staging, prod**).
You can configure your configuration properties in **application-{profile}.properties** where profile can be
**dev, qa, staging** and **prod**. Then you just need to enable the desired profile for Spring Boot to read values
from that specific profile properties file.

Spring Boot is very flexible to customize those conventions based on your project or team preferences.

### Dependencies Version Management

Typically, Spring Boot applications inherit from **spring-boot-starter-parent** which is configured with all
the compatible library versions so that you don't have to hunt for checking which library version is compatible
with which version of Spring. You can check **pom.xml** of **org.springframework.boot:spring-boot-dependencies** module to see
what are all the libraries pre-configured.

### Production Ready Monitoring Capabilities

Monitoring is an important aspect of any application running in production. Spring Boot provides production ready
monitoring capabilities via **Actuator**. You can get the application runtime information like **Memory usage, Disk Space, HealthCheck
of various components** etc via Actuator REST endpoints. Actuator uses **Micrometer** under-the-hood which you can use
to export all those application metrics to various monitoring services like **Prometheus, Datadog, Influx** etc.

### Embedded Servers Support

Traditionally, Java based web applications are built as **war** files and then deployed on a servlet containers or
application servers like **Tomcat, Wildfly, WebSphere** etc. The modern approach is to embed the server runtime within
the application itself so that you can take the artifact and run it without having to install and configure server externally.

Spring Boot provides support for embedding servlet containers like **Tomcat, Jetty, Undertow** and you can customize
various server properties in server-independent way. Spring Boot also provides various server-specific
customization properties as well.

### Spring Ecosystem

Spring Boot has a huge ecosystem of projects to support wide variety of application types.

* **SpringMVC & Spring WebFlux**: You can build traditional web applications and REST APIs using **SpringMVC** or **Spring WebFlux**.
* **Spring Data**: Spring Data provides high-level abstraction over ORM and NoSQL data access libraries so that you don't need to implement boilerplate CRUD operations again and again.
* **Spring Security**: You can implement Authentication and Authorization using Spring Security. It also supports implementing **OAuth 2.0** based security.
* **Spring Batch**: You can build robust batch applications using SpringBatch.
* **Spring Integration**: Spring Integration implements many of the **Enterprise Integration Patterns** which you can use for integration with 3rd party services.
* **Spring Cloud**: Spring Cloud provides support for building Cloud Native applications following [12 Factor Application](https://12factor.net/) principles
* **Spring Cloud Streams**: You can build Data Pipelines for processing Stream based Data Sources like Kafka.

And, there are many more interesting projects. See https://spring.io/projects for more Spring ecosystem projects.

Let's build a simple REST API application using Spring Boot and explore some of its features.

## Creating a Spring Boot Application

Spring Boot applications can be created either using [Spring Initializr](https://start.spring.io/) or
[IntelliJ IDEA Ultimate](https://www.jetbrains.com/idea/) or [Spring Tool Suite](https://spring.io/tools)
or [NetBeans IDE](https://netbeans.apache.org/) with [Spring Boot Plugin](https://github.com/AlexFalappa/nb-springboot).

{{< box tip >}}
You may be interested in [Better Alternatives To Generate Spring Boot Applications](https://www.youtube.com/watch?v=2E8r-HVLZGY)
{{< /box >}}

We are going to use [Spring Initializr](https://start.spring.io/) and provide the following project metadata.

* **Project**: Maven Project
* **Language**: Java
* **Spring Boot**: 3.1.2
* **Project Metadata**:
  * **Group**: com.sivalabs
  * **Artifact**: spring-boot-helloworld
  * **Name**: spring-boot-helloworld
  * **Description**: Spring Boot HelloWorld
  * **Package name**: com.sivalabs.helloworld
  * **Packaging**: jar
  * **Java**: 17
* Click on **ADD DEPENDENCIES** and add **Spring Web**, **Lombok** starters.
* Click on **GENERATE**

The generated Spring Boot application will be downloaded.
You can extract the zip file and import the project into your IDE.
I will be using IntelliJ IDEA, but you can use any of your favorite IDE.

{{< box info >}}
**Spring Boot Tutorial Series GitHub Code Repository:**
https://github.com/sivaprasadreddy/spring-boot-tutorials-blog-series
{{< /box >}}

## Simple REST API using Spring Boot

What we are going to build is a simple REST API with one endpoint **GET /api/hello?name={name}** which returns the JSON response
**{ "greeting" : "Hello {name}"}**. We don't want to hard-code the greeting prefix **"Hello"**, we want it to be configurable.

Let us start with creating response model class **GreetingResponse** as follows:

**src/main/java/com/sivalabs/helloworld/GreetingResponse.java**
```java
record GreetingResponse(String greeting){}
```

Next, create **ApplicationProperties** class to bind the application configuration parameters.

**src/main/java/com/sivalabs/helloworld/ApplicationProperties.java**
```java
import lombok.Getter;
import lombok.Setter;
import org.springframework.boot.context.properties.ConfigurationProperties;

@ConfigurationProperties(prefix = "app") // (1)
@Setter
@Getter
public class ApplicationProperties {
    private String greeting = "Hello";
    private String defaultName = "World";
}
```

* **(1)** We are binding the properties in **application.properties** file with common prefix **"app."** into **ApplicationProperties** class fields.

Create **GreetingService** and implement **sayHello(String name)** method as follows:

**src/main/java/com/sivalabs/helloworld/GreetingService.java**
```java
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Service;

@Service // (1)
@RequiredArgsConstructor // (2)
public class GreetingService {
    private final ApplicationProperties properties;  // (3)

    public String sayHello(String name) {
        String s = name == null ? properties.getDefaultName(): name;
        return String.format("%s %s", properties.getGreeting(), s);
    }
}
```

* **(1)** Declare **GreetingService** as Spring bean using **@Service** annotation.
* **(2)** Lombok's **@RequiredArgsConstructor** annotation will generate a constructor with all the **final** properties. In this case, it will generate a constructor with **ApplicationProperties** argument.
* **(3)** Injecting the **ApplicationProperties** instance as a dependency of **GreetingService** bean.

Create **HelloWorldController** and implement the **GET /api/hello** API endpoint as follows:

**src/main/java/com/sivalabs/helloworld/HelloWorldController.java**
```java
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@RestController // (1)
@RequiredArgsConstructor
@Slf4j // (2)
public class HelloWorldController {
    private final GreetingService greetingService;

    @GetMapping("/api/hello") // (3)
    public GreetingResponse sayHello(
            @RequestParam(name = "name", required = false) String name) {
        log.info("Say Hello to Name: {}", name);
        String greeting = greetingService.sayHello(name);
        return new GreetingResponse(greeting);
    }
}
```

* **(1)** Declares the class as Spring Controller with request handler methods.
* **(2)** Using Lombok's **@Slf4j** annotation to automatically create a SLF4J Logger instance instead of manually creating as **private static final Logger log = LoggerFactory.getLogger(HelloWorldController.class);**
* **(3)** The **sayHello()** method is annotated with **@GetMapping("/api/hello")** indicating it as a request handler method for **HTTP GET /api/hello** URL.

Let's configure the properties in **src/main/resources/application.properties** as follows:

```shell
app.greeting=Hello
app.default-name=World
```

Finally, we need to enable the configuration properties binding using **@EnableConfigurationProperties** as follows:

**src/main/java/com/sivalabs/helloworld/SpringBootHelloWorldApplication.java**
```java
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.context.properties.EnableConfigurationProperties;

@SpringBootApplication
@EnableConfigurationProperties({ApplicationProperties.class}) // (1)
//@ConfigurationPropertiesScan // (2)
public class SpringBootHelloWorldApplication {

    public static void main(String[] args) {
        SpringApplication.run(SpringBootHelloWorldApplication.class, args);
    }
}
```

* **(1)** Explicitly enabling the configuration properties binding for **ApplicationProperties** class. If we have more such configuration-binding classes, we can list them all.
* **(2)** Instead of explicitly specifying all the configuration-binding classes, we can use **@ConfigurationPropertiesScan** annotation to scan for all the classes that are annotated with **@ConfigurationProperties**.

We can run the application from the IDE by simply running the **main()** method in **Spring BootHelloWorldApplication**.

## Run application using Maven and Gradle
Spring Boot Maven and Gradle plugins provide the ability to run the application without requiring to build the artifact(jar or war).

Maven:

```shell
./mvnw spring-boot:run
```

Gradle:

```shell
./gradlew bootRun
```

## Run application as a FatJar
We can build the Spring Boot application as a fat-jar and run it using **java -jar** command.

**Maven:**

```shell
$ ./mvnw clean package
$ java -jar target/spring-boot-helloworld-0.0.1-SNAPSHOT.jar
```

**Gradle:**

```shell
$ ./gradlew clean build
$ java -jar build/libs/spring-boot-helloworld-0.0.1-SNAPSHOT.jar
```

We can verify the API endpoint using cURL as follows:

```shell
$ curl http://localhost:8080/api/hello
{"greeting":"Hello World"}

$ curl http://localhost:8080/api/hello?name=Siva
{"greeting":"Hello Siva"}
```

{{< box info >}}
**Spring Boot Tutorials**

You can find more Spring Boot tutorials on [Spring Boot Tutorials]({{% relref "/pages/spring-boot-tutorials" %}}) page.
{{< /box >}}

## Summary
We have learned about Spring Boot, its ecosystem and some of its core features.
We have created a simple REST API using Spring Boot and tested using cURL.

# Creating a Blog using Hugo, Asciidoc, and Netlify


In this tutorial, I will explain the process for setting up a blog 
using the static site generator [Hugo](https://gohugo.io/), writing content using **Markdown** 
and [Asciidoctor](https://docs.asciidoctor.org/), 
and deploy on [Netlify](https://www.netlify.com/).

<!--more-->


## Why should you have a blog?
For a software developer, having a well-maintained blog opens up a lot of opportunities.
First of all, you can write down your learnings and keep it as a quick reference for yourself.
But, having a blog with quality content brings lots of opportunities, such as:

* While writing a post on a topic, you tend to explore more to write accurate content, 
  which immensely helps you get more clarity on the subject.
* Well-known book publishers usually approach good bloggers to write books with them.
  If you are looking to write a book with an established publisher, this is a gateway.
* Your blog helps to create a good social profile. During interviews, 
  interviewers tend to explore your social presence and having a blog with good content 
  may help to already have a positive impression on you. 
  To be a good developer, you don't need to have a blog though :wink:
* You can share your knowledge by writing it down once and simply share the link instead of repeating yourself.

## Choose a blogging platform
There are a wide range of blogging platforms to choose from such as **Blogger**, **WordPress**, 
**Medium**, **hashnode** etc.
And there are **Static Site Generators** (SSG) like [Hugo](https://gohugo.io/), 
[Jekyll](https://jekyllrb.com/), [Gatsby](https://www.gatsbyjs.com/), etc 
using which you can generate your blog as static content and host it on 
[GitHub Pages](https://pages.github.com/), [Netlify](https://www.netlify.com/), 
[Vercel](https://vercel.com/) etc.

Some of the reasons I prefer SSG and **Hugo** are:

* I like writing content in Markdown/Asciidoc, push it to Git repo workflow than using WYSIWYG editors.
* I want to own my content and ability to switch the hosting platform if needed.
* Hugo's installation is very simple and build speed is just amazing.

If you are good with frontend development, then tools like **Gatsby**, **Nextjs**, **VuePress** etc are also good choices.
But, I just prefer a working solution with minimal effort, and Hugo fits the bill perfectly.

In this post, I will explain step-by-step instructions on how you can create a blog and deploy it on Netlify.

{{< box info >}}
**Blog Code Repository:**
https://github.com/sivaprasadreddy/hugo-asciidoc-blog
{{< /box >}}

## Create a new blog using Hugo
Hugo has excellent [documentation](https://gohugo.io/documentation/), 
and we can simply follow the instructions to create a blog.

You can install hugo following your [OS specific instructions](https://gohugo.io/installation/).
In my case, I am using MacOS and I installed hugo using **brew**.

You can follow [QuickStart](https://gohugo.io/getting-started/quick-start/) instructions to create a blog.

```shell
$ brew install hugo
$ hugo new site hugo-blog
$ cd hugo-blog
```

You can choose any of the available [themes](https://themes.gohugo.io/).
We can add a theme to our blog either adding as a git submodule 
or simply downloading the theme as a zip, extract it and put it under **themes** folder.

Because of my prior experience with git submodules, I prefer to go with downloading the zip and add it manually :wink:

Let's use [Ananke](https://github.com/theNewDynamic/gohugo-theme-ananke) theme for our blog.
Download the repo as a Zip, extract and copy code into **hugo-blog/themes/ananke** folder.

{{< box info >}}
**config.toml vs hugo.toml**

As of now, when you create a new site, the configuration file is created with name as **config.toml**.
But going forward, the [preferred](https://gohugo.io/getting-started/configuration/#hugotoml-vs-configtoml) name is **hugo.toml**.
So rename **config.toml** in the root folder to **hugo.toml** if you are using version 0.110.0 or higher.
{{< /box >}}

Next, add **`theme = "ananke"`** to **config.toml** file. 
Remember the theme name should match with your theme folder name under **hugo-blog/themes/**.

Now we are ready to add content to our blog.

### Add content(posts) using Markdown
The default and most commonly used format for writing content is Markdown.
So, let's start with creating a post using Markdown.

```shell
hugo new posts/my-first-post.md
```

This will create the file in the **/content/posts** directory. 
Open the file with your editor and update the content as follows:

```markdown
---
title: "My First Post"
date: 2023-07-20T09:03:20-08:00
draft: false
url: /my-first-post
---

Your blog post introduction goes here.

## Heading 1
content

## Show me the code

{{</* highlight java */>}}
class Hello {
    String hello() {
        return "Hello"
    }
}
{{</* /highlight */>}}


## Summary
post summary here
```

**Run the blog:**

You can start the hugo blog locally as follows:

```shell
# -D for drafts, -F for future date posts
hugo server -D -F
```
This usually starts the blog on http://localhost:1313/. 

You should be able to see the blog 
and navigate to the first post and see the code snippet with syntax highlighting too.

## Deploy on Netlify
Deploying your Hugo based blog on Netlify is fairly simple.
Create your account on Netlify if you don't have one and setup a Team.

First, create **netlify.toml** file in the root folder with the following content.

```toml
[build]
    publish = "public"
    command = "hugo"

[build.environment]
    HUGO_VERSION = "0.110.0"

[context.production.environment]
    HUGO_ENV = "production"

[context.deploy-preview]
    command = "hugo -D -F -b $DEPLOY_PRIME_URL/"

[context.branch-deploy]
    command = "hugo -D -F -b $DEPLOY_PRIME_URL/"
```

Then you can follow the instructions below to deploy your blog on Netlify.

* Navigate to Sites -> Add new site -> Import an existing project -> Deploy with GitHub
* Configure the Netlify app on GitHub -> Select repository
* Deploy your_repo_name

This will deploy your site with a random generated name.

You can customize the name in **Site configuration** -> **Change site name**

Then your blog should be available on https://sitename.netlify.app

## Using Asciidoc instead of Markdown
Markdown is good enough for most cases. 
But if you need a more powerful format, or you just prefer Asciidoc over Markdown 
then you can use the [Asciidoc format also](https://gohugo.io/content-management/formats/#list-of-content-formats) for writing content.

You can install **asciidoctor** following your [OS-specific instructions](https://docs.asciidoctor.org/asciidoctor/latest/install/).
Once installed, you should be able to run the command **asciidoctor --version** and see the version. 

Once **asciidoctor** is installed on your system, you can simply write content using **.adoc** files using asciidoctor syntax.

Create **content/posts/my-second-post.adoc** file with the following content:

```shell
---
title: "This is my second post"
date: 2023-07-20T09:39:58+05:30
draft: false
categories:
  - Java
tags:
  - java
  - spring-boot
---
:source-highlighter: rouge
:rouge-css: style
:rouge-style: thankful_eyes

In this guide, you will learn how to write HelloWorld program in Java.

== Prerequisites
* Java 17+
* Your favorite IDE (Intellij IDEA, Eclipse, NetBeans, VS Code)

== Say Hello in Java

Here is how to write code snippet using Asciidoc format.

[source,java]
----
class Hello {
    String hello() {
        return "Hello";
    }
}
----

== Summary
To learn more interesting things, visit https://www.sivalabs.in
```

In order to enable the execution of asciidoctor by Hugo, you need to configure the following in **config.toml** file:

```toml
[security.exec]
    allow = ['^asciidoctor$']
```

Now if you start your blog locally, you should be able to see the second post with syntax highlighting working.

### Adding Admonitions support
I really like Asciidoctor [Admonitions](https://docs.asciidoctor.org/asciidoc/latest/blocks/admonitions/) support to highlight some content as **NOTE, TIP, IMPORTANT, CAUTION, WARNING**.
We can configure Asciidoctor to use [FontAwesome](https://fontawesome.com/) to show Admonition icons.

In order to do that, we need to:

* Add FontAwesome CSS
* Add admonition css

Asciidoctor by default uses a [default stylesheet](https://docs.asciidoctor.org/asciidoctor/latest/html-backend/default-stylesheet/) which contains CSS styling classes for admonitions.
We can extract admonition styles into **adoc.css** file as follows:

```css
.admonitionblock > table {
    border-collapse: separate;
    border: 0;
    background: none;
    width: 100%
}

.admonitionblock > table td.icon {
    text-align: center;
    width: 80px
}

.admonitionblock > table td.icon img {
    max-width: none
}

.admonitionblock > table td.icon .title {
    font-weight: bold;
    font-family: "Open Sans", "DejaVu Sans", sans-serif;
    text-transform: uppercase;
    font-size: 2.0em;
}

.admonitionblock > table td.content {
    padding-left: 1.125em;
    padding-right: 1.25em;
    border-left: 1px solid #dddddf;
    color: rgba(0, 0, 0, .6)
}

.admonitionblock > table td.content > :last-child > :last-child {
    margin-bottom: 0
}

.admonitionblock td.icon [class^="fa icon-"] {
    font-size: 2.5em;
    text-shadow: 1px 1px 2px rgba(0, 0, 0, .5);
    cursor: default
}

.admonitionblock td.icon .icon-note::before {
    content: "\f05a";
    color: #0066a1
}

.admonitionblock td.icon .icon-tip::before {
    content: "\f0eb";
    text-shadow: 1px 1px 2px rgba(155, 155, 0, .8);
    color: #111
}

.admonitionblock td.icon .icon-warning::before {
    content: "\f071";
    color: #bf6900
}

.admonitionblock td.icon .icon-caution::before {
    content: "\f06d";
    color: #bf3400
}

.admonitionblock td.icon .icon-important::before {
    content: "\f06a";
    color: #bf0000
}
```

For adding FontAwesome CSS, we can use CDN reference.

How to add additional CSS files to your blog depends on your theme.
You can directly add the **<link ..>** in **&lt;head&gt;** section in your theme partial files.
If your theme supports adding additional CSS files via configuration, you can use that approach.

In our case, **Ananke** theme [supports adding additional CSS files](https://github.com/theNewDynamic/gohugo-theme-ananke#custom-css) via configuration.

Let's put **adoc.css** file in **/assets/ananke/css/** folder and add the following configuration in **config.toml** file:

```toml
[params]
    custom_css = ["https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.0/css/all.min.css", "adoc.css"]
```

Now you can specify asciidoctor attribute **:icons: font** and use admonitions in **my-second-post.adoc** file as follows:

```shell
---
title: "This is my first post"
date: 2023-07-20T09:39:58+05:30
draft: false
---
:source-highlighter: rouge
:rouge-css: style
:rouge-style: thankful_eyes
:icons: font

[NOTE]
====
This is a NOTE
====

[CAUTION]
====
This is a CAUTION
====

[WARNING]
====
This is a WARNING
====

[TIP]
====
This is a TIP
====

[IMPORTANT]
====
This is IMPORTANT
====
```

Now you should be able to see the nice icons as follows:

{{< figure src="/images/asciidoc-admonitions.webp" alt="asciidoc-admonitions" >}}


Instead of repeating the asciidoctor attributes in every post, we can configure them in **config.toml** as follows:

```toml
[markup]
    [markup.asciidocExt]
        [markup.asciidocExt.attributes]
            allow-uri-read = ""
            source-highlighter = "rouge"
            rouge-css = "style"
            rouge-style = "thankful_eyes"
            imagesdir = "/images"
            icons = "font"
```

Now you can remove the following asciidoctor attributes from **my-second-post.adoc** file.

```shell
:source-highlighter: rouge
:rouge-css: style
:rouge-style: thankful_eyes
:icons: font
```

### Including code snippets from remote locations
In addition to including the code snippets using asciidoctor syntax, 
we can also include code from remote code repository.

To use this feature, we need to configure the following in **config.toml** file:

```toml
[security.http]
    methods = ['(?i)GET|POST']
    urls = ['.*']
```

You can limit the urls to allow only certain URL patterns for security reasons.

Now, we can include code snippet from a remote file as follows:

```shell
[source,java]
----
include::https://raw.githubusercontent.com/sivaprasadreddy/techbuzz/main/techbuzz/src/main/java/com/sivalabs/techbuzz/TechBuzzApplication.java[]
----
```

You should be able to see the code snippet from remote repository.

## Configure Asciidoc support on Netlify
In order to support asciidoctor on Netlify, we need to create **Gemfile** to 
install **asciidoctor** and **rouge** gems as follows:

```ruby
source "https://rubygems.org"

gem 'asciidoctor', '~> 2.0', '>= 2.0.18'
gem "rouge"
```

Now run the following commands to create **Gemfile.lock** file.

```shell
$ gem install bundler
$ bundle install
```
This will help in using same versions in both local and on Netlify.

Now commit and push all the changes to GitHub and Netlify will automatically rebuild your site and deploy the changes.

{{< box info >}}
**Resources:**

**Blog Code Repository:**
https://github.com/sivaprasadreddy/hugo-asciidoc-blog

**Live Site:**  https://hugo-asciidoc-blog.netlify.app/
{{< /box >}}

## Summary
We have learned how to set up a blog using Hugo, Markdown/Asciidoctor and deploy it on Netlify.
We also explored a few asciidoctor configurations that will help in writing rich content.

I hope this helps. Happy blogging :smile:
# Kubernetes Tutorials


### Kubernetes
* [Getting Started with Kubernetes]({{< relref "2021-09-01-getting-started-with-kubernetes.md" >}})
* [Kubernetes - Releasing a new version of the application using Deployment Rolling Updates]({{< relref "2021-09-07-kubernetes-deployment-rolling-updates.md" >}})
* [Kubernetes - Blue/Green Deployments]({{< relref "2021-09-14-kubernetes-blue-green-deployments.md" >}})
* [Kubernetes - Exposing Services to outside of Cluster using Ingress]({{< relref "2021-09-22-kubernetes-ingress.md" >}})

# Microservices Tutorials


### Microservices
* [Part 1 : MicroServices : Spring Boot & Spring Cloud Overview]({{< relref "2018-03-02-microservices-using-springboot-spring-cloud-part-1-overview.md" >}})
* [Part 2 : MicroServices : Configuration Management with Spring Cloud Config and Vault]({{< relref "2018-03-05-microservices-part-2-configuration-management-spring-cloud-config-vault.md" >}})
* [Part 3 : MicroServices : Spring Cloud Service Registry and Discovery]({{< relref "2018-03-08-microservices-springcloud-eureka.md" >}})
* [Part 4 : MicroServices : Spring Cloud Circuit Breaker using Netflix Hystrix]({{< relref "2018-03-12-spring-cloud-netflix-circuit-breaker.md" >}})
* [Part 5 : MicroServices : Spring Cloud Zuul Proxy as API Gateway]({{< relref "2018-03-15-microservices-part-5-spring-cloud-zuul-proxy-as-api-gateway.md" >}})
* [Part 6 : MicroServices : Distributed Tracing with Spring Cloud Sleuth and Zipkin]({{< relref "2018-03-20-microservices-part-6-distributed-tracing-with-spring-cloud-sleuth-and-zipkin.md" >}})

# MyBatis Tutorials


### MyBatis

* [MyBatis Tutorial: Part1 – CRUD Operations]({{< relref "2012-10-21-mybatis-tutorial-part1-crud-operations.md" >}})
* [MyBatis Tutorial: Part-2: CRUD operations Using Annotations]({{< relref "2012-10-21-mybatis-tutorial-part-2-crud-operations-using-annotations.md" >}})
* [MyBatis Tutorial: Part 3 – Mapping Relationships]({{< relref "2012-10-21-mybatis-tutorial-part-3-mapping-relationships.md" >}})
* [MyBatis Tutorial : Part4 – Spring Integration]({{< relref "2012-10-24-mybatis-tutorial-part4-spring-integration.md" >}})

# Spring Boot Tutorials


### Spring Boot

* [Getting Started with Spring Boot]({{< relref "01-getting-started-with-spring-boot.md" >}}) 
* [Spring Boot Testing Tutorial]({{< relref "02-spring-boot-testing-tutorial.md" >}}) 
* [Spring Boot Application Configuration Tutorial]({{< relref "03-spring-boot-application-configuration-tutorial.md" >}})
* [Spring Boot Logging Tutorial]({{< relref "04-spring-boot-logging-tutorial.md" >}})
* [Spring Boot Profiles Tutorial]({{< relref "05-spring-boot-profiles-tutorial.md" >}})
* [Spring Boot JdbcTemplate Tutorial]({{< relref "06-spring-boot-jdbctemplate-tutorial.md" >}})
* [Spring Boot Database Transaction Management Tutorial]({{< relref "07-spring-boot-database-transaction-management-tutorial.md" >}})
* [Spring Boot Flyway Database Migration Tutorial]({{< relref "08-spring-boot-flyway-database-migration-tutorial.md" >}})
* [Spring Boot REST API Best Practices - Part 1 : Implementing Get Collection API]({{< relref "09-spring-boot-rest-api-tutorial-part-1.md" >}})
* [Spring Boot REST API Best Practices - Part 2 : Implementing Create and Update APIs]({{< relref "10-spring-boot-rest-api-tutorial-part-2.md" >}})
* [Spring Boot REST API Best Practices - Part 3 : Implementing FindById and DeleteById APIs]({{< relref "11-spring-boot-rest-api-tutorial-part-3.md" >}})
* [Spring Boot REST API Best Practices - Part 4 : Exception Handling in REST APIs]({{< relref "12-spring-boot-rest-api-tutorial-part-4.md" >}})
* [Why Spring Boot is so popular and how to learn Spring Boot effectively?]({{< relref "2018-02-22-why-springboot-so-popular-how-to-learn-springboot.md" >}})
* [Why Spring Boot?]({{< relref "2016-03-13-why-springboot.md" >}})
* [How Spring Boot AutoConfiguration magic works?]({{< relref "2016-03-13-how-springboot-autoconfiguration-magic.md" >}})
* [Getting Started with Spring Boot in Intellij IDEA Community Edition]({{< relref "2016-09-07-getting-started-springboot-intellij-idea-community-edition.md" >}})
* [Spring Boot : Working with MyBatis]({{< relref "2016-03-14-springboot-working-with-mybatis.md" >}})
* [Creating Custom Spring Boot Starter for Twitter4j]({{< relref "2016-04-08-creating-custom-springboot-starter-for.md" >}})
* [Spring Boot Messaging with RabbitMQ]({{< relref "2018-02-20-springboot-messaging-rabbitmq.md" >}})
* [Session Management using Spring Session with JDBC DataStore]({{< relref "2018-02-07-session-management-using-spring-session-jdbc-datastore.md" >}})
* [CI/CD for Spring Boot applications using Travis-CI]({{< relref "2018-01-24-ci-cd-springboot-applications-using-travis-ci.md" >}})
* [Spring Boot Best Practices]({{< relref "2019-05-14-spring-boot-best-practices.md" >}})
* [Testing Spring Boot Applications]({{< relref "2019-10-07-spring-boot-testing.md" >}})
* [Spring Boot Integration Testing using TestContainers Starter]({{< relref "2020-02-03-spring-boot-integration-testing-using-testcontainers-starter.md" >}})
* [Remote debugging Spring Boot application]({{< relref "2020-05-31-remote-debugging-spring-boot-application.md" >}})
* [Retrying Method Execution using Spring AOP]({{< relref "2016-01-05-retrying-method-execution-using-spring-aop.md" >}})

### Spring Security OAuth 2.0
* [Part 1: Getting familiar with OAuth 2 concepts]({{< relref "2023-09-22-spring-security-oauth2-tutorial-part1.md" >}})
* [Part 2: Authorization Code Flow]({{< relref "2023-09-24-spring-security-oauth2-tutorial-part2.md" >}})
* [Part 3: Client Credentials Flow]({{< relref "2023-09-25-spring-security-oauth2-tutorial-part3.md" >}})
* [Part 4: Authorization Code Flow with PKCE]({{< relref "2023-09-27-spring-security-oauth2-tutorial-part4.md" >}})
* [Part 5: Implicit & Resource Owner Password Credentials Flows]({{< relref "2023-09-27-spring-security-oauth2-tutorial-part5.md" >}})
* [Part 6: Microservices Sample Project Setup]({{< relref "2023-09-28-spring-security-oauth2-tutorial-part6.md" >}})
* [Part 7: Securing Spring MVC Client Application]({{< relref "2023-09-28-spring-security-oauth2-tutorial-part7.md" >}})
* [Part 8: Securing Resource Server]({{< relref "2023-10-02-spring-security-oauth2-tutorial-part8.md" >}})
* [Part 9: Invoking Secured Resource Server APIs from Client Application]({{< relref "2023-10-06-spring-security-oauth2-tutorial-part9.md" >}})
* [Part 10: Service to Service Communication using Client Credentials Flow]({{< relref "2023-10-09-spring-security-oauth2-tutorial-part10.md" >}})

### Spring Cloud

* [Introduction to Spring Cloud Config Server]({{< relref "2017-08-14-spring-cloud-tutorials-introduction-to-spring-cloud-config-server.md" >}}) 
* [Auto Refresh Config Changes using Spring Cloud Bus]({{< relref "2017-08-14-spring-cloud-tutorials-auto-refresh-config-changes-using-spring-cloud-bus.md" >}}) 

# SivaLabs Weekly Newsletter - 2023, Week 28


## What I Learned This Week

* I have published [Tomato Architecture - A Pragmatic Approach to Software Design](https://sivalabs.in/tomato-architecture-pragmatic-approach-to-software-design/) article 
  sharing my thoughts around how to build an application without over-engineering it.
* I have shared [My Thoughts on DevOps, Platform Engineering, and FullStack Developer](https://sivalabs.in/my-thoughts-on-devops-platform-engineering-and-fullstack-developer/) 
  talking about the challenges of being a Fullstack Developer.
* Published a video on [Testing AWS Service Integrations using LocalStack and Testcontainers](https://www.youtube.com/watch?v=OPQEjpX-yio)
* [Tom Hombergs](https://twitter.com/TomHombergs) published the 2nd Edition of 
  [Get Your Hands Dirty on Clean Architecture: Build 'clean' applications with code examples in Java](https://www.amazon.in/Your-Hands-Dirty-Clean-Architecture/dp/180512837X/) book
  which I highly recommend to read.
* If you are working on Go projects, then the [Air](https://github.com/cosmtrek/air) tool comes really handy to have Live Reload upon code changes.
  You can see an example [application](https://github.com/sivaprasadreddy/bookmarks-go).
* In recent years, a common approach being followed is building backend API and frontend as SPA separately.
  In some cases, though they are built as separate applications, they are bundled as single deployable 
  using [frontend-maven-plugin](https://github.com/eirslett/frontend-maven-plugin).
  Then I came across [Quinoa - A Quarkus extension to create Modern UI with no hassle](https://quarkus.io/blog/quinoa-modern-ui-with-no-hassle/).
  Looks very interesting to me.
* Spring 6.1 is introducing **RestClient** as an alternative to **RestTemplate** which provides the fluent API similar to Reactive **WebClient**.
  Read this [announcement](https://spring.io/blog/2023/07/13/new-in-spring-6-1-restclient) for more info.

Keep learning. Cheers.

# My Thoughts on DevOps, Platform Engineering, and FullStack Developer


In the software development world, once in a while a new ideology or process or technology 
will be introduced. And, the whole developer community goes crazy on that shiny new stuff.
A decade ago it was Agile/Scrum, then DevOps, then SRE and now Platform Engineering.

Somehow, I got curious about what is all the fuss about Platform engineering and read a bit about it, 
and I felt **"Oh, people realized that DevOps is a good idea but impractical, and they are correcting it now"**.

<!--more-->


Though these ideologies/processes like Agile, DevOps, etc are created with good intentions, 
somehow they are misunderstood and applied wrongly and end up with not so good results.

For example, take Agile. It was supposed to be about **interactions over processes**, 
**Customer collaboration** etc. 
But the unfortunate ground reality about Agile/Scrum is all the rituals like **daily-standup, 
backlog grooming, sprint-planning, sprint-review, retros** and other gazillion meetings.

Same with DevOps. Every single DevOps book and conference talk starts with DevOps is all about culture shift, 
not about Jenkins, CI/CD and Kubernetes. And yet we have DevOps Engineer job openings in every organization.

FullStack Developer is another interesting thing. Now most of the people mention that 
they are fullstack developers, which is great. Even before this term became famous, 
people used to do fullstack development.

When I started my career in 2006, I used to do both front-end and back-end except CSS.
There was a web designer who creates the skeleton with all the nice looking CSS,
and we took the templates and build both back-end and front-end.

But back then, things were very different. To be a "fullstack developer" 
you just need to know one backend programming language, HTML, JavaScript, CSS and jQuery.
We are good to go.

But now things are very different. JavaScript world is exploded with a billion new features, libraries, 
SPA frameworks, NodeJS, npm/yarn/pnpm, ReactJS and it's 278 close cousins, NextJS and NuxtJS etc etc.

I didn't feel learning some of these new things that difficult, because I didn't have to learn them all at once.
I learned one by one over the years, so it wasn't that difficult.

But imagine if someone who is just starting their career and wants to be a FullStack Developer. 
The number of things they need to be familiar with is very scary.

Again, the theoretical idea of "FullStack Developer" is "who can build a feature end-to-end 
with desire to learn on a need-basis."
In theory, it is great, but in reality you have "jack of all, master of none" and your application code reflects the same.

Maybe this is ok for many organizations and for certain parts of the applications.
I mean, maybe it's ok if your UI created by backend developer is not pixel perfect.
But, for some applications, maybe implementing UI with high-performance is very critical because 
most of their users use the application from mobile devices.
In my experience, I have seen only a handful of people are actually good at both front-end and back-end development.

Now people seem to realize that "DevOps" is putting too much burden of both
building the application and managing the infrastructure on the development team.
Platform Engineering is to solve that problem where a team of people who are good at 
Infrastructure Management will build the infrastructure and Development team focus on 
building the application.

I believe letting people do what they do best in a collaborative manner is better than 
having "jack of all, master of none" fullstack developers building a mediocre application.

Just imagining, in a true Agile team:
* Have a Platform Engineering team who provide the necessary infrastructure for development teams
* Have backend developers who are really good at backend development
* Have frontend developers who are really good at frontend development
* Have a DevOps developer who coordinates with the PE team to do what is necessary to put the application in production

It's just that you need to have (backend + frontend + DevOps developer) to be part of the same team.
They need to work collaboratively instead of in isolation.

Isn't that what Agile 1st principle is: **Individuals and interactions over processes and tools**.

Maybe in the near future, we may realize having **developers with specialization working collaboratively** 
is better than having **developers who know something about everything but nothing in-depth**. 
Who knows :-)


# Tomato Architecture - A Pragmatic Approach to Software Design


After a couple of years into software development, I wanted to improve my skills by learning more about Software Architecture and Design. 

<!--more-->


As you might guess, various sources suggested learning:
* [Design Patterns](https://refactoring.guru/design-patterns)
* [Clean Architecture](https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html)
* [Onion Architecture](https://jeffreypalermo.wpcomstaging.com/2008/07/the-onion-architecture-part-1/)
* [Hexagonal Architecture](https://alistair.cockburn.us/hexagonal-architecture/)
* [Ports & Adapters Architecture](http://wiki.c2.com/?PortsAndAdaptersArchitecture)
* [DDD (Domain Driven Design)](https://martinfowler.com/bliki/DomainDrivenDesign.html)

I have read various books and blog posts to learn these concepts, 
and I feel **Clean/Onion/Hexagonal/Ports&Adapters Architectures are very similar 
with a common goal of "protecting the core domain logic" from "external dependencies"**.

## The challenges in learning DDD, Clean/Onion/Hexagonal Architectures
I started reading [DDD Blue book](https://www.amazon.com/exec/obidos/ASIN/0321125215/domainlanguag-20) and I couldn't go too far with too much abstract explanation.
Then I read [DDD Red book](https://www.amazon.com/Implementing-Domain-Driven-Design-Vaughn-Vernon/dp/0321834577) and it's easier read than Blue book, but still many concepts are not very clear to me.
Then I read [Domain Driven Design Quickly](https://www.infoq.com/minibooks/domain-driven-design-quickly/) book available on InfoQ.

I understand the need for using ubiquitous language and modeling the business logic where it belongs etc.
But overall, I was a bit confused wondering am I already following DDD???

In my Java/Spring applications, I have been following many concepts like **Entities, Aggregates, Value Objects, Repositories, Application Services** etc.
But some people were saying this is **Anemic Domain Model** with the **Transaction Script Pattern**, and it is bad.
Then, if I ask how to rewrite this following proper DDD, they won't give an answer. 
They tell you to read one of the 3 books mentioned above.

And, there is another group of people where if you ask how to apply DDD, then they suggest using
the whole pack of **CQRS**, **Event Driven Architecture**, **Event Sourcing**, etc.

One fine day, I realized that many people simply parrot what they read without understanding it.

> **Litmus Test:**
> Ask them to explain something without telling "Martin Fowler / Uncle Bob/ Eric Evans said so".

## Motivations for Tomato Architecture
Following are some of the motivations that make me think about coming up with 
**Pragmatic Guidelines for Software Architecture**.

### There is No Silver Bullet Architecture
Duh, tell me something we don't already know, you might say.

True, we all know that, and yet we try to apply the silver bullet solutions 
(Clean / Onion / Hexagonal / Ports&Adapters Architectures) wherever possible.

A SaaS Product might need to be designed with a loose coupling in mind so that you can offer a choice to your clients 
which database they want to use.

But when an enterprise application is solving their own specific problem, then trying to introduce an abstraction 
with a hope that some day in the future, they can switch to another database/message-broker 
is unnecessary and over-engineering in many cases.

### Wrong pursuit of Testability
I don't know whether it is one of core principe or a by-product, but many people say
**Clean/Onion/Hexagonal/Ports&Adapters Architectures enables us to test our application using Unit Tests without depending on 
external services like databases, FileSystem, Message Brokers, etc.**

Maybe this _"Testability using Unit Tests without depending on external services like databases, FileSystem, Message Brokers, etc"_ 
is a good thing for certain types of applications. But in my experience, most of the enterprise business applications 
heavily depend on various external services like databases, FileSystem, Message Brokers, etc. 

No matter how many 1,000s of unit tests you have, you will not have confidence in your software
without having an Integration Test suite which actually talks to those real external dependencies.

So, I would say introducing the abstractions/indirections so that you can "unit test" your application 
without using any external services may not be a bright idea.
Instead, I would prefer to remove those abstractions and write more integration tests 
to ensure the correctness of the application functionality.

### Simplicity and Readability Wins in the Long Run
I hope we all will at least agree on this. Typically, enterprise software systems run for a long time(decades).
Simple and readable code is much easier to maintain and enhance in the long run.

Usually, developers join the team and leave after a while and new members join.
In such an environment, keeping the architecture as simple and boring as possible is very beneficial than 
making the project a playground for the over-enthusiastic developers/architects to flourish their resumes.

## Introducing Tomato Architecture

{{< figure src="/images/tomato-arc-logo.webp" alt="tomato-architecture-logo" >}}

There is absolutely nothing new in the [Tomato Architecture](https://github.com/sivaprasadreddy/tomato-architecture) that I am going to describe now.
It is just a **few good coding practices and architectural decisions to keep things simple** 
and bring back some sanity to prevent over-engineering.

Again, "Tomato" in the "Tomato Architecture" doesn't mean anything. Just like "Hexagon" in "Hexagonal Architecture" doesn't mean anything.
Don't dwell too much on the name.

> **IMPORTANT:**
> I have mostly worked on building enterprise applications solving their specific business problems.
> The guidelines I am going to propose as part of Tomato Architecture are aimed at such applications.

### Key Principles
* Think what is best for your software over blindly following suggestions by popular people.
* Strive to keep things simple instead of over-engineering the solution by guessing the requirements for the next decade.
* Do R&D, pick a technology and embrace it instead of creating abstractions with replaceability in mind.
* Make sure your solution is working as a whole, not just individual units.

### Architecture Diagram

{{< figure src="/images/tomato-architecture.webp" alt="tomato-architecture" >}}

### Implementation Guidelines

#### 1. Package by feature
A common pattern to organize code into packages is by splitting based on technical layers such as controllers, services, repositories, etc.
If you are building a Microservice which is already focusing on a specific module or business capability, then this approach might be fine.

If you are building a monolith or modular-monolith then it is strongly recommended to first split by features instead of technical layers.

For more info read: https://phauer.com/2020/package-by-feature/

#### 2. Keep "Application Core" independent of delivery mechanism (Web, Scheduler Jobs, CLI)
The Application Core should expose APIs that can be invoked from a main() method.
In order to achieve that, the "Application Core" should not depend on its invocation context.
Which means the "Application Core" should not depend on any HTTP/Web layer libraries.
Similarly, if your Application Core is being used from Scheduled Jobs or CLI then
any Scheduling logic or CLI command execution logic should never leak into Application Core.

#### 3. Separate the business logic execution from input sources (Web Controllers, Message Listeners, Scheduled Jobs etc)

The input sources such as Web Controllers, Message Listeners, Scheduled Jobs, etc should be a thin layer extracting the data
from request and delegate the actual business logic execution to "Application Core".

**DON'T DO THIS**

```java
@RestController
class CustomerController {
    private final CustomerService customerService;
    
    @PostMapping("/api/customers")
    void createCustomer(@RequestBody Customer customer) {
       if(customerService.existsByEmail(customer.getEmail())) {
           throw new EmailAlreadyInUseException(customer.getEmail());
       }
       customer.setCreateAt(Instant.now());
       customerService.save(customer);
    }
}
```

**INSTEAD, DO THIS**

```java
@RestController
class CustomerController {
    private final CustomerService customerService;
    
    @PostMapping("/api/customers")
    void createCustomer(@RequestBody Customer customer) {
       customerService.save(customer);
    }
}

@Service
@Transactional
class CustomerService {
   private final CustomerRepository customerRepository;

   void save(Customer customer) {
      if(customerRepository.existsByEmail(customer.getEmail())) {
         throw new EmailAlreadyInUseException(customer.getEmail());
      }
      customer.setCreateAt(Instant.now());
      customerRepository.save(customer);
   }
}
```

With this approach, whether you try to create a Customer from a REST API call or from a CLI,
all the business logic is centralized in Application Core.

**DON'T DO THIS**

```java
@Component
class OrderProcessingJob {
    private final OrderService orderService;
    
    @Scheduled(cron="0 * * * * *")
    void run() {
       List<Order> orders = orderService.findPendingOrders();
       for(Order order : orders) {
           this.processOrder(order);
       }
    }
    
    private void processOrder(Order order) {
       ...
       ...
    }
}
```

**INSTEAD, DO THIS**

```java
@Component
class OrderProcessingJob {
   private final OrderService orderService;

   @Scheduled(cron="0 * * * * *")
   void run() {
      List<Order> orders = orderService.findPendingOrders();
      this.processOrders(orders);
   }
}

@Service
@Transactional
class OrderService {

   public void processOrders(List<Order> orders) {
       ...
       ...
   }
}
```

With this approach, you can decouple order processing logic from scheduler
and can test independently without triggering through Scheduler.

#### 4. Don't let the "External Service Integrations" influence the "Application Core" too much
From the Application Core we may talk to database, message brokers or 3rd party web services, etc.
Care must be taken such that business logic executors not heavily depend on External Service Integrations.

For example, assume you are using Spring Data JPA for persistence, and
from your **CustomerService** you would like fetch customers using pagination.

**DON'T DO THIS**

```java
@Service
@Transactional
class CustomerService {
   private final CustomerRepository customerRepository;

   PagedResult<Customer> getCustomers(Integer pageNo) {
      Pageable pageable = PageRequest.of(pageNo, PAGE_SIZE, Sort.of("name"));
      Page<Customer> cusomersPage = customerRepository.findAll(pageable);
      return convertToPagedResult(cusomersPage);
   }
}
```

**INSTEAD, DO THIS**

```java
@Service
@Transactional
class CustomerService {
   private final CustomerRepository customerRepository;

   PagedResult<Customer> getCustomers(Integer pageNo) {
      return customerRepository.findAll(pageNo);
   }
}

@Repository
class JpaCustomerRepository {

   PagedResult<Customer> findAll(Integer pageNo) {
      Pageable pageable = PageRequest.of(pageNo, PAGE_SIZE, Sort.of("name"));
      return ...;
   }
}
```

This way any persistence library changes will only affect repository layer only.

#### 5. Keep domain logic in domain objects
If you have domain object state change methods that affect only that object or a method to calculate something
from the state of the object, then those methods belong to that domain object.

**DON'T DO THIS**

```java

class Cart {
    List<LineItem> items;
}

@Service
@Transactional
class CartService {

   CartDTO getCart(UUID cartId) {
      Cart cart = cartRepository.getCart(cartId);
      BigDecimal cartTotal = this.calculateCartTotal(cart);
      ...
   }
   
   private BigDecimal calculateCartTotal(Cart cart) {
      ...
   }
}
```

**INSTEAD, DO THIS**

```java

class Cart {
    List<LineItem> items;

   public BigDecimal getTotal() {
      ...
   }
}

@Service
@Transactional
class CartService {

   CartDTO getCart(UUID cartId) {
      Cart cart = cartRepository.getCart(cartId);
      BigDecimal cartTotal = cart.getTotal();
      ...
   }
}
```

#### 6. No unnecessary interfaces
Don't create interfaces with the hope that someday we might add another implementation for this interface.
If that day ever comes, then with the powerful IDEs we have now it is just a matter of extracting the interface in a couple of keystrokes.

If the reason for creating an interface is for testing with Mock implementation,
we have mocking libraries like Mockito which is capable of mocking classes without implementing interfaces.

So, unless there is a good reason, don't create interfaces.

#### 7. Embrace the framework's power and flexibility
Usually, the libraries and frameworks are created to address the common requirements that are required for majority of the applications.
So, when you choose a library/framework to build your application faster, then you should embrace it.

Instead of leveraging the power and flexibility offered by the selected framework,
creating an indirection or abstraction on top of the selected framework with the hope
that someday you might switch the framework to a different one is usually a very bad idea.

For example, Spring Framework provides declarative support for handling database transactions, caching, method-level security etc.
Introducing our own similar annotations and re-implementing the same features support
by delegating the actual handling to the framework is unnecessary.

Instead, it's better to either directly use the framework's annotations or compose the annotation with additional semantics if needed.

```java
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Transactional
public @interface UseCase {
   @AliasFor(
        annotation = Transactional.class
   )
   Propagation propagation() default Propagation.REQUIRED;
}
```

#### 8. Test not only units, but whole features

We should definitely write unit tests to test the units(business logic), by mocking external dependencies if required.
But it is more important to verify whether the whole feature is working properly or not.

Even if our unit tests are running in milliseconds, can we go to production with confidence? Of course not.
We should verify the whole feature is working or not by testing with the actual external dependencies such as database or message brokers.
That gives us more confidence.

I wonder this whole idea of "We should have core domain completely independent of external dependencies" philosophy
came from the time when testing with real dependencies is very challenging or not possible at all.

Luckily, we have better technology now (ex: [Testcontainers](https://testcontainers.com/)) to test with real dependencies.
Testing with real dependencies might take slightly more time, but compared to the benefits, that's a negligible cost.

## Summary
The **Tomato Architecture** guidelines and principles came from my own experience of working with 
enterprise business applications with unnecessary complex abstractions. 

As I already mentioned, there is nothing new in these guidelines and I believe it's a good thing.
No need to learn one more fancy thing. 
The whole idea is to keep it simple, and I can tell you from my own experience that working with simple codebases is a joy.

In the beginning of our careers, we all have that craving for building complex systems with fancy and trendy technologies.
But if you stay longer in software development, you will start appreciating simple and boring stuff more and more.

# SivaLabs Weekly Newsletter - 2023, Week 27


## SivaLabs Weekly Newsletter - 2023 Week 27

* I came to know about [T3 App](https://create.t3.gg/) which claims to be **"The best way to start a full-stack, typesafe Next.js app"**
* Also, came across [PocketBase](https://pocketbase.io/) which is **"Open Source backend for your next SaaS and Mobile app in 1 file"**
* Switched from **iTerm2** to [warp Terminal](https://www.warp.dev/), and I am liking it very much.
* I have written [Sharing Thoughts and Knowledge via Twitter vs Blogging](https://www.sivalabs.in/sharing-thoughts-and-knowledge-via-twitter-vs-blogging/) article.
* Published a new blog post [Running Testcontainers tests using GitHub Actions and Testcontainers Cloud](https://www.atomicjar.com/2023/06/running-testcontainers-tests-using-github-actions/). Try it out by creating a Free Testcontainers Cloud account.
* If you want to master testing Spring Boot applications and prefer reading a book over watching videos, then I highly recommend [Testing Spring Boot Applications Demystified](https://leanpub.com/testing-spring-boot-applications-demystified)
* Though I said **"I decided Go is not my cup of tea and decided to move on"** in [My Plans for 2023](https://www.sivalabs.in/my-plans-for-2023/), for some reason I needed to brush up my Go skills. 
  As part of that, I have implemented a small [app using Go, Gin, Pgx, Zap, Viper and Testcontainers](https://github.com/sivaprasadreddy/bookmarks-go).
* Published a video on [Getting Started with Testcontainers for Go](https://www.youtube.com/watch?v=PbV31aE-asU)
* I am not using Chat GPT much myself. I am interested to see how we can use it, so gonna watch [Building Java Stuff with ChatGPT - Episode 1](https://www.youtube.com/watch?v=I2wXog37PXY)

Keep learning. Cheers.

# Sharing Thoughts and Knowledge via Twitter vs Blogging


SocialMedia is a double-edged sword. We can put it to good use or become addicted to it and suffer.
The challenge is, it is hard to know when it became addiction. 
Most of the time we get to know that we are addicted to it once we started suffering.

<!--more-->


I use social media, especially [Twitter](https://twitter.com/sivalabs), very much mainly to keep up with what is going on in the tech world.
I must say Twitter helped me in several ways throughout my career.

I usually share my learning on https://sivalabs.in and post it on Twitter, and this gave decent visibility to me and my blog.
**Most importantly, Twitter helped me to connect with amazing people in the Java Community.**

However, lately I see some weirdness on Twitter which makes rethink about "Is Twitter still the right platform to share my thoughts or knowledge?"

### Too much negativity
First of all, people on Twitter are so eager to tell you that "You are wrong", "Your opinion is wrong, and my opinion is right".
I know that different people have different opinions. But that doesn't mean that "others are wrong" just because "you have a different opinion".

I agree that we should be open to criticism, but if you end up debating/clarifying for every damn thing, it is a waste of time.

### Debates without understanding the context
This happens quite a lot on Twitter, especially when talking about "Software Architecture".

Different people have different opinions based on their own experiences.
Someone might have worked on very complex systems, and for them DDD, Clean/Hexagonal Architecture is a good fit.
Someone else might have worked on applications which don't have super complex business logic but handles huge traffic.
For them, Clean/Hexagonal Architecture or DDD might seem overkill. Both of them are right.

But when someone shares their opinion on Twitter, there is a good chance to miss the context and endlessly debate why others are wrong.
I believe one of the reasons is due Twitter's 280-character limit people could not share more context of their opinion.

### Conversations with sociopaths
You may accidentally end up having a conversation with a sociopath who doesn't have basic manners, commonsense or any sense of humour.
Those shitheads from then onwards wait for you tweet anything and then start replying with their bullshit.
Thanks to Twitter for providing Block/Mute options.

----------------------------------------

So, considering all this, I believe blogging is much better to share your thoughts.
As there is no character limit, it would be easy to provide more context on why you arrived at a certain opinion.
For the people who want to reply also has plenty of opportunity to share their opinion and optionally explain why they have a contradicting opinion.

This reminds me of the good old days of DZone back in 2010.
Back then, people used to write blog posts and share their articles on aggregator sites like DZone or JavaCodeGeeks etc.
Then people used to share their views and even debate in a very respectable manner.
I vividly remember people like Howard M. Lewis Ship (author of Tapestry framework), [Jilles van Gurp](https://dev.to/jillesvangurp) etc used to discuss many topics in-depth from which I learned a lot.

## Personal interactions
Recently I kind of stopped sharing opinions and debating on anything because of above mentioned reasons.
Instead, I started connecting with people directly 1-on-1.

I met 3 people in-person whom I connected via Twitter online and we had a great time talking about various things ranging from software development to movies.
Also, I connected with a couple of people online via Zoom calls who live in Germany/USA. It feels good to connect with fellow passionate software developers.

This approach feels much better than debating on Twitter without a shared context.

So, I decided to write more on blog and share on Twitter for reach and engage in discussion only on blog comment section.

I know people habituated to read small tweets and quickly reply (often knee-jerk reactions) than thinking through and engage in a meaningful conversations.
But again they are not the people I/you may want to engage with, so losing those people is absolutely fine.

If you want to reach out to me, you can get in touch with me on https://sivalabs.in or sivalabs.in@gmail.com.

Cheers.


# A Tip for Debugging Tricky Software Bugs or Issues


While building software, once in a while we face some weird issue which is supposed to just work fine 
according to our understanding, but it is not working, and we don't have any clue. 

<!--more-->


We spent hours and hours trying to understand what the hell is going wrong and why it is not working.
Even worse, the exact same code is working fine in a small demo application. 
We all have been into this situation at some point in our careers, right?

After being into this kind of situation several times, I usually follow an approach to debug and fix the issue, 
which I am going to share in this article.

Typically, these kinds of tricky issues happen because of a combination of several things.

For example, if a Java library is working fine in a small demo project but not in your actual client project, 
then that could be because of the other libraries in your classpath.

But figuring out exactly which combination of libraries or configuration is causing the issue 
in a large project where hundreds of libraries are being used is very difficult.

So, whenever I am working on a project, I **create a small demo project with the same tech stack as my client project**, without implementing any features.
Then I try to implement the same use-case, in its simplest way, which is not working in my client project.
This helps me to figure out whether that library is actually working in the way I am expecting or not.
If it's working fine, then I keep adding other libraries/configurations to my demo project that are being used in my client project **one by one**
until I get the same issue.

This helps me to nail down which combination of libraries/configurations is causing the issue.
Most of the time, in Java projects, this could be because of conflicting library versions pulled transitively from different libraries.

Once you find out the root cause, fixing is not that difficult with the help of Google and StackOverflow.
Most likely, you may not be the first one who faced this problem. 
Somebody might have faced a similar issue already and figured out a solution or a workaround.

I work with [Testcontainers](https://testcontainers.com/) quite a bit, and sometimes I need to reproduce some issues reported by other developers to help them fix the issue.
In order to do that, first I need to reproduce that issue locally. 

That is the reason I created https://github.com/sivaprasadreddy/testcontainers-samples repository 
with commonly used technologies in combination with commonly used services (databases, message brokers etc).

These samples helped a lot while exploring the 
[new features introduced in Spring Boot 3.1.0](https://docs.spring.io/spring-boot/docs/3.1.0-SNAPSHOT/reference/htmlsingle/#features.testing.testcontainers), 
and I was able to find out some bugs and reported them too.

So, the bottom line is, **Always have a reproducer application with the same tech stack**.

## Bonus Tip

#### Make sure you are updating the file that you are thinking you are updating.

This might sound silly, but it is not very uncommon to do this mistake. 
We may have some configuration file somewhere in our filesystem, 
and for some unknown reason we make a copy of that file.
Then, we keep updating the wrong file and banging our heads shouting at the computer "Why don't you work"!!!

Before ruling out by saying "I am smart enough not to make this mistake", 
make sure you are updating the file that you are thinking you are updating :-).

Happy debugging!
# Should you use Lombok? Or, is it bad for you?


I'm a happy [Lombok](https://projectlombok.org/) library user. I found this library a decade ago, and I've been using it ever since.
I don't remember facing any major problems with it so far.

However, I see a few people that I greatly admire and learn from often complain that Lombok is very bad, and you should avoid it.
Obviously, I am curious to know what am I missing? I know a couple of scenarios where using Lombok brings some challenges.
Apart from it, I really enjoy using the Lombok library.

<!--more-->


## Why I use Lombok?
I, like most others, spend a lot of time reading code and I like to read concise code.
I know I can generate the **setters**, **getters**, **toString()** etc from the IDE. 
But if I can have the same result by adding a couple of annotations, then I prefer that because it is easier to read.
I don't have to think about, are these simple setters/getters, or somebody sneaked some logic into these setters/getters.

Mostly, I use the following Lombok annotations:
* **@Setter, @Getter**
* **@NoArgsConstructor, @AllArgsConstructor, @RequiredArgsConstructor**
* **@Builder**

I know there are features in Lombok like **@SneakyThrows**, **@EqualsAndHashCode** etc which I don't usually use.
Implementing **equals()** and **hashCode()** should be a conscious choice, and I prefer either generating from the IDE 
with appropriate fields selected or writing it by hand.

I'm well aware of the Java Records, and I use them when appropriate and use normal classes for other scenarios.
Also, if the Lombok generated setters/getters/toString() etc is not what you want, you can write your own and Lombok will back off.

* I never noticed any compilation slowness because of Lombok.
* I never experienced CI pipelines becoming slower because of Lombok.
* Lombok doesn't make me write anemic models, it's my choice.

## Problems with Lombok that I know of
* **Using @Data with JPA entities:** 
  When you have JPA entities with bi-directional relationships, then you may encounter StackOverflowErrors 
  if you blindly use **@Data** or **@ToString** or **@EqualsAndHashcode()**. 
  The solution is to understand why it happens and using the library properly.
* **Using with other annotation processors:** 
  This is a real pain and a good reason to avoid Lombok. Some annotation processors don't go well together (ex: [ErrorProne](https://errorprone.info/)) 
  and some do (ex: [MapStruct](https://mapstruct.org/)).
* **Lombok plugin not working with IDEs:** I remember facing a lot of difficulties installing Lombok plugin in Eclipse/STS. 
  I never faced any problem installing Lombok plugin in Intellij IDEA, but whenever a new version of IDE is released, 
  then Lombok used to break. But, now Lombok is bundled with IDE OOTB; these days I'm not facing those difficulties.  

## My reasons to continue using Lombok 
After seeing "Lombok is bad for you" tweets from a few well-known people in the community, 
even though I don't experience any challenges, I [de-lomboked a project](https://github.com/sivaprasadreddy/devzone/commit/f80ed9b634dbdae57aebd4df80bec970c01c5810) to see whether it is any better. 

It didn't bring me any joy, didn't make anything better. I had the exact same end result with plenty of boilerplate/verbose code.

**One common reason people say to avoid Lombok, which makes me giggle, BTW is: "You don't know what Lombok is doing under-the-hood".**

> Because all the developers know how Spring Data JPA findByUserNameAndPassword() exactly works.
> Everyone knows exactly how GraalVM turning java byte code into native code. Everyone knows exactly how Kubernetes Persistent Volumes work.

We don't have any problem using all the magic provided by those frameworks, but Lombok is bad for you. hmmm, I'm not convinced.

## Conclusion
As always, anyone can misuse a library or a framework and the remedy is "Education".
Learning how to use and how NOT to use will help.

I am all ears to know the practical problems you faced with Lombok so that I know when to avoid Lombok.

For example: 
* Hey, we were using lombok-1.18.8, and it has some bug which is slowing down the CI pipeline.
* We are using Lombok and MapStruct together, and in "this specific scenario" lombok generated code won't work.

Instead, "Hey, I know it is bad, and you have to take my word for a fact" doesn't really help.
Maybe Lombok is awful. Maybe not. 
Sharing the specific reasons will really help people to understand why Lombok, or any library for that matter, is good or bad.
If it is just a personal preference or a gut feeling, it is hard for others to feel it.

# Announcing My "Spring Boot - The Missing Guide" Video Series on YouTube


### TLDR;
I am happy to announce that I am starting the **Spring Boot - The Missing Guide** video series on my [SivaLabs YouTube Channel](https://www.youtube.com/c/sivalabs).
This **Spring Boot - The Missing Guide** video series is for those who need help understanding Spring Boot's magic.
In this series, I will explain the core concepts used in Spring and Spring Boot with practical examples.

<!--more-->


Also, I will give a deep-dive explanation of Spring Boot's AutoConfiguration and how to customize the default configuration.
Throughout the series, we will build a REST API using Spring Data JPA, and along the way, I will explain how to use various features of Spring Boot.

> **BONUS:** 
> you will learn how to test the application using **JUnit 5**, [Testcontainers](https://www.testcontainers.org/), **RestAssured** and the testing best practices.

### Spring Boot - The Missing Guide: 1 - Introduction

{{< youtube 3tTS8rwV7zQ >}}

### Longer version
I have been working with Java and Spring Boot for several years and with many teams with different skill sets.

I observed that some developers were having difficulty with Spring Boot mainly because they were unfamiliar with Java and Spring.
When they directly start with Spring Boot without any prior experience with Spring, many things look magical, and they couldn't understand how Spring Boot actually work.
Some of them are graduates just starting their careers, and some come from different programming language backgrounds.

So, I am making a video series on unpacking the Spring Boot's magic by explaining how Spring Boot works.

### Who are the target audience?
* People who are new to Spring/Spring Boot
* People starting with Spring Boot coming from a different programming language background
* People who have been using Spring Boot for some time but don't know how it works behind the scenes

If you are already a Spring Boot Expert, you might only learn a little from this series, 
but if you can watch and provide me feedback, that would be great.

### Like & Share
If you like to watch this series please subscribe and if you think any of your friends can benefit from this series please share it.

# Why I think Go is more verbose than Java

Few months ago I asked on Twitter "why you think Java is complex?" to understand others perspective.

<!--more-->


{{< x user="sivalabs" id="1509905489911762950" >}}

And, I got many responses (opinions, insights) 
and I compiled all the responses into this blog post [My attempt to understand why people perceive Java as complex](https://www.sivalabs.in/my-attempt-to-understand-why-people-perceive-java-as-complex/).
The intention is purely to understand the other's perspective, not to prove "Java is not verbose".

Back then (April 2022) somebody posted this article on Reddit and the usual thing happened on Reddit.
And, then couple of days back also somebody [posted it on Reddit](https://www.reddit.com/r/java/comments/102cfti/my_attempt_to_understand_why_people_perceive_java/) again, and it is making rounds again one more time.

I am always surprised when someone says Java is more verbose than Go.
If someone compares Java with Python, Ruby, JavaScript/TypeScript and say Java is verbose, yes totally.
Java is way more verbose than Python, Ruby, JavaScript/TypeScript. But I don't think Java is more verbose than Go.

So, I tweeted this :-)

{{< x user="sivalabs" id="1610446679970545664" >}}

Of course some would disagree, not at all unexpected.

But to understand where I am coming from, we need to take a step back and understand my point of view.

* I am not comparing which is better or feature rich at the overall language level. I am only talking about verbosity of language syntax because that is what many people are complaining about Java.
* I have started working with Go a couple of years back. My first impressions are:
  * It is a fresh breath of air. No annotations magic, no 10 levels of abstractions. :white_check_mark:
  * Go performance is top-notch. :white_check_mark:
  * Most of the common needs like Json parsers, testing, formatting tools etc are baked into standard library itself. :white_check_mark:
  * Not many types of collections support, no generics...bummer. :thumbsdown:
  * And, I didn't like Go way of error handling at all. Yes, I read "Errors are first class values" and still I don't like it. :rage:

But Go's simplicity stuck with me. The usual Go way is not to over-engineer code with unnecessary abstractions.
In fact my small stint with Go changed the way I think. 
I stopped writing code thinking "okay, I will add this abstraction here so that in future if we need....", instead I start coding for today's known requirements.

And, I kept thinking despite lacking many features that you find in most of the modern languages such as generics, rich collections support etc how Go became so popular and widely adopted??
I kinda understand why people like Go (or I think so) and I wrote a blog post with my thoughts [Code simplicity by Abstraction vs Verbosity](https://www.sivalabs.in/code-simplicity-by-abstraction-vs-verbosity/)

**By now it should be clear that I don't have anything against Go and in fact I like many things about Go.
Having said that I still feel Go is more verbose than Java mainly because of error handling.**

Whether I like it or not Go error handling is what it is, and I am not the only one [who doesn't like Go's error handling](https://github.com/ksimka/go-is-not-good).

There is no point in debating over opinions because everyone is right in their point of view.
So, I just shared two code snippets of how typical transaction handling is done in Java and Go.

Let us take a look at the usual way of transaction handling in Go and Java.

### Transaction handling in Go

```go
func (c CustomerService) createCustomer(customer Customer) error {
    // Get a Tx for making transaction requests.
    tx, err := db.BeginTx(ctx, nil)
    if err != nil {
        return err
    }
    // Defer a rollback in case anything fails.
    defer tx.Rollback()
    
    err := customerRepository.save(tx, customer)
    if err != nil {
        return err
    }
    err = addressRepository.saveCustomerAddress(tx, customer.Address)
    if err != nil {
        return err
    }
    // Commit the transaction.
    if err = tx.Commit(); err != nil {
        return err
    }
    return nil
}
```

### Transaction handling in Java(Spring Boot and JdbcTemplate)

```java
@Transactional
void createCustomer(Customer customer) {
    customerRepository.save(customer);
    addressRepository.saveCustomerAddress(customer.Address);
}
```

Transaction handling works just fine in both Go and Java cases. 
Spring Boot takes care starting the transaction and commit/rollback depending on any errors occurred.
I know Java is not Spring Boot, but the code would be same even with other frameworks like Quarkus and Micronaut too.
And, Go code will be same/similar even if I use any framework also because of the Go way of error handling.

Let's take another typical Go code reading from database:

```go
func (b postRepo) GetPosts() ([]Post, error) {
	var posts []Post
	query := "select id, title, content, created_time FROM posts"
	rows, err := b.db.Query(query)
	if err != nil {
		return nil, err
	}
	defer rows.Close()
	for rows.Next() {
		var post = Post{}
		err = rows.Scan(&post.Id, &post.Title, &post.Content, &post.CreatedTime)
		if err != nil {
			return nil, err
		}
		posts = append(posts, post)
	}
	err = rows.Err()
	if err != nil {
		return nil, err
	}
	return posts, nil
}
```

To my eyes this is super verbose mainly because of error handling.

If Go has error handling like any other programming language giving an option to developer where they want to handle error, 
the same code could have been as follows:

```go
func (b postRepo) GetPosts() ([]Post) {
	var posts []Post
	query := "select id, title, content, created_time FROM posts"
	rows := b.db.Query(query)
	for rows.Next() {
		var post = Post{}
		rows.Scan(&post.Id, &post.Title, &post.Content, &post.CreatedTime)
		posts = append(posts, post)
	}
	return posts
}
```
Sometimes exactly which statement throws error doesn't matter, and it can be treated as an overall failure and I want handle it in same way.
In the above method, it doesn't matter which lines throws error I want to log the error and return empty slice.
If I want to handle a special case like DB lock errors etc then I would specifically catch those errors and act accordingly(maybe do a retry).
But because of the current way of Go's error handling mechanism, there is no way but to add errors checks all over the place.

### 1. Is it unfair comparison? 
Some people feel it is unfair comparison and said I should also include Spring's Transaction handling code to compare.
I respectfully disagree with that.

As an application developer we use programming language and many frameworks/libraries to achieve our goals.
If the chosen framework works as expected with good enough performance, all I care about is the code "I need to write" for my application.
I don't have to dig into Spring's internal transaction handling logic or Gin's internal routing logic.
The comparison should be between the code we need to write.

### 2. How about other aspects?
My comparison was only about "verbosity". When it comes to other aspects I like Go a lot.
* Building a native binary with Go is a breeze. While GraalVM is an option in Java world, it is nowhere as good as Go's support IMO.
* When it comes to build and runtime performance, I find Go applications are much faster with less memory footprint.

> Fun fact: 
> 
> I was [building an application](https://github.com/orgs/sivalabs-bookstore/repositories) with around 7 services using Spring Boot and when I try to run them in Docker I couldn't because of memory/CPU consumption is too high.
> I replaced some services with Go implementation, and they run smoothly with very less memory consumption.

I am not saying Go is more verbose than Java based on my gut feeling or my bias towards Java.
I am simply showing code samples to convey why I feel Go is more verbose than Java.
If you think its incorrect assumption and show some code samples then I am glad to learn and improve my understanding.

# Conclusion
So, it all comes down to tradeoffs. No language is perfect in every aspect, and we have to choose the tech stack which ticks most checkboxes for our specific needs.
Just parroting "Language X is dead or Language Y is verbose" just because you read these memes/jokes on internet doesn't bring any value.
Try it and see for yourself whether those languages/frameworks are as good/bad as they say and make an informed opinion.

# My Plans for 2023


First of all, **Wish you a very happy new year and I hope this year will bring more joy and happiness into your lives.**

I would like to share how was 2022 for me and what my plans are for 2023.

<!--more-->


## Highlights of 2022
The year 2022 is very eventful for me and lots of interesting things happened.

### 1. "How to Become a Software Architect" series with in28minutes and JavaBrains
I collaborated with in28minutes Ranga and JavaBrains Koushik and did a "How to Become a Software Architect" series

{{< youtubepl id="PLBBog2r6uMCS0FX6Ym_8NYeTNykUF5970" >}}

### 2. Restarted my YouTube channel with "Spring Boot Tips - 10 part series" 
I started my YouTube channel few years ago and after making a couple of videos I stopped for some reason.
I restarted making videos with my favourite topic "Spring Boot" and made a 10 Part series on "Spring Boot Tips".

{{< youtubepl id="PLuNxlOYbv61jFFX2ARQKnBgkMF6DvEEic" >}}

### 3. Created "Spring Boot + Kubernetes Tutorial" 25-Part Series on YouTube
With the overwhelming positive feedback I got from "Spring Boot Tips" series, 
I made another 25-Part series "Spring Boot + Kubernetes Tutorial" Series demonstrating 
how to build a FullStack Spring Boot + ReactJS application from end-to-end and deploy on Kubernetes.

{{< youtubepl id="PLuNxlOYbv61h66_QlcjCEkVAj6RdeplJJ" >}}

### 4. I joined AtomicJar as Developer Advocate
If you know me via Twitter/SivaLabs Blog etc then you probably know that I like [Testcontainers](https://www.testcontainers.org/) very much for testing.
I got the wonderful opportunity to join [AtomicJar](https://www.atomicjar.com/) as a **Developer Advocate** and work with my favourite technology Testcontainers.

Read [Why I am excited to join AtomicJar](https://www.atomicjar.com/2022/11/why-i-am-excited-to-join-atomicjar/) to know more.

## Plans for 2023
I know plans don't go as planned always, but it gives some direction.
Following are few things I am planning to do in 2023.

### 1. Helping devs to efficiently use Testcontainers
At [AtomicJar](https://www.atomicjar.com/) we, Developer Advocate Team, planned many things that would help developers to learn how to write better tests using [Testcontainers](https://www.testcontainers.org/).
Also, we are focusing on sharing some "Best Practices" on using Testcontainers in various environments (local, CI etc).

### 2. "Spring Boot : The Missing Guide" YouTube Series
After doing **Spring Boot Tips** and **Spring Boot + Kubernetes Tutorial** Series, many people asked me whether I have any course that covers Spring Boot.

I am planning to make a new series to cover:
* The "must know" concepts before starting with Spring Boot
  * Dependency Injection, Front Controller, Template Design Patterns
  * Programming to Interfaces, Loose Coupling
* Building a Spring Boot CRUD application using InMemory Store
* Introducing Spring Data JPA
  * Using Pagination, Projections
  * Flyway DB Migrations
* Spring Boot CRUD with Spring Data JPA
* Building UI using Spring Boot + Thymeleaf
* Introducing Spring Security
* Secure the application using Spring Security
* Testing using Testcontainers
* Converting package structure from package-by-layer to package-by-feature
* Should you adopt Clean/Hexagonal Architecture?
* Decoupling the external service interactions
* Applying Single Responsibility Principle

I hope this video course will be useful for a lot of people.

### 3. Learning something new
It is important to keep updating your skills in IT to be relevant. 
I thought of learning **GoLang** more in-depth. I kinda have a love/hate relationship with Go.
While I am in love with some features of Go, I can't stand it's verbosity, especially it's error handling.
I thought I can get used to it, but no matter how hard I try I am not getting the joyful feeling.
So, **I decided Go is not my cup of tea and decided to move on**.

Maybe if I have time, I will learn **Rust**.

## Personal Stuff
* I am 39 years old now and what used to be a **"nice to follow"** healthy habits became **"must follow"** now. So I got to change my food habits, reduce screen time and take care of both physical and mental health.
* I feel people on social media are becoming more judgemental. Thanks to some people, I learned my lesson of **"not everyone deserves a reply"**.
* I have already realized that **being passionate about technology** is nice but **being balanced** is more important. In the name of **passion** don't screw up your health.
* **Screw all those personality development and productivity improvement bullshit.** It is absolutely ok to just chill doing nothing. I worked hard so that I can relax when I want to. 
  I won't fall into these **productivity gurus** trap of "every second has to be productive".

## Conclusion
Well, just relax, do stuff as good as you can and enjoy the ride.

# Spring Boot 3 : Error Responses using Problem Details for HTTP APIs

Spring Framework 6 implemented the **Problem Details for HTTP APIs** specification, [RFC 7807](https://www.rfc-editor.org/rfc/rfc7807.html). 
In this article we will learn how to handle exceptions in Spring Boot 3 REST API(which uses Spring Framework 6) and provide error responses using **ProblemDetails** API.

<!--more-->


> We are going to reuse the Spring Boot 3 sample application [spring-boot-jpa-crud-demo](https://github.com/sivaprasadreddy/spring-boot-jpa-crud-demo) which we used for [Using Java Records with Spring Boot 3](https://www.sivalabs.in/using-java-records-with-spring-boot-3/) article.

Assume we have the following REST API endpoints to create a bookmark and fetch a bookmark by id.

```java

@RestController
@RequestMapping("/api/bookmarks")
@RequiredArgsConstructor
public class BookmarkController {
    private final BookmarkService service;

    @PostMapping
    public ResponseEntity<Bookmark> save(@Valid @RequestBody Bookmark payload) {
        Bookmark bookmark = new Bookmark(null, payload.title(), payload.url(), Instant.now());
        return ResponseEntity.status(HttpStatus.CREATED).body(service.save(bookmark));
    }
    
    @GetMapping("/{id}")
    public ResponseEntity<Bookmark> getBookmarkById(@PathVariable Long id) {
        return service.getBookmarkById(id)
                .map(ResponseEntity::ok)
                .orElseThrow(() -> new BookmarkNotFoundException(id));
    }
}
```

And, the **BookmarkNotFoundException** is a typical **RuntimeException** as follows:

```java
public class BookmarkNotFoundException extends RuntimeException {

    public BookmarkNotFoundException(Long bookmarkId) {
        super("Bookmark with id: "+ bookmarkId+" not found");
    }
}
```

Now, when you make an API call to create a bookmark with invalid data (title and url are required) using the following cURL 
```shell
curl --location --request POST 'http://localhost:8080/api/bookmarks' \
--header 'Content-Type: application/json' \
--data-raw '{ "title":"", "url":"" }'
```

then you will get a default Spring Boot error response as follows:

```json
{
    "timestamp": "2022-11-30T04:42:14.282+00:00",
    "status": 400,
    "error": "Bad Request",
    "trace": "org.springframework.web.bind.MethodArgumentNotValidException: Validation failed for argument [0] in public org.springframework.http.ResponseEntity<com.sivalabs.bookmarks.domain.Bookmark> com.sivalabs.bookmarks.web.BookmarkController.save(com.sivalabs.bookmarks.domain.Bookmark) with 2 errors: [Field error in object 'bookmark' on field 'url': rejected value []; codes [NotEmpty.bookmark.url,NotEmpty.url,NotEmpty.java.lang.String,NotEmpty]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [bookmark.url,url]; arguments []; default message [url]]; default message [Url is mandatory]] [Field error in object 'bookmark' on field 'title': rejected value []; codes [NotEmpty.bookmark.title,NotEmpty.title,NotEmpty.java.lang.String,NotEmpty]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [bookmark.title,title]; arguments []; default message [title]]; default message [Title is mandatory]] 
            at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.resolveArgument(RequestResponseBodyMethodProcessor.java:144)
            at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:122)
            at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:181)
            ....
            ....
            ",
    "message": "Validation failed for object='bookmark'. Error count: 2",
    "errors": [
        {
            "codes": [
                "NotEmpty.bookmark.url",
                "NotEmpty.url",
                "NotEmpty.java.lang.String",
                "NotEmpty"
            ],
            "arguments": [
                {
                    "codes": [
                        "bookmark.url",
                        "url"
                    ],
                    "arguments": null,
                    "defaultMessage": "url",
                    "code": "url"
                }
            ],
            "defaultMessage": "Url is mandatory",
            "objectName": "bookmark",
            "field": "url",
            "rejectedValue": "",
            "bindingFailure": false,
            "code": "NotEmpty"
        },
        {
            "codes": [
                "NotEmpty.bookmark.title",
                "NotEmpty.title",
                "NotEmpty.java.lang.String",
                "NotEmpty"
            ],
            "arguments": [
                {
                    "codes": [
                        "bookmark.title",
                        "title"
                    ],
                    "arguments": null,
                    "defaultMessage": "title",
                    "code": "title"
                }
            ],
            "defaultMessage": "Title is mandatory",
            "objectName": "bookmark",
            "field": "title",
            "rejectedValue": "",
            "bindingFailure": false,
            "code": "NotEmpty"
        }
    ],
    "path": "/api/bookmarks"
}
```

> Prior to Spring Framework 6 many organizations use Zalando's [problem-spring-web](https://github.com/zalando/problem-spring-web) for returning RFC 7807 compliant error responses.
> If you want to learn How to use Problem Spring Web Library you can watch my video here

{{< youtube hVfajuyEJMQ >}}

Now let us see how to use ProblemDetails API to return RFC 7807 compliant responses.

## 1. Enable RFC 7807 responses
To enable RFC 7807 responses we need to create a global exception handler using **@ControllerAdvice** by extending **ResponseEntityExceptionHandler**.

```java
@RestControllerAdvice
public class GlobalExceptionHandler extends ResponseEntityExceptionHandler {

}
```

Now when you make the same API call you will get the RFC 7807 compliant response with Header **"Content-Type=application/problem+json"** as follows:

```json
{
    "type": "about:blank",
    "title": "Bad Request",
    "status": 400,
    "detail": "Invalid request content.",
    "instance": "/api/bookmarks"
}
```

The **ResponseEntityExceptionHandler** implements **@ExceptionHandler** methods for most of the Spring MVC built-in exceptions such as **MethodArgumentNotValidException**, **ServletRequestBindingException**, **HttpRequestMethodNotSupportedException**, etc.
So, the **MethodArgumentNotValidException** is handled by the exception handler method and the appropriate error response is returned.

## 2. Handling Custom Exceptions
Let us see how to handle our own custom exceptions and return RFC 7807 compliant response using ProblemDetails API.

Let's make an HTTP API call fetch a bookmark with a non-existing id
```shell
curl --location --request GET 'http://localhost:8080/api/bookmarks/111'
```

then you will get the following response:

```json
{
    "timestamp": "2022-11-30T04:34:42.002+00:00",
    "status": 500,
    "error": "Internal Server Error",
    "trace": "com.sivalabs.bookmarks.domain.BookmarkNotFoundException: Bookmark with id: 111 not found  
              at com.sivalabs.bookmarks.web.BookmarkController.lambda$getBookmarkById$0(BookmarkController.java:31)
              at java.base/java.util.Optional.orElseThrow(Optional.java:403)
              at com.sivalabs.bookmarks.web.BookmarkController.getBookmarkById(BookmarkController.java:31)
              at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              ....,
              ....
              ",
    "message": "Bookmark with id: 111 not found",
    "path": "/api/bookmarks/111"
}
```

To customize the error response we can create an ExceptionHandler in **GlobalExceptionHandler** and use ProblemDetail to return a customized response.

```java
@RestControllerAdvice
public class GlobalExceptionHandler extends ResponseEntityExceptionHandler {

    @ExceptionHandler(BookmarkNotFoundException.class)
    ProblemDetail handleBookmarkNotFoundException(BookmarkNotFoundException e) {
        ProblemDetail problemDetail = ProblemDetail.forStatusAndDetail(HttpStatus.NOT_FOUND, e.getMessage());
        problemDetail.setTitle("Bookmark Not Found");
        problemDetail.setType(URI.create("https://api.bookmarks.com/errors/not-found"));
        return problemDetail;
    }
}
```

Now when you make the API call to fetch bookmark with non-existing id then you will get the following response:

```json
{
  "type": "https://api.bookmarks.com/errors/not-found",
  "title": "Bookmark Not Found",
  "status": 404,
  "detail": "Bookmark with id: 111 not found",
  "instance": "/api/bookmarks/111"
}
```
In addition to the standard fields **type, title, status, detail, instance** we can also include **custom properties** as follows:

```java
@RestControllerAdvice
public class GlobalExceptionHandler extends ResponseEntityExceptionHandler {

    @ExceptionHandler(BookmarkNotFoundException.class)
    ProblemDetail handleBookmarkNotFoundException(BookmarkNotFoundException e) {
        ProblemDetail problemDetail = ProblemDetail.forStatusAndDetail(HttpStatus.NOT_FOUND, e.getMessage());
        problemDetail.setTitle("Bookmark Not Found");
        problemDetail.setType(URI.create("https://api.bookmarks.com/errors/not-found"));
        problemDetail.setProperty("errorCategory", "Generic");
        problemDetail.setProperty("timestamp", Instant.now());
        return problemDetail;
    }
}
```

We have added 2 custom properties **errorCategory** and **timestamp** which will be included in the response as follows:

```json
{
    "type": "https://api.bookmarks.com/errors/not-found",
    "title": "Bookmark Not Found",
    "status": 404,
    "detail": "Bookmark with id: 111 not found",
    "instance": "/api/bookmarks/111",
    "errorCategory": "Generic",
    "timestamp": "2022-11-30T05:21:59.828411Z"
}
```

In addition to **ProblemDetail**, you can also return an instance of **ErrorResponse** which is a contract to expose HTTP error response details including HTTP status, response headers, and a body in the format of RFC 7807.

> All Spring MVC exceptions such as MethodArgumentNotValidException, ServletRequestBindingException, HttpRequestMethodNotSupportedException, etc. implement ErrorResponse.

```java
@RestControllerAdvice
public class GlobalExceptionHandler extends ResponseEntityExceptionHandler {

    @ExceptionHandler(BookmarkNotFoundException.class)
    ErrorResponse handleBookmarkNotFoundException(BookmarkNotFoundException e) {
        return ErrorResponse.builder(e, HttpStatus.NOT_FOUND, e.getMessage())
                .title("Bookmark not found")
                .type(URI.create("https://api.bookmarks.com/errors/not-found"))
                .property("errorCategory", "Generic")
                .property("timestamp", Instant.now())
                .build();
    }
}
```

## 3. Custom Exceptions extending ErrorResponseException
Instead of implementing **@ExceptionHandler** methods for our custom exceptions, we can extend from **ErrorResponseException** and just throw the Exception.

```java
import org.springframework.http.HttpStatus;
import org.springframework.http.ProblemDetail;
import org.springframework.web.ErrorResponseException;

import java.net.URI;
import java.time.Instant;

public class BookmarkNotFoundException extends ErrorResponseException {

    public BookmarkNotFoundException(Long bookmarkId) {
        super(HttpStatus.NOT_FOUND, asProblemDetail("Bookmark with id "+ bookmarkId+" not found"), null);
    }

    private static ProblemDetail asProblemDetail(String message) {
        ProblemDetail problemDetail = ProblemDetail.forStatusAndDetail(HttpStatus.NOT_FOUND, message);
        problemDetail.setTitle("Bookmark Not Found");
        problemDetail.setType(URI.create("https://api.bookmarks.com/errors/not-found"));
        problemDetail.setProperty("errorCategory", "Generic");
        problemDetail.setProperty("timestamp", Instant.now());
        return problemDetail;
    }
}
```

By making **BookmarkNotFoundException** extending **ErrorResponseException** we can simply throw **BookmarkNotFoundException** and 
SpringMVC will handle it and return the error response in RFC 7807 compliant format without requiring us to implement **@ExceptionHandler** method in **GlobalExceptionHandler**.

## Conclusion
By using Spring Framework's ProblemDetails API for error responses we can standardize on the response format which will be very beneficial
when you have a large number of microservices communicating with each other.
# Using Java Records with Spring Boot 3


Records were introduced in Java 14 as a preview feature and became a standard feature with JDK 16.
Records are a concise representation of immutable data class. 

<!--more-->


Prior to Records this is how we usually create an immutable class.

```java

import java.util.Objects;

class Person {
    private final Long id;
    private final String name;

    public Person(Long id, String name) {
        this.id = id;
        this.name = name;
    }

    public Long getId() {
        return this.id;
    }

    public String getName() {
        return this.name;
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) {
            return true;
        }
        if (o == null || getClass() != o.getClass()) {
            return false;
        }
        Person person = (Person) o;
        return Objects.equals(id, person.id) && Objects.equals(name, person.name);
    }

    @Override
    public int hashCode() {
        int result = id != null ? id.hashCode() : 0;
        result = 31 * result + (name != null ? name.hashCode() : 0);
        return result;
    }

    @Override
    public String toString() {
        return "Person{" + "id=" + id + ", name='" + name + '\'' + '}';
    }
}
```

While most of the time we usually generate **equals()**, **hashCode()** and **toString()** using either IDE generation or using **Lombok**, it adds more noise to the code.
The same Person class can be written as a Record as follows:

```java
public record Person(Long id, String name){ }
```

That's it. The **equals()**, **hashCode()** and **toString()** methods will be auto-generated for records.
However, note that getters don't follow the usual **getId(), getName()** pattern. 
Instead, it will generate accessor methods as **person.id()** and **person.name()**.

## Using Records with Spring Boot 3
[Spring Boot 3](https://spring.io/blog/2022/11/24/spring-boot-3-0-goes-ga) is released on 24-Nov-2022 which requires Java 17+.
Let us see how and where can we use Records with Spring Boot.

### Binding Application Properties
If you are familiar with Spring Boot application properties binding to a class, this look like this:

```java
import jakarta.validation.constraints.Max;
import jakarta.validation.constraints.Min;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.validation.annotation.Validated;

@ConfigurationProperties(prefix = "app")
@Validated
class ApplicationProperties {
    @Min(1)
    @Max(100)
    private int pageSize;
    
    public int getPageSize() {
        return pageSize;
    }
    public void setPageSize(int pageSize) {
        this.pageSize = pageSize;
    }
}
```

Spring Boot 2.2.0 introduced support for **ConstructorBinding** which can be used to bind properties to an immutable class.

```java
import jakarta.validation.constraints.Max;
import jakarta.validation.constraints.Min;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.context.properties.bind.ConstructorBinding;
import org.springframework.validation.annotation.Validated;

@ConfigurationProperties(prefix = "app")
@Validated
public class ApplicationProperties {
    @Min(1)
    @Max(100)
    private final int pageSize;

    @ConstructorBinding
    public ApplicationProperties(int pageSize) {
        this.pageSize = pageSize;
    }
    public int getPageSize() {
        return pageSize;
    }
}
```

Most likely you want your **ApplicationProperties** object to be immutable, and hence Records is a good choice here.
So, we can make **ApplicationProperties** as a Record and use it as follows:

```java
import jakarta.validation.constraints.Max;
import jakarta.validation.constraints.Min;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.validation.annotation.Validated;

@ConfigurationProperties(prefix = "app")
@Validated
public record ApplicationProperties(
        @Min(1)
        @Max(100)
        int pageSize
) {
}
```

This is very concise and also prevents accidentally modifying the configuration properties values.

## Binding to HTTP Request/Response Payloads
We usually create DTO classes with setters and getters to bind incoming HTTP request payload because frameworks need a way to bind the request payload to the class properties.

Spring Boot by default uses [Jackson](https://github.com/FasterXML/jackson) library to convert request/response payloads to/from JSON and Jackson 2.12 introduced support for Records.
So, we can use Records to bind incoming request payloads and also return records as response.

Here is a Record with Bean Validation constraints applied:

```java
import jakarta.validation.constraints.NotEmpty;
import java.time.Instant;

public record Bookmark(
        Long id,
        @NotEmpty(message = "Title is mandatory")
        String title,
        @NotEmpty(message = "Url is mandatory")
        String url,
        Instant createdAt) {
}
```

We can use the Bookmark record with SpringMVC controller as follows:

```java
import com.sivalabs.bookmarks.domain.Bookmark;
import com.sivalabs.bookmarks.domain.BookmarkService;
import jakarta.validation.Valid;
import lombok.RequiredArgsConstructor;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;

import java.time.Instant;

@RestController
@RequestMapping("/api/bookmarks")
@RequiredArgsConstructor
public class BookmarkController {
    private final BookmarkService service;

    @PostMapping
    public ResponseEntity<Bookmark> save(@Valid @RequestBody Bookmark bookmark) {
        Bookmark savedBookmark = service.save(bookmark);
        return ResponseEntity.status(HttpStatus.CREATED).body(savedBookmark);
    }
}
```

Here we are binding the JSON request payload to Bookmark record and also returning the Bookmark record as response payload which Jackson will convert to JSON.

> You can find a sample project demonstrating the Java records usage with Spring Boot 3 at https://github.com/sivaprasadreddy/spring-boot-jpa-crud-demo

# Conclusion
Java records are very useful to model the immutable data carrier objects with very concise syntax.
However, records are not a silver bullet solution for every scenario. There could be cases where a regular class is better suited than records.

# Rethinking About My Social Media Usage

Social media is a double-edged sword. Depending on how you use it, it can be really helpful or become a source of pressure and frustration.
I don't use Facebook and Instagram much, but I use [Twitter](https://twitter.com/sivalabs) and recently started using [LinkedIn](https://www.linkedin.com/in/siva-prasad-reddy-katamreddy/) too.

<!--more-->


Twitter is my primary source of technical information. I follow many good people in Java Community.
Twitter is a great platform to connect with those awesome people and I learn a lot from what they share.
We also exchange our views on many technical and non-technical aspects and learn from each other.

Considering [how things are going with Twitter now](https://twitter.com/search?q=twitter%20layoffs), I am rethinking about my social media usage.
Many people are thinking of moving to other platforms like https://mastodon.social/ as an alternative to Twitter.
I don't know how the future holds for Twitter, but I think there are few things I should change regarding my social media usage.

## 1. Prefer writing a blog post over writing Twitter threads
Tweets are a very convenient way to exchange a small piece of information than writing a blog post.
But sometimes the piece of information we want to share feels too much for a blog post but can't fit in a tweet. 
Then we usually go for writing a thread. But recently Twitter threads become annoying and (mis)used for attracting more engagement.
Personally I feel it is easier to read a blog post than a twitter thread.
In addition to that, sometimes it is difficult to put our message along with context and people may misunderstand, and you end up adding more context in subsequent threads.

So, I decided to write a blog post whenever it makes sense than tweeting. 
Of course, I would share the link to my post on Twitter too.

## 2. Worthiness of Conversations
We, as human beings, express our feelings. Especially during and after this pandemic and WorkFromHome, loneliness became a real problem.
Sharing your joy or excitement or frustration over Zoom is not the same as doing it face-to-face.

You may learn something new, and you are excited to share it with someone.
Or, you are struggling with some tricky problem for hours and there is no one sitting next to you to share your frustration.
Then we may express those feelings, opinions or frustrations on social media like Twitter. 
As long as it is not offending anybody or belittling someone it shouldn't be a problem.
But, if people have that much understanding the world would be a better place.

**Imagine I am sitting in a restaurant having my dinner and heard someone sitting at the next table telling his friend "This is the best ice cream in the world".**

**Then, as a normal person, my reaction would be "Wow, that ice cream must be very delicious, maybe I should try it too". Or, if I think there is some other flavour of ice cream that I think is even better than what he is eating then I may think "He didn't know about THAT other ice cream which is much more delicious".
And, if I feel that person is a nice and approachable one then I might even tell him "Sir, I glad you are enjoying your ice cream, if possible try 'THAT' ice cream which is available in SO-AND-SO place, it is even more tasty. You will definitely like it".**

**But, what I wouldn't do is telling him "How dare you to say this is the best ice cream in the world. Did you eat all the ice creams in the world to pass such a blanket statement. 
You are insulting all other ice cream manufacturers. I won't let you get away with this, I am going to expose your true face to the world by removing your mask".**

**If you think no one would behave like that crazy then think again :-)**

![someone-is-wrong-on-internet](/images/someone-is-wrong-on-internet.webp "someone-is-wrong-on-internet")

Life is too short to waste by engaging in conversations with those kind of people.
So, I learned that it is very important to know whether someone is worthy of having a conversation or not.

**I do have my own opinions which are different from others and at times I disagree with some people that I greatly admire and respect.
But what we don't do is disrespect each other and do personal attacks because we are mature enough to respect others' opinions.**

## 3. I own my content
One of my primary uses of Twitter is bookmarking references to good resources (articles, code repositories, etc.) using **Like** so that I can read them later.
If worst comes to worst, I may lose all that information. 
Earlier I started writing [What I Learned in Week](https://www.sivalabs.in/what-i-learned-2020-week-14/) but stopped for some reason.
I am going to restart that so that I don't have to worry about losing that data.

Happy learning & sharing :-)

# Book Review : Cloud Native Spring in Action


I got a chance to review the upcoming [Cloud Native Spring in Action](https://www.manning.com/books/cloud-native-spring-in-action) book by [Thomas Vitale](https://twitter.com/vitalethomas) and here is my review of the book.

<!--more-->


![Cloud Native Spring in Action](/raw-images/cnsia-book-cover.jpeg "Cloud Native Spring in Action")

## TLDR:
If you know the basics of Spring Boot and want to master the advanced concepts and also understand what "Cloud Native" means and
how to build & deploy production grade Spring Boot applications on Kubernetes then this is the book you are looking for.


## Longer version:

There are many books on Spring Boot that are aimed towards complete beginners to intermediate skilled developers.
But **Cloud Native Spring in Action** book's target audience is developers who have basic knowledge of Spring Boot and are looking forward to mastering the advanced concepts.

* The 1st chapter introduces you to the concept of "Cloud Native" application development and set the stage for coming chapters.
  If you are already familiar with **Cloud Native**, **Kubernetes** and **Spring Boot** then probably you can quickly skim through it.

* From the 2nd chapter onwards you can learn how to build Spring Boot based applications, containerize them using Buildpacks support and deploy them on Minikube cluster.

* Then onwards author shows how to build a [BookStore](https://github.com/PolarBookshop) application using microservices architecture using Spring Boot, SpringCloud and Kubernetes.

* Author clearly explained how to build microservices using relational databases like Postgresql using JDBC and R2DBC with Flyway migration tool.

* The next chapters teach you how to implement resiliency, fault-tolerant applications using:
  * Spring WebFlux and ProjectReactor
  * Resilience4j and SpringCloud CircuitBreaker
  * Event driven microservices using RabbitMQ
  * Serverless functions using Spring Cloud Functions

* My favorite topic/chapter among all is about implementing API Gateway using Spring Cloud Gateway.
  The author explained how to implement API Gateway with resiliency/fault tolerance using Resilience4j,
  and added security using OAuth 2.0 with KeyCloak.

* The author also covered a lot of concepts around deployment using:
  * GitHub Actions to build CI pipeline
  * docker-compose based local dev environment setup
  * Creating Kubernetes YAML manifest files and also using Kustomize to prepare production grade deployment manifests.

* Monitoring and Observability is very crucial in production. The author explained how to use Prometheus, Grafana, Loki, OpenTelemetry and Tempo to set up the Observability stack.

* Finally, concluded with complete production deployment setup using Kustomize, ArgoCD and GitHub Actions.

## A couple of things that could be improved
* Many people are currently using Spring Cloud modules which can be replaced with some Kubernetes features such as Service Registry, Ribbon Load Balancer, etc.
  Having a mapping from Spring Cloud features to Kubernetes would help the migration path. Something like https://dzone.com/articles/deploying-microservices-spring-cloud-vs-kubernetes
* KeyCloak server setup is done via CLI commands, but for beginners it would be easy to do it from UI. And, then showing how to export the realm and creating KeyCloak server using that realm config would have been great.

## Conclusion
> Overall the **Cloud Native Spring in Action** is a must-read for every developer who is interested in learning to build production grade Spring Boot applications and deploy them on the Cloud.

The sample code at https://github.com/PolarBookshop can be taken as a reference for building Cloud Native Spring Boot applications.

# How (not) to ask for Technical Help?


We, software developers, have a great advantage when it comes to getting help from others.
Irrespective of how much experience you have you will need help from others. 
And, there are plenty of good people out there willing to help you by means of answering on StackOverflow, writing blog posts, making video tutorials, etc. They are taking their personal time to help strangers whom they may never meet in person, kudos to all of them.

<!--more-->


But, in order to help you they need some context of the problem, and you need to approach them in reasonable ways.

Here I would like to share my thoughts on **what are the better ways to ask for help?** and what are **some patterns that you should avoid while asking for help**.

## How NOT to ask for help
### 1. Use the right medium/channel

**StackOverflow**, **GitHub Issues** etc are specifically designed for this purpose, where you can share code snippets and ask for help.

**Twitter DMs**, **WhatsApp messages** are terrible ways to post a bunch of code snippets and ask for help.
It is very hard for the people to make sense of code on Twitter DMs and WhatsApp. Use the right medium/channel.

### 2. Asking "I need XYZ, it is very urgent"
Those who want to help others may also have a regular day job, they have their own priorities, families to take care of.
What is urgent for you may not be urgent for them, especially when they are offering their help for FREE.

Just adding "It is very urgent" doesn't help much, in fact that may have a negative impact also.

### 3. Asking for whole project implementation as HELP
Some people ask **"Could you please send me a microservices based sample project with OAuth 2.0 security and Angular front-end?"**

This is not "help", that is a whole project work for which people usually charge few hundred dollars.

If you try to implement that project yourself and got stuck somewhere then people would help you to resolve that particular issue. 
But if you ask them to build the whole application for you then there is very less chance you may get that "HELP".

### 4. Asking for help with "It's not working" as the whole details
Occasionally I come across questions on StackOverflow as follows:

* I am using Spring Security and I am getting 403 error. Please help.
* I am using Spring Boot and when I start the application it is throwing error. Why?
* I am getting CORS issue even after configuring CORS. How to resolve it?

These are not just the titles, these are the whole questions. No code snippets, no Exception StackTraces, nothing.

Just imagine, somebody asked you for help with just that info, would you be able to help? No right?

**Share the context, share some code snippets or exception you are getting** so that others can get some insights into the problem and can help you. 

Just asking for help with 10000 feet view of the problem won't get you any answers. 

## How to ask for help

### 1. Share context, software versions, actual problem you are facing
Many times we try to solve a problem and come up with some approach which involves steps 1, 2 and 3.
We may get stuck at step 2 or 3 and ask for help how to resolve that particular issue.
This is called [The XY Problem](https://xyproblem.info/)

Sometimes giving the overall context, not just the problem you are facing, really helps others to understand the context of the issue. 
There may be a better alternative approach than what you are trying to follow which can only be realized if you share the full context. 

Also, it is a good practice to share what OS, language/framework versions you are using to reproduce the problem before solving it.

### 2. Share what you have tried already
You might have already tried in few ways to solve the issue before asking others for help.
Sharing all those details will spare others time from trying those same ways again.

### 3. Provide minimal reproducible example
Sometimes it is the very minor details, which we take it for granted, causing the issue.
The best way to get help is to create a minimal reproducible example project and share it

Here is a StackOverflow article on [How to create a Minimal, Reproducible Example](https://stackoverflow.com/help/minimal-reproducible-example)

Happy learning and sharing :-)

# My all-time favourite 20 inspiring Tech Talks

We, software developers, aspire to become better by learning new tools and techniques.
In addition to that, I find listening to the experts and learning from their experiences is a great way to upskill ourselves.

<!--more-->


We may want to learn a new technology, or want to start our own start-up company or learn why some hyped technologies failed.
Also, we may want to learn some behavioural aspects like how to be empathetic to others, be a better mentor to juniors etc.

I listen to a lot of tech talks by some awesome technologists and try to learn from them.
Here I would like to share some of my all-time favourite talks with you in this article.

## 1. Things I Wish I'd Known By Rod Johnson

{{< youtube cZHZm8JgTBk >}}

## 2. The Well-Balanced Programmer (J.B. Rainsberger )
{{< youtube XhN6fJYG81A >}}

## 3. Scaling Yourself • Scott Hanselman
{{< youtube FS1mnISoG7U >}}

## 4. Building Careers with Empathy with Scott Hanselman
{{< youtube 6eSpILu7wpg >}}

## 5. It's Important to be a Full Stack Developer | Scott Hanselman
{{< youtube Bx17ywo-5nM >}}

## 6. If I knew then what I know now - Scott Hanselman
{{< youtube YI34UIMgkxs >}}

## 7. Kelsey Hightower fireside chat at DevRelCon San Francisco
{{< youtube wCwLOHMtkpo >}}

## 8. RailsConf 2019 - Opening Keynote by David Heinemeier Hansson
{{< youtube VBwWbFpkltg >}}

## 9. Managers are from Mars, Developers are from Venus by Ted Neward
{{< youtube 2lXuYHQmjKk >}}

## 10. Do not walk away from Complexity, Run - Venkat Subramaniam
{{< youtube nZcLHkORdHE >}}

## 11. Continued Learning: The Beauty of Maintenance - Kent Beck
{{< youtube 3gib0hKYjB0 >}}

## 12. "Good Enough" Architecture • Stefan Tilkov
{{< youtube PzEox3szeRc >}}

## 13. "Simple Made Easy" - Rich Hickey
{{< youtube LKtk3HCgTa8 >}}

## 14. TDD, Where Did It All Go Wrong (Ian Cooper)
{{< youtube EZ05e7EMOLM >}}

## 15. Functional Programming in 40 Minutes • Russ Olsen
{{< youtube 0if71HOyVjY >}}

## 16. Jessica Kerr "Functional Principles for Object Oriented Development"
{{< youtube pMGY9ViIGNU >}}

## 17. Rob Pike, "Public Static Void"
{{< youtube 5kj5ApnhPAE >}}

## 18. The Why of Go
{{< youtube bmZNaUcwBt4 >}}

## 19. OAuth 2.0 and OpenID Connect (in plain English)
{{< youtube 996OiexHze0 >}}

## 20. What the heck is the event loop anyway? | Philip Roberts
{{< youtube 8aGhZQkoFbQ >}}

What are your favourite talks?

# Announcing My "Spring Boot Tips Video Series" on YouTube


### TLDR;
I am happy to announce that I am going to create **"Spring Boot Tips Video Series"** on my [SivaLabs YouTube Channel](https://www.youtube.com/c/SivaLabs).
This **"Spring Boot Tips Series"** is to help people who are new to [Spring Boot](https://spring.io) to learn **how to use Spring Boot** 
with a keen focus on **understanding how it works behind the scenes**.

<!--more-->


### Longer version
I have been working with Spring Boot for several years, and I have written a couple of books on it too.
Also, I have been giving online training on Spring Boot for students and working professionals too.
A common thing I noticed is for the people who are completely new to Spring Boot, it looks like a magic.
Even people, who have been working with Spring Boot for some time, know how to use it but don't know how it works behind the scenes.

It is quite understandable though. Spring Boot is kind of framework on top of frameworks 
abstracting away too much from developer to increase developer productivity.

Recently I gave Spring Boot training to a group of people, and I noticed the same pattern.

**Few questions from the students:**

> How do you know for which library dependency we need to include `<version>` in pom.xml and for which library we don't have to?

> Why shouldn't I use @SpringBootTest for my unit tests?

> There are so many options to configure application properties. Which one to use? Are there any best practices around it?

> How Spring Boot AutoConfiguration works? Can we create our own starters as well?

> How AOP works? What are some practical usages of AOP?

and many more...

As I have written a few articles on these topics earlier, I shared those articles.

* [How Spring Boot AutoConfiguration magic works?](https://www.sivalabs.in/how-springboot-autoconfiguration-magic/)
* [Testing Spring Boot Applications](https://www.sivalabs.in/spring-boot-testing/)
* [Spring Boot Best Practices](https://www.sivalabs.in/spring-boot-best-practices/)
* [Retrying Method Execution using Spring AOP](https://www.sivalabs.in/retrying-method-execution-using-spring-aop/)

They found these articles very useful and a couple of students asked me why don't you create videos explaining things like this?
I liked that idea of creating such videos because that would help me to simply ask them to watch those videos instead of me explaining the same thing again and again.
So, I decided to make this "Spring Boot Tips Video Series".

### Who are the target audience?
* People who are new to Spring Boot
* People who are using Spring Boot for some time but don't know how it works behind the scenes

If you are already a Spring Boot Expert then you might not learn much from this series, 
but if you can spend some time to watch and provide me feedback that would be great.

### Spring Boot Tips : Part 1 - DependencyManagement
In this first part of "Spring Boot Tips" Series, I have explained how dependency management works in Spring Boot Maven Project.

{{< youtube 2dPon1G5S-M >}}

### Like & Share
If you like to watch this series please subscribe and if you think any of your friends can benefit from this series please share it.

# My attempt to understand why people perceive Java as complex


I work as an Architect/TechLead/SeniorDeveloper depending on client engagement. 
And, **Java** is the primary programming language that I have used most of my career.
In recent years I got an opportunity to work with **Go**, **NodeJS** and **.Net Core** technologies too.
I find Java, especially with **Spring Boot**, a solid platform to build enterprise grade applications.

<!--more-->


Occasionally there comes a discussion of technology selection, and I noticed some people don't prefer/like Java.
Also, on social media I see a lot of jokes on Java's complexity. 
While I enjoy the humour out of it, it makes me wonder why so many people perceive Java as complex??!!
An annoying thing is some people who just started their careers and never worked with Java at all also have a strong opinion that "Java is complex"..

Some experienced developers also have the opinion that "Java is complex" based on their prior 2004 timeframe experience.
It is understandable though. How many of us who worked with JSF 1.x pledged to never touch a JSF project again in our lifetime??!!
Even though JSF 2 is lot better, still some people don't like to try JSF again. I think same is happening for "Java" as a whole.

But, **it would be a mistake just ruling out it is all because of past bad experience and not understanding why some people see Java as complex**.
Maybe there are problems which we are not seeing them as problems because we are habituated to it so much that we don't see them as problems anymore.
Also, there may be better programming language/platforms which we are not realizing.
It would be good to hear from the people so that we can make better decisions while picking tech stack for next project. 

So I asked this on Twitter:

{{< x user="sivalabs" id="1509905489911762950" >}}

I got plenty of replies from newbies, experts and some industry legends as well.

The interesting thing is along with criticism on Java issues, there is a lot of appreciation to Java platform as well.

I tried to categorize all those replies and added my view on it. 

So, here it goes :-)


## 1. Poor Getting Started Experience
As a developer with more than 15+ years of Java development experience I know my way around Java ecosystem.
I know which flavor of JDK I want to install, how to manage multiple JDK versions, what IDEs are available, 
build tools, the commonly used libraries etc.

Let's take a look at how a newbie getting started with Java experience looks like:

* I want to install Java and my initial google search makes me land on https://www.oracle.com/java/technologies/downloads/
* I am not sure whether Oracle JDK is free to use or commercial one, I want to use free open-source one.
* Upon exploring further I learned there are free alternatives like **OpenJDK, AdoptOpenJDK, Amazon Correto, Microsoft OpenJDK, Zulu, Temurin**. 
  What are the differences among those variations? God knows, anyway let me go with OpenJDK.
* Installed OpenJDK, set JAVA_HOME and added JAVA_HOME/bin to PATH, and I am good to go.
* Opened VS Code, created a .java file, copy-pasted infamous HelloWorld program and did **javac HelloWorld.java** and **java HelloWorld** ... voila...
* Let me write a test. Google suggested to use **JUnit**.
* How to use JUnit library? Again upon a bit of googling most of the articles showed using JUnit with **Maven** or **Gradle**.
* What is Maven/Gradle? Okay, they are build tools. Let me install Maven/Gradle, done..cool.
* How to create a Maven/Gradle project? Most of the tutorials are showing project creation using IntelliJ IDEA or Eclipse. So now I need to install an IDE.
* After creating a Maven/Gradle project from the IDE it seems downloading half of the internet for my HelloWorld project...okay no problem..I have good internet connection.

It's not as easy as it can be, but it's not so much terrible either. Also, many of these tasks are one time setup.
However, there are some uncertainties, and gives an impression of Java needs complex setup and can't work with Java without heavy IDE setup.

Compare this to "Getting started experience" of a JavaScript/NodeJS developer.
* Install NodeJS based on your OS
* Build stuff
```shell
$ npm init --yes
$ npm install express
$ npm start
$ npm test
$ npm build
```

NodeJS comes with a built-in default package-manager **npm**, a way to manage 3rd party dependencies. 
If I am not happy with **npm** then I can go for **yarn** or **pnpm**, but I got the toolchain to perform common tasks out of the box.

This is way easier getting started experience compared to Java.

### How Java Pros do it?
Luckily there is much easier way to get started with Java using [SDKMAN](https://sdkman.io/).

```shell
$ curl -s "https://get.sdkman.io" | bash
$ source "$HOME/.sdkman/bin/sdkman-init.sh"
$ sdk list java
$ sdk install java 17-open
$ sdk install maven
$ sdk install gradle
$ sdk install jbang
$ gradle init <- For gradle project
$ mvn archetype:generate -DgroupId=com.mycompany -DartifactId=my-app -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false <- for creating maven project
```

SDKMAN also provide support for installing various other tools like **Maven**, **Gradle**, **JBang** etc.
If you want to play with Java for learning purpose I would recommend using **[JBang](https://www.jbang.dev/)**.

While most of the Java developers use full-blown powerful IDES like **Intellij IDEA** or **Eclipse** or **NetBeans**, you can work with Java using **VS Code** too.

## 2. Core Standard Library is not enough
Another common complaint was Java's Core Standard Library is not enough and many tasks need some external library, and it's true.

* There is no built-in testing library, no built-in JSON (un)marshalling support, etc.
* While Java NIO is a big improvement, still File I/O is complex and verbose in Java compared to other languages.
* No built-in (Kotlin like) support for immutable data structures

### My view
* Java has good Standard Library such as Collections, Streams, Java NIO etc. 
* File I/O, JDBC etc still feels very low-level and verbose
* There are good 3rd party libraries like **commons-lang, commons-io, Jackson, Gson, Vavr, Eclipse Collections** to perform these tasks
* Adding a 3rd party library doesn't look like a big problem to me.
* The challenge is when there are too many choices it confuses newbies which one is right to pick.

## 3. Java is Verbose
This is the most common remark on Java that **Java is the most verbose language**.  
This simply reveals that you haven't yet worked with Go language :-)

### My view
Yes, people hate setters, getters etc but I don't see it as a deal-breaker. There is **Lombok** and **Records** to avoid writing boilerplate.
By using right libraries it is possible to write not-too-verbose Java code. 

In fact, I strongly feel Java has the right balance of verbosity vs conciseness.
Usually Java based enterprise applications are maintained for years and what is considered as "verbose" is what actually helping people to understand it.
I heard stories from my friends who inherited Scala/NodeJS codebases developed 3 years ago, and they have stories to tell about verbosity vs conciseness.

One of the reply says it all:
> I'll take legacy Java over legacy Node.js any day.

## 4. Complex "enterprisy" stuff
Another common theme was complex enterprise stuff like **ClassLoaders, JNDI, OSGI, JMX, Reflection, AOP, ByteWeaving, Dynamic Proxies, application servers** etc.

### My view
A decade ago I worked with EJB 2 with ear packaging, and I know how frustrating to deal with ClassLoader issues.
Also, I had first-hand experience dealing with class loader issues while deploying a Spring application in Application Server like JBoss/WildFly.

Luckily I am not working with EJBs, OSGI, JMX, application servers in the last 7 or 8 years.
In the modern Java world the applications are being built as fat jars and the frameworks are taking care of doing AOP, ByteWeaving, Dynamic Proxies etc.

Unless you are still working on legacy applications, I don't think you will be dealing with these issues with modern Java tech stack. 

## 5. Complexity for Library authors
The power users, library authors deal with whole new level of complexity compared to typical business application developers.

Some challenges that are mentioned with Java for advanced users are:

* Class loading, 
* Type erasure, 
* Exception semantics for functional interfaces
* Reflection
* Primitives
* Nulls
* Default mutability
* Multi-threading
* Universal equality

I can't say much about this because I haven't built any complex libraries.

In addition to these few more concerns are:
* High memory footprint for microservices
* [Streaming arch layered on an aging ecosystem w poorly understood ramifications](https://stackoverflow.com/questions/21163108/custom-thread-pool-in-java-8-parallel-stream)

## 6. Too much abstraction
This is unarguably the most frustrating part for newbies. 
While the modern frameworks make it easy to build applications for those who know what is happening behind the scenes, it is all black magic for newbies. 
Most of the frameworks are abstractions on top of abstractions on top of abstractions. 
When they try to learn a little more about something it is like peeling an onion.

## 7. Poor/Non-native FP Support
While Java adds some functional programming support from Java 8 onwards, Java is definitely not designed as Functional Programming language at its core.
Some people mentioned they can build software using FP in much better way than using OOP. 
You can apply some FP concepts like Pure functions, Immutability in Java as well, but it won't be as natural as in FP languages like Clojure or Haskell.

## 8. Miscellaneous reasons
* Higher chance of working with Legacy applications
* Bad interview experiences

The reason I asked "why you think Java is complex?" is to understand others point of view, and I got more visibility on why some people feel so.
While I don't agree with few things and some are not a deal-breaker for me, it definitely helped me to understand some genuine concerns.

IMO, any language/framework/library is created to solve one primary problem and along the way it may get some more non-primary features as well.
Java is designed as Object-Oriented language, and it will not be as good as Clojure or Haskell if you want it to be a Functional Programming Language.
Each programming language has it's niche areas like Python for Machine Learning, Go/Rust for Systems Programming, Java/NodeJS for enterprise application development etc.

Java's niche is in building large enterprise applications, not to print "Hello World!!!" in the console.
The Java ecosystem grew addressing the enterprise needs, and improving developer productivity. 

In a typical enterprise business application you need to build complex business processes by integrating with several other external systems, 
you need to interact with relational databases, NoSQL databases, messaging systems, cache providers, Cloud Services etc.

Show me a more productive platform to build such enterprise applications rapidly? My money is on **Java/Spring Boot/Quarkus/Micronaut**.

Yes, you need to learn/know a shitload of things to make effective use of such rapid application development platform. 
That is the price you may need to pay to achieve that productivity.

But, that is not the only approach. You can go in another direction of picking the only-needed libraries and stitch them together by yourself.
Add cross-cutting logic all over the place, add monitoring capabilities, write cloud service adapters by yourself. 
This may result in good light-weight system that requires minimal cognitive load for developers.

# Summary
Don't simply discard any programming language/platform just because there are "Language X is complex" memes/jokes on the internet.
Don't just keep repeating "Language X is complex" because you heard someone else saying it based on their 2004 timeframe experience.
Try it yourself, you may like it or may not. It's worth taking a look at old things to know if they are still bad, or if they improved a lot over the years.

# Code simplicity by Abstraction vs Verbosity

Java is and has been the primary programming language I have used throughout my career.
If I want to build something quick for a prototype or if I need to pick the tech stack with a tight deadline then Java is my first choice.
Especially after Java 8 it becomes more and more feature rich and powerful. 

<!--more-->


On top of it with frameworks like [Spring Boot](https://spring.io/projects/spring-boot), [Quarkus](https://quarkus.io/) and [Micronaut](https://micronaut.io/) building enterprise grade applications becomes a breeze.
**When people from Non-Java community complain about Java being verbose and complex, I actually don't get it.**

In the recent years I got a chance to work with [Go](https://go.dev/) and [DotNet Core](https://dotnet.microsoft.com/en-us/) in my official projects.
In addition to that I worked with **NodeJS** and **Python** a bit for my personal use.

I understand why many people like **NodeJS** and **Python** because they are kind of **Batteries Included**.
For the same task in Java you might need to include external libraries whereas in Python/NodeJS the standard library itself provide most of the commonly used features in a non-verbose way.
If you are familiar with **Kotlin** then you feel right at home while working with **C#**. 

Go is an interesting language and I have shared [my opinion on Go](https://www.sivalabs.in/golang-from-a-java-developer-perspective/) a while ago when I first worked with it.

**My first impression of Go**

{{< figure src="/images/java-is-verbose.webp" >}}

Have you ever seen a movie and doesn't like it much but a couple of scenes are top-notch which makes you watch the same movie again just to see those awesome scenes?
I had that kind of experience with Go. I didn't like it much when I first used it, but few things about Go stuck with me. One of it being Simplicity.

Okay, let's take a look at the following Java/Spring Boot code.

```java
@Service
@Transactional
@RequiredArgsConstructor
public class CustomerService {
    private final CustomerRepository customerRepository;
    private final AddressRepository addressRepository;

    public void createCustomer(Customer customer) {
        customerRepository.save(customer);
        addressRepository.saveCustomerAddress(customer.getAddress());
    }
}
```

This snippet of code shouldn't need any explanation for any Java developer who used Spring Boot at-least once.
But there is a lot going on here.

* The **CustomerService** is marked as a Spring bean which can be injected into other Spring beans
* Since Spring 4.x no need to add **@Autowired** annotation on constructor. If there is only one constructor it will be automatically used to create the bean.
* The class is annotated with **@Transactional** which means when someone calls **createCustomer()** method a DB transactions will be started and committed upon successful completion.
* If **customerRepository.save()** or **addressRepository.saveCustomerAddress()** method throws any **RuntimeException** then transaction will be automatically rolled-back.
* If **CustomerService.createCustomer()** is being called from another transactional method from a different class then instead of starting a new transaction it will run in the parent transaction.

While this looks familiar and easy to read, there are lots of things happening under the hood.

> To fully understand how this simple method behave, one needs to have all the above-mentioned **"How Spring & Spring Boot works behind the scenes"** knowledge.

Here I am showing Spring Boot code as an example, but it is the same for Quarkus or Micronaut frameworks also.

Let's take a look at the following Go code which does same as above:

```golang
type CustomerService struct {
   customerRepository CustomerRepository
   addressRepository AddressRepository
}

func (c CustomerService) createCustomer(customer Customer) error {
    // Get a Tx for making transaction requests.
    tx, err := db.BeginTx(ctx, nil)
    if err != nil {
        return err
    }
    // Defer a rollback in case anything fails.
    defer tx.Rollback()
    
    err := customerRepository.save(tx, customer)
    if err != nil {
        return err
    }
    err = addressRepository.saveCustomerAddress(tx, customer.Address)
    if err != nil {
        return err
    }
    // Commit the transaction.
    if err = tx.Commit(); err != nil {
        return err
    }
    return nil
}
```

Well, this is a typical Go implementation of the same functionality we implemented above.
Assuming it does what it intends to do, at a first glance this is what I feel:

* It's a lot verbose with error check for every 3rd statement
* DB transaction handling is all over the place, and we may need to repeat the same for all the DB interacting methods.
* Business logic and low-level technical details(error/transaction handling) are tangled

> But, there is no hidden-magic. A 20+ years experienced dev can read and understand this code in the same as 
a six-month experienced grad can read and understand it.

**I know, I know...You are shouting at me "there is no magic...it is magic only if you don't understand how it works"**

This point leads me to the next point I want to discuss.
But first let's not jump to the conclusions of which language is better because that is not the intention of this article.

## What is the biggest challenge developers facing during software development?
We all know that we spend more time reading existing code than writing new code. 
In a typical long-term enterprise application development, existing developers leave and new developers join the project and the biggest challenge is making sense of existing code. If the code is simple then it is easy to understand. 

**But quantifying "code simplicity" is not easy.** What looks simpler to you, might look complex to me and vice-versa. 
So we(developers and community) try our best to find ways to write code as simple as possible.
Different communities take different approach based on the context, timing, need etc.

Here is my opinion on how Java and Go take a different approach to attain code simplicity.

### Java's approach : Abstractions
I feel Java's standard library is too low-level and verbose to perform many common tasks like reading a file, making an HTTP call etc.
To avoid the boilerplate and verbosity, libraries like **commons-lang**, **Guava** etc are created providing higher-level abstractions shielding all the low-level details from developer.

On top of it, many enterprise application needs a common set of features like **logging**, **configuration**, **monitoring** etc. 
So frameworks are created to address these common needs abstracting away the common application level boilerplate code.

> Over the time there are abstractions on top of abstractions on top of abstractions...

Well, those abstractions help you build the application faster by taking care of application-level boilerplate code and letting you focusing on business logic.

> The only expectation from the developers is "Understand how those abstractions work under the hood and be productive with the frameworks/libraries" 

**In my view this is where things are going wrong terribly.** 

Now every company wants fullstack developers who can write front-end, back-end and a bit of infra-as-code too.
There is no time to learn properly how all these abstractions work under the hood. 
To meet the delivery timelines developers want to get things done as quickly as possible.
Add these 2 annotations over there and these 4 lines of configuration over here and if it is working then it's done. 
If encountered any error, apply the first suggestion from stackoverflow and if it works then it's done.
The work is done but without any clue on how it actually works. 

Over the time this mode of working puts a fear in the head saying "Java is complex".

### Go's approach : Clarity/Verbose over Clever
The Go community prefer to keep code simple by means of "no-magic" which means no annotation, no reflection, no ORMs etc.
There are good number of libraries for ORM, Http Routers etc but Go community prefers "If at all possible stick with Standard Library" approach.

Go community seems to prefer writing more clear and verbose code than creating clever higher abstractions.
The best part is with Go's standard library you can achieve most of the application needs without needing external dependencies, of course with more lines of code.

> The trade-off for verbosity is "Anybody with a bit of Go familiarity can easily read any idiomatic Go code and understand it"

If you are a Go admirer by now you might get pissed-off reading "verbosity" again and again and ask what do you mean by verbosity?
Well, the error checks for almost every 3rd line, the way we can check if a map contains a key etc all looks verbose to me. Personal opinion.

Based on my limited experience with Go, I am liking it so much because of it's "boring" nature.
To learn Go I am reading many articles and open source codebases on GitHub and there aren't too many new things to learn.

**After seeing few Go code repositories I was like 'meh, the next one will be similar with same core features with different business logic'.
From the enterprise long-term application development point of view this is an awesome thing. Less magic to learn and easy to onboard new developers.**

## How can we make Java "Simpler"?
Java is the preferred choice in large enterprises for a reason. Speed of development and quick to market.
I can confidently say you can create production grade Java application using Spring Boot/Quarkus/Micronaut in few hours, not in weeks or months.
As these frameworks already provides the application skeleton with batteries included all you need to do is write business logic.

If I can take few ideas from Go and brought them to Java, specifically to Spring Boot, then those would be:

* No ORMs unless it is simple CRUD application. I prefer to spend the one-time cost of writing more code than recurring debugging cost. I would prefer [JOOQ](https://www.jooq.org/) over **JPA**.
* I know backwards compatibility is one of the key reasons for Java/Spring Boot's success. But I like to strip a few things from Spring Boot
  * Support for XML config, I don't see any greenfield projects using XML config. XML support could be extracted as a separate module.
  * No need for [loading config properties from 14 different sources](https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.external-config)
  * Some simpler ways to use and debug AOP ( if you ever tried to debug AOP code then you know what I am talking about)
  * **Quarkus** like **live-reloading** and **continuous testing**
* Do we really need **"com.mycompany.myproject.mymodule"** packaging structure for private business applications? For opensource libraries yes, but for business applications??

## Summary
Once you start working with multiple languages you may start questioning the status quo. 
You may look at things a bit differently and bring back some good habits to your beloved programming language.
I think "Code Simplicity" is an undervalued trait of software development and, we need to put more focus on code readability and simplicity.

If you are a Java developer who never tried Go, I highly recommend taking a look at Go. 
You may or may not like Go entirely but there will be few things you will fall in love with.

Having said all this, **"Java is and will always be my favourite language"** and it feels good to learn things from other languages.

Wish you all a very happy new year :-)

# How I dealt with burn out?


A month ago I was constantly feeling stressed out, tired and not getting proper sleep. To put it simply I got burned out.
The interesting thing is it is not because of my official work pressure. 
Currently, I am working with a cool client, we are delivering the features on time and the team is friendly...I have nothing to complain about my official work.
So I thought (and tweeted) it might be because of the things I do in addition to my regular job.

<!--more-->


{{< x user="sivalabs" id="1455547804114296834" >}}

Apart from my regular job, I do teaching, blogging and also always learn something new every day.
As I was feeling stressed regularly I realised something is going wrong, and I decided to take a step back and fix this before it gets worse.

As a first step I did 2 things to think clearly.

1. Stop learning anything new that is not required for my official work
2. Stay away from Twitter (more on why staying away from twitter helped me later)


## How the problem started?
Usually I prefer to be prepared for my upcoming work. So if I know that in my next project we are going to use **GoLang** I try to learn **GoLang** even before project starts.
We all know we can't learn everything and be ready for any future task. But I try to learn as much as I can.

And this is what I have been following since the beginning of my career. In a typical 8-hour work day I spent minimum 3 hours for learning... every day.
Because of this continuous learning I was able to complete my official work on time and has plenty of time to learn more...and the cycle continued...

This approach of futuristic learning helped me in several ways:

1. I could learn new topics with a free mind without any deadline
2. As I already know how to implement a feature, when I pick up the task I was able to focus more on how to do it in a better way
3. I could give reasonable estimates

But the problem with this approach is **I never know "how much is good enough to say I know this framework/library"**.
If I have a task at hand then after finishing it I can feel that accomplishment because I was able to complete that task.
But in this futuristic learning process there is no such a milestone to feel that kind of accomplishment. 

No matter how much I learn, the days are still ending with a feeling of **"there is more to learn"**.

Couple of months ago I got really excited about Kubernetes and I learned a bit about it and some of commonly used k8s tools.
Kubernetes, [Pods, Deployments, Services, Ingress](https://www.sivalabs.in/getting-started-with-kubernetes/), [Minikube, K3S, KinD, Skaffold](https://github.com/sivaprasadreddy/geeksclub-microservices-spring-boot/tree/main/k8s), KNative, OpenShift, Minishift, CodeReady Containers, GitOps, [ArgoCD](https://github.com/sivaprasadreddy/geeksclub-gitops) ...and the rabbit hole goes on and on...
In addition to that I got curious about [Quarkus](https://github.com/sivaprasadreddy/geeksclub-microservices-quarkus), [OAuth using Keycloak](https://github.com/sivaprasadreddy/spring-boot-keycloak), Tanzu Community Edition etc. 

I learned a great deal about these topics, but everyday I go to bed with a feeling of **I am not sure whether I learned enough of these and there could be more to learn**.
But how can I validate whether I learned **"good enough"** of these technologies???

**Then I got a bright idea:**

To build an application with a bunch of **Microservices** and **React/Vue** based SPA UI, secure them with **OAuth Keycloak**, create infrastructure using **AWS CDK** or **Pulumi** or **Terraform** and deploy them on **AWS** or **Azure**. 
Then setup **CI/CD infrastructure** using **Kubernetes**, **Jenkins**/**Github Actions** and **ArgoCD**.

But the problem is to run all these things I need better hardware. I have one MacBook Pro with 16GB RAM, another Mac with 8GB RAM and one Lenovo Laptop with 8GB RAM. 
I thought of buying a beefy laptop to do all these stuff.

{{< x user="sivalabs" id="1451184999525339142" >}}

Sounds like a good plan and I am ready for the adventure.

The next day I was having my breakfast and out of nowhere I got the feeling that **"What the fuck I am doing!!! Why I am trying to do all the things which is supposed to be doing by an entire IT department?? What I am trying to prove and to whom??"**

I don't know exactly what is wrong, but I realized I am doing something terribly wrong.

## The Root Cause Analysis
Once I realized I am doing something wrong, I wisely took the decision of taking a step back and unplug myself with these crazy ideas and think through it.

**1. Fear Of Missing Out (FOMO)**: Not everyone gets to work on latest and greatest technologies all the times. When you see everyone around you talking how they scaled their app using the latest technologies it infuses a bit of fear of getting outdated. So, to compensate the lack of opportunity to work with those cool technologies I was thinking of doing all those crazy things.

**2. Not knowing how much learning is "good enough"**: As I mentioned above, my futuristic learning process doesn't tell me how much is good enough and still leaving me with the thought of "there is more to know".

**3. Social Media**: While Social Media is helpful in many ways it can also add more stress to you. There are 16-year old front-end devs lecturing you **"how to live a better life"**, the so-called **"your REST API is not really REST API"** or **"You are doing DDD wrong"** backend gatekeepers. Looking at these people you may start wondering **"Is everybody figured out how to live life and do software engineering except me??"**

{{< figure src="/images/diversions-simple.webp" >}}

## The Remedy
Once I understood what is going on, I took few steps before things getting worse.

* I just stopped learned anything new that is not required for my official job
* I took a break from Twitter which is the only social media app that is adding more pressure to me

Guess what? With these 2 simple steps I got 4 extra free hours in a day :-) I spent the time with my son and listening to my favorite music and sleeping more.
I also watched my favorite shows (FRIENDS and Game of Thrones) one more time. 

**What about my futuristic learning which helped me throughout my career?**

I just needed to make a small change in the process. Instead of having an open-ended goal like **"I will learn OAuth today"** I have more specific goal like **"Today I will learn OAuth using Client Credentials"**.
With this small change, once I complete my learning task for that day I stop and end the day with a satisfaction of **"I learned OAuth using Client Credentials today"**.

## Summary
The fun fact of life is **"You most likely won't discover anything ground-breaking new. You may re-discover the true meaning of what you already read in your 6th grade".**

We all studied the phrase **"Health is wealth"** from our childhood days itself. But we forget it and trap into the rat-race. 
While working crazy hours to achieve your career goals looks great, but not at the cost of health. Having balance is more important. 

There is more to life. Take a break and go to places. **The worst thing that could happen to you is having pocket full of money but not enough energy to spend it.**

# Kubernetes - Exposing Services to outside of Cluster using Ingress


In the previous article [Kubernetes - Blue/Green Deployments]({{< relref "2021-09-14-kubernetes-blue-green-deployments.md" >}}) we have learned how to release a new version of application using **Blue/Green Deployments strategy**.

<!--more-->


* [1. Getting Started with Kubernetes]({{< relref "2021-09-01-getting-started-with-kubernetes.md" >}})
* [2. Kubernetes - Releasing a new version of the application using Deployment Rolling Updates]({{< relref "2021-09-07-kubernetes-deployment-rolling-updates.md" >}})
* [3. Kubernetes - Blue/Green Deployments]({{< relref "2021-09-14-kubernetes-blue-green-deployments.md" >}})
* [4. Kubernetes - Exposing Services to outside of Cluster using Ingress]({{< relref "2021-09-22-kubernetes-ingress.md" >}})

In this article, we are going to learn how to use **Ingress** to expose HTTP and HTTPS routes from outside the cluster to services within the cluster.

So far in this Kubernetes article series we have seen how to create **Deployments** and expose them as **Services** and access them in different ways based on Service type.
If it is a **ClusterIP** type Service then we ssh into Minikube and access the application. If it is a **NodePort** type Service then we accessed the application using Node IP (Minikube IP) and node port. 

But we don't expect the end-users to remember IP addresses and port numbers to access our services, right? We use easy to remember domain names such as **google.com**, **stackoverflow.com** etc which are mapped to IP Addresses. 

In Kubernetes world we can use **Ingress** to route the traffic based on various rules.

Let's say we have 2 versions of our sample application k8s-boot-demo, and we want to run both versions, and we want to make them available as follows:

* http://k8sdemo.com/v1/ should provide access to version v1
* http://k8sdemo.com/v2/ should provide access to version v2
* http://api.k8sdemo-v1.com/ should provide access to version v1
* http://api.k8sdemo-v2.com/ should provide access to version v2

Here we need to map same domain name with different path prefixes to different services. 
Also, we need to map two different domain names to different services.

Let us see how can use Ingress to achieving this requirement.

### 1. Configure Domain names in hosts file

As we don't own those domain names let us configure them locally in our hosts file (/etc/hosts) for local testing.
Get the Minikube IP and map the domain names to that IP.

```shell
$ minikube ip
192.168.99.103
```

**/etc/hosts**

```shell
255.255.255.255 broadcasthost
::1 localhost

192.168.99.103 k8sdemo.com
192.168.99.103 api.k8sdemo-v1.com
192.168.99.103 api.k8sdemo-v2.com
```

### 2. Deploy Ingress Controller on Kubernetes Cluster
Unlike other controllers Kubernetes doesn't come with a Ingress Controller pre-installed.
We need to select the Ingress Controller that is supported by the Cloud platform and depending on our choice/needs.
Please see https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/ to get a list of supported Ingress Controllers.

As we are using Minikube and minikube already comes with Nginx Ingress Controller as an Addon, we just need to enable it.

```shell
$ minikube addons enable ingress
```

You can learn more about Nginx Controller at https://kubernetes.github.io/ingress-nginx/

### 3. Deploy services

We are going to deploy the same service we built and used in previous articles. We will deploy both versions v1 and v2.
They can be same application or different application, doesn't matter.

Create k8s-boot-demo-v1 manifest file **app-v1.yaml** as follows: 

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-boot-demo-v1-deployment
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: k8s-boot-demo-v1
  template:
    metadata:
      labels:
        app: k8s-boot-demo-v1
    spec:
      containers:
        - name: k8s-boot-demo-v1
          image: sivaprasadreddy/k8s-boot-demo:v1
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              port: 8080
              path: /actuator/health/readiness
          livenessProbe:
            httpGet:
              port: 8080
              path: /actuator/health/liveness
          startupProbe:
            httpGet:
              port: 8080
              path: /actuator/health/readiness
            initialDelaySeconds: 5

---
apiVersion: v1
kind: Service
metadata:
  name: k8s-boot-demo-v1-service
spec:
  type: ClusterIP
  selector:
    app: k8s-boot-demo-v1
  ports:
    - name: app-port-mapping
      protocol: TCP
      port: 8080
      targetPort: 8080
```

Create k8s-boot-demo-v2 manifest file **app-v2.yaml** as follows:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-boot-demo-v2-deployment
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: k8s-boot-demo-v2
  template:
    metadata:
      labels:
        app: k8s-boot-demo-v2
    spec:
      containers:
        - name: k8s-boot-demo-v2
          image: sivaprasadreddy/k8s-boot-demo:v2
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              port: 8080
              path: /actuator/health/readiness
          livenessProbe:
            httpGet:
              port: 8080
              path: /actuator/health/liveness
          startupProbe:
            httpGet:
              port: 8080
              path: /actuator/health/readiness
            initialDelaySeconds: 5

---
apiVersion: v1
kind: Service
metadata:
  name: k8s-boot-demo-v2-service
spec:
  type: ClusterIP
  selector:
    app: k8s-boot-demo-v2
  ports:
    - name: app-port-mapping
      protocol: TCP
      port: 8080
      targetPort: 8080
```

Deploy our services as follows:

```shell
$ kubectl apply -f app-v1.yaml
$ kubectl apply -f app-v2.yaml
$ kubectl get pods
```

### 4. Deploy Ingress Resources
Finally, the key part where we configure the rules mapping the domain names to Kubernetes services.

**ingress.yaml**

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: k8s-boot-demo-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
    -  host: "k8sdemo.com"
       http:
         paths:
           - pathType: Prefix
             path: "/v1"
             backend:
               service:
                 name: k8s-boot-demo-v1-service
                 port:
                   number: 8080
           - pathType: Prefix
             path: "/v2"
             backend:
               service:
                 name: k8s-boot-demo-v2-service
                 port:
                   number: 8080

    - host: "api.k8sdemo-v1.com"
      http:
        paths:
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: k8s-boot-demo-v1-service
                port:
                  number: 8080

    - host: "api.k8sdemo-v2.com"
      http:
        paths:
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: k8s-boot-demo-v2-service
                port:
                  number: 8080
```

**Explanation of Ingress manifest:**

* We have configured **k8sdemo.com** host and used multiple paths configurations with different path prefixes to map to different services.
* We have configured **api.k8sdemo-v1.com** host and map it to Service version v1
* We have configured **api.k8sdemo-v2.com** host and map it to Service version v2

You can use suffixes and wildcards to fine tune the mappings, for more details see https://kubernetes.io/docs/concepts/services-networking/ingress/.

Also, different Ingress Controllers use different configurations to configure its inner workings. 
We have used **nginx.ingress.kubernetes.io/rewrite-target: /** annotation to [map to Target URI where the traffic must be redirected.](https://github.com/kubernetes/ingress-nginx/blob/main/docs/examples/rewrite/README.md)

Finally, we can test our services by using domain names as follows:

```shell
curl http://k8sdemo.com:80/v1/
{"app":"K8S Spring Boot Demo","version":"v1","hostName":"k8s-boot-demo-v1-deployment-657df6b8df-jqgj7"}
curl http://k8sdemo.com:80/v2/
{"version":"v2","app":"K8S Spring Boot Demo","hostName":"k8s-boot-demo-v2-deployment-55b6dfd4d4-69sgz"}
curl http://api.k8sdemo-v1.com:80/
{"app":"K8S Spring Boot Demo","version":"v1","hostName":"k8s-boot-demo-v1-deployment-657df6b8df-jqgj7"}
curl http://api.k8sdemo-v2.com:80/
{"version":"v2","app":"K8S Spring Boot Demo","hostName":"k8s-boot-demo-v2-deployment-55b6dfd4d4-69sgz"}
```

## Summary

We have seen how to use Ingress to expose the Services to outside of Kubernetes Cluster.
As we have used Minikube for local development and it already has Nginx Controller support as an Addon it is relatively easy to setup/enable Ingress.

But in production setup usually Kubernetes Administrators will configure the Ingress Controller based on the Cloud Platform and available choices.
We can use wildcards also to configure the mapping instead of listing down each host name to Service mapping.
# Kubernetes - Blue/Green Deployments


In the previous article [Kubernetes - Releasing a new version of the application using Deployment Rolling Updates]({{< relref "2021-09-07-kubernetes-deployment-rolling-updates.md" >}}) we have learned how to release a new version of application using **Deployment Rolling Updates** and how to use **Readiness** and **Liveness** probes to route traffic to only fully initialized containers.

<!--more-->


* [1. Getting Started with Kubernetes]({{< relref "2021-09-01-getting-started-with-kubernetes.md" >}})
* [2. Kubernetes - Releasing a new version of the application using Deployment Rolling Updates]({{< relref "2021-09-07-kubernetes-deployment-rolling-updates.md" >}})
* [3. Kubernetes - Blue/Green Deployments]({{< relref "2021-09-14-kubernetes-blue-green-deployments.md" >}})
* [4. Kubernetes - Exposing Services to outside of Cluster using Ingress]({{< relref "2021-09-22-kubernetes-ingress.md" >}})

In this article, we are going to learn:
* What are the **problems with Kubernetes Rolling Updates**?
* What is **Blue/Green Deployments strategy**?
* How to **release a new version of application using Blue/Green Deployments strategy**?

## 1. What are the problems with Kubernetes Rolling Updates?
We have seen how we can use RollingUpdate strategy which is supported by Kubernetes out-of-the-box to deploy a new version of the application.
But it poses some problems on the application architecture and design such as:

* Application should be able to gracefully handle both versions in compatible way
* The application database changes should be backward compatible
* If the application has upstream/downstream service integrations then the data exchange formats should be backward compatible

While it is generally a good idea to keep new versions backward compatible we also need to keep in mind the cost of it.
You may have to add feature toggles and additional configuration to handle both versions at the same time which definitely increase the complexity and tech debt.

So, instead of immediately rolling out the new version by replacing the old version how about deploying the new version along side current version and test it thoroughly for a while and once we are confident with new version changes redirect the application traffic to new version? That is exactly what Blue/Green Deployment Strategy suggests.  

## 2. What is Blue/Green Deployments strategy?

Blue/Green deployment is a technique that reduces downtime and complexity by running two separate environments called Blue(current version) and Green(new version). At any moment of time only one of the environments is live and serves all the traffic. Once all the testing is done in Green(new version) environment then Green will be made as the Live environment and Blue becomes the Stand-By environment.


## 3. How to release a new version of application using Blue/Green Deployments strategy?
Unlike RollingUpdate strategy, Kubernetes doesn't have out-of-the-box support for Blue/Green Deployment Strategy.
However, achieving Blue/Green Deployments with Kubernetes is very easy using **labels**.

Let's say we have deployed our application with version v1 using the following Kubernetes manifest and is live now.

**k8s/deployment-blue.yaml**

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-boot-demo-deployment-blue
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: k8s-boot-demo
      version: v1
      color: blue
  template:
    metadata:
      labels:
        app: k8s-boot-demo
        version: v1
        color: blue
    spec:
      containers:
        - name: k8s-boot-demo
          image: sivaprasadreddy/k8s-boot-demo:v1
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
```

* You can notice that our pod is associated with 3 labels **app: k8s-boot-demo, version: v1, color: blue**.
* The deployment is also configured with **selector.matchLabels** using same labels **app: k8s-boot-demo, version: v1, color: blue**.

Now let's create a Service to expose our version v1 deployment.

**k8s/service-live.yaml**

```yaml
apiVersion: v1
kind: Service
metadata:
  name: k8s-boot-demo-service
spec:
  type: NodePort
  selector:
    app: k8s-boot-demo
    version: v1
  ports:
    - name: app-port-mapping
      protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 30090
```

* Nothing much new here and the important thing to notice here is the **selector** labels. Only **app: k8s-boot-demo, version: v1** labels are associated with Service but not **color**.

Now we can deploy our resources and access them via NodePort as follows:

```shell
$ kubectl apply -f k8s/deployment-blue.yaml
$ kubectl apply -f k8s/service-live.yaml
$ minikube ip
192.168.99.103
$ curl 192.168.99.103:30090/api/info
{"version":"v1","app":"K8S Spring Boot Demo","hostName":"k8s-boot-demo-deployment-blue-7459fc4bd8-ptmxx"}
```

So far so good, we have our Blue environment up and running. 

Now let's deploy our new version v2 in a **Green** environment.

**k8s/deployment-green.yaml**

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-boot-demo-deployment-green
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: k8s-boot-demo
      version: v2
      color: green
  template:
    metadata:
      labels:
        app: k8s-boot-demo
        version: v2
        color: green
    spec:
      containers:
        - name: k8s-boot-demo
          image: sivaprasadreddy/k8s-boot-demo:v2
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
```

* You can notice that our pod is associated with 3 labels **app: k8s-boot-demo, version: v2, color: green**.
* The deployment is also configured with **selector.matchLabels** using same labels **app: k8s-boot-demo, version: v2, color: green**.

Now let's create a Service to expose our version v2 deployment.

**k8s/service-preprod.yaml**

```yaml
apiVersion: v1
kind: Service
metadata:
  name: k8s-boot-demo-service-preprod
spec:
  type: NodePort
  selector:
    app: k8s-boot-demo
    version: v2
  ports:
    - name: app-port-mapping
      protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 30092
```

The important thing to note here is we are creating a separate service named **k8s-boot-demo-service-preprod** and with selector labels **app: k8s-boot-demo, version: v2** and with **nodePort: 30092**.

```shell
$ kubectl apply -f k8s/deployment-green.yaml
$ kubectl apply -f k8s/service-preprod.yaml
$ kubectl get all
$ minikube ip
192.168.99.103
$ curl 192.168.99.103:30090/api/info
{"hostName":"k8s-boot-demo-deployment-blue-7459fc4bd8-rk8kt","app":"K8S Spring Boot Demo","version":"v1"}
$ curl 192.168.99.103:30092/api/info
{"version":"v2","app":"K8S Spring Boot Demo","hostName":"k8s-boot-demo-deployment-green-d7b94fdc5-5xxgw"}
```

Now our green environment is also up and running.

We are able to access and test our new version and as everything looks fine we want to make our green environment as Live.
All we need to do is to update label **version: v1** to **version: v2** in **k8s/service-live.yaml** and apply changes.

**k8s/service-live.yaml**

```yaml
apiVersion: v1
kind: Service
metadata:
  name: k8s-boot-demo-service
spec:
  type: NodePort
  selector:
    app: k8s-boot-demo
    version: v2
  ports:
    - name: app-port-mapping
      protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 30090
```

```shell
$ kubectl apply -f k8s/service-live.yaml
$ minikube ip
192.168.99.103
$ curl 192.168.99.103:30090/api/info
{"version":"v2","app":"K8S Spring Boot Demo","hostName":"k8s-boot-demo-deployment-green-d7b94fdc5-bhv9c"}
$ curl 192.168.99.103:30092/api/info
{"version":"v2","app":"K8S Spring Boot Demo","hostName":"k8s-boot-demo-deployment-green-d7b94fdc5-bhv9c"}
```

That's it. Now the service is pointing to the pods running version v2 of our application. And, immediately any requests to live Service will be pointing to new version.

## Summary
In this article we have learned how to release a new version using Blue/Green Deployments strategy to reduce the complexity of having to support both old and new version of the application at the same time.

So far we have been using a single node Minikube cluster and exposing the service using NodePort so that we can access the application using Minikube IP Address and NodePort. But in reality we will have multiple worker nodes and using NodePort is not a feasible approach. To overcome this issue we can use Ingress where we can access our services using a domain name instead of node IP addresses and port numbers. We will learn how to use Ingress in next article.

# Kubernetes - Releasing a new version of the application using Deployment Rolling Updates


In the previous article [Getting Started with Kubernetes]({{< relref "2021-09-01-getting-started-with-kubernetes.md" >}}) we have learned how to deploy a Spring Boot application on Kubernetes(Minikube) using **Deployment** and **Service** primitives. We also looked at how to access our application using **NodePort** on Minikube IP address.

<!--more-->


* [1. Getting Started with Kubernetes]({{< relref "2021-09-01-getting-started-with-kubernetes.md" >}})
* [2. Kubernetes - Releasing a new version of the application using Deployment Rolling Updates]({{< relref "2021-09-07-kubernetes-deployment-rolling-updates.md" >}})
* [3. Kubernetes - Blue/Green Deployments]({{< relref "2021-09-14-kubernetes-blue-green-deployments.md" >}})
* [4. Kubernetes - Exposing Services to outside of Cluster using Ingress]({{< relref "2021-09-22-kubernetes-ingress.md" >}})

In this article, we are going to learn:
* How to release a new version of application using **Deployment Rolling Updates**?
* How to use **Readiness** and **Liveness** probes to route traffic to only fully initialized containers?

## 1. Build new version of the application
If you look at the previous article Spring Boot application code we have used **@Value** to bind the **app.version** property and hard coded app title **"K8S Spring Boot Demo"**.
Let's update the code to use **@ConfigurationProperties** instead of **@Value** and make **app.title** also a configurable parameter.

**ApplicationProperties.java**

```java
package com.sivalabs.k8sbootdemo;

import lombok.Getter;
import lombok.Setter;
import org.springframework.boot.context.properties.ConfigurationProperties;

@ConfigurationProperties(prefix = "app")
@Setter
@Getter
public class ApplicationProperties {
    private String version;
    private String title;
}
```

**K8sBootDemoApplication.java**

```java
package com.sivalabs.k8sbootdemo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.context.properties.EnableConfigurationProperties;

@SpringBootApplication
@EnableConfigurationProperties(ApplicationProperties.class)
public class K8sBootDemoApplication {

	public static void main(String[] args) {
		SpringApplication.run(K8sBootDemoApplication.class, args);
	}
}
```

**src/main/resources/application.properties**

```shell
spring.application.name=k8s-boot-demo
management.endpoints.web.exposure.include=*
app.version=v2
app.title=K8S Spring Boot Demo
```

**AppInfo.java**

```java
package com.sivalabs.k8sbootdemo;

import com.fasterxml.jackson.annotation.JsonPropertyOrder;
import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;

@Setter
@Getter
@NoArgsConstructor
@AllArgsConstructor
@JsonPropertyOrder({"version", "app", "hostName"})
public class AppInfo {
    private String version;
    private String app;
    private String hostName;
}
```

**DemoController.java**
```java
package com.sivalabs.k8sbootdemo;

import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

import java.time.LocalDateTime;

@RestController
@RequiredArgsConstructor
@Slf4j
public class DemoController {
    private final ApplicationProperties properties;

    @GetMapping({"", "/api/info"})
    public AppInfo apiInfo()
    {
        log.info("Request for apiInfo at : {}", LocalDateTime.now());
        return new AppInfo(properties.getVersion(), properties.getTitle(), getHostname());
    }
    ...
    ...
}
```

Finally, in **pom.xml** update **<docker.tag>v1</docker.tag>** to **<docker.tag>v2</docker.tag>**, build and push the new image with **v2** tag to DockerHub using following command:

```shell
$ ./mvnw clean package jib:build
```

## 2. Deploy application of version v1 

First let us deploy the current v1 version of the application using Deployment and Service primitives as follows:

**app.yaml**

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-boot-demo-deployment
spec:
  replicas: 6
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: k8s-boot-demo
  template:
    metadata:
      labels:
        app: k8s-boot-demo
    spec:
      containers:
        - name: k8s-boot-demo
          image: sivaprasadreddy/k8s-boot-demo:v1
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: k8s-boot-demo-service
spec:
  type: NodePort
  selector:
    app: k8s-boot-demo
  ports:
    - name: app-port-mapping
      protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 30090
```

**Explanation of Deployment Strategy**:

* We are using **RollingUpdate** strategy (which is default) for our Deployment. The **RollingUpdate** strategy creates the new pods with new container images and then terminate the old pods. 
* **There won't be complete downtime** as there will always be pods (old and new) accepting requests
* **For some time both old and new pods will be running parallelly** so application needs to be able to handle it
* Another Deployment strategy option is **Recreate** which will delete all the existing pods first and then create new pods, so there will be application downtime.

Let's deploy the application using following command and access the application:

```shell
$ kubectl apply -f app.yaml
$ kubectl get all
$ minikube ip
192.168.99.103
$ curl 192.168.99.103:30090/api/info
{"version":"v1","app":"K8S Spring Boot Demo","hostName":"k8s-boot-demo-deployment-8595cc898b-lznhf"}
```

{{< figure src="/images/k8s-p2-1.webp" >}}

## 3. Deploy new version v2 using Deployment Rolling Updates

As our new version v2 image is already pushed to DockerHub, all we need to do is update **app.yaml** to use **sivaprasadreddy/k8s-boot-demo:v2** image.
But before that let's open one terminal window and run the CURL command to continuously call the API endpoint as follows:

```shell
$ while true; do curl 192.168.99.103:30090/api/info; sleep 1; done;
```
{{< figure src="/images/k8s-p2-2.webp" >}}

Now, update **app.yaml** to use **sivaprasadreddy/k8s-boot-demo:v2** as image and apply the changes in a new terminal window.

```shell
$ kubectl apply -f app.yaml
$ kubectl get all
```

{{< figure src="/images/k8s-p2-3.webp" >}}

You can see the old pods being Terminated and new pods getting created.

Now if you take a look at the first Terminal window where we are calling API continuously you can see some errors as follows:

{{< figure src="/images/k8s-p2-4.webp" >}}

For some calls we are getting response successfully and for some it is failing with **curl: (7) Failed to connect to 192.168.99.103 port 30090: Connection refused** error.

The reason is when we are calling API using Service NodePort Kubernetes is forwarding the requests to all the pods with label **app: k8s-boot-demo** without checking whether the application is fully initialized and ready to accept the requests or not.

We can use **Readiness and Liveness probes** to overcome this problem.

## 4. Using Readiness and Liveness probes

The **Readiness** state indicates whether the application is ready to accept client requests or not. If the Readiness state is unready, Kubernetes will not route traffic to this instance(pod).

The **Liveness** state of an application indicates whether the internal state is valid or not. If Liveness is broken, this means that the application itself is in a failed state and cannot recover from it.
In this case, Kubernetes will restart the application instance.

You can read more about Readiness and Liveness probes here:
* [Liveness and Readiness Probes with Spring Boot](https://spring.io/blog/2020/03/25/liveness-and-readiness-probes-with-spring-boot)
* [Configure Liveness, Readiness and Startup Probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)

Spring Boot Actuator provides the support for Readiness and Liveness probes out-of-the-box.
**Spring Boot can automatically detect that the application is running in a Kubernetes environment and enable the probes.**

But if you are running the application locally you need to enable the probes explicitly using following configuration:

```shell
management.endpoint.health.probes.enabled=true
```

You can check the Readiness and Liveness of our Spring Boot application by accessing 
[http://192.168.99.103:30090/actuator/health/readiness](http://192.168.99.103:30090/actuator/health/readiness) and [http://192.168.99.103:30090/actuator/health/liveness](http://192.168.99.103:30090/actuator/health/liveness) and you will see the response as follows:

```json
{
  "status": "UP"
}
```

Now we can configure Kubernetes Liveness and Readiness probes using these Actuator endpoints as follows.

```yaml {hl_lines=["7-8","23-30"]}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-boot-demo-deployment
spec:
  replicas: 6
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: k8s-boot-demo
  template:
    metadata:
      labels:
        app: k8s-boot-demo
    spec:
      containers:
        - name: k8s-boot-demo
          image: sivaprasadreddy/k8s-boot-demo:v2
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              port: 8080
              path: /actuator/health/readiness
          livenessProbe:
            httpGet:
              port: 8080
              path: /actuator/health/liveness
```

We have updated the Pod definition within the Deployment to include **readinessProbe** and **livenessProbe** fields pointing to Readiness and Liveness HTTP endpoints.

Now let's test this by deleting the existing deployment, deploy version v1 and then update to v2.

```shell
$ kubectl delete deployment.apps/k8s-boot-demo-deployment
$ vi app.yaml //update version to v1
$ kubectl apply -f app.yaml
$ while true; do curl 192.168.99.103:30090/api/info; sleep 1; done;
```

In a 2nd Terminal window update version to v2 in app.yaml and apply the changes.

```shell
$ vi app.yaml //update version to v2
$ kubectl apply -f app.yaml
```

Now if you check in the first terminal you won't see **curl: (7) Failed to connect to 192.168.99.103 port 30090: Connection refused** errors anymore.

In addition to **readinessProbe** and **livenessProbe**, Kubernetes also support **startupProbe.initialDelaySeconds** property which can be used to delay the probes checking for certain period. This comes handy if your application takes some time to fully initialize.

```yaml {hl_lines=["16-17"]}
spec:
  containers:
    - name: k8s-boot-demo
      image: sivaprasadreddy/k8s-boot-demo:v2
      imagePullPolicy: Always
      ports:
        - containerPort: 8080
      readinessProbe:
        httpGet:
          port: 8080
          path: /actuator/health/readiness
      livenessProbe:
        httpGet:
          port: 8080
          path: /actuator/health/liveness
      startupProbe:
        initialDelaySeconds: 5
        httpGet:
          port: 8080
          path: /actuator/health/liveness
```

We have learned how to deploy a new version using **RollingUpdate** strategy. What if there is an issue with new version and we want to rollback to previous version?

## 5. Rolling back to previous version

We can check the deployment rollout history as follows:

```shell
$ kubectl rollout history deployment.apps/k8s-boot-demo-deployment

deployment.apps/k8s-boot-demo-deployment
REVISION  CHANGE-CAUSE
1         <none>
2         <none>
```

We can see there are 2 revisions of this deployment. We can see more details about a particular deployment revision as follows:

```shell
$ kubectl rollout history deployment.apps/k8s-boot-demo-deployment --revision=2

deployment.apps/k8s-boot-demo-deployment with revision #2
Pod Template:
  Labels:	app=k8s-boot-demo
	pod-template-hash=64d545b968
  Containers:
   k8s-boot-demo:
    Image:	sivaprasadreddy/k8s-boot-demo:v2
    Port:	8080/TCP
    Host Port:	0/TCP
    Liveness:	http-get http://:8080/actuator/health/liveness delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:	http-get http://:8080/actuator/health/readiness delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:	<none>
    Mounts:	<none>
  Volumes:	<none>
```

Now we can rollback to previous version i.e, v1 as follows:

```shell
$ kubectl rollout undo deployment.apps/k8s-boot-demo-deployment
```

Or you can rollback to a specific revision as follows:

```shell
$ kubectl rollout undo deployment.apps/k8s-boot-demo-deployment --to-revision=1
```

## Summary
While using Rollout strategy for Deployments is easy, it expects the application to work properly even when both the old and new versions are running parallelly.
This might be a big challenge when it involves database schema changes, external service integrations etc. 

We can use **Blue/Green Deployment strategy** to handle this problem. We will see how we can perform Blue/Green deployments in the next article.


# Getting Started with Kubernetes


[Docker](https://www.docker.com/) and [Kubernetes](https://kubernetes.io/) has revolutionized how we build and run the applications. 
In recent years containerization becomes new normal, and many organizations start using Kubernetes as Container Orchestration Platform.

<!--more-->


I was curious about Kubernetes and [played with it a bit 3 years ago](https://github.com/sivaprasadreddy/spring-boot-tutorials/tree/master/spring-boot-k8s-demo) but I thought let's wait and see is it going to be a real thing or just another over hyped technology. Now I see Kubernetes everywhere, especially where microservice architecture is used for their systems. So, I finally decided to learn it properly and share my learning along the way.

* [1. Getting Started with Kubernetes]({{< relref "2021-09-01-getting-started-with-kubernetes.md" >}})
* [2. Kubernetes - Releasing a new version of the application using Deployment Rolling Updates]({{< relref "2021-09-07-kubernetes-deployment-rolling-updates.md" >}})
* [3. Kubernetes - Blue/Green Deployments]({{< relref "2021-09-14-kubernetes-blue-green-deployments.md" >}})
* [4. Kubernetes - Exposing Services to outside of Cluster using Ingress]({{< relref "2021-09-22-kubernetes-ingress.md" >}})

In this article we will learn:

* Creating a docker image from a Spring Boot application
* Understanding the Need for Kubernetes
* Local kubernetes setup using Minikube
* Overview of Kubernetes Objects
* Run the Spring Boot app in a Pod
* Scaling the application using Deployment
* Exposing the Deployment as a Service

## 1. Creating a docker image from a Spring Boot application
We are going to build a simple Spring Boot application which we are going to use for this tutorial.
Go to [Spring Initializr](https://start.spring.io/) and select **Web, Actuator and Lombok** starters and generate the application.
You can click [Here](https://start.spring.io/#!type=maven-project&language=java&platformVersion=2.5.4&packaging=jar&jvmVersion=11&groupId=com.sivalabs&artifactId=k8s-boot-demo&name=k8s-boot-demo&description=Demo%20project%20for%20Spring%20Boot&packageName=com.sivalabs.k8sbootdemo&dependencies=lombok,web,actuator) to generate the application with desired starters already selected.

Our Spring Boot application is going to be a very simple one as our goal here is to learn Kubernetes, not exploring Spring Boot's super powers.

Create a REST Controller as follows:

```java
package com.sivalabs.k8sbootdemo;

import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

import java.net.InetAddress;
import java.time.LocalDateTime;
import java.util.Map;

@RestController
@Slf4j
public class DemoController {

    @Value("${app.version}")
    private String version;

    @GetMapping({"", "/api/info"})
    public Map<String, String> apiInfo()
    {
        log.info("Request for apiInfo at : {}", LocalDateTime.now());
        return Map.of("app", "K8S Spring Boot Demo",
                "hostName", getHostname(),
                "version", version);
    }

    @GetMapping("/api/terminate")
    public String terminate()
    {
        log.info("Request for terminate at : {}", LocalDateTime.now());
        System.exit(1); //give the developer a medal for writing this line in a web app
        return "I'll be BACK";
    }

    private String getHostname() {
        String hostName = "Unknown";
        try {
            InetAddress addr = InetAddress.getLocalHost();
            hostName = addr.getHostName();
        } catch (Exception e) {
            log.error("Error in getting hostname", e);
        }
        return hostName;
    }
}
```

Configure the following properties in **src/main/resources/application.properties**

```shell
spring.application.name=k8s-boot-demo
management.endpoints.web.exposure.include=*
app.version=v1
```

We can use **Spring Boot Maven Plugin** itself to build the Docker image or we can use **Jib plugin**.
Let's use [Jib Maven plugin](https://github.com/GoogleContainerTools/jib/tree/master/jib-maven-plugin) to build the Docker Image and push it to DockerHub.

Add the following plugin in pom.xml

```xml
<properties>
    <java.version>11</java.version>
    <dockerhub.username>sivaprasadreddy</dockerhub.username>
    <docker.tag>v1</docker.tag>
</properties>
<build>
  <plugins>
    ...
    ...
    <plugin>
        <groupId>com.google.cloud.tools</groupId>
        <artifactId>jib-maven-plugin</artifactId>
        <version>2.7.0</version>
        <configuration>
            <from>
                <image>gcr.io/distroless/java:11</image>
            </from>
            <to>
                <image>${dockerhub.username}/${project.artifactId}:${docker.tag}</image>
            </to>
            <container>
                <ports>
                    <port>8080</port>
                </ports>
            </container>
        </configuration>
    </plugin>
  <plugins>
</build>
```

Before pushing the image to DockerHub, first you need to authenticate with DockerHub.

```shell
$ docker login
$ ./mvnw clean package jib:build
```

Now the docker image will be created and pushed to your DockerHub registry.

You can quickly verify it using the following command:

```shell
$ docker run -p 8080:8080 sivaprasadreddy/k8s-boot-demo:v1 
```

Now we should be able to access http://localhost:8080/api/info and see the following JSON response:

```json
{
    "version": "v1",
    "app": "K8S Spring Boot Demo",
    "hostName": "61490b6bb05d"
}
```

## 2. Understanding the Need for Kubernetes

In the previous step we were able to build a docker image from our Spring Boot application and also run it using Docker.
We have already verified that it is working fine by invoking an API endpoint.

Let's call another endpoint that we created `http://localhost:8080/api/terminate`

We won't see any response and if we check the console you can notice that the container exited.
If we look at the code of `/api/terminate` API handler method we are calling `System.exit(1)` which is killing the container.

> Yes, that's a poor man's simulation of container crashed due to some problems like `OutOfMemory` etc... :wink:

Here the problem is once the container is crashed our application is dead. It won't restart itself or spin up another container automatically without human intervention. While this is fine for a demo, but for running in production this is not acceptable.

**In production environment we want to make sure always the application is up and running and also we want to run N instances to serve the high traffic. This is where Container Orchestration tools like Kubernetes come into picture.**

We can instruct Kubernetes to run 5 instances of our application and Kubernetes will take care of running 5 instances all the time.
If one container got crashed due to any issue then Kubernetes takes care of spinning up another container to make sure 5 containers are running as instructed.

That's enough of theory, let's get our hands on Kubernetes.

## 3. Local kubernetes setup using Minikube

There are few options to run a Kubernetes Cluster locally such as [Minikube](https://minikube.sigs.k8s.io/docs/start/), [Kind](https://kind.sigs.k8s.io/), [K3s](https://k3s.io/) etc. I felt **Minikube** is the easiest way to get up and running compared to others.

### Install Minikube
Let's install Minikube which can be used to create a single node kubernetes cluster locally.
You can checkout the [Minikube Docs](https://minikube.sigs.k8s.io/docs/start/) and follow the installation steps based on your OS.
For MacOS you can simply install using **brew install minikube**

Once minikube is installed create a cluster using the following command:

```shell
$ minikube start --memory 4096 --driver=virtualbox
```
### Install kubectl
We need to install **kubectl** commandline tool which will be used to communicate with the kubernetes cluster.
Follow the instructions at https://kubernetes.io/docs/tasks/tools/ to install kubectl based on your OS.

For MacOS you can simply install using **brew install kubectl**

Verify the installation:

```shell
$ kubectl version
$ minikube version
```

Now that we have the required tools installed, before jumping onto running containers we need to understand about few Kubernetes Objects.

## 4. Overview of Kubernetes Objects

There are many Kubernetes Objects that we commonly use such as **Pod, Deployment, Service, ConfigMap, Secrets, Persistent Volumes, Jobs, CronJobs, ServiceAccount** etc. Instead of dumping everything at once on your head we will explore them as and when required.

* **Pod:** A Pod is the smallest deployable unit which encapsulates one or more containers. Most of the times there will be only one container in a pod but there are cases where we need to run multiple containers in a single pod. 

* **Deployment:** A Deployment takes care of maintaining the desired state such as "there should be 3 replicas of this pod". Deployment uses ReplicaSets to scale up or down based on the desired state. Deployments can also be used to perform rolling updates, Blue/Green Deployments etc.

* **Service:** A Service exposes the deployment by providing a single interface (IP Address) to a set of Pods effectively working as a LoadBalancer.

Knowing about these 3 Object types is enough to start with and we will learn about other Object types along the way.

## 5. Run the Spring Boot app in a Pod

There are 2 ways we can deploy kubernetes objects:

1. **Imperative way** using **kubernetes run** or **kubernetes create** commands.
2. **Declarative way** using **YAML Manifest** files

While imperative way comes handy to run adhoc commands to explore things, Declarative way is more maintainable in real projects. So we are going to follow Declarative way using YAML files.

**k8s/pod.yml**

```yaml
kind: Pod
apiVersion: v1
metadata:
  name: k8s-boot-demo-pod
spec:
  containers:
    - name: k8s-boot-demo
      image: sivaprasadreddy/k8s-boot-demo:v1
      imagePullPolicy: Always
      ports:
        - containerPort: 8080
```

**Explanation of Pod definition:**

* We are defining the object type as **Pod** using **kind** property
* We are giving a name to the pod in metadata section. We can also define the namespace and label etc which are optional.
* We are defining the container details as part of spec, for now we have only one container.

Let's deploy the pod on Minikube cluster:

```shell
$ kubectl apply -f k8s/pod.yaml
$ kubectl get pods -o wide
```

{{< figure src="/images/k8s-p1-1.webp" >}}


```shell
$ kubectl describe pods k8s-boot-demo-pod // to see more details of the pod
$ kubectl logs k8s-boot-demo-pod //to view logs of the pod
```

We are able to see our pod deployed successfully and see all the pod details using **kubectl describe** command, and we can even see our Spring Boot application logs.

But we can't invoke any API endpoint as Pod is running within the container only and it is not accessible from outside the cluster.

Well, there is a way. We can ssh into minikube and from there we can invoke the API.

```shell
$ minikube ssh
$ curl 172.17.0.5:8080/api/info
```

{{< figure src="/images/k8s-p1-2.webp"  >}}

Cool, we are able to call our API Endpoint. Let's try to call **/api/terminate** endpoint.

```shell
$ minikube ssh
$ curl 172.17.0.5:8080/api/terminate
```

When the **/api/terminate** endpoint is called container got crashed and Kubernetes restarted the container as the default **restartPolicy** is **Always**. Now if you see the pod details it is in **Running** state and **RESTARTS** is **1**. 

Let's update **restartPolicy** to **Never** and see what happens when a container crashed.

**k8s/pod.yml**

```yaml
kind: Pod
apiVersion: v1
metadata:
  name: k8s-boot-demo-pod
spec:
  restartPolicy: Never
  containers:
    - name: k8s-boot-demo
      image: sivaprasadreddy/k8s-boot-demo:v1
      imagePullPolicy: Always
      ports:
        - containerPort: 8080
```

Only certain fields are allowed to update for an existing pod and **restartPolicy** is not one of them. 
So let's first delete the pod and recreate it.

```shell
$ kubectl delete -f k8s/pod.yaml
$ kubectl apply -f k8s/pod.yaml
$ kubectl get pod k8s-boot-demo-pod -o wide
```

Now if we ssh into minikube and invoke **/api/terminate** endpoint and see the pod status it will show as **Error** and the spring boot application is no longer running.

We learned how to run a pod and how it behaves when a container got crashed depending on **restartPolicy** setting.

Now, let us see how we can scale up or scale down the number of pods using **Deployments**.

## 6. Scaling the application using Deployment

We can use Kubernetes Deployment to make sure N replicas of pods available all the time. In addition to that Deployments can also be used to perform **rolling updates, Blue/Green deployments** etc. But in this article we will just see how we can use Deployments to run 3 replicas of Spring Boot application.

**k8s/deployment.yaml**

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-boot-demo-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: k8s-boot-demo
  template:
    metadata:
      labels:
        app: k8s-boot-demo
    spec:
      containers:
        - name: k8s-boot-demo
          image: sivaprasadreddy/k8s-boot-demo:v1
          ports:
            - containerPort: 8080
```

**Explanation of the Deployment Manifest:**

* We are defining the object type as **Deployment** using kind property
* If you notice **spec.template** child section of **Deployment** it is same as Pod definition with a **label** added.
* The **restartPolicy** property of Pod is removed because for a Deployment the only valid value is **Always**.
* The association between Deployment and Pods are through matching labels. The pad has label **app: k8s-boot-demo** 
 and Deployment's **spec.selector.matchLabels** has same label **app: k8s-boot-demo**.
* We have specified the number of replicas we want to have using **spec.replicas** field.

```shell
$ kubectl delete -f k8s/pod.yaml
$ kubectl apply -f k8s/deployment.yaml
$ kubectl get all
```

{{< figure src="/images/k8s-p1-3.webp"  >}}

We can see **3 Pods, 1 ReplicaSet and 1 Deployment** got created. 

Now let us delete a Pod and see what happens.

```shell
$ kubectl delete pod k8s-boot-demo-deployment-5cd5446684-d7h82
$ kubectl get all
```

Now you can notice a new a pod got created automatically to match the current state (2 pods) to the desired state(3 pods). 

We can easily scale up and down either changing the replicas in **deployment.yaml** or using the **kubectl scale** command.

```shell
$ kubectl scale deployment k8s-boot-demo-deployment --replicas=5
$ kubectl get all
```

Let us see what happens if a Pod crashed. As we did earlier get one Pod's IP address, ssh into minikube and call /api/terminate endpoint.

You should see the pod automatically restarted to match the desired replica count.

We are able to scale up or down using Deployments, but we are still calling APIs on specific pods.

In the next section we will see how to use Kubernetes **Services** which act as a **LoadBalancer** and provide a way to call our APIs from outside the cluster.

## 7. Exposing the Deployment as a Service

Let's create a Service definition using YAML file as follows:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: k8s-boot-demo-service
spec:
  type: NodePort
  selector:
    app: k8s-boot-demo
  ports:
    - name: app-port-mapping
      protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 30090
```

**Explanation of the Service Manifest:**

* We are defining the object type as **Service** using **kind** property
* The association between Service and Pods are through matching labels. The pad has label **app: k8s-boot-demo**
  and Service's **spec.selector** has same label **app: k8s-boot-demo**. So, this Service will act as a load balancer to all the pods which has label **app: k8s-boot-demo**. 
* We have specified the type as **NodePort** which exposes the port **30090** on Cluster Node and forward the requests to Pod's 8080 port which in turn forwards to Container's 8080 port.

```shell
$ kubectl apply -f k8s/deployment.yaml
$ kubectl apply -f k8s/service.yaml
$ kubectl get all
```

{{< figure src="/images/k8s-p1-4.webp"  >}}

We can access the service using minikube IP on nodePort 30090.

```shell
$ minikube ip
192.168.99.103
$ curl 192.168.99.103:30090/api/info
$ while true; do curl 192.168.99.103:30090/api/info; sleep 1; echo "";  done;
```

You can see requests are serving from different pods by checking the **hostName** in the response.

{{< figure src="/images/k8s-p1-5.webp"  >}}

## Tips
* Use Kubectl autocomplete https://kubernetes.io/docs/reference/kubectl/cheatsheet/#kubectl-autocomplete
* Instead of writing Manifest YAML files from scratch you can generate them as follows:

```shell
$ kubectl run k8s-boot-demo --image=sivaprasadreddy/k8s-boot-demo:v1 --dry-run=client -o yaml
```
This will print the following Pod definition yaml in console.

```yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: k8s-boot-demo
  name: k8s-boot-demo
spec:
  containers:
  - image: sivaprasadreddy/k8s-boot-demo:v1
    name: k8s-boot-demo
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {} 
```
Using **--dry-run=client -0 yaml** we can generate the YAML without actually creating such resource.

Similarly, we can generate manifest YAML for Deployment as well.

```shell
$ kubectl create deployment k8s-boot-demo-deployment --image=sivaprasadreddy/k8s-boot-demo:v1 --dry-run=client -o yaml > deployment.yaml
```
This will write the following Deployment manifest YAM to **deployment.yaml** file.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: k8s-boot-demo-deployment
  name: k8s-boot-demo-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: k8s-boot-demo-deployment
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: k8s-boot-demo-deployment
    spec:
      containers:
      - image: sivaprasadreddy/k8s-boot-demo:v1
        name: k8s-boot-demo
        resources: {}
status: {}
```
* You can use [Lens](https://k8slens.dev/) as a GUI client for Kubernetes
* If you are terminal person then [k9s](https://github.com/derailed/k9s) or [KDash](https://github.com/kdash-rs/kdash) comes very handy

## Summary
We have covered some basics of getting started with Kubernetes and learn how to use **Pods, Deployments and Services**.
But we just scratched the surface, there are plenty of things to know. Though we are able to call our API using Service NodePort, in a multi-node cluster environment we might want to use **Ingress** mapping a domain name to a services deployed on multiple nodes.

In the next part we will see how to release a new version of the application using **Deployment Rolling Updates**.
# Get Super Productive with Intellij File Templates


[Intellij IDEA](https://www.jetbrains.com/idea/) is the most popular IDE for **Java** development and there are tons of features to explore and become more productive.
We all know about [Live Templates](https://www.jetbrains.com/help/idea/using-live-templates.html) which can be used to expand a small expression to common code snippets.

Recently I came to know about **File Templates**, and I realized I can put it to good use and be more productive.

<!--more-->


First let us discuss the problem I was trying to solve with File Templates.

I work with **Spring Boot** a lot, and I work on implementing REST APIs in most of my projects.

So how do I create a REST Controller?

* Create a Class, say **CustomerController**
* Add **@RestController** to make a Spring REST Controller
* Add **@RequestMapping("/api")** so that common prefix need not be repeated for every handler method in that controller
* Add Lombok's **@RequiredArgsConstructor** and **@Slf4j** to generate a Constructor with all the final fields and to have SLF4J logger.

This is pretty standard pattern that I follow, but it is too much manual typing.
I always wished if there is an option to create a Spring Rest Controller directly similar to how we create a Class.

Intellij IDEA File Templates can help to solve this problem.

## Creating Single File Templates

Let's create a Single File Template to create Spring REST Controller.

* Go to Preferences -> Editor -> File and Code Templates
* On Files tab click on + (Create Template)
* Enter Name as : **Spring REST Controller**, and leave Extension as **java**
* In the Content text box add the following content

```java
#if (${PACKAGE_NAME} && ${PACKAGE_NAME} != "")package ${PACKAGE_NAME};#end

import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;

import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/api")
@RequiredArgsConstructor
@Slf4j
public class ${NAME} {

}

```

* Click OK

Nothing magical here, the first line is a **Velocity** template expression.
We are checking whether you are creating this class in a package or in default package then adding package statement if required.

* **${PACKAGE_NAME}** represents the package name in which you are creating this class.
* **${NAME}** represents the class name you are going to provide.

There are many more [Predefined template variables](https://www.jetbrains.com/help/idea/file-template-variables.html#predefined_template_variables) if you want to use.

Now if we go to create a new Class we can see a new option **Spring REST Controller**.

{{< figure src="/images/spring-rest-controller-ft.webp" height="450" width="480" >}}

If I give the class name as **ProductController** then class will be created with following content.

```java
package com.sivalabs.demo;

import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;

import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/api")
@RequiredArgsConstructor
@Slf4j
public class ProductController {

}
```

Wooohoooo!!!

Why to just stop at Controller alone, let's create template for Spring Data JPA Repositories and JPA Entity as well.

### Spring Data JPA Repository Template
Usually I name Spring Data JPA Repositories as **&lt;EntityName&gt;Repository**. Let's create a template for that.

```java
#if (${PACKAGE_NAME} && ${PACKAGE_NAME} != "")package ${PACKAGE_NAME};#end
#set ($index = $NAME.indexOf('Repository'))
#set ($ENTITY= $NAME.substring(0, $index))

import org.springframework.data.jpa.repository.JpaRepository;

public interface ${NAME} extends JpaRepository<$ENTITY, Long> {
}
```

Except those **#set** directives everything else looks same only.
Those **#set** statements are Velocity directives, and we are defining variables here.
We are stripping of 'Repository' from the interface name and storing it in $ENTITY variable and using it as generic argument for **JpaRepository**.

### JPA entity Template
Let's create a bare minimum JPA Entity structure with Lombok's **@Setter** and **@Getter**

```java
#if (${PACKAGE_NAME} && ${PACKAGE_NAME} != "")package ${PACKAGE_NAME};#end

import lombok.Getter;
import lombok.Setter;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.persistence.Table;

@Setter
@Getter
@Entity
@Table(name = "${NAME}")
public class ${NAME} {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

}
```

Cool, now whenever you try to create a class you see these 3 options:

{{< figure src="/images/multiple-single-fts.webp" height="450" width="480" >}}

This is already looks nice, but **how about creating all Controller, Service, Repository and Entity in one go**?
Fantastic, isn't it?  Intellij IDEA also support Templates with multiple files.

## Templates with multiple files
We can create a parent template and add child templates to it so that when you generate a class with parent template it will generate all child template classes as well.

Let's first define what is the requirement for us.

I want to generate API boilerplate code by just giving the JPA Entity name. 
Then IDE should generate the following:

* JPA entity class **some.parent.package.entities.MyEntity**
* Spring Data JPA Repository **some.parent.package.repositories.MyEntityRepository**
* Spring Service class **some.parent.package.services.MyEntityService**  
* Rest Controller **some.parent.package.controllers.MyEntityController**

### 1. Create a Parent Template

Create a Parent Template and give name as **Generate Boilerplate For Entity** and add the following in the content area.

```java
#if (${PACKAGE_NAME} && ${PACKAGE_NAME} != "")package ${PACKAGE_NAME};#end

#set( $CamelCaseName = "$NAME.substring(0,1).toLowerCase()$NAME.substring(1)" )

class Dummy$NAME{}
```

> It seems there is a limitation that Parent template should generate a class, otherwise it is failing. So generating a Dummy$NAME class.

We are setting template variable **$CamelCaseName** which is a camelCased value of entity.

Ex: If entity name is **UserAccount** then **$CamelCaseName** value is **userAccount**.

### 2. Create Child Template for JPA Entity

* Select the parent template and click on **Create Child Template File** button
* Enter File name as **entities/${NAME}**
* Enter Content as follows

```java
#if (${PACKAGE_NAME} && ${PACKAGE_NAME} != "")package ${PACKAGE_NAME};#end

import lombok.Getter;
import lombok.Setter;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.persistence.Table;

@Setter
@Getter
@Entity
@Table(name = "${NAME}")
public class ${NAME} {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;
    
}
```

We have given the File name as **entities/${NAME}** so that entity will be created in **entities** sub-package under the selected package.

### 2. Create Child Template for Spring Data JPA Repository

* Select the parent template and click on Create Child Template File button
* Enter File name as **repositories/${NAME}Repository**
* Enter Content as follows

```java
#if (${PACKAGE_NAME} && ${PACKAGE_NAME} != "")package ${PACKAGE_NAME};#end

import org.springframework.data.jpa.repository.JpaRepository;
import ${PACKAGE_NAME}.entities.${NAME};

public interface ${NAME}Repository extends JpaRepository<${NAME}, Long> {
}
```


### 3. Create Child Template for Service

* Select the parent template and click on Create Child Template File button
* Enter File name as **services/${NAME}Service**
* Enter Content as follows

```java
#if (${PACKAGE_NAME} && ${PACKAGE_NAME} != "")package ${PACKAGE_NAME};#end
#set( $CamelCaseName = "$NAME.substring(0,1).toLowerCase()$NAME.substring(1)" )

import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;

import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

import ${PACKAGE_NAME}.repositories.${NAME}Repository;

@Service
@Transactional
@RequiredArgsConstructor
@Slf4j
public class ${NAME}Service {
    private final ${NAME}Repository ${CamelCaseName}Repository;
    
}
```

### 4. Create Child Template for REST Controller

* Select the parent template and click on Create Child Template File button
* Enter File name as **controllers/${NAME}Controller**
* Enter Content as follows

```java
#if (${PACKAGE_NAME} && ${PACKAGE_NAME} != "")package ${PACKAGE_NAME};#end
#set( $CamelCaseName = "$NAME.substring(0,1).toLowerCase()$NAME.substring(1)" )

import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;

import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import ${PACKAGE_NAME}.services.${NAME}Service;

@RestController
@RequestMapping("/api")
@RequiredArgsConstructor
@Slf4j
public class ${NAME}Controller {
    private final ${NAME}Service ${CamelCaseName}Service;
    
}
```

>If you are a keen observer you should have noticed that we are repeating the **$CamelCaseName** expression in multiple child templates.
According to [Intellij docs](https://www.jetbrains.com/help/idea/templates-with-multiple-files.html) variables defined in Parent template should have been available in Child templates. 
But, unfortunately they are rendering as empty values for me...probably a bug.

We are all set, now. Let's test it now.

If we go and try to create a class we will see the following option:

{{< figure src="/images/multiple-file-templates.webp" height="450" width="480" >}}

Now if you give the class name as **Customer** and select **Generate Boilerplate for Entity** then it will generate all Entity, Repository, Service and Controller.

This is amazing, and we can go on to create templates for other common classes we create.

## Few feature requests
There are few things that I would like Intellij IDEA to improve upon **File and Code templates**.

1. Ability to generate Multiple files without requiring Parent should generate a class or interface restriction. Because of this restriction we had to create that Dummy class.
2. Ability to show or hide the child templates in New Class Dialog. Also, child template names are being shown as **Generate Boilerplate for Entity.java.child.0**, **Generate Boilerplate for Entity.java.child.1** etc. If there is a way to give an alias for the child templates it would be nice.


## How about a bonus tip?

You can refactor **@RequestMapping(method=GET, value="...")** into **@GetMapping("...")** as follows:

{{< x user="sivalabs" id="1377637211651203082" >}}

{{< figure src="/images/structural-search.webp" height="450" width="720" >}}

Happy coding :-)

# Few things I learned the hard way in 15 years of my career


I started my career as a Java developer in 2006, and it is almost 15 years that I have been continuing as a software developer.
Along the way, I have experienced some amazing things, and a lot of shitty things as well. 
As I am not a super smart person by birth, I ended up learning many things in the hard way. 

In this article, I would like to share some of my learnings.

<!--more-->


1. **You will always be working in a time and resources-constrained environment**. 
   So those who can get things done within the given constraints will succeed. 
   Your ideals won't help you because you are not living in an ideal world.

2. You can brag about **Clean Code, Clean Architecture, Domain Driven Design, TDD, BDD,** etc all day long, 
   but what management wants to see is releasing the software on time as per plan. 
   If you talk about the importance of **Code Quality**, **Long term maintainability** they will talk about 
   **Need of the hour**, **Time to market**, etc. In this argument, you can't win, trust me.

3. **Everyone has a great suggestion for someone else's problems**. 
   When they face the very same situation, they may struggle more than you. 
   So, just because someone gave you clever advice don't assume that everyone else is able to handle 
   these difficulties smoothly, and only you are struggling.

4. Someday you too, yeah YOU....the passionate developer, get bored with writing code. 
   **Have some non-coding habits** too so that you won't go crazy not knowing what to do with your free time. 

5. **Age is NOT just a number**. With age several things change. 
   When you are a bachelor you have all the time in the world to play with all the cool stuff. 
   But once you get married and have children then things change, and priorities change. 
   You won't have that much time you used to have. So use your time wisely when you still have it. 
   **Learn as much as possible in the early days of your career because later you will be busy learning 
   how to talk diplomatically** :wink:.

6. You will not get tired if you are doing something that you love. 
   So, **do something that you really LOVE at least for a few years in your career**. 
   Later you can go on and write boring REST APIs to earn money :money_mouth_face:.

7. **While having a debate with strongly opinionated people don't try to explain why your approach is better. 
   They won't listen anyway**. Ask how their approach works in the cases you feel doesn't work.

8. Your super clean code written in the modern coolest language will become a legacy in 
   a few years and will be rewritten in something else. 
   **Your good relationship with co-workers will help you throughout your career**.

9. Before saying, "This existing application code is crap, and we should first rewrite it from scratch in a cleaner way 
   before adding new features", take a moment and have a good look at the code you wrote a few years ago. 
   Yes, that code of yours also may look crappy, but you know under what circumstances you have written that. 
   The same may be the case for other's code. **Understand the circumstances without judging too early**.

10. With a bit of **commonsense and googling skills** you can become a decent developer. 
    Learn **how to debug and narrow down the problem** then you will become an awesome developer.

11. Learn to say **I don't know** more often and then see how simple life is.

12. If you got accepted/rejected in an interview, don't take it entirely upon yourself. 
    There may be many things at play. Interviewers might select you because they have rejected the last 12 candidates, 
    and there is pressure to go easy in evaluation. 
    Interviewers might reject you because they don't want a highly skilled developer career to burn into ashes 
    in their shitty company. So take it easy. 

13. While it may sound very cool to hear **"Your team heavily depends on you"**, you are actually digging your grave...very slowly.

14. Are you eager to become a Tech Lead or an Architect so that you will have the power to design the perfect system 
    in the way you always wanted to?. Well, sorry to be the one telling you that you may end up sitting in long boring meetings, 
    trying to convince the people who don't even want to hear you, writing the documents in Confluence etc 
    more than getting your hands-on system design.

15. It is very hard to think about improving the performance of that one tricky query when your neck is paining severely. 
    So, **first go get the proper work desk setup before it is too late**.

16. Your monthly paycheck is not entirely the result of your own hard work; it also includes lots of help from open-source developers, 
    blog posts, StackOverflow answers. Don't forget to thank the people who helped you.
 
17. If at all possible **help somebody to build their career**, that is more satisfying than being the highest paid developer in your team.

18. Have some long-term career goals so that you won't lose career focus due to temporary day-to-day frustrations. 

19. While it feels good to be in an echo chamber where everyone will agree to everything you say, 
    **have someone who can tell you that you are wrong on your face**.

20. **If you realize that the work culture is shitty, and you can't change anything there, then get the hell out of there
    as quickly as you can**. Staying in such environment for too long will make you lose self-confidence.

21. We don't have an infinite amount of time to learn and master every programming language, framework, and tool. 
    So, learn what may help you in the near future, and you can learn other things if need be. 
    Follow Purpose-Driven Learning than just randomly tinkering around with everything that appeared on Tech Twitter or Hacker News.

22. As you gain more experience, you will be spending more time with people than code. 
    So, if you are an **"I hate people management"** type, then understand that there is no way around. 
    So try to improve soft skills.

23. [Learn how to ask for help](https://stackoverflow.com/help/minimal-reproducible-example). 
    If you are asking someone to help you then provide as many inputs as you can and specify the exact problem you are facing. 
    Let me tell you, simply messaging "I want to build a highly scalable system. How?" 
    or "I am getting 403 error from OAuth 2.0. Why?" won't get you any answers.

24. Life is too short to make all the mistakes by yourself and learn from them. **Find a good mentor and get some guidance**.

25. **Learn to say NO**. It is very difficult, especially in India, to say NO for unrealistic deadlines because many managers 
    can't even conceive the idea of hearing NO from a subordinate. But over the time they understand when you say NO mean NO. 
    Even they start respecting you if you keep delivering on time when you say YES.

26. **Your past glory won't carry forward to your new company**. You have to prove yourself in new organization 
    before expecting them to realize your worth.

27. You need to ignore certain unpleasant things at work. Yes, your fellow Java developer who doesn't even know 
    how to set JAVA_HOME environment variable will get an "Employee Of The Year" award, another person may get "Quality Champion" 
    award just because they share a cigarette lighter with a manager in smoking zone. 
    There are certain things that are beyond your control and focusing on them only demotivates you at work. 
    Learn to ignore those things, not easy...but need to learn.

28. If properly used, Social Media platforms like Twitter can help you to learn from experts, build network and 
    become part of a larger community. But, make sure to take breaks and think for yourself about what you want to do 
    instead of doing what everyone else is doing.

29. On weekends, it is perfectly ok to stay away from Computer, just chilling without doing anything as opposed to what 
    "Productivity Gurus" tells you on Twitter :-)

30. It is very difficult to focus on work when having health issues, so take care of health as a high priority always. 
    Some simple habits like drinking more water, taking regular breaks and daily walking help a lot.

#### What are your experiences? Please share in the comments.

# Tips to work at traditional enterprise organizations as consultant/contractor (and save your ass)


I worked at different types of IT organizations including small 20 people company, world's largest bank,
start-ups etc. Working at a service based company is both very interesting and at times frustrating depending on the client.
On the positive side you get to work on different types of domains/applications, different tech stacks,
and you may get to learn variety of organizational cultures. On the downside you may end up working with 
**"Typical traditional enterprise organizations"** which makes you question your career choices!!!

<!--more-->


If you are about to say dude, **"Typical traditional enterprise organizations" doesn't always mean bad workplaces**.
Yes, I totally agree with you and I have once worked at such a large organization which I still consider "happiest days of my career".
So, there can be "large enterprise organizations" which have great work culture, 
but we are going to talk about the other segment of "Typical traditional enterprise organizations" in this article.

So, what I am referring to as "Typical traditional enterprise organizations"? 
Excluding those very few amazing large organizations, most of the remaining organizations that I worked with have some common qualities, 
hence I used the word **"Typical"**.

These **"Typical traditional enterprise organizations"** 

* Assumes **they are following "Agile" because they are using JIRA and having stand-ups every day**.
* Have **siloed Dev and QA departments** usually reporting to different managers, 
  each team doesn't have a clue on what other team is doing, only come to work together 2 months before production release.
* **Hard to change anything at all because you will need 4 levels of approvals** for any small change in the process.
* If any new joiner suggests any improvement over the current way of doing things, 
  they will be tortured so much that leaves the person with only 2 choices: 
    1) Forget about improvements and become like one of them 
    2) Leave the organization
  
I guess you get the picture of what kind of organizations I am talking about.

**Working at such organizations itself is a challenge. Working at such organization as a vendor/contractor/consultant is even more horrible.**

> There are some simple universal rules of nature:
>
> * Roses are red
> * Lillies are white
> * Customer is king
> * If something goes wrong it is vendor's fault.

So, here are few things I noticed in such types of organizations over and over again.
I tried to handle these problems in different ways and finally settled on few approaches.
These are my approaches based on my personality type and experience. So it may or may not suits you.

> A young one says "Never give up". A wise one says "Know when to give up. Not all the battles are worth fighting for."

### 1. The search for "Declaration of Production Readiness"
You may think playing the role of a project manager or scrum master is easy. No, it's not.
What you may not know is, in these kind of organizations they usually write a **Declaration of Production Readiness** document for each project 
and hides it somewhere in their office building. 

{{< figure src="/images/scroll-letter.webp" height="250" width="280" >}}

The Scrum Master is supposed to find it and do all the things mentioned in that letter 
for the project to be considered as **"Production Ready"**. 
Most of the time, Scrum Masters find that document just 3 weeks before production deadline.
Even more interesting thing is, the "Declaration of Production Readiness" document of all the projects contains same thing:

* Should have tests with minimum 80% code coverage
* Performance test suite proving your APIs are returning response with in max 3 seconds
* Should not use any libraries with known [CVE vulnerabilities](https://cve.mitre.org/)

Once the Scrum Master heroically found the "Declaration of Production Readiness" document, 
they will question the dev team "Why didn't you have minimum 80% test coverage", and you will be tasked to write tests with in a week.

You might be in a shock because whenever you bring up "we are not able to write enough tests due to timelines" 
the very same Scrum Master told you not to worry about tests, QA team will take care of automation testing.

> **Lesson:** Setup code quality checks and linting tools from the beginning of the project so that you don't have to work crazy hours at the end.

### 2. GIVEN "development going on" WHEN "assigned a JIRA card" THEN "ask for clear requirements"
How many times you worked on JIRA cards which contains just the "title"? 
In the Sprint after Sprint mad rush they simply create JIRA cards with nothing but a title and ask you to work on it.
If your scrum master is a **"one day trained certified scrum master"** then there might be description in the card as:
> As a user I should be able to [Whatever is there in title appears here]

You might have verbal discussion with BA/product owner, and they might ask you to start working on it.
But during testing, they might realize they haven't thought about many other scenarios.
Instead of accepting it's a miss from all of us as a team, they question you "why didn't you cover those scenarios?"

> **Lesson:** Never pick up card without having clear requirements. 
If they didn't put any details in the card then ask product owner or scrum master for details and put acceptance criteria, 
what is in scope, what is out of scope clearly and get it reviewed by BA and then start working on it. 
If they insist on starting to work on it without having clear understanding of the card then respectfully disagree.
> 
> Make sure you have the following details in the card before picking it up:
>
> * Detailed description of requirement
> * What is in scope and out of scope
> * Dependencies on other tasks
> * Input and output formats
> * How to handle error scenarios

### 3. Live by rules

You might have read the [Agile Manifesto](https://agilemanifesto.org/) and try to follow few of the things mentioned in the manifesto.

> Individuals and interactions over processes and tools

Stop right there...they may tolerate if you don't deliver on-time, but not following processes is considered as a crime.
Just because Agile Manifesto said **"Responding to change over following a plan"**, 
when you find a bug and quickly fix it and create a PR in 10 minutes then 
you might expect a pat on the shoulder but all you get is a question "Why did you work on a non planned task?"

> **Lesson:** If you found a bug then create a JIRA ticket and let the team know it and tell them you know how to fix it and if they agreed to work on it then fix it.

### 4. Keep project related discussion in official channels
When you raise a DevOps ticket for some resource it may take 2 to 4 days to fulfill that request.
If you have a friend in DevOps team that might be resolved in 20 minutes. 
Having good relationships with fellow team members and other teams greatly helps in many ways.

However, when you are working as a consultant/contractor things will be little different, 99% of the times you will be treated as an outsider only.
In this situation it is better to make every conversation regarding project in the official communication channels only.

If there is an issue going on like deployment failures etc then if you get a direct message from one of the client side team member then it is time to be cautious.
If they ask you questions like **how are you?**, **where are you originally from?** etc then you should be definitely cautious :-).

If there is an issue from their side(ex: devops team) and if they asked you **is there a way to fix from your side(ex: dev team) without going through all "escalation chain"** 
and suppose you know a way of fixing it, you might think of HELPING them and say "YES" then you are done.

While this conversation sounds like YOU ARE HELPING THEM, the way it is going to be conveyed to upper management is 
"the dev team missed some configuration, and they are working on the fix. We(devops team) are blocked on that." 

> **Lesson:** If somebody approached you to do a temporary fix or something then tell them you are happy to help and 
let's take this conversation in the official channel so that team also aware of this and if they have any concerns they will let us know.

### 5. If it's not your responsibility then don't put your finger in that

We are all familiar with the following **"It's not my job"** image, aren't we?

{{< figure src="/images/not-my-job.webp" height="300" width="380" >}}

Yes, in software development world we use this image to represent a bad software engineer attitude.
But I feel this is exactly how I need to be when working at a bad workplace.

Suppose, you came across a very poor(read as shitty) deployment process followed by DevOps team, and you know a far better way.
As someone who genuinely want to improve the overall development process you might propose a new way. 
The scrum master and other team members might have liked the approach whereas DevOps team is not so happy about it because they need to learn something new.

Any new/better way needs some amount of reading and understanding of new way of doing it.
But they (DevOps team)  try to apply it without properly understanding it, missing the required configuration or mis-configuring the details which eventually resulting in errors, more deployment cycles etc.

And, the conclusion would be, as you might have already guessed, **"The issues are because of the new approach proposed by those vendors"**.

Your project manager or scrum master might agree with you at cafeteria or parking lot that the issue happened because of their DevOps team fault, but they have to backup their team in front of everyone.

> **Lesson:** Before proposing a better way think twice is the organization/team members are open for suggestions or not.

## Summary
I worked at some places where the colleagues are super talented and getting your PR approved is like an achievement.
They shred your PullRequest(PR) to 1000 pieces pointing out every minor details. 
But, at the end of the day I feel great because I became little better at what I do for a living.

On the other hand, working at those **"Typical traditional enterprise organizations"** I feel exhausted by the end of the day even though there are absolutely no technical challenges.
Having to be cautious all the times because of the "blame game" culture sucks the energy out of you.
But being cautious is better than becoming a fool.

I hope this article would be helpful to avoid some common mistakes.
# My 2020 year review and plans for 2021


I would like to share how was my year 2020 and what I am planning to do in 2021.
As we all know 2020 was a rough year due to COVID-19, lockdown and all, but 2020 is a good year for me in many ways.
It used to take minimum 2 to 3 hours of daily commute for office, now that I have been working from home I am able to use that time for other activities.

<!--more-->


### Following are few notable things for me in 2020:

* Learned and [worked on GoLang]({{< relref "2020-07-02-golang-from-a-java-developer-perspective.md" >}}) for a project.
* Created Yeoman based [Spring Boot application Generator]({{< relref "2020-01-29-creating-yeoman-based-springboot-generator.md" >}}).
* Created [Localstack Spring Boot Starter]({{< relref "2020-09-16-localstack-spring-boot-starter-tutorial.md" >}})
* Reviewed Phillip's [Testing Spring Boot Applications Masterclass Course]({{< relref "2020-10-15-philip-testing-spring-boot-applications-masterclass-review.md" >}}) and learned many testing tips along the way.
* Used [HypersistenceOptimizer]({{< relref "2021-01-02-improve-jpa-performance-using-hypersistence-optimizer.md" >}}) created by [Vlad Mihalcea](https://twitter.com/vlad_mihalcea) to improve JPA applications and got mind blown away by its fantastic developer experience.
* I taught Java, Spring Boot and few other things to someone, and she is really doing well at her job. A [proud moment for me as a mentor]({{< relref "2020-06-09-my-proud-moment-as-a-mentor.md" >}}).

### Few lessons learnt:

* I have been angry and disappointed for several years about "Loud mouths gets more credit than hardworking talented people" in IT.
  Finally, I kind of [understood how things work in real world](https://twitter.com/sivalabs/status/1331047522073133058) and make peace with it. I no longer angry about it anymore.
  
* We might be following some awesome developers on Twitter and looking at so many cool things they talk/blog about 
  we may assume that IT has advanced so much, and we are stuck with some old stuff. 
  But, the reality is very few companies only actually using that cool stuff.
  For example, you might see people talking about their GitOps deployment infrastructure where for every branch they spin up infra on Kubernetes using Terraform & Ansible, verify everything and teardown after done with that feature.
  Whereas, the reality is many companies still has one EC2 instance created for QA server and deploy the new features on Tuesday only.
  So, don't be overwhelmed by what you see on **Tech Twitter**.
  
* No need to share every damn thought of yours on Twitter. Sometimes we might not be able to express our thoughts properly within 280 characters and people might misunderstand easily.
  Or, some people think they already know everything and act as self-appointed internet police. It's not worth engaging in debates on social media with such people.

* Somehow we look for that magical moment of "Grand Success" in our career/life only to realize that "magical" moment is temporary and lasts in just few hours or days.
  Happiness comes from small things that happens every day in our lives. We should learn to enjoy these small things.


## My plans for 2021
I know things won't go as planned, nevertheless it is necessary to plan so that we can at least try to go in that direction.

### 1. No frontend development
Occasionally I get very excited about the frontend frameworks like React, Angular and Vue and I learn them, practice them 
by writing some small applications and that's it.
As my official work mostly involves backend development, I don't get to work on frontend that much.
When I look at them again after few months its all new again.
So, I decided to put my enthusiasm for frontend development aside completely.

### 2. Production Readiness
My perspective on software development keeps changing over the years. 
Until recently I used to heavily focus on development side of things such as programming languages, frameworks, libraries.
Now I think more about how easily we can run it in production with minimal effort while still achieving the desired outcome.
So, my focus is gradually shifting to **how to make applications production ready?**. 
That's why I want to learn more about **deployment automation and monitoring**.

In 2021, I would like to get good grips on following things:

* Deployment using **Terraform/AWS CDK/Pulumi**
* Commonly used **AWS Services like S3, SQS, RDS, Lambda, ECS**
* Monitoring using **Prometheus, Grafana**
* Log management using **ELK/EFK/Loki**

Apart from these I am very interested in [Deno](https://deno.land/), but I most likely won't spend much time on Deno this year.

### 3. Blogging
I wish to write more blog posts but other priorities keep coming up. I will try to write more posts may be on deployment and monitoring.

In addition to blogging, I like to share my learning through tweets, sample applications etc.
You can follow me on Twitter at [@sivalabs](https://twitter.com/sivalabs) and GitHub at [sivaprasadreddy](https://github.com/sivaprasadreddy).

# Improve JPA application performance using HypersistenceOptimizer


In Java world, [Hibernate/JPA](https://hibernate.org/) is the most popular ORM framework. 
Also, JPA/Hibernate is a [very controversial topic](https://www.reddit.com/r/java/comments/5nz5nq/reasons_jpa_and_hibernate_should_be_phased_out/) 
because [some people](https://www.reddit.com/r/java/comments/53p253/how_hibernate_almost_ruined_my_career/) [don't like](https://www.reddit.com/r/java/comments/6prjad/java_shower_thoughts_when_people_say_high/) it at all.
It is very understandable though. 

Many people start using JPA/Hibernate with minimal knowledge and keep adding logic (read as "annotations") as and when required to get the job done.
While checking the overall application performance, most of the time people realize it's because of poor performance of persistence layer. 

<!--more-->


Some people simply blame Hibernate saying it's generating/executing too many unoptimized queries. 
Some people try to learn JPA/Hibernate a little more to optimize their code and overwhelmed by the sheer volume of things to know in order to properly use JPA/Hibernate. 

After using ORM frameworks like JPA/Hibernate and SQL oriented persistence frameworks like [MyBatis](https://mybatis.org/mybatis-3/) over a decade this is how I felt about JPA/Hibernate.

{{< x user="sivalabs" id="1337038528723578880" >}}

Compared to JPA/Hibernate, the SQL oriented frameworks like [MyBatis](https://mybatis.org/mybatis-3/), [JOOQ](https://www.jooq.org/) may need more coding effort.
However, we have full control over what queries we want to execute, easy to understand what is going on and less time in debugging.

On the other hand, JPA/Hibernate provides lots of features but requires a significant amount of time to learn in order to efficiently use it.
It is very easy to shoot yourself in the foot with JPA. 
Just make child collection fetch strategy as **EAGER** to fix **LazyLoadingException** (like many newbies do), 
use **List** instead of **Set** and try to add/remove one element from collection and see the number of SQL queries generated.

Not only this, you need to understand **ID Generator Strategies**, **Mapping strategies**, **JPQL**, **Join Strategies**, **Batch Operations**, **LockModes** etc to properly use JPA/Hibernate.
Mastering JPA/Hibernate is definitely not an easy task and takes significant time and requires assistance from experts.
But, there is a tremendous reward for learning all these concepts which is **ultra productivity with minimal code**.

## Introducing Hypersistence Optimizer
Anybody working with JPA/Hibernate surely ends up spending good amount of time on https://vladmihalcea.com/ blog which is One Stop Shop for anything related JPA/Hibernate.
[Vlad Mihalcea](https://twitter.com/vlad_mihalcea) built a product called [Hypersistence Optimizer](https://vladmihalcea.com/hypersistence-optimizer/) to provide "Expert's Assistance" for any JPA/Hibernate related issues.

Basically, Hypersistence Optimizer profiles your JPA/Hibernate code and give you a report of issues.
Not just reporting issues, but also explain what is the issue and how to fix it.

Let's see how to use HypersistenceOptimizer in a Spring Boot application.

You can get trial version or buy full version of HypersistenceOptimizer at https://vladmihalcea.com/hypersistence-optimizer/.
You can follow the [Installation Guide](https://vladmihalcea.com/hypersistence-optimizer/docs/installation-guide/) to setup HypersistenceOptimizer depending on your application type.

For a simple Spring Boot application we can setup as follows:

* Configure dependencies

**build.gradle**

```groovy
implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
runtimeOnly 'com.h2database:h2'
implementation "io.hypersistence:hypersistence-optimizer:2.2.0"
```

**pom.xml**
```xml
<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-jpa</artifactId>
    </dependency>
    <dependency>
        <groupId>com.h2database</groupId>
        <artifactId>h2</artifactId>
    </dependency>
    <dependency>
        <groupId>io.hypersistence</groupId>
        <artifactId>hypersistence-optimizer</artifactId>
        <version>2.2.0</version>
    </dependency>
</dependencies>
```

* Copy `hypersistence-optimizer-2.2.0/config/META-INF` folder into your project `src/test/resources` folder.

* Configure `HypersistenceOptimizer` in your test config code`(src/test/java)`. 
  
```java
package com.sivalabs.devzone;

import io.hypersistence.optimizer.HypersistenceOptimizer;
import io.hypersistence.optimizer.core.config.JpaConfig;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import javax.persistence.EntityManagerFactory;

@Configuration
public class HypersistenceConfiguration {
    @Bean
    public HypersistenceOptimizer hypersistenceOptimizer(EntityManagerFactory entityManagerFactory) {
        return new HypersistenceOptimizer(new JpaConfig(entityManagerFactory));
    }
}
```

* Configure following in your `logback.xml`

```xml
<logger name="Hypersistence Optimizer" level="info"/>
```

* Write test

```java
package com.sivalabs.devzone;

import com.sivalabs.devzone.domain.models.LinksDTO;
import com.sivalabs.devzone.domain.services.LinkService;
import io.hypersistence.optimizer.HypersistenceOptimizer;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.data.domain.PageRequest;
import org.springframework.data.domain.Pageable;
import org.springframework.data.domain.Sort;

import static org.assertj.core.api.Assertions.assertThat;
import static org.junit.jupiter.api.Assertions.assertTrue;

@SpringBootTest
public class ApplicationTest {
    @Autowired
    private HypersistenceOptimizer hypersistenceOptimizer;

    @Autowired
    private LinkService linkService;

    @Test
    void shouldReturnPagedLinks() {
        Pageable pageable = PageRequest.of(0, 10, Sort.by(Sort.Direction.DESC, "createdAt"));
        LinksDTO linksDTO = linkService.getAllLinks(pageable);
        assertThat(linksDTO.getData()).isNotEmpty();
    }

    @AfterEach
    void afterEach() {
        assertTrue(hypersistenceOptimizer.getEvents().isEmpty());
    }
}
```

Here the key part is, after each test we are verifying whether there are any issues found by HypersistenceOptimizer and make sure no issues found.

For example, let's assume we have a JPA entity with an enum as follows:

```java
@Entity
@Table(name = "users")
public class User {
    @Id
    private Long id;

    @Column(nullable = false)
    @Enumerated(EnumType.STRING)
    private RoleEnum role;
}
```

When you run the test you can see the following info in log and test will fail.

```
2021-01-02 12:18:53.127  WARN 49626 --- [    Test worker] Hypersistence Optimizer                  : MAJOR - EnumTypeStringEvent - The [role] enum attribute in the [com.sivalabs.devzone.domain.entities.User] entity uses the EnumType.STRING strategy, which has a bigger memory footprint than EnumType.ORDINAL. For more info about this event, check out this User Guide link - https://vladmihalcea.com/hypersistence-optimizer/docs/user-guide/#EnumTypeStringEvent
2021-01-02 12:18:53.151  INFO 49626 --- [    Test worker] org.hibernate.dialect.Dialect            : HHH000400: Using dialect: org.hibernate.dialect.H2Dialect
2021-01-02 12:18:53.163  WARN 49626 --- [    Test worker] Hypersistence Optimizer                  : 1 issues were found: 0 BLOCKER, 0 CRITICAL, 1 MAJOR, 0 MINOR
```

You can see that Hypersistence Optimizer found an issue with our Entity mappings and gives us the link https://vladmihalcea.com/hypersistence-optimizer/docs/user-guide/#EnumTypeStringEvent to know more about the issue 

>    ### EnumTypeStringEvent
>    The Java Enum can be mapped in two ways with JPA and Hibernate. By default, the Enum ordinal is used to materialize the Enum value in the database. However, to make it more readable, some developers choose to store the Enum name instead, which has a higher memory and disk footprint.
>    For more details about this topic, check out [this article](https://vladmihalcea.com/the-best-way-to-map-an-enum-type-with-jpa-and-hibernate/).

This is the pattern you will notice while using Hypersistence Optimizer. 

**You run the tests, Hypersistence Optimizer will detect if there are any issues and gives a brief description what is the issue and why it is an issue.
Then one of Vlad's blog post is linked to show you how to resolve that issue. Rinse and repeat.**

After using Hypersistence Optimizer for a couple of weeks this is how I feel about working with JPA and Hypersistence Optimizer.

{{< x user="sivalabs" id="1337374516511559681" >}}

## Summary

While we enjoy so many open source frameworks and libraries for free, we should realize cost vs effort impact on software delivery.
On any given day I would suggest going with an off the shelf Security solution like [Okta](https://www.okta.com/), [Keycloak](https://www.keycloak.org/) etc rather than building your own OAuth Security solution.
It is wise for a company to buy a licence for the IDE if it drastically increases developer productivity.

In the same way, I would strongly suggest getting a Hypersistence Optimizer licence if you are building a JPA based application and performance is a key success factor.
While using Hypersistence Optimizer you not only fix your application issues but also gradually become a master of JPA/Hibernate itself.
Thanks Vlad for building such an awesome product and "Developer Experience" of using Hypersistence Optimizer is just amazing.

# Quirks of Spring's @TestConfiguration

If you know me you know that I am a big fan of Spring ecosystem.
I have been using Spring framework since 2007, and I am pretty familiar with many of its features.

Even if we are familiar with some technology once in a while we get stuck with small issues and end up spending hours and hours figuring out why something is not working as expected.

<!--more-->


Spring framework is very flexible, and usually there are multiple ways to achieve the same thing.
Most of the times all different ways result in same behaviour, but sometimes it may exhibit different behaviour.

Recently I faced one such issue with Spring's `@TestConfiguration` feature, and I would like to share my learnings.

Let's start with creating a simple Spring Boot application, no need to add any starters, but you can add Lombok to avoid boilerplate.

Create a simple `MyService` class as follows:

```java
package com.sivalabs.myapp;

import lombok.RequiredArgsConstructor;

@RequiredArgsConstructor
public class MyService {
    private final String content;

    public String getContent() {
        return content;
    }
}
```

Create a Spring Configuration class `AppConfig` and register `MyService` as a bean.

```java
package com.sivalabs.myapp;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class AppConfig {
    @Bean
    public MyService myService() {
        return new MyService("content-prod");
    }
}
``` 

The main entrypoint class `Application.java`

```java
package com.sivalabs.myapp;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class Application {
    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }
}
```

Write the following Spring Boot integration test `ApplicationTest` as follows:

```java
package com.sivalabs.myapp;

import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

@SpringBootTest
class ApplicationTest {
    @Autowired
    private MyService myService;
    
    @Test
    void contextLoads() {
        System.out.println(myService.getContent());
    }
}
```

There is nothing special so far and as you might have expected when you run the test it will print **content-prod**.

In our Spring Boot applications we might want to have different(mock) configurations for testing.

> One such usecase is we might want to use Localstack docker container for testing AWS related functionality.
For production use we register AmazonS3, AmazonSQS etc beans which talks to real AWS services 
whereas for testing we can configure those beans EndpointURI pointing to localstack http://localhost:4566.

Let's see how we can use Spring's `@TestConfiguration` feature to configure beans for testing.

As a good developer you can take a good look at the official documentation 
[Detecting Test Configuration](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-testing-spring-boot-applications-detecting-config)
 before starting our exciting journey.

From docs:

```
If you want to customize the primary configuration, you can use a nested @TestConfiguration class. 
Unlike a nested @Configuration class, which would be used instead of your application's primary configuration, 
a nested @TestConfiguration class is used in addition to your application's primary configuration.
```
#### Case 1: Using top-level @TestConfiguration class

Create `TestConfig` class as follows:
 
```java
package com.sivalabs.myapp;

import org.springframework.boot.test.context.TestConfiguration;
import org.springframework.context.annotation.Bean;

@TestConfiguration
public class TestConfig {
    @Bean
    public MyService myService() {
        return new MyService("content-testconfig-toplevel");
    }
}
```

Now if you run ApplicationTest again then you will see **content-prod** printed in the console.

Again, if you read the docs,

```
@TestConfiguration can be used on an inner class of a test to customize the primary configuration. 
When placed on a top-level class, @TestConfiguration indicates that classes in src/test/java should not be picked up by scanning. 
You can then import that class explicitly where it is required.
```

Ok, so if you are using `@TestConfiguration` for a top-level class it won't be picked up automatically, we need to import it explicitly.

```java
package com.sivalabs.myapp;

import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.context.annotation.Import;

@SpringBootTest
@Import(TestConfig.class)
class ApplicationTest {
    @Autowired
    private MyService myService;
    
    @Test
    void contextLoads() {
        System.out.println(myService.getContent());
    }
}
```

If you run the test now, you will get the following error:

```
A bean with that name has already been defined in class path resource [com/sivalabs/myapp/TestConfig.class] and overriding is disabled.

Action:

Consider renaming one of the beans or enabling overriding by setting spring.main.allow-bean-definition-overriding=true
```

Let's add `spring.main.allow-bean-definition-overriding=true` to `src/main/resources/application.properties` and run the test again.

You might be expecting **content-testconfig-toplevel** but **content-prod** gets printed.

Don't know why `MyService` defined in `TestConfig` didn't take priority.
Let's try adding `@Primary` to tell Spring to take this bean with high priority.

```java
package com.sivalabs.myapp;

import org.springframework.boot.test.context.TestConfiguration;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Primary;

@TestConfiguration
public class TestConfig {
    @Bean
    @Primary
    public MyService myService() {
        return new MyService("content-testconfig-toplevel");
    }
}
```

Run the test again, but still **content-prod** gets printed.

**Siva, you said you are good at Googling**

![You are goddamn right](/images/youre-goddamn-right.webp "You are goddamn right")

> For a software developer, searching for solutions on StackOverflow, GitHub issues, blogs, forums etc is an invaluable skill.

After a bit of googling someone on StackOverflow said, while overriding beans using `@TestConfiguration` the bean name should be different from actual bean name.
We can change bean name either by naming the method as `myServiceMock()` or using `@Bean(name = "myServiceMock")`.

```java
package com.sivalabs.myapp;

import org.springframework.boot.test.context.TestConfiguration;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Primary;

@TestConfiguration
public class TestConfig {
    @Bean
    @Primary
    public MyService myServiceMock() {
        return new MyService("content-testconfig-toplevel");
    }
}
```

Run the test again, now you can see **content-testconfig-toplevel** gets printed in the console.

#### Case 2: Using nested @TestConfiguration class
Instead of using top-level `@TestConfiguration` class we can also use nested class as follows:

```java
package com.sivalabs.myapp;

import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.test.context.TestConfiguration;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Primary;

@SpringBootTest
class ApplicationTest {
    @Autowired
    private MyService myService;
    
    @Test
    void contextLoads() {
        System.out.println(myService.getContent());
    }

    @TestConfiguration
    static class TestConfig {
        @Bean
        @Primary
        public MyService myServiceMock() {
            return new MyService("content-testconfig-nested");
        }
    }
}
```

Run the test again, now you can see **content-testconfig-nested** gets printed in the console.

Now comes the interesting part. Remove `@Primary` annotation and rename `myServiceMock()` to `myService()` and run the test again.
It just works fine and **content-testconfig-nested** gets printed in the console.

> The `@TestConfiguration` is working differently when used on top-level class and nested-class.

#### Case 3: Using nested @TestConfiguration class from Parent class
It's common to have a `AbstractBaseIntegrationTest` with all the configurations for integration tests and inherit it.

Let's create `AbstractBaseIntegrationTest` as follows:

```java
package com.sivalabs.myapp;

import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.test.context.TestConfiguration;
import org.springframework.context.annotation.Bean;

@SpringBootTest
public class AbstractBaseIntegrationTest {

    @TestConfiguration
    static class TestConfig {
        @Bean
        public MyService myService() {
            return new MyService("content-testconfig-parent-nested");
        }
    }
}
```

Update `ApplicationTest` to extend `AbstractBaseIntegrationTest` as follows:

```java
package com.sivalabs.myapp;

import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;

class ApplicationTests extends AbstractBaseIntegrationTest {

    @Autowired
    private MyService myService;

    @Test
    void contextLoads() {
        System.out.println(myService.getContent());
    }
}
```

If you run the test now you can see **content-prod** gets printed in the console.
So, the nested `@TestConfiguration` class from the parent class is not being used.

One quick way to make it work is by adding `@ContextConfiguration` annotation to `AbstractBaseIntegrationTest`.

```java
import org.springframework.test.context.ContextConfiguration;

@SpringBootTest
@ContextConfiguration
public class AbstractBaseIntegrationTest {

    @TestConfiguration
    static class TestConfig {
        @Bean
        public MyService myService() {
            return new MyService("content-testconfig-parent-nested");
        }
    }
}
```

#### Case 4: Using top-level @TestConfiguration class from Parent class
This approach is kind of combination of Case 2 and Case 3.

We can create top-level `@TestConfiguration` class as follows:

```java
package com.sivalabs.myapp;

import org.springframework.boot.test.context.TestConfiguration;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Primary;

@TestConfiguration
public class TestConfig {
    @Bean
    @Primary
    public MyService myServiceMock() {
        return new MyService("content-testconfig-toplevel");
    }
}
```

Import `TestConfig` in `AbstractBaseIntegrationTest` as follows:

```java
package com.sivalabs.myapp;

import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.context.annotation.Import;

@SpringBootTest
@Import(TestConfig.class)
public class AbstractBaseIntegrationTest {

}
```

Finally, extend our IntegrationTest from `AbstractBaseIntegrationTest` class.

```java
package com.sivalabs.myapp;

import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;

class ApplicationTests extends AbstractBaseIntegrationTest {

    @Autowired
    private MyService myService;

    @Test
    void contextLoads() {
        System.out.println(myService.getContent());
    }
}
```

If you run the test now you can see **content-testconfig-toplevel** gets printed in the console.

As you have noticed, using `@TestConfiguration` class in different ways is exhibiting different behaviours.
I hope Spring team will streamline the `@TestConfiguration` behaviour in future releases :-)

# JOOQ Tutorials


### jOOQ

* [jOOQ Tutorial - 1 : Getting Started]({{< relref "2023-10-12-spring-boot-jooq-tutorial-part-1.md" >}})
* [jOOQ Tutorial - 2 : Implementing CRUD Operations]({{< relref "2023-10-16-spring-boot-jooq-tutorial-part-2.md" >}})
* [jOOQ Tutorial - 3 : Fetching One-to-One Relationships]({{< relref "2023-10-19-spring-boot-jooq-tutorial-part-3.md" >}})
* [jOOQ Tutorial - 4 : Fetching One-to-Many Relationships]({{< relref "2023-10-23-spring-boot-jooq-tutorial-part-4.md" >}})
* [jOOQ Tutorial - 5 : Fetching Many-to-Many Relationships]({{< relref "2023-10-26-spring-boot-jooq-tutorial-part-5.md" >}})

# Tutorials


### Spring AI

* [Spring AI : Chat with LLMs]({{< relref "2026-07-20-spring-ai-chat-with-llms.md" >}})
* [Spring AI : Structured Output]({{< relref "2026-07-22-spring-ai-structured-output.md" >}})
* [Spring AI : Chat Memory]({{< relref "2026-07-25-spring-ai-chat-memory.md" >}})

### Spring Boot

* [Getting Started with Spring Boot]({{< relref "01-getting-started-with-spring-boot.md" >}})
* [Spring Boot Testing Tutorial]({{< relref "02-spring-boot-testing-tutorial.md" >}})
* [Spring Boot Application Configuration Tutorial]({{< relref "03-spring-boot-application-configuration-tutorial.md" >}})
* [Spring Boot Logging Tutorial]({{< relref "04-spring-boot-logging-tutorial.md" >}})
* [Spring Boot Profiles Tutorial]({{< relref "05-spring-boot-profiles-tutorial.md" >}})
* [Spring Boot JdbcTemplate Tutorial]({{< relref "06-spring-boot-jdbctemplate-tutorial.md" >}})
* [Spring Boot Database Transaction Management Tutorial]({{< relref "07-spring-boot-database-transaction-management-tutorial.md" >}})
* [Spring Boot Flyway Database Migration Tutorial]({{< relref "08-spring-boot-flyway-database-migration-tutorial.md" >}})
* [Spring Boot REST API Best Practices - Part 1 : Implementing Get Collection API]({{< relref "09-spring-boot-rest-api-tutorial-part-1.md" >}})
* [Spring Boot REST API Best Practices - Part 2 : Implementing Create and Update APIs]({{< relref "10-spring-boot-rest-api-tutorial-part-2.md" >}})
* [Spring Boot REST API Best Practices - Part 3 : Implementing FindById and DeleteById APIs]({{< relref "11-spring-boot-rest-api-tutorial-part-3.md" >}})
* [Spring Boot REST API Best Practices - Part 4 : Exception Handling in REST APIs]({{< relref "12-spring-boot-rest-api-tutorial-part-4.md" >}})
* [Why Spring Boot is so popular and how to learn Spring Boot effectively?]({{< relref "2018-02-22-why-springboot-so-popular-how-to-learn-springboot.md" >}})
* [Why Spring Boot?]({{< relref "2016-03-13-why-springboot.md" >}})
* [How Spring Boot AutoConfiguration magic works?]({{< relref "2016-03-13-how-springboot-autoconfiguration-magic.md" >}})
* [Creating Custom Spring Boot Starter for Twitter4j]({{< relref "2016-04-08-creating-custom-springboot-starter-for.md" >}})
* [Spring Boot Messaging with RabbitMQ]({{< relref "2018-02-20-springboot-messaging-rabbitmq.md" >}})
* [Session Management using Spring Session with JDBC DataStore]({{< relref "2018-02-07-session-management-using-spring-session-jdbc-datastore.md" >}})
* [Spring Boot Best Practices]({{< relref "2019-05-14-spring-boot-best-practices.md" >}})
* [Testing Spring Boot Applications]({{< relref "2019-10-07-spring-boot-testing.md" >}})
* [Spring Boot Integration Testing using TestContainers Starter]({{< relref "2020-02-03-spring-boot-integration-testing-using-testcontainers-starter.md" >}})
* [Remote debugging Spring Boot application]({{< relref "2020-05-31-remote-debugging-spring-boot-application.md" >}})
* [Retrying Method Execution using Spring AOP]({{< relref "2016-01-05-retrying-method-execution-using-spring-aop.md" >}})

### Spring Security OAuth 2.0 Tutorials
* [Spring Security OAuth 2 Tutorial - 1 : Getting familiar with OAuth 2 concepts]({{< relref "2023-09-22-spring-security-oauth2-tutorial-part1.md" >}})
* [Spring Security OAuth 2 Tutorial - 2 : Authorization Code Flow]({{< relref "2023-09-24-spring-security-oauth2-tutorial-part2.md" >}})
* [Spring Security OAuth 2 Tutorial - 3 : Client Credentials Flow]({{< relref "2023-09-25-spring-security-oauth2-tutorial-part3.md" >}})
* [Spring Security OAuth 2 Tutorial - 4 : Authorization Code Flow with PKCE]({{< relref "2023-09-27-spring-security-oauth2-tutorial-part4.md" >}})
* [Spring Security OAuth 2 Tutorial - 5 : Implicit & Resource Owner Password Credentials Flows]({{< relref "2023-09-27-spring-security-oauth2-tutorial-part5.md" >}})
* [Spring Security OAuth 2 Tutorial - 6 : Microservices Sample Project Setup]({{< relref "2023-09-28-spring-security-oauth2-tutorial-part6.md" >}})
* [Spring Security OAuth 2 Tutorial - 7 : Securing Spring MVC Client Application]({{< relref "2023-09-28-spring-security-oauth2-tutorial-part7.md" >}})
* [Spring Security OAuth 2 Tutorial - 8 : Securing Resource Server]({{< relref "2023-10-02-spring-security-oauth2-tutorial-part8.md" >}})
* [Spring Security OAuth 2 Tutorial - 9 : Invoking Secured Resource Server APIs from Client Application]({{< relref "2023-10-06-spring-security-oauth2-tutorial-part9.md" >}})
* [Spring Security OAuth 2 Tutorial - 10 : Service to Service Communication using Client Credentials Flow]({{< relref "2023-10-09-spring-security-oauth2-tutorial-part10.md" >}})

### Spring Cloud

* [Introduction to Spring Cloud Config Server]({{< relref "2017-08-14-spring-cloud-tutorials-introduction-to-spring-cloud-config-server.md" >}}) 
* [Auto Refresh Config Changes using Spring Cloud Bus]({{< relref "2017-08-14-spring-cloud-tutorials-auto-refresh-config-changes-using-spring-cloud-bus.md" >}}) 

### Microservices
* [Part 1 : MicroServices : Spring Boot & Spring Cloud Overview]({{< relref "2018-03-02-microservices-using-springboot-spring-cloud-part-1-overview.md" >}})
* [Part 2 : MicroServices : Configuration Management with Spring Cloud Config and Vault]({{< relref "2018-03-05-microservices-part-2-configuration-management-spring-cloud-config-vault.md" >}})
* [Part 3 : MicroServices : Spring Cloud Service Registry and Discovery]({{< relref "2018-03-08-microservices-springcloud-eureka.md" >}})
* [Part 4 : MicroServices : Spring Cloud Circuit Breaker using Netflix Hystrix]({{< relref "2018-03-12-spring-cloud-netflix-circuit-breaker.md" >}})
* [Part 5 : MicroServices : Spring Cloud Zuul Proxy as API Gateway]({{< relref "2018-03-15-microservices-part-5-spring-cloud-zuul-proxy-as-api-gateway.md" >}})
* [Part 6 : MicroServices : Distributed Tracing with Spring Cloud Sleuth and Zipkin]({{< relref "2018-03-20-microservices-part-6-distributed-tracing-with-spring-cloud-sleuth-and-zipkin.md" >}})

### Kubernetes
* [Getting Started with Kubernetes]({{< relref "2021-09-01-getting-started-with-kubernetes.md" >}})
* [Kubernetes - Releasing a new version of the application using Deployment Rolling Updates]({{< relref "2021-09-07-kubernetes-deployment-rolling-updates.md" >}})
* [Kubernetes - Blue/Green Deployments]({{< relref "2021-09-14-kubernetes-blue-green-deployments.md" >}})
* [Kubernetes - Exposing Services to outside of Cluster using Ingress]({{< relref "2021-09-22-kubernetes-ingress.md" >}})

### jOOQ

* [jOOQ Tutorial - 1 : Getting Started]({{< relref "2023-10-12-spring-boot-jooq-tutorial-part-1.md" >}})
* [jOOQ Tutorial - 2 : Implementing CRUD Operations]({{< relref "2023-10-16-spring-boot-jooq-tutorial-part-2.md" >}})
* [jOOQ Tutorial - 3 : Fetching One-to-One Relationships]({{< relref "2023-10-19-spring-boot-jooq-tutorial-part-3.md" >}})
* [jOOQ Tutorial - 4 : Fetching One-to-Many Relationships]({{< relref "2023-10-23-spring-boot-jooq-tutorial-part-4.md" >}})
* [jOOQ Tutorial - 5 : Fetching Many-to-Many Relationships]({{< relref "2023-10-26-spring-boot-jooq-tutorial-part-5.md" >}})

### MyBatis

* [MyBatis Tutorial: Part1 – CRUD Operations]({{< relref "2012-10-21-mybatis-tutorial-part1-crud-operations.md" >}})
* [MyBatis Tutorial: Part-2: CRUD operations Using Annotations]({{< relref "2012-10-21-mybatis-tutorial-part-2-crud-operations-using-annotations.md" >}})
* [MyBatis Tutorial: Part 3 – Mapping Relationships]({{< relref "2012-10-21-mybatis-tutorial-part-3-mapping-relationships.md" >}})
* [MyBatis Tutorial : Part4 – Spring Integration]({{< relref "2012-10-24-mybatis-tutorial-part4-spring-integration.md" >}})

### Thoughts & Opinions

* [Few Things I learned in the HardWay in 15 years of my career]({{< relref "2021-08-11-few-things-i-learned-the-hardway-in-15-years-of-my-career.md" >}})
* [My Life as a Software Developer in India]({{< relref "2015-07-06-my-life-as-software-developer-in-india.md" >}})
* [A Developers Perspective on Spring vs JavaEE]({{< relref "2015-06-15-a-developers-perspective-on-spring-vs-javaee.md" >}})
* [Can MicroServices Architecture Solve All Your Problems?]({{< relref "2014-11-30-can-microservices-architecture-solve-all-your-problems.md" >}})
* [Clean Code: Don't mix different levels of abstractions]({{< relref "2013-12-29-clean-code-dont-mix-different-levels-of-abstractions.md" >}})
* [Keep The Code Clean: WatchDog & SpotTheBug Approach]({{< relref "2012-08-28-keep-code-clean-watchdog-spotthebug.md" >}})
* [Are frameworks making developers dumb?]({{< relref "2011-08-22-frameworks-making-developers-dumb.md" >}})
* [10 things to become an outstanding Java developer]({{< relref "2011-01-26-10-things-to-become-outstanding-java.md" >}})

# My Thoughts on CleanCode, Simplicity, Automation and Empathy


When I observe old people I notice one interesting thing. They don't usually react aggressively like teenagers do.
Even when they have to deal with difficulties in life they deal with it calmly. 
Even when some people undermine them they simply give a smile and move on rather than trying to prove they are wrong.
**I wonder is growing older makes them wise?**

<!--more-->


Looking at some people I know who is in their 50's, I can confidently say **No, becoming older doesn't make them wiser automatically** :-)

But, I guess they have seen enough things in their lifetime, and they understood that **this too will pass** and 
that's why they handle difficulties in gracious way as opposed to committing suicides like many teenagers are doing these days even for small things like failing in exams.
I think they have more data points to think rationally rather than going by emotion or following cult.

> I am in my late 30's now and there is a significant change in the way I see things now. Did I become wiser?
Looking at the number of stupid decisions I make on daily basis, nah...I didn't become so much wiser...but still better than I used to be.

As years pass by, my thoughts on many aspects of software development changed radically. 
Like many passionate developers I have high regards for writing clean code, creating better software designs, automating the boring stuff etc.
But, what has changed is I start applying them depending on the context rather than by rule.

## Clean Code
I don't think anybody would disagree that writing clean code is very important. 
We have so many fantastic books, wonderful talks by knowledgeable people on how to write clean code.
Writing clean code takes more time and effort, but it will pay off in the long run. 
So, can we conclude that we should always aim for writing clean code? 

Younger myself would have said "Yes, we should always aim for writing clean code". But, now I would say "It depends".

Just imagine, there is an event scheduled to happen and want to open online registrations.
Would I bother to implement the application following all clean code principles, TDD, BDD etc etc.
No, I would just quickly build something to get it working. I might not even bother to write tests, optimise DB queries etc.
I know it's only for a short term use, and I don't think investing time and effort for great code quality with 100% code coverage is worth it.

Try telling a young passionate developer that "You no need to aim for Clean Code for all kinds of applications" :-)

## Simplicity
As a young developer I loved complexity. Yes, I loved writing complex code. That made me feel better about myself.
If someone looks at my code and say it is easy to understand then I felt very disappointed.
I wanted people to look at my complex code solving a complex problem and say "wow, dude its complex... but, you nailed it".

So, how to make things complex?? Well, for a start we can apply all the design patterns we read last week.
Then we can make everything configurable reading properties files from 16 different sources. 
Then, add some abstractions to accommodate futuristic enhancements. You got yourself a "Senior Software Engineer" badge.

After implementing all that complex design and looking at a usecase most of it looks unnecessary. 
But, we tell ourselves that the beauty of this design will be realized when new requirements come up.
Sadly 90% of the time the design feels complex for every single usecase and 
when a new enhancement came up we realize the current design doesn't support it.

After going through this vicious cycle again and again, now I crave for simplicity.
I love to code for the requirement at hand. If a similar requirement came up 2nd time I copy paste the code and refactor the necessary logic.
When a similar requirement came up 3rd time then only I think of coming up with an abstraction. 
At this stage I have clear data points to create a meaningful abstraction rather than creating futuristic imaginary abstractions.

Keep things simple, implement just enough logic for the business requirement. 
Now, I feel proud when someone says "I looked at your PR, and it's easy to understand".

I prefer copy pasting 10 lines of code with minor changes instead of creating meaningless abstractions.
In fact the skill of a developer will be known based on code simplicity because writing simple code is the hardest thing.

## Automation & Empathy
I love automating boring stuff. I use bash alias to avoid typing long commands, I create application generator to speed up development process,
I create reusable modules to avoid boilerplate coding. Everytime I automate something I pat myself on shoulder :-)

The entire world is moving on so fast with all the technology revolution. Now we have automated tools for almost everything.
We can comfortably sit in our chair and talk to someone miles away through video calls, pay our bills without having to stand in queues.

We, humans, never get satisified. We want more automation. We want our gadgets to tell us to wish our parents on their birthdays.
If possible we want our gadgets to automatically send the wishes with fancy graphics and stuff. 
We want the technology to complete sentences for us while writing email. 

We don't like people directly calling us on our phones. #Slack-me.
We don't have time for any of this bloody human interactions.. we have more important things to do such as learning the next buzzword technology.
**During this amazing transformation we lost a quality called Empathy.**

Few months ago I worked with a client where they have a semi-automated system where they perform certain activities manually.
The manual work involves lifting of heavy objects, measuring the radius etc and is done by mostly not so educated people.
Our job is to automate that manual process also using Machine Learning and AI so that we can reduce the number of folks from 19 to 2.
Very well, from the organization point of view it reduces human error and increase efficiency. 

**You may ask, shouldn't we use these amazing technologies (ML/AI) for something greater good like 
fighting climate change, finding out cures for deadly diseases instead of applying it for petty things??!!!**

What happens to those poor not so well-educated workers?. They are doing this work for decades, now all of a sudden they became redundant.
But who cares!! 

If you ask this question you may end up listening to **ToughMan** quotes: **Stronger will ive, weaker will die...that's how nature works**.
Yes, it is easy for someone who is earning 6 digits salary monthly to preach how nature works and throw fancy quotes from "Sapiens" book.
It takes a lot of empathy to step into their shoe and realize how hard it is.

During this project we met an ML/AI expert to solve our problem, and the expert not only blabbered for hours without giving solution but also gave a sneak peak of his dream ML project.
He wants to put a sensor in Fritz, scan vegetables and place order on BigBasket automatically whenever it reaches to a threshold. 
I can't think of a better use of ML/AI :-).

Well, I am not saying abandon all the technology advancements and live in a cave. But, don't forget the very reason why we are inventing all this technology.
The very promise of technology is boasted as "Bringing people together". If you can't show some empathy you can never get closer to anyone.

While interviewing someone understand that the candidate is in a lot of pressure and may not come up with spontaneous answers on the spot.
If any of your colleague who is usually good at work is not doing very well lately, try to understand if they are going through any troubles in personal life.
Don't impart your "Leave home at home" wisdom that you gained by reading a self-help book last night.

## Summary

When I think of what I am missing because of COVID-19, one thing is clear.
I can surround myself with all fancy gadgets, but without personal interaction with other people life doesn't feel good.
Being highly skilled might make you more productive at work, but the only way to get closer to peoples heart is having empathy.
  
Be human, be humble.


# Philip's Testing Spring Boot Applications Masterclass Course Review


If you ever worked on a large project which doesn't have any tests then you know how painful it is to work on such codebase.

> You know what is more painful? Working on a project which has plenty of USELESS tests written for the sake of passing minimum code coverage requirement.

<!--more-->


Being a Professional Software Developer means more than solving a problem. One needs to learn how to write maintainable and testable code, 
write meaningful tests, know what to test and what not to test, how to evolve the codebase backing with a good set of test suite.

When it comes to Java land, Spring/Spring Boot is the de facto framework for building web and enterprise applications.
One of the key factor for the success of Spring/Spring Boot is its fantastic support for Testing.
Spring is 17+ years old and evolved over the years with more and more features added over the time which resulted in the same thing can be achieved in many ways.

Most of the developers who are new to Spring Boot face the challenge of knowing what is the current recommended approach to solve a specific problem, same applies to testing.

The [Testing Spring Boot Applications Masterclass Course](https://rieckpil.de/testing-spring-boot-applications-masterclass/) is created by [Philip Riecks](https://twitter.com/rieckpil) to specifically teach how to test Spring Boot applications.
I got an opportunity to review the course I find it wonderful for learning how to test Spring Boot applications leveraging modern testing frameworks and libraries.

[![Testing Spring Boot Applications Masterclass](/images/TSBAM.webp "Testing Spring Boot Applications Masterclass")](https://rieckpil.de/testing-spring-boot-applications-masterclass/)

I can recommend this wonderful course for many reasons:

* **Concise and to the point:** 
The first thing I liked about the course is there is no beating around the bush. 
Phillip explains the concept to the point with practical code examples.

* **Backed by a realistic project, no helloworld examples:** 
Throughout the course the concepts are explained by taking a realistic example application rather than using basic helloworld style examples.

* **Covered most commonly used tools/libraries:** 
The course teaches you how to leverage the modern tools and technologies such as **JUnit5**, **Testcontainers**, **Wiremock** etc 
for testing complex applications.

* **Covered everything you need for testing a typical Spring Boot application:** 
In this course almost everything is covered(**JUnit5, Mockito, WebLayer, DataLayer, Security, Testcontainers, Wiremock, 
WebClient, Localstack, Selenide** etc) for testing a Spring Boot application.

In this fast paced software development world, we have everything except time to learn everything from the official documentation 
of the languages/frameworks/libraries. We can certainly take advantage of the amazing courses like this which explains 
the concepts in an easy to understand way to learn things faster. 

I would highly recommend [Testing Spring Boot Applications Masterclass Course](https://rieckpil.de/testing-spring-boot-applications-masterclass/) for anybody working with Spring Boot applications.
 
# LocalStack Spring Boot Starter Tutorial


I was working on a Spring Boot application which is planned to deploy on AWS and, we were using S3, SQS and RDS services.

[LocalStack](https://github.com/localstack/localstack) provides an easy-to-use test/mocking framework 
for developing AWS based Cloud applications. 

<!--more-->


I want to use Localstack mock services instead of using real AWS services for two purposes:

* To run integration tests
* To run application locally

We can use [Testcontainers](https://www.testcontainers.org/modules/localstack/) to spin up a *Localstack* docker container, 
but we need to configure Amazon service clients like `AmazonS3`, `AmazonSQSAsync` which is typical boilerplate that we copy-paste from project to project.
Instead of copy-pasting the code snippets, creating a Spring Boot starter which autoconfigures the Amazon service clients is a better approach and less error prone.
So, I have created [localstack-spring-boot-starter](https://github.com/sivalabs/localstack-spring-boot-starter)

## How to use localstack-spring-boot-starter?
The ideal way to use **localstack-spring-boot-starter** is to enable it with a specific profile such as `integration-test` or `local` 
then activate that profile based on the need.

> You can checkout the example applications on how to use localstack-spring-boot-starter https://github.com/sivalabs/localstack-spring-boot-starter/tree/master/examples

### 1. Project Setup

You can go to https://start.spring.io/ and generate a Spring Boot application and then add the following dependencies.

```xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
		 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
						http://maven.apache.org/maven-v4_0_0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>com.sivalabs</groupId>
	<artifactId>localstack-spring-boot-sample</artifactId>
	<packaging>jar</packaging>
	<version>1.0-SNAPSHOT</version>

	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>2.3.3.RELEASE</version>
	</parent>

	<properties>
		<java.version>1.8</java.version>
		<aws-java-sdk.version>1.11.852</aws-java-sdk.version>
		<testcontainers.version>1.14.3</testcontainers.version>
		<localstack-starter.version>0.0.2</localstack-starter.version>
	</properties>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>

	<dependencyManagement>
		<dependencies>
			<dependency>
				<groupId>org.testcontainers</groupId>
				<artifactId>testcontainers-bom</artifactId>
				<version>${testcontainers.version}</version>
				<type>pom</type>
				<scope>import</scope>
			</dependency>
		</dependencies>
	</dependencyManagement>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>
		<dependency>
			<groupId>org.testcontainers</groupId>
			<artifactId>localstack</artifactId>
		</dependency>
		<dependency>
			<groupId>io.github.sivalabs</groupId>
			<artifactId>localstack-spring-boot-starter</artifactId>
			<version>${localstack-starter.version}</version>
		</dependency>
		<dependency>
			<groupId>com.amazonaws</groupId>
			<artifactId>aws-java-sdk</artifactId>
			<version>${aws-java-sdk.version}</version>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
		</dependency>
	</dependencies>

</project>
```

We have added `org.testcontainers:localstack`, `com.amazonaws:aws-java-sdk` and `localstack-spring-boot-starter` dependencies.

### 2. AWS Configuration

Create a Spring configuration class for configuring AWS client beans for production and non-production environments as follows:

```java
import io.github.sivalabs.localstack.EnableLocalStack;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;

@Configuration
public class AppConfig {

    @Configuration
    @Profile("aws")
    public static class AwsConfig {
        private static final Regions REGION = Regions.US_EAST_1;
        
        @Bean
        @Primary
        public AmazonSQSAsync amazonSQSAsync() {
            return AmazonSQSAsyncClientBuilder.standard()
                .withRegion(REGION)
                .build();
        }
    
        @Bean
        @Primary
        public AmazonS3 amazonS3() {
            return AmazonS3ClientBuilder.standard()
                .withRegion(REGION)
                .build();
        }
    }

    @Configuration
    @Profile({"integration-test", "local"})
    @EnableLocalStack
    public static class LocalStackAwsConfig {

    }
}
```

In our AppConfig configuration class, we have two nested configuration classes, one for production(`aws`) and another one for non-prod(`integration-test` and `local`).
The `AwsConfig` is associated with `aws` profile and `LocalStackAwsConfig` is associated with `integration-test` and `local`.
The most important thing to notice here is that, `LocalStackAwsConfig` class is annotated with `@EnableLocalStack` which actually triggers the Localstack auto-configuration.

### 3. Localstack configuration

We need to specify which AWS services we want to use and, we can enable or disable specific service as follows:

```bash
localstack.enabled=true # default true
localstack.services=S3,SQS
localstack.s3.enabled=true # default true
localstack.sqs.enabled=false # default true
```

As of now, the following services are auto-configured using **AWS Java SDK V1**:
`SQS, S3, SNS, DYNAMODB, DYNAMODBSTREAMS, KINESIS, IAM, LAMBDA, CLOUDWATCH, SECRETSMANAGER`

### 4. Use AutoConfigured AWS Services
Now with Localstack auto-configuration we can inject `AmazonS3`, `AmazonSQS`, `AmazonSQSAsync` etc and use them in our application.

```java
@Service
@RequiredArgsConstructor
public class S3Service {
    private final AmazonS3 amazonS3;

    public Bucket createBucket(String bucketName) {
        return amazonS3.createBucket(bucketName);
    }
    
    public S3Object getObject(String bucketName, String key) {
        return amazonS3.getObject(bucketName, key);
    }

    public PutObjectResult store(String bucketName, String key, String value) {
        return amazonS3.putObject(bucketName, key, value);
    }
}
```

### 5. Run Integration Tests
We can write integration test by enabling `integration-test` profile using `@ActiveProfiles("integration-test")`.

```java
import com.amazonaws.services.s3.model.S3Object;
import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

import java.io.ByteArrayInputStream;

import static org.assertj.core.api.Assertions.assertThat;

@SpringBootTest
@ActiveProfiles("integration-test")
@Slf4j
class S3ServiceTest {
    private static final String bucketName = "mybucket";

    @Autowired
    private S3Service s3Service;

    @Test
    void shouldStoreAndRetrieveDataFromS3Bucket() {
        s3Service.createBucket(bucketName);
        log.info("Created S3 Bucket: {}", bucketName);
        s3Service.putObject(bucketName, "my-key-1", "my-value-1");
        S3Object s3Object = s3Service.getObject(bucketName, "my-key-1");
        assertThat(s3Object.getObjectContent()).hasSameContentAs(new ByteArrayInputStream("my-value-1".getBytes()));
    }
}
```

### 6. Run application locally
As we have enabled Localstack auto-configuration using **local** profile, we can start our application by enabling **local** profile as follows:

`java –jar -Dspring.profiles.active=local app.jar`

We can also run the application from the IDE itself by setting environment variable SPRING_PROFILES_ACTIVE=local.

### 7. How to use with Spring Cloud AWS?
We can use `localstack-spring-boot-starter` with Spring Cloud AWS as well. To run Spring Cloud AWS application locally we need to disable following properties:

```bash
cloud.aws.region.auto=false
cloud.aws.stack.auto=false
```

### 8. Examples
* [Minimal Spring Boot application](https://github.com/sivalabs/localstack-spring-boot-starter/tree/v1.0.0/examples/localstack-spring-boot-sample)
* [SpringCloud AWS application](https://github.com/sivalabs/localstack-spring-boot-starter/tree/v1.0.0/examples/localstack-spring-cloud-aws-sample)

## 9. What's next?
As of now `localstack-spring-boot-starter` supports auto-configuration of AWS services based on AWS Java SDK v1.
Support for AWS Java SDK v2 is in progress. You can checkout the Java SDK v2 support in https://github.com/sivalabs/localstack-spring-boot-starter/tree/v1.0.0 branch.
Also, support for creating S3 buckets, SQS queues on application start up will also be in next release.

## 10. Want to contribute?
You are most welcome to contribute to this starter by implementing auto-configuration for more services supported by LocalStack.
You can also use this starter and report any bugs. Adding more example applications is also a nice thing to help others to get started.

I hope `localstack-spring-boot-starter` is useful and if you like it [give it a star](https://github.com/sivalabs/localstack-spring-boot-starter).

# All the resources you ever need as a Java & Spring application developer


Nowadays software development is a challenging task as the number of programming languages, frameworks, libraries are increasing rapidly day by day.
Luckily, the number of resources to learn on the internet for free also increased a lot. There are a ton of free resources to learn almost everything on the internet.
There are lots of official documentation, blog posts, Udemy courses, YouTube videos etc.

<!--more-->


As a backend developer who primarily work on JVM based languages/frameworks like **Java, Kotlin, Spring Boot** etc, 
following are a collection of resources I usually follow to learn new things. Thought of sharing with you, it could be helpful for you too.

## Spring
* [This Week in Spring](https://spring.io/blog) series by [Josh Long](https://twitter.com/starbuxman) 
* [SpringDeveloper](https://www.youtube.com/channel/UC7yfnfvEUlXUIfm8rGLwZdA) YouTube Channel 
* [Spring I/O](https://www.youtube.com/channel/UCLMPXsvSrhNPN3i9h-u8PYg) YouTube Channel 

### Blogs
* If you like reading lengthy in-depth articles, then https://www.marcobehler.com/ by [Marco Behler](https://twitter.com/MarcoBehler) is the place to go.
* [SpringAcademy](https://www.youtube.com/c/springacademy) YouTube Channel by [Maciej Walkowiak](https://twitter.com/maciejwalkowiak)
* For high quality Spring content go to https://reflectoring.io/ by [Tom Hombergs](https://twitter.com/TomHombergs)
* Another nice blog to learn Spring is https://rieckpil.de/ by [Philip Riecks](https://twitter.com/rieckpil) 
and his [YouTube Channel](https://www.youtube.com/channel/UCEbZTcePN9QwMr1Lr1SED0Q)
* If you want to lean Spring Boot, SpringCloud and Kubernetes then https://piotrminkowski.com/ by [Piotr Mińkowski](https://twitter.com/piotr_minkowski) 
and his [YouTube Channel](https://www.youtube.com/channel/UCAieNgran7umupT_D50KaGw)
* There is nothing you can't find about Spring Boot on [HowToDoInJava](https://howtodoinjava.com/) by [Lokesh Gupta](https://twitter.com/HowToDoInJava)
* Of course https://www.baeldung.com/ by [Eugen](https://twitter.com/baeldung)
* Another interesting blog https://shekhargulati.com/ by [Shekhar Gulati](https://twitter.com/shekhargulati). 
Don't miss his "A Handcrafted Monthly Newsletter For Software Developers" series.
* [SpringFramework Guru](https://springframework.guru/) by [spring_guru](https://twitter.com/spring_guru) 
and his [YouTube Channel](https://www.youtube.com/channel/UCrXb8NaMPQCQkT8yMP_hSkw)
* [Dan Vega](https://www.danvega.dev/) by [therealdanvega](https://twitter.com/therealdanvega) 
and his [YouTube Channel](https://www.youtube.com/user/danvegadotorg)
* [In28minutes](https://www.youtube.com/channel/UCLz7LG4YVi7_iyk4yOARcxA) by Ranga Karanam
* [JavaGuides](https://www.javaguides.net/) by [Ramesh Fadatare](https://twitter.com/GuidesJava)
* http://dhaval-shah.com/ by [Dhaval Shah](https://twitter.com/dhaval201279)
* https://deepu.tech/ by [Deepu K Sasidharan](https://twitter.com/deepu105)
* [JavaRevisited](https://medium.com/javarevisited) by [Javinpaul](https://twitter.com/javinpaul)
* This blog https://www.petrikainulainen.net/blog/ was my go to reference by [Petri Kainulainen](https://twitter.com/petrikainulaine).
I guess he is busy with other interesting things in life lately :-).
* [Telusko YouTube Channel](https://www.youtube.com/channel/UC59K-uG2A5ogwIrHw4bmlEg)
* [Java Brains YouTube Channel](https://www.youtube.com/channel/UCYt1sfh5464XaDBH0oH_o7Q)
* [IntelliJ IDEA YouTube Channel](https://www.youtube.com/channel/UC4ogdcPcIAOOMJktgBMhQnQ)
* https://blog.frankel.ch/
* https://www.callicoder.com/
* https://blog.jdriven.com/
* https://deinum.biz/blog/
* https://mrhaki.blogspot.com/
* https://dimitr.im/
* https://microservices.io/
* https://developer.okta.com/blog/
* https://blog.codecentric.de/en/
* https://blog.gojekengineering.com/

## SQL, JPA, Hibernate, JOOQ

* https://vladmihalcea.com/ by [Vlad Mihalcea](https://twitter.com/vlad_mihalcea)
* https://thorben-janssen.com/ by [Thorben Janssen](https://twitter.com/thjanssen123) and his [YouTube Channel](https://www.youtube.com/channel/UCYeDPubBiFCZXIOgGYoyADw)
* Anghel Leonard articles on DZone https://dzone.com/users/196910/anghelleonard.html
* [JOOQ Blog](https://blog.jooq.org/) by [Lukas Eder](https://twitter.com/lukaseder)

## Web Development

* [Traversy Media](https://www.youtube.com/channel/UC29ju8bIPH5as8OGnQzwJyA)
* [Codevolution](https://www.youtube.com/channel/UC80PWRj_ZU8Zu0HSMNVwKWw)
* [freeCodeCamp.org](https://www.youtube.com/channel/UC8butISFwT-Wl7EV0hUK0BQ)
* [Academind](https://www.youtube.com/channel/UCSJbGtTlrDami-tDGPUV9-w)
* [Programming with Mosh](https://www.youtube.com/channel/UCWv7vMbMWH4-V0ZXdmDpPBA)
* [Techsith](https://www.youtube.com/channel/UCbGZKLIHpox2l0whz6_RYyg)
* [Fun Fun Function](https://www.youtube.com/channel/UCO1cgjhGzsSYb1rsB4bFe4Q)
* [The Net Ninja](https://www.youtube.com/c/TheNetNinja/)

## DevOps

* [Stack Simplify DevOps](https://www.youtube.com/c/StackSimplify/videos)
* [JavaHomeCloud](https://www.youtube.com/c/JavaHomeCloud/videos)
* [Agent of Change](https://www.youtube.com/c/AgentofChange-RajdeepSaha/videos)
* [KnowledgeIndia AWS Azure Tutorials](https://www.youtube.com/channel/UCzpHRBVnkzBfSsXostYuW1g)
* http://www.devopsbookmarks.com/
* https://serversforhackers.com/

I am sure I might have missed many other wonderful blogs that I haven't come across.
Please comment if you know any other cool blogs :-)

# GoLang from a Java developer perspective


Gone are the days we can call ourselves as Java developer, .NET developer, Python developer etc.
Nowadays we might need to work with multiple languages to some extent such as Python for scripting, Go for CLI utilities etc.

Out of personal interest I started learning [Go Language](https://golang.org/) 3 months ago and I am using Go for our current application. In this article I would like to share my first impression on Go language.

<!--more-->


## 1. Happy Beginning

Few years ago I thought of switching from WordPress to a static generator for my blog and tried [Jekyll](https://jekyllrb.com/) and [Hugo](https://gohugo.io/).
Hugo is really blazing fast and I learned that Hugo is built using GoLang. So, I already have a good opinion on Go.

Also, I keep hearing lot of good things about Go and in the recent years many developers prefer Go over Java for MicroServices development. Although that hype seems to be settled down lately.

### Go is a simple language

Go is a very simple language to learn. Go doesn't include too many features, so it is easy to learn compared to other languages. I would say in a week you can be comfortably program in Go. You can learn about Go **datatypes, control structures, collections, functions, error handling** in a couple of days and get familiar with **go modules, structs, interfaces, pointers** and **Go routines/channels** in another 4 or 5 days.

I find the following resources very helpful to get started with Go:

- https://golangbot.com/learn-golang-series/
- https://gobyexample.com/
- https://www.callicoder.com/categories/golang/
- https://quii.gitbook.io/learn-go-with-tests/

### Batteries included

In a typical application development we do things like **reading/writing files**, **parsing JSON data**, **work with databases**, **create HTTP REST APIs** etc. Go language has built-in support for all of these. In addition to this, Go also includes support for unit testing, HTTP testing in the standard library itself.

### Out of the box tooling

Go provides nice tooling support for code formatting, test code coverage checks, benchmarking etc.

- [Go Tooling in Action](https://www.youtube.com/watch?v=uBjoTxosSys)
- [gofmt](https://golang.org/cmd/gofmt/)
- [Go Test Coverage](https://blog.golang.org/cover)
- [Go Test Benchmarks](https://golang.org/pkg/testing/#hdr-Benchmarks)

## 2. WTF Moments

Nowadays we have many wonderful programming languages like Java, Kotlin, Python, JavaScript, Go, Rust to name a few. If I have to select a programming language for a real application(not pet projects) I would at least expect some features that are available in most of the programming languages. So, I expected some of the commonly used features in Go and I am very surprised to know that Go doesn't have them.

### Very very limited collections

- Storing a collection of unique values is such a very common requirement and most of the languages provides **Set** collection to support that. Go doesn't have any built-in support for **Set** type collection. As there is no **Set** datatype in Go, we mostly ended up using map to store the values as keys.

- Getting all the keys or values from a map is also a very common use-case. Go doesn't have any **map.keys()** or **map.values()** to support that. You have to iterate the map and collect the keys/values in a slice.

- Slices are similar to Lists, but not exactly Lists. If you think you can check whether a slice has an element by **mySlice.contains(value)**, think again. There is no **.contains()** method on slice, you have to loop over slice elements and check.

- The **map** datatype by definition stores key-value pairs, so you might think at least **map** might have **.containsKey()** method. Sorry. To check if a key exists in map you have to write the following code:

```go
if val, ok := mymap["foo"]; ok {
    //do something here
}
```

{{< figure src="/images/why-cat.webp" alt="But, Why?"  width="350" height="300" >}}

### Error handling

There is no concept of Exceptions in Go unlike in Java or Ruby etc. In Go, a function can return multiple values. A common pattern is to return a pair of (result, error) from a function.

The following code demonstrate how errors are handled in idiomatic Go.

```go
func doSomething(n int) (int, error) {
  if n == -1 {
    return 0, errors.New("Are you crazy?!!")
  }
  return n * n, nil
}

func iWishIGetPaidByLinesOfCode() {
  square, err := doSomething(3)
  if err != nil {
    log.Error("You think each function returning a value and error is cool. "+ err.Error())
  }
  //do something useful with square
}
```

Ok, instead of throwing Exceptions and handling them in Catch blocks we are returning the errors as first class values. After all the code doesn't look that bad, isn't.

Let's take a little more complex example of reading a file line by line and insert the data into database.

```go
db, err := getDbConnection()
if err != nil {
  log.Fatal(err)
  return
}
defer db.Close()
data, err := ioutil.ReadFile("test.txt")
if err != nil {
    fmt.Println("File reading error", err)
    return
}
for _, line := range data.lines() { //just assume there is .lines() which returns slice of strings
  stmt, err := db.Prepare("INSERT INTO users(name) VALUES(?)")
  if err != nil {
    log.Error(err)
    return
  }
  res, err := stmt.Exec(line)
  if err != nil {
    log.Error(err)
    return
  }
  lastId, err := res.LastInsertId()
  if err != nil {
    log.Error(err)
    return
  }
  rowCnt, err := res.RowsAffected()
  if err != nil {
    log.Error(err)
    return
  }
  log.Printf("ID = %d, affected = %d\n", lastId, rowCnt)
}
```

When I take a glance at this code all I see is error checks with the actual business logic stuffed somewhere in between.

Yes, you can argue that handling errors are part of actual workflow. But there must be a better way of dealing with the errors.

I believe the language should give the developer a provision to define the scope of successful or failure of an operation and let the developer handle each error individually or as a whole as they seems fit.

In the above example, suppose if I don't care whether the whole method failed due to reading file, talking to database or while getting LastInsertId I consider it as a failure and just log the error and return. Then I want to implement the same functionality as follows:

```go
func loadUsers() {
  try {
    db := getDbConnection()
    defer db.Close()
    data := ioutil.ReadFile("test.txt")
    for _, line := range data.lines() {
      stmt := db.Prepare("INSERT INTO users(name) VALUES(?)")
      res := stmt.Exec(line)
      lastId := res.LastInsertId()
      rowCnt := res.RowsAffected()
      log.Printf("ID = %d, affected = %d\n", lastId, rowCnt)
    }
  } catch(err) {
    log.Error("Something terrible happened. Error: "+err.Error())
    return err
  }
}
```

If I want to handle DB connection errors differently I would handled them as follows:

```go
func loadUsers() {
  var db *sql.DB

  try {
    db = getDbConnection()
    defer db.Close()
    }
  } catch(err) {
    log.Error("DB connection Error: "+err.Error())
    return err
  }

  try {
    data := ioutil.ReadFile("test.txt")
    for _, line := range data.lines() {
      stmt := db.Prepare("INSERT INTO users(name) VALUES(?)")
      res := stmt.Exec(line)
      lastId := res.LastInsertId()
      rowCnt := res.RowsAffected()
      log.Printf("ID = %d, affected = %d\n", lastId, rowCnt)
    }
  } catch(err) {
    log.Error("Something terrible happened. Error: "+err.Error())
    return err
  }
}
```

In Go most of the standard library functions return a value and error and while writing idiomatic Go code your code is full of error checks. That might look perfectly fine for many, but for my personal taste that's terrible.

### Funky Date Formatting

Kidnap a software developer and put a gun to his/her head and tell them to format a date object to print **year-month-day** in only one try.

{{< figure src="/images/go-date-format.webp" alt="Go Date Formatting" >}}

I believe most of the developers will bet their life to **dateObject.format("yyyy-mm-dd")**. This notation is so common in many programming languages. But in Go, we need format date object (time.Time) using **dateObject.Format("2006-01-02")**.

In Go we need to use the magical date **Mon Jan 2 15:04:05 MST 2006** for [formatting/parsing dates](https://programming.guide/go/format-parse-string-time-date-example.html).

### Go feels low-level and verbose

As of now Go doesn't have support for **Generics** or **Streams**. Lack of support for Generics and Streams combined with lack of decent collection types and operations result in writing lot of boilerplate code. Add error handling into this mix and voila...

I really wish I get paid by number of lines of code while working on Go projects :-)

{{< figure src="/raw-images/money-tears.gif"  width="350" height="300" >}}

## 3. Retrospection and Realization

After going through all these WTF moments, I started thinking how come Go is so popular!!?? A programming language simply doesn't become that popular without having some good reasons (of course the hipster developers want to work on something new for every six months...that's a different story). There must be something really good about Go which makes it so popular. So, I started thinking what I am missing??!!!.

Sometimes people use programming languages/tools for totally different purposes than what their creators originally created for.
So, I thought of hearing from the Go Lang creators and I found this wonderful talk by Rob Pike.

{{< youtube rFejpH_tAHM >}}

After watching the talk I kind of got answers to some of my questions.

### Go is designed to be simple

It seems Go is not created to compete with any existing popular languages. So, they(Go creators) are not in the race of providing all the features of popular languages plus some more. They want to create a simple programming language with minimal features. Sometimes the simplicity may result in verbosity.

For example, lets take a look at the following Java/Spring code.

```java
@Transactional
public void creteUser(User user) {
  this.validateUser(user);
  this.userRepository.save(user);
  this.addressRepository.save(user.getAddress());
}
```

This method looks simple, isn't it. But to fully understand this method's runtime behaviour one needs to know Spring's magic.
First, the **@Transactional** annotation start a DB transaction if it is not already a part of an existing transaction. Then, if the method completes normally the DB transaction gets committed automatically, of if there is any **RuntimeException** occurs DB transaction gets rolled back automatically. Any of these details are not visible in the code.

Lets look at Go version of the same method.

```go
function createUser(user User) {
  db, err := getConnection()
  if err != nil {
    log.Fatal(err)
  }

  ctx := context.Background()
  tx, err := db.BeginTx(ctx, nil)
  if err != nil {
    log.Fatal(err)
  }

  _, err = tx.ExecContext(ctx, "INSERT INTO users (name, gender) VALUES (...)")
  if err != nil {
    tx.Rollback()
    return
  }

  _, err = tx.ExecContext(ctx, "INSERT INTO addresses (street, city) VALUES (...)")
  if err != nil {
    tx.Rollback()
    return
  }

  err = tx.Commit()
  if err != nil {
    log.Fatal(err)
  }
}
```

Anyone with little bit of programming background can understand (at least conceptually) what is going on in here.
Yes, it is verbose, but clear. No background magic going on here. Sometimes a bit of verbosity is better than smart and clever code.

### Go is designed to solve specific problems

I believe usually programming languages are created by its creator to solve a specific type of problems that they are facing. I don't think Rob Pike or other Go creators were writing any e-commerce web applications, or fancy REST API and they created Go to solve those web app development challenges. To my guess they might have been working on building low-latency, high-performance servers, low level protocol implementations etc using C++ or some other low level programming languages and they were not happy with C++. At least that's what I understood from their talks.

So, comparing Go with Java or Ruby might be apples to oranges comparison. If I want to build a web application may be I am more productive with Spring Boot or Ruby On Rails than Go.

When I try to learn a new language I looks for sample applications on Github to learn how people usually do something in that language. When I search for Go sample projects, most of the projects are like **Docker, Kubernetes, CLI clients** etc and I hardly find any typical web applications. That doesn't mean you can't use Go for web apps or REST API, but it may not be right tool for those type of applications especially if productivity is a key factor.

## 4. Summary

While learning a new language we might unknowingly compare with our favourite programming language and keep comparing whether the new language is having all the features of our current favourite language. If we keep comparing everything then it is hard to see the beauty of the new language. We should explore the new language and learn its strengths and weaknesses.

While learning Go, I keep comparing it with Java, Spring Boot combo which is the best tech stack in my opinion. I shouldn't have compared Go with Java + Spring Boot.

Also, we shouldn't hang on to little things like **Funky Date Formatting** :-) How hard it is to remember **01 is for month, 02 is for day, 03 is for hour** etc. May be it is better than **"yyyy-mm-dd"** format. How many times we get confuse about does **MM** represents Month or Minutes??!! With Go's notation there is no such confusion.

Java has vast collection library, generics, streams, huge ecosystem of frameworks and libraries. While using Java + Spring Boot I can simply add a bunch of starter which mostly take care of boilerplate code and start working on business logic.

I really get surprised when people keep saying Java is old, java is verbose, Java is all about creating Factories, Builders etc etc. Yes, the frameworks(Spring) might have Factories, Builders etc but I don't even remember when was the last time I had to create Factory in my business application. If anyone think Java is same as it was in EJB 1.x days then please do yourself a favour and take another look at Java.

To summarize, I like Go, but it's not in my top 3 favourites. I choose Java over Go for most of the applications. But if I ever want to create a CLI application then I don't even think for a second, I will just use Go with [Cobra](https://github.com/spf13/cobra).

# My proud moment as a mentor


As software developers, most of us implicitly assume gaining strong technical skills might make us successful.
Many of us assume being good at cutting edge technologies, mastering keyboard shortcuts, being able to work on 
front-end, back-end and infrastructure etc etc will make us successful in our careers.

Yes, the above mentioned skills definitely help, but just being good at tech skills might not make us successful.
There is more to it. 

<!--more-->


Let me share you the story of my friend who restarted her career after a break of 3 years.
I would say she really did amazing, and I would like to share her story.

She started her career as a software developer few years ago, primarily worked on a commercial product.
While using that product she never had to use any of typical programming tools and technologies like HTML, JS, CSS, Git, Java etc.
She is a software engineer who likes to work from 9 to 5 and doesn't like to spend rest of the time on computers at all.
She had to take a break from career for some personal reasons, and meanwhile the product she used to work on went out of business too.
After 3 years she thought of restarting her career and want to pick a stream which has more job opportunities.

She asked me to help her to figure out which technologies to learn and if possible to teach her those technologies.
I suggested her to learn Java, Spring Boot, ReactJS because there are more job opportunities for these technologies.

I started teaching her Core Java, HTML, JS, CSS. She was able to pick up these fine.
Then I started teaching her Maven, Git, jQuery, NPM etc and parallelly giving a 30,000 feat overview on Java frameworks like
Servlets, Tomcat, Hibernate, Spring. She got overwhelmed by the number of technologies, tools required to build a web application.

![Overwhelming](/images/overcoming.webp "Overwhelming")

That is the time I realized that I didn't get overwhelmed by the number of technologies because I learned all those things over the years.
But, nowadays if somebody wants to get into software development it's really hard due to the number of tools and technologies to be familiar with.

After 4 months of daily practice she started looking for a job and luckily she got a job opportunity as well.
She started working on a project in which they used Java, Groovy, Spring Boot, AngularJS 1.x, ReactJS, Vagrant, Git, Maven, Mockito,
Postgresql, Docker just to name a few.

Just imagine how intimidating it would be for someone who is new to software development to pick up these many technologies!!
In the beginning she couldn't even understand what they are talking about when the team was talking about Vagrant, Git Cherrypick, docker volumes etc.

Fast forward 6 months, her team lead gave feedback to her that:

> If a task is assigned to her he can forget about it and hope it's done.
If she is in the call with client he(team lead) doesn't pay much attention because he is sure she will get all the required details to do the work".
And, most importantly client side folks prefer talking to her because she grasps things so fast and so well.

If this is not an amazing thing what can be!!!

So, how this magical transformation happened??!!

Let us take a step back and think about what management(Manager, team lead etc) expects from a Developer?

In big enterprise organizations, developers don't work on shiny new things every day.
The challenge is not in learning cutting edge technologies, or mastering all front-end, back-end and DevOps tools.
The challenge is in getting things done with the limited available resources in given short time.
Many times we might not have enough documentation, the SME's might not be available all the times.
We might need to follow up with SMEs, business folks, talk to other teams in order to get the information required to do our job.

Based on my observation, following qualities made her a good software developer.

**Learn on need basis:** As and when she come across new things (at day to day work, team discussion etc) she noted them down and came back to me to learn about them.

**Don't hesitate to ask questions:** When she is in calls with team or clients, 
she won't hesitate to ask for more details, repeat something if she couldn't understand what they said until she is clear about the requirement.

**Don't compromise on timelines:** Good luck trying to convince her for less timelines. She is very very careful while giving estimates.
If she is not sure she can complete the task in given timelines then she openly say it without any hesitation.

Though occasionally her team lead got pissed off at her about not accepting for less timelines, 
but later slowly he realized that those tasks actually needs more time and appreciated for being strict about timelines.

**Get things done**: She is a task master. Once a task is assigned to her she will make sure that it is moved to DONE lane.
She follows up with the people, gather more details, ask lot of questions, seek other's help and finally get it done.

Who wouldn't like to have such team members??!!!

I can't express how happy I am to see her growing in her career, and I wish her all the best :-)

> Though strong technical skills definitely help, to become a successful software developer 
you no need to be someone who is passionate about technologies, works on pet projects off the hours, master the IDE shortcuts.
Overall, what I am trying to say is "Persistence is the key. Getting things done mentality is what management looks for in their team members".

# Remote debugging Spring Boot application


We all know how to run Spring Boot application in debug mode from our favorite IDEs(IntelliJ IDEA, Eclipse etc) and debug the code.
Recently I had to debug a Spring Boot application which is running on a remote server. 

There is already lot of information on the Internet on how to remote debug java applications, but some approaches didn't work for me.
So, here I would like to share my findings.

<!--more-->


## If application is running on localhost
Suppose you are running your application on localhost only and remote debug from your IDE.

```shell script
$ export JAVA_OPTS='-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8787'
$ java $JAVA_OPTS -jar myapp.jar
```

> You might have seen debug options specified as "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8787".
> But in the newer versions of Java the preferred approach is "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8787"

Now from Intellij IDEA you can remote debug the application by creating remote debug configuration by 
`Run -> Edit Configurations... -> Add New Configuration (Command + N) -> Enter Host and Port values`

![RemoteDebuggingIntellijIdea](/images/java-remote-debug.webp "RemoteDebuggingIntellijIdea")

After creating debug configuration click on Debug, and you should see the following message:

> `Connected to the target VM, address: 'localhost:8787', transport: 'socket'`

## If application is running on remote host
Suppose you need to run the application on a remote host and debug from your localhost.

**On Remote Server:**

```shell script
$ export JAVA_OPTS='-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8787'
$ java $JAVA_OPTS -jar myapp.jar
```

I followed the same approach as mentioned above and in debug configuration I gave the remote server's IP address instead of `localhost`.
But, when I click on Debug I got the following error:

> `unable to open debugger port (198.xxx.xxx.xx:8787): java.net.ConnectException "Connection refused"`

After a bit of googling I finally resolved the issue by starting the application on remote server as follows:

```shell script
$ export JAVA_OPTS='-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=0.0.0.0:8787'
$ java $JAVA_OPTS -jar myapp.jar
```

Note we gave the address as **address=0.0.0.0:8787** instead of **address=8787**.

With this I am able to remote debug the application in the same way I did when I run it on localhost.

By the way, we can also run our Spring Boot application in debug mode as follows without creating jar file:

> `mvn spring-boot:run -Dspring-boot.run.jvmArguments="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8787"`

Happy debugging :-)
# What I Learned in Week-14, 2020


It's been tough time for everyone due to the Corona virus situation. 
Staying at home for weeks and weeks is not easy, but it is very necessary thing to do so that we don't spread the virus.

However, if I look on the bright side I used to spend minimum 3 hours for office commute everyday.
Now I am using that time for various other activities including learning some new things.

### So, here is what I learned in the last week:

* I have published a new article on [How to Imposing Code Structure Guidelines using ArchUnit](https://sivalabs.in/impose-architecture-guidelines-using-archunit/)

* I use **Spring Security JWT based authentication** for most of my personal projects. 
Instead of copy-pasting the same code again and again I created a [Spring Boot JWT Security Starter](https://github.com/sivalabs/spring-boot-jwt-security-starter)

* Though Spring Boot is very easy, understanding the internal working mechanism of it is a bit challenging and overwhelming.
**Marco Behler** published the following new articles covering Spring framework, Spring Boot and Spring Security 
and I highly recommend to read them.

    * [What is Spring Framework? From Dependency Injection to Web MVC](https://www.marcobehler.com/guides/spring-framework)
    * [What is Spring Boot? Autoconfigurations In-Depth](https://www.marcobehler.com/guides/spring-boot)
    * [Spring Security: Authentication and Authorization In-Depth](https://www.marcobehler.com/guides/spring-security)

* If you are trying to learn **Deploying Spring Boot + SpringCloud Applications on Kubernetes** then 
I highly recommend this course https://udemy.com/course/kubernetes-crash-course-for-java-developers/ 
by **Ranga Karanam** [In28Minutes](https://twitter.com/In28Minutes). 
A very practical and hands-on course covering from basics to advanced topics.

* If your application is heavily relying upon **Spring ecosystem** you may also want to follow **Spring style coding/formatting standards**. 
In that case this [spring-javaformat-maven-plugin](https://github.com/spring-io/spring-javaformat) might come handy.
Also, you can take a look at [Prettier Java](https://github.com/jhipster/prettier-java) by **JHipster** team.

We may not get the opportunity to work on many things in our official projects. 
For example, I never get a chance to setup **SSL/TLS certificates**, never worked on setting up **database cluster** etc.
Similarly I have some gray areas in some concepts like **OAuth**, **SQL query optimization** etc.
So, I thought of why not use this available time to brush up on these topics. 
I found the following wonderful resources to learn those concepts. 

### Database Query Optimization using EXPLAIN ANALYZE
* [The Postgres Performance Tuning Manual: Query Plans](https://blog.gojekengineering.com/the-postgres-performance-tuning-manual-query-plans-52a023c2342d)
* [Index Columns for LIKE in PostgreSQL](https://niallburkley.com/blog/index-columns-for-like-in-postgres/)
* [EXPLAIN documentation](https://www.postgresql.org/docs/9.1/sql-explain.html)

### OAuth 2.0 & Open ID Connect
* [OAuth 2.0 and OpenID Connect (in plain English)](https://www.youtube.com/watch?v=996OiexHze0)
* [Security Patterns for Microservice Architectures](https://developer.okta.com/blog/2020/03/23/microservice-security-patterns)

I would strongly suggest to read articles on [Okta Developer Blog](https://developer.okta.com/blog/) 
and subscribe to [OktaDev Youtube Channel](https://www.youtube.com/channel/UC5AMiWqFVFxF1q9Ya1FuZ_Q) 
for learning about application security.

### Pluralsight Free April Month
Just in case you haven't already heard, **Pluralsight** is offering [**FREE ACCESS for April 2020 month**]((https://t.co/9Son2fPnwG?amp=1)).
Happy learning :-)

# Imposing Code Structure Guidelines using ArchUnit

While building the software, we all agree, as a team, to follow a set of guidelines which are typically considered as best practices.
But during the development, developers might violate those guidelines unknowingly or ignorance.
Typically, we rely upon **code reviews** or code quality checking tools like **SonarQube**, **PMD**, etc. to check for such violations.
But some of the guidelines could be opinionated decisions which might not be able to automate using SonarQube, PMD etc.

<!--more-->


For example, typically I would like to follow the guidelines mentioned below for my Java based applications:

1. **Follow 3-tier layering structure** (Web, Service, Repository layers) where any layer can only talk to the immediate lower layer and lower layer must not talk to upper layer.
   i.e, Web layer can talk to Service layer, Service layer can talk to Repository layer. But Repository layer can't talk to Service or Web layer, Service layer can't talk to Web layer.

2. If the application is big, we might want to follow **Package-By-Feature** where only the Web and Service components are public and rest of the components should be package-private in each feature package.

3. While using Spring dependency injection, **Don't use Field-based injection** and prefer Constructor-based injection.

Like this, there could be many guidelines we want to follow. The good news is we can impose these guidelines as verifiable JUnit tests using [ArchUnit](https://www.archunit.org/).

![ArchUnit](/images/ArchUnit-Logo.webp "ArchUnit")

Here is the [ArchUnit UserGuide](https://www.archunit.org/userguide/html/000_Index.html)

Let us see how we can use ArchUnit for testing our architecture guidelines.

Add the following **archunit-junit5** dependency.

```xml
<dependency>
    <groupId>com.tngtech.archunit</groupId>
    <artifactId>archunit-junit5</artifactId>
    <version>1.2.1</version>
    <scope>test</scope>
</dependency>
```

Let us see how we can apply various guidelines that I mentioned above.

### Rule 1: Services and Repositories should not talk to Web layer

```java
package com.sivalabs.moviebuffs;

import com.tngtech.archunit.core.domain.JavaClasses;
import com.tngtech.archunit.core.importer.ClassFileImporter;
import com.tngtech.archunit.core.importer.ImportOption;
import org.junit.jupiter.api.Test;
import static com.tngtech.archunit.lang.syntax.ArchRuleDefinition.*;
import static com.tngtech.archunit.library.Architectures.layeredArchitecture;

class ArchTest {
   JavaClasses importedClasses = new ClassFileImporter()
           .withImportOption(ImportOption.Predefined.DO_NOT_INCLUDE_TESTS)
           .importPackages("com.sivalabs.moviebuffs");
   
    @Test
    void servicesAndRepositoriesShouldNotDependOnWebLayer() {
      noClasses()
          .that().resideInAnyPackage("com.sivalabs.moviebuffs.core.service..")
            .or().resideInAnyPackage("com.sivalabs.moviebuffs.core.repository..")
          .should()
            .dependOnClassesThat()
            .resideInAnyPackage("com.sivalabs.moviebuffs.web..")
          .because("Services and repositories should not depend on web layer")
          .check(importedClasses);
    }

}
```

The ArchUnit provides DSL that's pretty clear to understand the intent of the test. The above test code reads like English sentence and easy to understand what we are testing.

### Rule 2: Should follow Layered Architecture

In a typical Spring Boot application, **Service** layer depends on **Repository** layer, **Web** and **Config** layers depends on **Service** layer. **We don't want Web or Config layer directly talk to Repository layer.**

We can impose that restriction with the following test.

```java
@Test
void shouldFollowLayeredArchitecture() {
  layeredArchitecture()
      .layer("Web").definedBy("..web..")
      .layer("Config").definedBy("..config..")
      .layer("Service").definedBy("..service..")
      .layer("Persistence").definedBy("..repository..")

      .whereLayer("Web").mayNotBeAccessedByAnyLayer()
      .whereLayer("Service").mayOnlyBeAccessedByLayers("Config", "Web")
      .whereLayer("Persistence").mayOnlyBeAccessedByLayers("Service")
      .check(importedClasses);
}
```

### Rule 3: Spring's @Autowired should NOT be used with Field based injection

```java
@Test
void shouldNotUseFieldInjection() {
    noFields()
      .should().beAnnotatedWith(Autowired.class)
      .check(importedClasses);
}
```

### Rule 4: Should follow Naming convention

We might want to follow some naming conventions like all service class names should end with **Service** etc.

```java
@Test
void shouldFollowNamingConvention() {
    classes()
        .that().resideInAPackage("com.sivalabs.moviebuffs.core.repository")
        .should().haveSimpleNameEndingWith("Repository")
        .check(importedClasses);

    classes()
        .that().resideInAPackage("com.sivalabs.moviebuffs.core.service")
        .should().haveSimpleNameEndingWith("Service")
        .check(importedClasses);
}
```

### Rule 5: Should use JUnit 5 only

We might want to use **JUnit 5** as our testing framework. But JUnit 4 dependency might have pulled into classpath as a transitive dependency (cough... **Testcontainers**... cough) and we might accidentally import JUnit4 classes/annotation such as **@Test**, **Assert** etc by mistake.

We can restrict the usage of JUnit 4 classes as follows:

```java
@Test
void shouldNotUseJunit4Classes() {
    JavaClasses classes = new ClassFileImporter()
        .importPackages("com.sivalabs.moviebuffs");

    noClasses()
        .should().accessClassesThat().resideInAnyPackage("org.junit")
        .because("Tests should use Junit5 instead of Junit4")
        .check(classes);

    noMethods().should().beAnnotatedWith("org.junit.Test")
        .orShould().beAnnotatedWith("org.junit.Ignore")
        .because("Tests should use Junit5 instead of Junit4")
        .check(classes);
}
```

I just described only a few of the possibilities, and you can get as creative as you want :wink:

Please read the official [ArchUnit UserGuide](https://www.archunit.org/userguide/html/000_Index.html) on what are all the cool things you can do with **ArchUnit**.

# What I Learned in Week-11, 2020


I come across many blog posts from various sources like [Twitter](https://twitter.com/sivalabs),
[StackOverflow](https://stackoverflow.com/), etc., and some of them are very helpful.
Also, in day-to-day work I learn a few new things, explore some new tools. 
Usually I keep track of this information using **Browser Bookmarks** and [Pocket](https://app.getpocket.com/).

I thought maybe this information could be useful for others too. So, I am going to share it weekly on my blog.

<!--more-->


* If you are looking for a detailed and practical guide for implementing Microservices using Spring Boot, then look no further:
[Java Microservices: A Practical Guide](https://www.marcobehler.com/guides/java-microservices-a-practical-guide)

* While using JPA, how many times do we want to see the generated SQL query with actual parameter values instead of placeholders?
We can use [p6spy-spring-boot-starter](https://github.com/gavlyukovskiy/spring-boot-data-source-decorator) to log the generated SQL statements with actual parameters.

* I am surprised at how easy it is to create CLI tools using **NodeJS** and
[inquirer.js](https://github.com/SBoudrias/Inquirer.js).
At work, I have created a simple CLI tool to build and deploy Spring Boot applications on PCF in just a couple of hours.
If you are interested, this is a great tutorial:
[Build a JavaScript Command Line Interface (CLI) with Node.js](https://www.sitepoint.com/javascript-command-line-interface-cli-node-js/)

* If you are just starting out with **GitHub Actions** or thinking of migrating from **TravisCI**, then:
[Continuous Integration of Java project with GitHub Actions](https://medium.com/faun/continuous-integration-of-java-project-with-github-actions-7a8a0e8246ef)

* If you are new to **Git**, then this post is definitely a good starting point covering various concepts of Git with pictorial representation:
[How to teach Git](https://rachelcarmena.github.io/2018/12/12/how-to-teach-git.html) 

* If your Spring Boot application is starting slowly on your MacOS, then this post has a solution for you:
[Fixing Slow Spring Boot Startup](https://blog.noizwaves.io/2017/09/02/slow-spring-boot-startup.html)

* **Docker** became part of day-to-day work, and once in a while we want to clean up unnecessary stuff:
[How To Remove Docker Containers, Images, Volumes, and Networks](https://linuxize.com/post/how-to-remove-docker-images-containers-volumes-and-networks/)

* You might have heard about **Hexagonal Architecture** and are wondering how to implement it using Spring, then this article is for you:
[Achieving framework-independent zen with the power of interfaces and hexagonal architecture](https://rskupnik.github.io/framework-independence-with-hexagonal-architecture)

* If you don't like **Setters and Getters** but you are using them just because of JSON to Object marshalling, then this is for you:
[Deserialize Immutable Objects with Jackson](https://www.baeldung.com/jackson-deserialize-immutable-objects)

* If you have some time to burn and want to explore interesting tech stuff:
[DEVURLS](https://devurls.com/)

Happy learning :-)

# Spring Boot Integration Testing using TestContainers Starter


One of the many reasons for the huge popularity of Spring and Spring Boot is
its great support for [Testing](https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#boot-features-testing).
We can write unit tests using [Mockito](https://site.mockito.org/) without requiring any Spring features. 
And, we can write Integration Tests using Spring testing support by creating Spring ApplicationContext. 

<!--more-->


> Read [Guide to Testing Spring Boot Applications](https://sivalabs.in/spring-boot-testing/)
    
While running integration tests, we might need to interact with external services like relational databases, NoSQL datastores, Kafka, etc. 
We can spin up those external services as Docker containers and run tests against them.

## Testcontainers

**From Testcontainers docs:**

> Testcontainers is a Java library that supports JUnit tests, providing lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container.

We can use Testcontainers to spin up a [Singleton docker container](https://www.testcontainers.org/test_framework_integration/manual_lifecycle_control/#singleton-containers) 
in a Spring Boot integration test as follows:

```java
@SpringBootTest
@ContextConfiguration(initializers = {UserServiceIntegrationTest.Initializer.class})
class UserServiceIntegrationTest {
    private static PostgreSQLContainer sqlContainer;
    
    static {
        sqlContainer = new PostgreSQLContainer("postgres:10.7")
                .withDatabaseName("integration-tests-db")
                .withUsername("sa")
                .withPassword("sa");
        sqlContainer.start();
    }

    static class Initializer implements ApplicationContextInitializer<ConfigurableApplicationContext> {
        public void initialize(ConfigurableApplicationContext configurableApplicationContext) {
            TestPropertyValues.of(
              "spring.datasource.url=" + sqlContainer.getJdbcUrl(),
              "spring.datasource.username=" + sqlContainer.getUsername(),
              "spring.datasource.password=" + sqlContainer.getPassword()
            ).applyTo(configurableApplicationContext.getEnvironment());
        }
    }

    @Autowired
    private UserService userService;
    
    @Test
    void shouldGetAllUsers() {
        // test userService.getAllUsers()
    }   

}
```

This is such a common requirement in Spring Boot applications, so the community built
[Testcontainers Spring Boot starter](https://github.com/testcontainers/testcontainers-spring-boot) to make it easier.

## Testcontainers Spring Boot starter
The Testcontainers Spring Boot starter depends on **spring-cloud-starter**. 
If you are NOT already using any SpringCloud starters in the application, then include **spring-cloud-starter** as a **test** dependency.

```xml
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter</artifactId>
    <scope>test</scope>
</dependency>
```

Now include the data service library depending on what data service you need.
For example, if you want to use a **Postgresql** docker container, then add the following dependency:

```xml
<dependency>
    <groupId>com.playtika.testcontainers</groupId>
    <artifactId>embedded-postgresql</artifactId>
    <scope>test</scope>
</dependency>
```
When you add the **embedded-postgresql** dependency, the following properties will be added to the Environment:

* embedded.postgresql.port
* embedded.postgresql.host
* embedded.postgresql.schema
* embedded.postgresql.user
* embedded.postgresql.password

We can use these properties to configure datasource properties for testing.

**Usually we want to spin up docker containers for integration tests but not for unit tests.
So we can disable it by default and only enable during integration tests by using profile based configuration.**

**src/test/resources/bootstrap.properties**

```properties
embedded.postgresql.enabled=false
```

**src/test/resources/bootstrap-integration-test.properties**

```properties
embedded.postgresql.enabled=true
spring.datasource.driver-class-name=org.postgresql.Driver
spring.datasource.url=jdbc:postgresql://${embedded.postgresql.host}:${embedded.postgresql.port}/${embedded.postgresql.schema}
spring.datasource.username=${embedded.postgresql.user}
spring.datasource.password=${embedded.postgresql.password}
```

Now we can run integration tests with **integration-test** profile using **@ActiveProfiles** as follows:

```java
@SpringBootTest
@ActiveProfiles("integration-test")
class UserServiceIntegrationTest {
    
    @Autowired
    private UserService userService;
    
    @Test
    void shouldGetAllUsers() {
        // test userService.getAllUsers()
    }   

}
```

We may want to use a specific version of a docker image, then you can configure it as follows:

**src/test/resources/bootstrap-integration-test.properties**

```properties
embedded.postgresql.dockerImage=postgres:10.7
embedded.postgresql.enabled=true
```

The Testcontainers Spring Boot starter already provides support for most commonly used containers like
Postgresql, MariaDB, MongoDB, Redis, RabbitMQ, Kafka, Elasticsearch, etc.

To my surprise, there is no direct support for MySQL as of now. But there is a simple workaround
for this as described here: https://github.com/testcontainers/testcontainers-spring-boot/issues/151.

# Creating Yeoman based Spring Boot Generator


I have been working with Spring and [Spring Boot](https://spring.io/projects/spring-boot) for many years, and
I needed to create a lot of Spring (Boot) applications for various reasons, like
[blog posts](https://sivalabs.in/categories/springboot/), [sample apps](https://github.com/sivaprasadreddy/spring-boot-tutorials),
[book sample code](https://github.com/sivaprasadreddy/beginning-spring-boot-2), and for my personal learning as well.
So, I needed some tool/mechanism to quickly create a Spring (Boot) application with the most commonly used configuration.

<!--more-->


I know we have the most popular [Spring Boot Initializer](http://start.spring.io/) to create Spring Boot applications.
But, it generates the application with only selected starter dependencies added and nothing more.

Most of the time, I needed to configure some common properties in the **application.properties** file, additional libraries like
**Testcontainers**, Zalando's **Problem Spring Web**, etc. So, I thought of automating this repetitive work in some way.

As a first attempt, I tried to automate this by creating various [Maven archetypes](https://github.com/sivaprasadreddy/maven-archetype-templates)
with various combinations of technologies. It helped me to some extent, but I had to refactor a few things after generating the application.
Also, I couldn't find a way to include some dependencies/configuration conditionally based on user input.

As a second attempt, I tried to extend [Spring Boot Initializer](https://github.com/sivaprasadreddy/spring-initializr-extensions)
by adding new features. I felt it was okayish but not productive enough. In addition to that, I needed to host it somewhere and
make sure it was running all the time.

And then I thought of creating a [Yeoman](https://yeoman.io/) based generator.
First, I came to know about Yeoman when AngularJS 1.x was hot, and there was an amazing Yeoman generator for
creating AngularJS applications. Later I stumbled upon [JHipster](https://www.jhipster.tech/), which blew my mind.
JHipster is amazing, and what you can build with JHipster in just 30 minutes is phenomenal.
However, there are certain JHipster features that do not fit my personal preferences, such as:

*   I like **jar** packaging
*   I like to use **spring-boot-starter-\*** rather than configuring individual libraries
*   I like to have an option to generate an application **without spring-security**
*   I prefer **Flyway** over **Liquibase**
*   I like to have only minimum and required configuration, i.e., **no AsyncConfiguration, LocaleConfiguration, CacheConfiguration, Logstash Logging**, etc.
*   I like **.properties** over **.yml**

So, I decided to write my own Spring Boot Generator using Yeoman, and
I started creating [generator-springboot](https://github.com/sivaprasadreddy/generator-springboot).
And, I have implemented the generator to generate a Spring Boot application with the following features:

*   Spring Boot REST API with jar type packaging
*   Generate CRUD operations along with Unit and Integration Tests
*   CORS configuration
*   Swagger UI Integration
*   Spring Data JPA integration with option to select databases like MySQL, PostgreSQL, MariaDB, etc.
*   Flyway or Liquibase data migration support
*   Spring Boot Actuator configuration
*   Integration with Config Server, Service Registry, Sleuth, Zipkin
*   TestContainers integration
*   JUnit 5
*   Docker configuration
*   Jenkinsfile

I thought of writing an article on how to create a Yeoman-based generator so that others can also create their own generators if interested.

First of all, there is wonderfully written [Official Documentation](https://yeoman.io/authoring/) on how to create our own Yeoman generator.
I strongly recommend going through it.

### How Yeoman Generators work?
To put it simply, a Yeoman generator works as follows:

*   Create template file(s) with placeholders.
*   Prompt the user for inputs.
*   Generate files by replacing the placeholders with user-provided values.

Okay, enough talk. Let's get to business.

## Create Spring Boot Yeoman Generator

There are some fundamental concepts to be understood which are explained clearly in the documentation, and repeating them here is redundant.
So, once again I strongly recommend going through this official documentation to get some basic understanding of concepts https://yeoman.io/authoring/index.html.

Create a directory called **generator-springboot** and create the following files and directories inside generator-springboot as follows:

```bash
+- generator-springboot/
   +- package.json
   +- generators/
      +- app/
         +- index.js
```

**package.json**

```json
{
  "name": "generator-springboot",
  "version": "0.0.1",
  "description": "A Yeoman generator for generating Spring Boot microservices",
  "files": ["generators"],
  "main": "index.js",
  "keywords": [
    "yeoman-generator",
    "java",
    "spring",
    "spring-boot",
    "microservice"
  ],
  "dependencies": {
    "yeoman-generator": "^4.0.1"
  }
}
```

**generators/app/index.js**

```js
"use strict";
const Generator = require("yeoman-generator");

module.exports = class extends Generator {
  method1() {
    this.log("method 1 just ran");
  }

  method2() {
    this.log("method 2 just ran");
  }
};
```

During development, we can symlink to the npm package to run the generator.
From the root of your generator project (**generator-springboot/** folder), type:

```bash
generator-springboot> npm link
```

And then run the generator as follows:

```bash
generator-springboot> yo springboot
```

It will execute both method1() and method2() and print the log statements on console.

As we have not explicitly mentioned any priority, the methods method1() and method2() ran with default priority as follows:

```js
"use strict";
const Generator = require("yeoman-generator");

module.exports = class extends Generator {
  default: {
    method1() {
        this.log("method 1 just ran");
    }

    method2() {
        this.log("method 2 just ran");
    }
  }
};
```

Yeoman defines the following priorities so that we can hook up our custom logic at appropriate priority executions.

The available priorities are (in running order):

1. **initializing** - Your initialization methods (checking current project state, getting configs, etc)
2. **prompting** - Where you prompt users for options (where you'd call this.prompt())
3. **configuring** - Saving configurations and configure the project (creating .editorconfig files and other metadata files)
4. **default** - If the method name doesn't match a priority, it will be pushed to this group.
5. **writing** - Where you write the generator specific files (routes, controllers, etc)
6. **conflicts** - Where conflicts are handled (used internally)
7. **install** - Where installations are run (npm, bower)
8. **end** - Called last, cleanup, say good bye, etc

Now let's create **pom.xml** template **pom.xml.tpl** and **src/main/java/com/mycompany/demo/Application.java** file in **generators/app/templates** folder as follows:

**generators/app/templates/pom.xml.tpl**

```xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
         http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.2.4.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.mycompany</groupId>
    <artifactId><%= appName %></artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name><%= appName %></name>

    <properties>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
    </dependencies>
</project>
```
Notice that we have the placeholders with the syntax **<%= variableName %>**.

**generators/app/templates/src/main/java/com/mycompany/demo/Application.java**

```java
package com.mycompany.demo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class Application {
    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }
}
```

Update **generators/app/index.js** as follows:

```js
"use strict";
const Generator = require("yeoman-generator");

module.exports = class extends Generator {
  constructor(args, opts) {
    super(args, opts);
  }

  initializing() {
    this.log("Generating Spring Boot Application");
  }

  prompting() {
    const prompts = [
      {
        type: "string",
        name: "appName",
        message: "What is the application name?",
        default: "myservice"
      }
    ];

    return this.prompt(prompts).then(answers => {
      this.appName = answers.appName;
    });
  }

  writing() {
    this.fs.copyTpl(
      this.templatePath("pom.xml.tpl"),
      this.destinationPath(this.appName + "/pom.xml"),
      {
        appName: this.appName
      }
    );

    this.fs.copy(
      this.templatePath("src"),
      this.destinationPath(this.appName + "/src")
    );
  }

  end() {
    this.log(`Application ${this.appName} generated successfully`);
  }
};
```

As per the priorities order, first the **initializing()** block gets executed, then **prompting()**, then **writing()**, and finally **end()**.
In **prompting()** we are asking the user to provide the application name and store it in the **appName** variable for later use.

In **writing()** we are generating the **pom.xml** file using the **pom.xml.tpl** template by replacing the **appName** placeholder with the user-provided value. Also, we are copying the entire **src** directory.

Instead of storing each user input in a separate variable and passing them explicitly to replace placeholders, we can store all the user-provided input values in a JSON object and pass it as follows:

```js
"use strict";
const Generator = require("yeoman-generator");

module.exports = class extends Generator {
  constructor(args, opts) {
    super(args, opts);
    this.configOptions = this.options.configOptions || {};
  }

  initializing() {
    this.log("Generating Spring Boot Application");
  }

  prompting() {
    const prompts = [
      {
        type: "string",
        name: "appName",
        message: "What is the application name?",
        default: "myservice"
      },
      {
        type: "list",
        name: "appType",
        message: "Do you want to use WebMVC or WebFlux?",
        choices: [
          {
            value: "webmvc",
            name: "WebMVC"
          },
          {
            value: "webflux",
            name: "WebFlux"
          }
        ],
        default: "webmvc"
      }
    ];

    return this.prompt(prompts).then(answers => {
      Object.assign(this.configOptions, answers);
    });
  }

  writing() {
    this.fs.copyTpl(
      this.templatePath("pomm.xml.tpl"),
      this.destinationPath(this.configOptions.appName + "/pom.xml"),
      this.configOptions
    );

    this.fs.copy(
      this.templatePath("src"),
      this.destinationPath(this.configOptions.appName + "/src")
    );
  }

  end() {
    this.log(
      `Application ${this.configOptions.appName} generated successfully`
    );
  }
};
```

We have added one more prompt of type **list** to select the **appType**.

Notice that we are initializing **configOptions** in the constructor and adding the user-provided prompt answers into configOptions. Later we are simply passing the configOptions object, which will contain all the placeholders.

Now, we can conditionally include either **webmvc** or **webflux** starter in **pom.xml.tpl** based on the user selection as follows:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
         http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.2.4.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.mycompany</groupId>
    <artifactId><%= appName %></artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name><%= appName %></name>

    <properties>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <%_ if (appType === 'webmvc') { _%>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <%_ } _%>
        <%_ if (appType === 'webflux') { _%>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-webflux</artifactId>
        </dependency>
        <%_ } _%>
    </dependencies>
</project>
```

Over time, the number of inputs we need from the user may grow. Instead of adding prompt questions in the same **index.js** file, we can create a separate **prompt.js** file and add all prompts in that file.

Create **generators/app/prompt.js** as follows:

```js
module.exports = {
  prompting
};

function prompting() {
  const done = this.async();
  const prompts = [
    {
      type: "string",
      name: "appName",
      message: "What is the application name?",
      default: "myservice"
    },
    {
      type: "list",
      name: "appType",
      message: "Do you want to use WebMVC or WebFlux?",
      choices: [
        {
          value: "webmvc",
          name: "WebMVC"
        },
        {
          value: "webflux",
          name: "WebFlux"
        }
      ],
      default: "webmvc"
    }
  ];

  this.prompt(prompts).then(answers => {
    Object.assign(this.configOptions, answers);
    done();
  });
}
```

Update **generators/app/index.js** as follows:

```js
const prompts = require('./prompts');

module.exports = class extends Generator {

    ...
    ...
    get prompting() {
        return prompts.prompting;
    }
}
```

Sometimes we might want to ask more questions based on a previous prompt answer.
We can conditionally prompt the user using **when** as follows:

```js
const prompts = [
    ...
    ...
    {
        type: 'confirm',
        name: 'sql',
        message: 'Do you want to use Spring Data Jpa?',
        default: true
    },
    {
        when: response => response.sql === true,
        type: 'list',
        name: 'databaseType',
        message: 'Which type of database you want to use?',
        choices: [
            {
                value: 'postgresql',
                name: 'Postgresql'
            },
            {
                value: 'mysql',
                name: 'MySQL'
            },
            {
                value: 'mariadb',
                name: 'MariaDB'
            }
        ],
        default: 'postgresql'
    }
]
```

### Composing with Sub-generator

We may want to add more and more features to our generator. Instead of adding all those features to a single generator, we can create sub-generators and invoke them from the main generator using **this.composeWith(...)**.

Suppose we have the main generator **app** and 3 other sub-generators: **microservice**, **config-server**, and **service-registry**.

```bash
+- generator-springboot/
   +- package.json
   +- generators/
      +- app/
         +- index.js
      +- microservice/
         +- index.js
      +- config-server/
         +- index.js
      +- service-registry/
         +- index.js
```

Now from the main app generator we can invoke sub-generators as follows:

**generator-springboot/generators/app/index.js**

```js
"use strict";
const Generator = require("yeoman-generator");

module.exports = class extends Generator {
  prompting() {
    const prompts = [
      {
        type: "list",
        name: "appType",
        message: "Which type of application you want to generate?",
        choices: [
          {
            value: "microservice",
            name: "Spring Boot MicroService"
          },
          {
            value: "config-server",
            name: "Spring Cloud Config Server"
          },
          {
            value: "service-registry",
            name:
              "Spring Cloud Eureka Server for Service Registry and Discovery"
          }
        ],
        default: "microservice"
      }
    ];
    return this.prompt(prompts).then(answers => {
      this.appType = answers.appType;
    });
  }

  default() {
    this.composeWith(require.resolve("../" + this.appType));
  }
};
```

Now when you run the main generator and select the **appType**, it will invoke the respective sub-generator.

### Invoking sub-generator directly

We can create sub-generators to add more features to an existing yeoman generated application.
For example, once we create a Spring Boot application we can use a entity sub-generator to generate a JPA entity.

We can invoke sub-generator directly as follows:

```bash
myservice> yo springboot:entity Product
```

### Using user configuration

We can save the user-selected options so that we can use them later while running sub-generators.
For example, if the user selected to use the FlywayDB migration tool, then while running the Entity generator we can check which DB migration option is selected and generate DB migration scripts accordingly.

We can save the user configuration in the main generator as follows:

**generators/app/index.js**

```js
module.exports = class extends Generator {

    ...
    ...
    configuring() {
        Object.assign(this.configOptions, constants);
        this.config.save(this.configOptions);
    }
}
```

**this.config.save()** method will store the user configuration in **.yo-rc.json** file.

**.yo-rc.json**

```json
{
  "generator-springboot": {
    "appName": "myservice",
    "packageName": "com.mycompany.myservice",
    "sql": true,
    "databaseType": "postgresql",
    "dbMigrationTool": "flywaydb",
    "features": [],
    "buildTool": "gradle",
    "packageFolder": "com/mycompany/myservice",
    "...": "...",
    "...": "..."
  }
}
```

We can also use **this.config.set()** to specify one key-value pair or an object of multiple keys/values.
Note that **save()** method is automatically called whenever we call **this.config.set()**.

Now we can use **this.config.getAll()** to load the configuration from **.yo-rc.json** file.

**generators/entity/index.js**

```js
module.exports = class extends Generator {
    ...
    ...
    writing() {
        const userConfig = this.config.getAll();
        if(userConfig.dbMigrationTool === "flywaydb") {
            //do something
        }
    }
}
```

We can also pass **Arguments** and **Options** to the generator. For example, for the entity sub-generator we want to pass the mandatory **entityName** as an argument and the optional **table-name** as an option.

**generators/entity/index.js**

```js
module.exports = class extends Generator {

    constructor(args, opts) {
        super(args, opts);
        this.configOptions = this.options.configOptions || {};

        this.argument("entityName", {
            type: String,
            required: true,
            description: "Entity name"
        });

        this.option('table-name', {
            type: String,
            desc: "Table name"
        })
    }

    writing() {
        const entityName = this.options.entityName;
        const tableName = this.options.tableName;
        ....
        ...
    }
}
```

Now we can invoke the entity sub-generator from within the generated application directory as follows:

```bash
myservice> yo springboot:entity Person --table-name persons
```

### Testing Yeoman generator

We can also unit test Yeoman generators by using **mocha**, **yeoman-test**, and **yeoman-assert** packages.

**package.json**

```json
{
  "name": "generator-springboot",
  "version": "0.0.6",
  "description": "A Yeoman generator for generating Spring Boot microservices",
  ...
  ...
  "scripts": {
    "test": "npm run test:unit -- test/**/*.spec.js test/*.spec.js --no-insight",
    "test:unit": "mocha --timeout 30000 --slow 0 --reporter spec"
  },
  ...
  ...
  "devDependencies": {
    "fs-extra": "7.0.1",
    "mocha": "6.1.4",
    "sinon": "7.2.5",
    "yeoman-assert": "3.1.1",
    "yeoman-test": "1.9.1"
  }
}

```

We can write unit test to invoke a generator using **yeoman-test** as follows:

**generator-springboot/test/microservice.spec.js**

```js
const path = require("path");
const assert = require("yeoman-assert");
const helpers = require("yeoman-test");

describe("Spring Boot Microservice Generator", () => {
  describe("Generate minimal microservice with Maven", () => {
    before(done => {
      helpers
        .run(path.join(__dirname, "../generators/microservice"))
        .withPrompts({
          appName: "mymicroservice",
          packageName: "com.mycompany.mymicroservice",
          packageFolder: "com/mycompany/mymicroservice",
          sql: false,
          buildTool: "maven"
        })
        .on("end", done);
    });

    it("creates expected default files for minimal microservice", () => {
      assert.file("pom.xml");
    });
  });
});
```
Of course, here I am doing a simple sanity check of whether pom.xml is generated or not. We can test much more if we want to.

Now we can run tests using `npm run test`.

I think we have covered sufficient ground to start building a Yeoman generator.

For reference, please check out my GitHub repository https://github.com/sivaprasadreddy/generator-springboot.

# Testing REST APIs using Postman and Newman


[**Postman**](https://www.getpostman.com/) is the most popular tool for testing REST APIs.
I guess most developers are familiar with how to use Postman to send various types (GET, POST, PUT, DELETE, etc.) of HTTP requests
with various types of payloads (form data, JSON, etc.).

<!--more-->


In addition to triggering one API request and verifying the response, I frequently come across some common needs, like:

*   Call an API and assert response
*   Parameterize variables based on the environment
*   Trigger a series of API calls to simulate some user flow
*   Trigger one API request with different inputs and assert expected response
*   Load test by simulating a load on the server by firing a series of requests repeatedly with a certain delay
*   Run a Postman collection to do a Smoke Test from a Build Pipeline

As I am already using Postman to test my REST APIs, I thought of exploring if I can do all these tasks using Postman itself, and it turns out I can.

So, I wanted to share a few Postman tips I've learned.

> I am using Postman to test my REST API implemented using **Spring Boot** and secured with **Spring Security JWT based authentication**.
> You can find the source code of this article at https://github.com/sivaprasadreddy/todo-list/tree/master/todo-api-spring-boot.
>
> The Postman collection files are at https://github.com/sivaprasadreddy/todo-list/tree/master/config/postman

While searching for a way to run a Postman collection programmatically,
I came across [Newman](https://learning.getpostman.com/docs/postman/collection_runs/command_line_integration_with_newman/),
which is a command-line Collection Runner for Postman. Newman is built on NodeJS, and we can install it globally to use it.

```bash
$ npm install -g newman
$ newman run mycollection.json
```

Or we can use Newman npm package as a library as mentioned here https://www.npmjs.com/package/newman#using-newman-as-a-library.

Ok, enough talk. Let's get to business.

First, open Postman and create a new collection, say "todolist-api".

## 1. Call an API and assert response
We want to make a call to Spring Boot Actuator Health endpoint and verify the HTTP Status Code and assert that the **status** is **UP**.

Add a new request with the URL http://localhost:8080/actuator/health with the GET method.
In the **Headers** tab, add a **Content-Type** header with the value **application/json**.

In **Tests** tab add the following code:

```javascript
pm.test("Check Application Health", function () {
    pm.response.to.have.status(200);
    var jsonData = pm.response.json();
    pm.expect(jsonData.status).to.eql("UP");
});
```

Here, we are using the Postman API to retrieve the response and verify the HTTP status code to be 200.
Also, the expected JSON response from the **/actuator/health** request is to have **{"status": "UP"}**.
So we are asserting that the JSON response field **status** is equal to **UP**.

We can just hit the Send button and see the Response body, and you can also see in the **Test Results** tab that the test has passed.

We can also run all the Requests (I know we have only one for now) by clicking on the Play button next to the Collection name and Run.
It will open up the collection runner window and show all the tests available to run.
We can Select/Deselect requests and click on the Run button to run the tests.

#### Running Postman collection using Newman

In Postman, click on the 3 dots next to the Collection name and export the Collection as a JSON file.
Let's say the collection JSON file name is **todolist.postman_collection.json**.

We can run the collection using Newman as follows:

```bash
$ newman run todolist.postman_collection.json
```

You can see various options you can pass to newman 
at https://learning.getpostman.com/docs/postman/collection_runs/command_line_integration_with_newman/#options

## 2. Trigger a series of API calls to simulate some user flow
Next, my requirement is that a user should be able to create a new Todo, update the newly created Todo, and finally delete that Todo.
Even before doing all this, users should be authenticated.
As we are using JWT based authentication, upon successful login, the user will get an **access_token**, and
for each subsequent request, the user should send the **access_token** as a Header.

Also, we need to capture the ID of the newly created Todo, which is an auto-generated primary key in the DB,
and use that ID for update and delete todo requests.

Now, add a new Request named **login** with the URL http://localhost:8080/api/auth/login and the POST method with the following JSON Payload:

```json
{
	"username": "admin@gmail.com",
	"password": "admin"
}
```

If you hit Send button you can see the response as follows:

```json
{
    "access_token": "some long sequence of alphanumeric chars",
    "expires_in": 604800
}
```

![Postman Collection](/images/postman-collection.webp "Postman Collection")

We want to capture this **access_token** and pass it in subsequent requests.
In order to do this, we need to run the requests in an environment so that we can set some variables in one request and reference them in subsequent requests.

In Postman, in the top right corner, there is an Environment dropdown; click on the gear icon and add a new environment called **todo-local-env**.

Now add the following code snippet in the **Tests** tab of the **login** request:


```javascript
pm.test("Authenticate user", function () {
    pm.response.to.have.status(200);
    var jsonData = JSON.parse(responseBody);
    pm.expect(jsonData.access_token).not.eq(undefined);
    postman.clearEnvironmentVariable("TodoApiAccessToken");
    postman.setEnvironmentVariable("TodoApiAccessToken", jsonData.access_token);
});
```

We are verifying whether the request is successful and setting the **TodoApiAccessToken** variable in the selected environment.

Now if you trigger the login request and look at **todo-local-env**, you can find the **TodoApiAccessToken** variable with the **access_token** value.

![Postman Environment Variables](/images/postman-env-vars.webp "Postman Environment Variables")

Now, create another request named **create-todo** with the URL http://localhost:8080/api/todos and the POST method with the following JSON payload:

```json
{
	"text": "Write a new blog post"
}
```

In the **Headers** tab, add a Header with the key **Authorization** and the value **Bearer {{TodoApiAccessToken}}**.
Here we are referring to the environment variable's value using the **{{variable_name}}** syntax.

In the **Tests** tab, add the following snippet:

```javascript
pm.test("Create New Todo", function () {
    pm.response.to.have.status(201);
    var jsonData = JSON.parse(responseBody);
    postman.clearEnvironmentVariable("todo_id");
    postman.setEnvironmentVariable("todo_id", jsonData.id);
});
```

Here we are capturing the new Todo's **id** value and setting the environment variable **todo_id**.

Next, add the **update-todo** Request with the URL http://localhost:8080/api/todos/{{todo_id}} and the PUT method with the following JSON payload:

```json
{
	"text": "Write a new blog post - updated"
}
```

We can add the following snippet in the **update-todo** request's **Tests** tab:

```javascript
pm.test("Update Existing Todo", function () {
    pm.response.to.have.status(200);
});
```

Finally, we can add a **delete-todo-by-id** request with the URL http://localhost:8080/api/todos/{{todo_id}} and the DELETE method.
Add the following snippet in the **delete-todo-by-id** request's **Tests** tab:

```javascript
pm.test("Delete Todo by Id", function () {
    pm.response.to.have.status(200);
    postman.clearEnvironmentVariable("todo_id");
});
```

Now we can run the entire collection as mentioned earlier, which will
[run all the requests in the order they appear in the collection](https://learning.getpostman.com/docs/postman/collection_runs/starting_a_collection_run/).

> IMPORTANT: We should select the environment, **todo-local-env** in our case, for the environment variable sharing to work.

#### Running collection using Newman with environment
In order to run the collection using Newman, we need to export the collection and also our environment.

Click on the **Manage Environments** gear icon in the top right corner and click on the **Download Environment** icon for **todo-local-env**, which will be downloaded as a JSON file.

Let's name it **todo-local-env.postman_environment.json**.

```bash
$ newman run todolist.postman_collection.json -e todo-local-env.postman_environment.json
```

### Organizing user flows 

We have created multiple requests (login, create-todo, update-todo, delete-todo) and have run the collection as a single flow.
But in our real applications, there will be many different user flows with different behaviors.

We can create folders within our collection representing a logical user flow and place the appropriate requests in that folder.

Then, we can run the requests of a specific folder using Newman as follows:

```bash
$ newman run todolist.postman_collection.json -e todo-local-env.postman_environment.json --folder todo-crud
```

## 3. Parameterize variables based on the environment
So far, I have been running my application locally, and it is accessible at http://localhost:8080/.
I could run my application in a Docker container, which is accessible at http://localhost:18080/.

Now, if I want to run my Postman collection against the Docker container instance, I need to change the base URL for all requests.
Instead of changing the URLs or any environment-specific parameters when switching between environments, we can configure those values in different environments and use those values in the request configurations.

Now click on **Manage Environments**, select **todo-local-env**, and add a variable **base_url** with the value **http://localhost:8080**.

Now change all the URLs from **http://localhost:8080/some-path** to **{{base_url}}/some-path**.

Next, create a new environment called **todo-docker-env** and add a variable **base_url** with the value **http://localhost:18080**.

Now we can simply select the desired environment and run the collection. 

## 4. Trigger API request with different inputs and assert expected responses
We may also need to call an API endpoint with different inputs and verify responses, such as performing a login with valid and invalid credentials and asserting the expected responses.

We can use **data files** to pass different inputs so that we don't have to change the request configuration.
We can pass data using JSON or CSV files.

For example, we can pass credentials using a CSV file as follows:

**todo-credentials.csv**

```csv
email,password,expected_http_code
admin@gmail.com,admin.200
dummy@gmail.com,secret,401
```

The first row is a header with variable names.
We can use the placeholders in the URL path or in the body as follows:

```json
{
	"username": "{{email}}",
	"password": "{{password}}"
}
```

In the **Tests** tab, we can refer to data values using **data.variableName** as follows:

```javascript
pm.test("Authenticate user", function () {
    pm.response.to.have.status(data.expected_http_status_code);
}); 
```


```bash
$ newman run todolist.postman_collection.json -e local.postman_environment.json --folder authentication --data todo-credentials.csv
```

## 5. Load testing
I was trying to do some simple load testing of my API, and I know I can use **Gatling** to do it.
But I just wanted a simple and quick solution. It turns out Postman provides that feature as well.

```bash
$ newman run todolist.postman_collection.json -e local.postman_environment.json -n 100 --delay-request 500
```

* The **-n** flag represents the **number of iterations to run**.
* The **--delay-request** represents the **delay (in ms) between requests**.

## 6. Run Postman collection to do Smoke Test from Build Pipeline
We can create a Postman collection to perform a smoke test after deploying the application.
We can simply install the Newman npm package and run the collection in our build pipeline.

You can checkout 

* [Integration with Travis CI](https://learning.getpostman.com/docs/postman/collection_runs/integration_with_travis/)
* [Integration with Jenkins](https://learning.getpostman.com/docs/postman/collection_runs/integration_with_jenkins/)

Thank you for reading the article; your feedback is welcome.
If you find this article useful, please share it.

# Testing Spring Boot Applications


**Spring Boot** is the most popular tech stack for building Java-based REST APIs.
In this tutorial, we will learn how to write tests for Spring Boot applications.

* Create Spring Boot Application
* Unit Testing using [JUnit 5](https://junit.org/junit5/docs/current/user-guide/) and [Mockito](https://site.mockito.org/)
* Integration Testing using [TestContainers](https://www.testcontainers.org/)
* Testing MicroService Integrations using [MockServer](http://www.mock-server.com/)

<!--more-->


As we all know, we write unit tests for testing a single component's (a class's) behavior, whereas we write integration tests for testing a feature that may involve interaction with multiple components.

Most of the time, one component will depend on other component(s), so while implementing unit tests, we should mock
the dependencies with the desired behavior using frameworks like [Mockito](https://site.mockito.org/).

So, the question is, how do we implement Unit Tests and Integration tests in a Spring Boot application? Read on :-)

> Sample application Code for this article can be found at https://github.com/sivaprasadreddy/spring-boot-tutorials/tree/master/testing/springboot-testing-demo

## Create Spring Boot Application

Let us consider a scenario where we are building a REST API to manage users.
If we follow a typical 3-tier layered architecture, we might have a JPA entity **User**, Spring Data JPA Repository **UserRepository**,
**UserService**, and **UserController** implementing CRUD operations as follows:

First of all, create a Spring Boot application with the following dependencies:

#### pom.xml
```xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
		 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
	http://maven.apache.org/maven-v4_0_0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>com.sivalabs</groupId>
	<artifactId>springboot-testing-demo</artifactId>
	<packaging>jar</packaging>
	<version>1.0-SNAPSHOT</version>
	<name>springboot-testing-demo</name>

	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>2.1.8.RELEASE</version>
		<relativePath/> <!-- lookup parent from repository -->
	</parent>
    
	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<java.version>1.8</java.version>
	</properties>

	<build>
		<plugins>						
			<plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
			<plugin>
				<artifactId>maven-failsafe-plugin</artifactId>
			</plugin>
		</plugins>
	</build>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-actuator</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-data-jpa</artifactId>
		</dependency>
		<dependency>
	        <groupId>org.springframework.boot</groupId>
	        <artifactId>spring-boot-devtools</artifactId>
	        <optional>true</optional>
	    </dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>org.projectlombok</groupId>
			<artifactId>lombok</artifactId>
			<optional>true</optional>
		</dependency>
		<dependency>
			<groupId>com.h2database</groupId>
			<artifactId>h2</artifactId>
			<scope>runtime</scope>
		</dependency>
		<dependency>
			<groupId>org.postgresql</groupId>
			<artifactId>postgresql</artifactId>
			<scope>runtime</scope>
		</dependency>
	</dependencies>
</project>
```

By default **spring-boot-starter-test** comes with **JUnit 4** as testing framework. 
We can **exclude JUnit4** and **add JUnit 5** dependencies as follows:

```xml
<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
        <exclusions>
            <exclusion>
                <groupId>junit</groupId>
                <artifactId>junit</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.junit.jupiter</groupId>
        <artifactId>junit-jupiter-engine</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.junit.jupiter</groupId>
        <artifactId>junit-jupiter-params</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>
```

Let us create JPA entity, repository, service and controller for User as follows:

#### User.java
```java
package com.sivalabs.myservice.entities;

import lombok.*;
import javax.persistence.*;
import javax.validation.constraints.NotEmpty;
import java.io.Serializable;

@Entity
@Table(name = "users")
@Setter
@Getter
@NoArgsConstructor
@AllArgsConstructor
public class User implements Serializable
{
	@Id
	@GeneratedValue(strategy = GenerationType.IDENTITY)
	private Long id;

	@NotEmpty(message = "Email should not be empty")
	@Column(nullable = false, unique = true, length = 100)
	private String email;

	@Column(nullable = false, length = 100)
	private String password;

	@Column(nullable = false, length = 100)
	private String name;

}
```

#### UserRepository.java
```java
package com.sivalabs.myservice.repositories;

import org.springframework.data.jpa.repository.JpaRepository;
import com.sivalabs.myservice.entities.User;
import org.springframework.data.jpa.repository.Query;
import java.util.Optional;

public interface UserRepository extends JpaRepository<User, Long>
{
	@Query("select u from User u where u.email=?1 and u.password=?2")
	Optional<User> login(String email, String password);

    Optional<User> findByEmail(String email);
}
```

#### UserService.java
```java
package com.sivalabs.myservice.services;

import com.sivalabs.myservice.exception.UserRegistrationException;
import com.sivalabs.myservice.repositories.UserRepository;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import com.sivalabs.myservice.entities.User;
import java.util.List;
import java.util.Optional;

@Service
@Transactional
public class UserService 
{
    private final UserRepository userRepository;

    @Autowired
	public UserService(UserRepository userRepository) {
		this.userRepository = userRepository;
	}

	public Optional<User> login(String email, String password)
	{
		return userRepository.login(email, password);
	}
	
	public User createUser(User user)
	{
		Optional<User> userOptional = userRepository.findByEmail(user.getEmail());
		if(userOptional.isPresent()){
			throw new UserRegistrationException("User with email "+ user.getEmail()+" already exists");
		}
		return userRepository.save(user);
	}

	public User updateUser(User user)
	{
		return userRepository.save(user);
	}

	public List<User> findAllUsers() {
    	return userRepository.findAll();
	}

	public Optional<User> findUserById(Long id) {
    	return userRepository.findById(id);
	}

	public void deleteUserById(Long id) {
    	userRepository.deleteById(id);
	}
	
}
```

#### UserController.java
```java
package com.sivalabs.myservice.web.controllers;

import com.sivalabs.myservice.entities.User;
import com.sivalabs.myservice.services.UserService;
import lombok.extern.slf4j.Slf4j;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.validation.annotation.Validated;
import org.springframework.web.bind.annotation.*;
import java.util.List;

@RestController
@RequestMapping("/api/users")
@Slf4j
public class UserController {

	private final UserService userService;

	public UserController(UserService userService) {
		this.userService = userService;
	}

	@GetMapping
	public List<User> getAllUsers() {
		return userService.findAllUsers();
	}

	@GetMapping("/{id}")
	public ResponseEntity<User> getUserById(@PathVariable Long id) {
		return userService.findUserById(id)
				.map(ResponseEntity::ok)
				.orElseGet(() -> ResponseEntity.notFound().build());
	}

	@PostMapping
	@ResponseStatus(HttpStatus.CREATED)
	public User createUser(@RequestBody @Validated User user) {
		return userService.createUser(user);
	}

	@PutMapping("/{id}")
	public ResponseEntity<User> updateUser(@PathVariable Long id, @RequestBody User user) {
		return userService.findUserById(id)
				.map(userObj -> {
					user.setId(id);
					return ResponseEntity.ok(userService.updateUser(user));
				})
				.orElseGet(() -> ResponseEntity.notFound().build());

	}

	@DeleteMapping("/{id}")
	public ResponseEntity<User> deleteUser(@PathVariable Long id) {
		return userService.findUserById(id)
				.map(user -> {
					userService.deleteUserById(id);
					return ResponseEntity.ok(user);
				})
				.orElseGet(() -> ResponseEntity.notFound().build());
	}
}
```

Nothing too fancy here, typical CRUD operations in a Spring Boot application.

### Exception Handling using Zalando Problem Web
We are going to use [Zalando Problem Web](https://github.com/zalando/problem-spring-web) [Spring Boot starter](https://github.com/zalando/problem-spring-web/tree/master/problem-spring-web) to handle
Exceptions so that it automatically converts application errors into JSON responses.

Just adding the following dependency is enough to start using Zalando Problem Web, and of course, you can customize it if you want to.

```xml
<problem-spring-web.version>0.25.0</problem-spring-web.version>
...
...
<dependency>
    <groupId>org.zalando</groupId>
    <artifactId>problem-spring-web-starter</artifactId>
    <version>${problem-spring-web.version}</version>
    <type>pom</type>
</dependency>
```

Now let us see how we can write Unit Tests and Integration Tests for this functionality.

## Unit Testing using JUnit 5 and Mockito
Let's start by writing unit tests for **UserService**.
*We should be able to write unit tests for UserService WITHOUT using any Spring features*.

We are going to create a mock **UserRepository** using **Mockito.mock()** and create a UserService instance using the mock UserRepository instance.

```java
package com.sivalabs.myservice.services;

import com.sivalabs.myservice.entities.User;
import com.sivalabs.myservice.exception.UserRegistrationException;
import com.sivalabs.myservice.repositories.UserRepository;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;

import java.util.Optional;

import static org.assertj.core.api.Assertions.assertThat;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.BDDMockito.given;
import static org.mockito.Mockito.*;

class UserServiceTest {

    private UserService userService;
    private UserRepository userRepository;

    @BeforeEach
    void setUp() {
        userRepository = mock(UserRepository.class);
        userService = new UserService(userRepository);
    }

    @Test
    void shouldSavedUserSuccessfully() {
        User user = new User(null, "siva@gmail.com","siva","Siva");
        given(userRepository.findByEmail(user.getEmail())).willReturn(Optional.empty());
        given(userRepository.save(user)).willAnswer(invocation -> invocation.getArgument(0));

        User savedUser = userService.createUser(user);

        assertThat(savedUser).isNotNull();

        verify(userRepository).save(any(User.class));
    }

    @Test
    void shouldThrowErrorWhenSaveUserWithExistingEmail() {
        User user = new User(1L, "siva@gmail.com","siva","Siva");
        given(userRepository.findByEmail(user.getEmail())).willReturn(Optional.of(user));

        assertThrows(UserRegistrationException.class, () -> {
            userService.createUser(user);
        });

        verify(userRepository, never()).save(any(User.class));
    }
}
```

I have created a **UserRepository** mock object and a **UserService** instance in the **@BeforeEach** method so that every test has a clean setup.
Here we are not using any Spring or Spring Boot testing features such as **@SpringBootTest** because we don't have to for testing the behavior of UserService.

I am not going to write tests for other methods because they are simply delegating the calls to **UserRepository**.

If you prefer to use annotation magic to create a mock **UserRepository** and inject that mock into **UserService**,
you can use **mockito-junit-jupiter** as follows: 

Add **mockito-junit-jupiter** dependency

```xml
<dependency>
    <groupId>org.mockito</groupId>
    <artifactId>mockito-junit-jupiter</artifactId>
    <scope>test</scope>
</dependency>
```

Use **@Mock** and **@InjectMocks** to create and inject mock objects as follows:

```java
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;

@ExtendWith(MockitoExtension.class)
class UserServiceAnnotatedTest {

    @Mock
    private UserRepository userRepository;
        
    @InjectMocks
    private UserService userService;

    ...
    ...
}
```

### Now, shall we write tests for UserRepository? hmmmm...

The **UserRepository** is an interface extending **JpaRepository**, and there is hardly any logic we implemented, and
we shouldn't be testing the Spring Data JPA framework, as I strongly believe the Spring Data JPA team has already tested it :-)

However, we added a couple of custom methods, one leveraging the Query naming convention **findByEmail()** and another with a custom JPQL query **login()**.
We should test these methods. If there are any syntactic errors, Spring Data JPA will throw errors on startup, but we should test the logical errors ourselves.

We could implement tests for **UserRepository** using Spring Boot's **@DataJpaTest** annotation with **in-memory database support**.
But running tests against an in-memory database might give a false impression that it will also work on a real production database.
So, I prefer to run tests against the production database type, in our case, PostgreSQL.

We can use [TestContainers support to spin up a PostgreSQL](https://www.testcontainers.org/modules/databases/postgres/) Docker container and run the tests pointing to that database.
However, I consider this an Integration Test rather than a Unit Test, as we are talking to a real database.
So, we will see how to write integration tests for `UserRepository` later.

### What about unit tests for Controller?
Yes, I want to write unit tests for the controller, and I want to check whether the REST endpoint is giving the proper
HTTP ResponseCode, returning the expected JSON, etc.

Spring Boot provides the **@WebMvcTest** annotation to test Spring MVC Controllers.
Also, **@WebMvcTest** based tests run faster as it will load only the specified controller and its dependencies, without loading the entire application.

While loading the Controller using **@WebMvcTest**, Spring Boot won't automatically load the Zalando Problem Web AutoConfiguration.
So, we need to configure **ControllerAdvice** as follows:

```java
package com.sivalabs.myservice.common;

import org.springframework.context.annotation.Profile;
import org.springframework.web.bind.annotation.ControllerAdvice;
import org.zalando.problem.spring.web.advice.ProblemHandling;

@Profile("test")
@ControllerAdvice
public final class ExceptionHandling implements ProblemHandling {

}
```

Now we can write tests for **UserController** by injecting a Mock **UserService** bean and invoke API endpoints using **MockMvc**.

As Spring Boot is creating the **UserController** instance, we are creating a mock **UserService** bean using Spring's **@MockBean** as opposed to plain Mockito's **@Mock**.

```java
package com.sivalabs.myservice.web.controllers;

import com.fasterxml.jackson.databind.ObjectMapper;
import com.sivalabs.myservice.entities.User;
import com.sivalabs.myservice.services.UserService;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTest;
import org.springframework.boot.test.mock.mockito.MockBean;
import org.springframework.http.MediaType;
import org.springframework.test.context.ActiveProfiles;
import org.springframework.test.web.servlet.MockMvc;
import org.zalando.problem.ProblemModule;
import org.zalando.problem.violations.ConstraintViolationProblemModule;

import java.util.ArrayList;
import java.util.List;
import java.util.Optional;

import static org.hamcrest.CoreMatchers.is;
import static org.hamcrest.Matchers.hasSize;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.BDDMockito.given;
import static org.mockito.Mockito.doNothing;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.*;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.*;

@WebMvcTest(controllers = UserController.class)
@ActiveProfiles("test")
class UserControllerTest {

    @Autowired
    private MockMvc mockMvc;

    @MockBean
    private UserService userService;

    @Autowired
    private ObjectMapper objectMapper;

    private List<User> userList;

    @BeforeEach
    void setUp() {
        this.userList = new ArrayList<>();
        this.userList.add(new User(1L, "user1@gmail.com", "pwd1","User1"));
        this.userList.add(new User(2L, "user2@gmail.com", "pwd2","User2"));
        this.userList.add(new User(3L, "user3@gmail.com", "pwd3","User3"));

        objectMapper.registerModule(new ProblemModule());
        objectMapper.registerModule(new ConstraintViolationProblemModule());
    }

    @Test
    void shouldFetchAllUsers() throws Exception {
        given(userService.findAllUsers()).willReturn(this.userList);

        this.mockMvc.perform(get("/api/users"))
                .andExpect(status().isOk())
                .andExpect(jsonPath("$.size()", is(userList.size())));
    }

    @Test
    void shouldFindUserById() throws Exception {
        Long userId = 1L;
        User user = new User(userId, "newuser1@gmail.com", "pwd", "Name");
        given(userService.findUserById(userId)).willReturn(Optional.of(user));

        this.mockMvc.perform(get("/api/users/{id}", userId))
                .andExpect(status().isOk())
                .andExpect(jsonPath("$.email", is(user.getEmail())))
                .andExpect(jsonPath("$.password", is(user.getPassword())))
                .andExpect(jsonPath("$.name", is(user.getName())))
        ;
    }

    @Test
    void shouldReturn404WhenFetchingNonExistingUser() throws Exception {
        Long userId = 1L;
        given(userService.findUserById(userId)).willReturn(Optional.empty());

        this.mockMvc.perform(get("/api/users/{id}", userId))
                .andExpect(status().isNotFound());

    }

    @Test
    void shouldCreateNewUser() throws Exception {
        given(userService.createUser(any(User.class))).willAnswer((invocation) -> invocation.getArgument(0));

        User user = new User(null, "newuser1@gmail.com", "pwd", "Name");
        this.mockMvc.perform(post("/api/users")
                .contentType(MediaType.APPLICATION_JSON_UTF8)
                .content(objectMapper.writeValueAsString(user)))
                .andExpect(status().isCreated())
                .andExpect(jsonPath("$.email", is(user.getEmail())))
                .andExpect(jsonPath("$.password", is(user.getPassword())))
                .andExpect(jsonPath("$.name", is(user.getName())))
                ;

    }

    @Test
    void shouldReturn400WhenCreateNewUserWithoutEmail() throws Exception {
        User user = new User(null, null, "pwd", "Name");

        this.mockMvc.perform(post("/api/users")
                .contentType(MediaType.APPLICATION_JSON_UTF8)
                .content(objectMapper.writeValueAsString(user)))
                .andExpect(status().isBadRequest())
                .andExpect(header().string("Content-Type", is("application/problem+json")))
                .andExpect(jsonPath("$.type", is("https://zalando.github.io/problem/constraint-violation")))
                .andExpect(jsonPath("$.title", is("Constraint Violation")))
                .andExpect(jsonPath("$.status", is(400)))
                .andExpect(jsonPath("$.violations", hasSize(1)))
                .andExpect(jsonPath("$.violations[0].field", is("email")))
                .andExpect(jsonPath("$.violations[0].message", is("Email should not be empty")))
                .andReturn()
        ;
    }

    @Test
    void shouldUpdateUser() throws Exception {
        Long userId = 1L;
        User user = new User(userId, "user1@gmail.com", "pwd", "Name");
        given(userService.findUserById(userId)).willReturn(Optional.of(user));
        given(userService.updateUser(any(User.class))).willAnswer((invocation) -> invocation.getArgument(0));

        this.mockMvc.perform(put("/api/users/{id}", user.getId())
                .contentType(MediaType.APPLICATION_JSON_UTF8)
                .content(objectMapper.writeValueAsString(user)))
                .andExpect(status().isOk())
                .andExpect(jsonPath("$.email", is(user.getEmail())))
                .andExpect(jsonPath("$.password", is(user.getPassword())))
                .andExpect(jsonPath("$.name", is(user.getName())));

    }

    @Test
    void shouldReturn404WhenUpdatingNonExistingUser() throws Exception {
        Long userId = 1L;
        given(userService.findUserById(userId)).willReturn(Optional.empty());
        User user = new User(userId, "user1@gmail.com", "pwd", "Name");

        this.mockMvc.perform(put("/api/users/{id}", userId)
                .contentType(MediaType.APPLICATION_JSON_UTF8)
                .content(objectMapper.writeValueAsString(user)))
                .andExpect(status().isNotFound());

    }

    @Test
    void shouldDeleteUser() throws Exception {
        Long userId = 1L;
        User user = new User(userId, "user1@gmail.com", "pwd", "Name");
        given(userService.findUserById(userId)).willReturn(Optional.of(user));
        doNothing().when(userService).deleteUserById(user.getId());

        this.mockMvc.perform(delete("/api/users/{id}", user.getId()))
                .andExpect(status().isOk())
                .andExpect(jsonPath("$.email", is(user.getEmail())))
                .andExpect(jsonPath("$.password", is(user.getPassword())))
                .andExpect(jsonPath("$.name", is(user.getName())));

    }

    @Test
    void shouldReturn404WhenDeletingNonExistingUser() throws Exception {
        Long userId = 1L;
        given(userService.findUserById(userId)).willReturn(Optional.empty());

        this.mockMvc.perform(delete("/api/users/{id}", userId))
                .andExpect(status().isNotFound());

    }

}
```

Now we have a good amount of unit tests testing various components of our application.
But still, there is a lot of chance for things to go wrong; maybe we have some property configuration issues, or we might have some errors in our DB migration scripts, etc.

So, let's write Integration Tests to have more confidence that our application is running properly.

## Integration Testing using TestContainers
Spring Boot provides excellent support for integration testing. We can use the **@SpringBootTest** annotation
to load the application context and test various components.

Let us start with writing integration tests for **UserController**.
As I mentioned earlier, we want to test using a PostgreSQL database instead of an in-memory database.

Add the following dependencies.

```xml
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>postgresql</artifactId>
    <version>1.11.3</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>junit-jupiter</artifactId>
    <version>1.11.3</version>
    <scope>test</scope>
</dependency>
```

We can use TestContainers support for JUnit 5 as mentioned here https://www.testcontainers.org/test_framework_integration/junit_5/.
However, starting and stopping Docker containers for every test or every test class might cause tests to run slowly.
So, we are going to use the Singleton Containers approach mentioned at https://www.testcontainers.org/test_framework_integration/manual_lifecycle_control/#singleton-containers

Let us create a base class **AbstractIntegrationTest** so that all our integration tests can extend it without repeating the common configuration.

```java
package com.sivalabs.myservice.common;

import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.test.util.TestPropertyValues;
import org.springframework.context.ApplicationContextInitializer;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.test.context.ActiveProfiles;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.web.servlet.MockMvc;
import org.testcontainers.containers.PostgreSQLContainer;

import static org.springframework.boot.test.context.SpringBootTest.WebEnvironment.RANDOM_PORT;

@Slf4j
@ActiveProfiles("test")
@SpringBootTest(webEnvironment = RANDOM_PORT)
@AutoConfigureMockMvc
@ContextConfiguration(initializers = {AbstractIntegrationTest.Initializer.class})
public abstract class AbstractIntegrationTest {

    @Autowired
    protected MockMvc mockMvc;

    @Autowired
    protected ObjectMapper objectMapper;

    private static PostgreSQLContainer sqlContainer;

    static {
        sqlContainer = new PostgreSQLContainer("postgres:10.7")
                .withDatabaseName("integration-tests-db")
                .withUsername("sa")
                .withPassword("sa");
        sqlContainer.start();
    }

    public static class Initializer
            implements ApplicationContextInitializer<ConfigurableApplicationContext> {
        public void initialize(ConfigurableApplicationContext configurableApplicationContext) {
            TestPropertyValues.of(
                    "spring.datasource.url=" + sqlContainer.getJdbcUrl(),
                    "spring.datasource.username=" + sqlContainer.getUsername(),
                    "spring.datasource.password=" + sqlContainer.getPassword()
            ).applyTo(configurableApplicationContext.getEnvironment());
        }
    }

}
```

We have used **@AutoConfigureMockMvc** to auto-configure **MockMvc**,
and **@SpringBootTest(webEnvironment = RANDOM_PORT)** to start the server on a random available port.

We have started **PostgreSQLContainer** and used **@ContextConfiguration(initializers = {AbstractIntegrationTest.Initializer.class})**
to configure the dynamic database connection properties.

Now we can implement an Integration Test for **UserController** as follows:

```java
package com.sivalabs.myservice.web.controllers;

import com.sivalabs.myservice.common.AbstractIntegrationTest;
import com.sivalabs.myservice.entities.User;
import com.sivalabs.myservice.repositories.UserRepository;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.MediaType;

import java.util.ArrayList;
import java.util.List;

import static org.hamcrest.CoreMatchers.is;
import static org.hamcrest.Matchers.hasSize;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.*;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.*;

class UserControllerIT extends AbstractIntegrationTest {

    @Autowired
    private UserRepository userRepository;

    private List<User> userList = null;

    @BeforeEach
    void setUp() {
        userRepository.deleteAll();

        userList = new ArrayList<>();
        this.userList.add(new User(1L, "user1@gmail.com", "pwd1","User1"));
        this.userList.add(new User(2L, "user2@gmail.com", "pwd2","User2"));
        this.userList.add(new User(3L, "user3@gmail.com", "pwd3","User3"));

        userList = userRepository.saveAll(userList);
    }

    @Test
    void shouldFetchAllUsers() throws Exception {
        this.mockMvc.perform(get("/api/users"))
                .andExpect(status().isOk())
                .andExpect(jsonPath("$.size()", is(userList.size())));
    }

    @Test
    void shouldFindUserById() throws Exception {
        User user = userList.get(0);
        Long userId = user.getId();

        this.mockMvc.perform(get("/api/users/{id}", userId))
                .andExpect(status().isOk())
                .andExpect(jsonPath("$.email", is(user.getEmail())))
                .andExpect(jsonPath("$.password", is(user.getPassword())))
                .andExpect(jsonPath("$.name", is(user.getName())))
        ;
    }

    @Test
    void shouldCreateNewUser() throws Exception {
        User user = new User(null, "user@gmail.com", "pwd", "name");
        this.mockMvc.perform(post("/api/users")
                .contentType(MediaType.APPLICATION_JSON_UTF8)
                .content(objectMapper.writeValueAsString(user)))
                .andExpect(status().isCreated())
                .andExpect(jsonPath("$.email", is(user.getEmail())))
                .andExpect(jsonPath("$.password", is(user.getPassword())))
                .andExpect(jsonPath("$.name", is(user.getName())));

    }

    @Test
    void shouldReturn400WhenCreateNewUserWithoutEmail() throws Exception {
        User user = new User(null, null, "pwd", "Name");

        this.mockMvc.perform(post("/api/users")
                .contentType(MediaType.APPLICATION_JSON_UTF8)
                .content(objectMapper.writeValueAsString(user)))
                .andExpect(status().isBadRequest())
                .andExpect(header().string("Content-Type", is("application/problem+json")))
                .andExpect(jsonPath("$.type", is("https://zalando.github.io/problem/constraint-violation")))
                .andExpect(jsonPath("$.title", is("Constraint Violation")))
                .andExpect(jsonPath("$.status", is(400)))
                .andExpect(jsonPath("$.violations", hasSize(1)))
                .andExpect(jsonPath("$.violations[0].field", is("email")))
                .andExpect(jsonPath("$.violations[0].message", is("Email should not be empty")))
                .andReturn()
        ;
    }

    @Test
    void shouldUpdateUser() throws Exception {
        User user = userList.get(0);
        user.setPassword("newpwd");
        user.setName("NewName");

        this.mockMvc.perform(put("/api/users/{id}", user.getId())
                .contentType(MediaType.APPLICATION_JSON_UTF8)
                .content(objectMapper.writeValueAsString(user)))
                .andExpect(status().isOk())
                .andExpect(jsonPath("$.email", is(user.getEmail())))
                .andExpect(jsonPath("$.password", is(user.getPassword())))
                .andExpect(jsonPath("$.name", is(user.getName())));

    }

    @Test
    void shouldDeleteUser() throws Exception {
        User user = userList.get(0);

        this.mockMvc.perform(
                delete("/api/users/{id}", user.getId()))
                .andExpect(status().isOk())
                .andExpect(jsonPath("$.email", is(user.getEmail())))
                .andExpect(jsonPath("$.password", is(user.getPassword())))
                .andExpect(jsonPath("$.name", is(user.getName())));

    }

}
```

The **UserControllerIT** tests look very similar to **UserControllerTest**, with the difference being how we load the ApplicationContext.
While using **@SpringBootTest**, Spring Boot will actually start the application by loading the entire application,
so that tests will fail if there is any misconfiguration.

Next, we are going to write a test for **UserRepository** using **@DataJpaTest**.
But we want to run tests against a real database, not with an in-memory database.
We can use **@AutoConfigureTestDatabase(replace=AutoConfigureTestDatabase.Replace.NONE)** to turn off using an in-memory database and use the configured database.

Let us create **PostgreSQLContainerInitializer** so that any repository tests can use this to configure dynamic PostgreSQL database properties.

```java
package com.sivalabs.myservice.common;

import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.test.util.TestPropertyValues;
import org.springframework.context.ApplicationContextInitializer;
import org.springframework.context.ConfigurableApplicationContext;
import org.testcontainers.containers.PostgreSQLContainer;

@Slf4j
public class PostgreSQLContainerInitializer
        implements ApplicationContextInitializer<ConfigurableApplicationContext> {

    private static PostgreSQLContainer sqlContainer;

    static {
        sqlContainer = new PostgreSQLContainer("postgres:10.7")
                .withDatabaseName("integration-tests-db")
                .withUsername("sa")
                .withPassword("sa");
        sqlContainer.start();
    }

    public void initialize (ConfigurableApplicationContext configurableApplicationContext){
        TestPropertyValues.of(
                "spring.datasource.url=" + sqlContainer.getJdbcUrl(),
                "spring.datasource.username=" + sqlContainer.getUsername(),
                "spring.datasource.password=" + sqlContainer.getPassword()
        ).applyTo(configurableApplicationContext.getEnvironment());
    }

}
```

Now we can create **UserRepositoryTest** as follows:

```java
package com.sivalabs.myservice.repositories;

import com.sivalabs.myservice.common.PostgreSQLContainerInitializer;
import com.sivalabs.myservice.entities.User;
import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.jdbc.AutoConfigureTestDatabase;
import org.springframework.boot.test.autoconfigure.orm.jpa.DataJpaTest;
import org.springframework.test.context.ContextConfiguration;
import javax.persistence.EntityManager;
import java.util.Optional;
import static org.assertj.core.api.Assertions.assertThat;

@Slf4j
@DataJpaTest
@AutoConfigureTestDatabase(replace= AutoConfigureTestDatabase.Replace.NONE)
@ContextConfiguration(initializers = {PostgreSQLContainerInitializer.class})
class UserRepositoryTest {

    @Autowired
    EntityManager entityManager;

    @Autowired
    private UserRepository userRepository;

    @Test
    void shouldReturnUserGivenValidCredentials() {
        User user = new User(null, "test@gmail.com", "test", "Test");
        entityManager.persist(user);
        
        Optional<User> userOptional = userRepository.login("test@gmail.com", "test");
        
        assertThat(userOptional).isNotEmpty();
    }
}
```

Well, I guess we learned something about how to write unit and integration tests using various Spring Boot features.

We are living in a Microservices world, and there is a high chance that our service might talk to other microservices.
How are we going to test those integration points? How are we going to verify the timeout scenarios?
Well, we can certainly use a Mock object and pray to God that it will work fine in production :-)
Or we can use libraries like **MockServer** to simulate service-to-service communication.

## Testing MicroService Integrations using MockServer
Assume from our application we want to fetch a GitHub profile of a user. We can use the GitHub REST API to fetch the user profile.
Also, we want to time out the call after 2 seconds, and if we don't get a response by that time, we want to return a default user response.

We can implement this using **Hystrix** as follows:

#### application.properties
```java
githuub.api.base-url=https://api.github.com
```

```java
package com.sivalabs.myservice.config;

import lombok.Data;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;

@Component
@Data
public class ApplicationProperties {

    @Value("${githuub.api.base-url}")
    private String githubBaseUrl;
}
```

Register a **RestTemplate** bean and enable the Hystrix Circuit Breaker using **@EnableCircuitBreaker**.

```java
package com.sivalabs.myservice;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.circuitbreaker.EnableCircuitBreaker;
import org.springframework.context.annotation.Bean;
import org.springframework.web.client.RestTemplate;

@SpringBootApplication
@EnableCircuitBreaker
public class Application
{
	public static void main(String[] args) {
		SpringApplication.run(Application.class, args);
	}

	@Bean
	public RestTemplate restTemplate() {
		return new RestTemplate();
	}
}
```

Create a **GithubUser** class which holds the response from the GitHub API.

```java
package com.sivalabs.myservice.model;

import com.fasterxml.jackson.annotation.JsonProperty;
import lombok.Data;

@Data
public class GithubUser {
    private Long id;
    private String login;
    private String url;
    private String name;
    @JsonProperty("public_repos")
    private int publicRepos;
    private int followers;
    private int following;
}
```

Create **GithubService** which talks to the GitHub REST API using **RestTemplate** as follows:

```java
package com.sivalabs.myservice.services;

import com.netflix.hystrix.contrib.javanica.annotation.HystrixCommand;
import com.netflix.hystrix.contrib.javanica.annotation.HystrixProperty;
import com.sivalabs.myservice.config.ApplicationProperties;
import com.sivalabs.myservice.model.GithubUser;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.web.client.RestTemplate;

@Service
@Slf4j
public class GithubService {
    private final ApplicationProperties properties;
    private final RestTemplate restTemplate;

    @Autowired
    public GithubService(ApplicationProperties properties, RestTemplate restTemplate) {
        this.properties = properties;
        this.restTemplate = restTemplate;
    }

    @HystrixCommand(fallbackMethod = "getDefaultUser", commandProperties = {
            @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "2000")
    })
    public GithubUser getGithubUserProfile(String username) {
        log.info("GithubBaseUrl:"+properties.getGithubBaseUrl());
        return this.restTemplate.getForObject(properties.getGithubBaseUrl() + "/users/" + username, GithubUser.class);
    }

    GithubUser getDefaultUser(String username) {
        log.info("---------getDefaultUser-----------");
        GithubUser user = new GithubUser();
        user.setId(-1L);
        user.setLogin("guest");
        user.setName("Guest");
        user.setPublicRepos(0);
        return user;
    }
}
```

Let us create a **GithubController** with an endpoint to return the user's GitHub profile.

```java
package com.sivalabs.myservice.web.controllers;

import com.sivalabs.myservice.model.GithubUser;
import com.sivalabs.myservice.services.GithubService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/api/github")
public class GithubController {

    private final GithubService githubService;

    @Autowired
    public GithubController(GithubService githubService) {
        this.githubService = githubService;
    }

    @GetMapping("/users/{username}")
    public GithubUser getGithubUserProfile(@PathVariable String username) {
        return githubService.getGithubUserProfile(username);
    }
}
```

We can use **MockServer** to simulate the dependent microservice responses so that we can verify our application's behavior in various scenarios.

We can use [TestContainers support to spin up MockServer](https://www.testcontainers.org/modules/mockserver/) Docker container as follows:

Add the following dependencies:

```xml
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>mockserver</artifactId>
    <version>1.11.3</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.mock-server</groupId>
    <artifactId>mockserver-netty</artifactId>
    <version>5.5.1</version>
    <scope>test</scope>
</dependency>
```

In **AbstractIntegrationTest** add **MockServerContainer** configuration as follows:

```java
import org.mockserver.client.MockServerClient;
import org.testcontainers.containers.MockServerContainer;

@Slf4j
@ActiveProfiles("test")
@SpringBootTest(webEnvironment = RANDOM_PORT)
@AutoConfigureMockMvc
@ContextConfiguration(initializers = {AbstractIntegrationTest.Initializer.class})
public abstract class AbstractIntegrationTest {
    ...
    ...
    private static MockServerContainer mockServerContainer;

    static {
        ....
        ....
        mockServerContainer = new MockServerContainer();
        mockServerContainer.start();
    }

    protected MockServerClient mockServerClient = new MockServerClient(
            mockServerContainer.getContainerIpAddress(),
            mockServerContainer.getServerPort());


    public static class Initializer
            implements ApplicationContextInitializer<ConfigurableApplicationContext> {
        public void initialize(ConfigurableApplicationContext configurableApplicationContext) {
            TestPropertyValues.of(
                    "spring.datasource.url=" + sqlContainer.getJdbcUrl(),
                    "spring.datasource.username=" + sqlContainer.getUsername(),
                    "spring.datasource.password=" + sqlContainer.getPassword(),
                    "githuub.api.base-url=" + mockServerContainer.getEndpoint()
            ).applyTo(configurableApplicationContext.getEnvironment());
        }
    }

}
```

Note that we are starting **MockServerContainer** and injecting the endpoint URL with **"githuub.api.base-url="+mockServerContainer.getEndpoint()**.
Also, we have created **MockServerClient**, which we are going to use for setting up the expected responses.

```java
package com.sivalabs.myservice.web.controllers;

import com.sivalabs.myservice.common.AbstractIntegrationTest;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.mockserver.model.Header;
import org.mockserver.verify.VerificationTimes;

import java.util.concurrent.TimeUnit;

import static org.hamcrest.CoreMatchers.is;
import static org.mockserver.model.HttpRequest.request;
import static org.mockserver.model.HttpResponse.response;
import static org.mockserver.model.JsonBody.json;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;

class GithubControllerIT extends AbstractIntegrationTest {

    @BeforeEach
    void setup() {
        mockServerClient.reset();
    }

    @Test
    void shouldGetGithubUserProfile() throws Exception {
        String username = "sivaprasadreddy";
        mockGetUserFromGithub(username);
        this.mockMvc.perform(get("/api/github/users/{username}", username))
                .andExpect(status().isOk())
                .andExpect(jsonPath("$.login", is(username)))
                .andExpect(jsonPath("$.name", is("K. Siva Prasad Reddy")))
                .andExpect(jsonPath("$.public_repos", is(50)))
                ;
        verifyMockServerRequest("GET", "/users/.*", 1);
    }

    private void mockGetUserFromGithub(String username) {
        mockServerClient.when(
            request().withMethod("GET").withPath("/users/.*"))
            .respond(
                response()
                    .withStatusCode(200)
                    .withHeaders(new Header("Content-Type", "application/json; charset=utf-8"))
                    .withBody(json("{ " +
                            "\"login\": \""+username+"\", " +
                            "\"name\": \"K. Siva Prasad Reddy\", " +
                            "\"public_repos\": 50 " +
                            "}"))
            );
    }

    private void verifyMockServerRequest(String method, String path, int times) {
        mockServerClient.verify(
                request()
                        .withMethod(method)
                        .withPath(path),
                VerificationTimes.exactly(times)
        );
    }
}
```

Note that we are setting up an expected JSON response for the **\<githuub.api.base-url\>/users/.\*** URL pattern on **mockServerClient**.
So, when we make a call to **http://localhost:8080/api/github/users/{username}**, GithubController will in turn call GithubService, which makes a call to
**\<githuub.api.base-url\>/users/{username}** and returns the mock JSON response that we set using **mockServerClient**.

We can also simulate failure and timeout scenarios as follows:

```java
@Test
void shouldGetDefaultUserProfileWhenFetchingFromGithubFails() throws Exception {
    mockGetUserFromGithubFailure();
    this.mockMvc.perform(get("/api/github/users/{username}", "dummy"))
            .andExpect(status().isOk())
            .andExpect(jsonPath("$.login", is("guest")))
            .andExpect(jsonPath("$.name", is("Guest")))
            .andExpect(jsonPath("$.public_repos", is(0)))
    ;
    verifyMockServerRequest("GET", "/users/.*", 1);
}

@Test
void shouldGetDefaultUserProfileWhenFetchingFromGithubTimeout() throws Exception {
    mockGetUserFromGithubDelayResponse();
    this.mockMvc.perform(get("/api/github/users/{username}", "dummy"))
            .andExpect(status().isOk())
            .andExpect(jsonPath("$.login", is("guest")))
            .andExpect(jsonPath("$.name", is("Guest")))
            .andExpect(jsonPath("$.public_repos", is(0)))
    ;
    verifyMockServerRequest("GET", "/users/.*", 1);
}

private void mockGetUserFromGithubDelayResponse() {
    mockServerClient.when(
        request().withMethod("GET").withPath("/users/.*"))
        .respond(response().withStatusCode(200).withDelay(TimeUnit.SECONDS, 10));
}

private void mockGetUserFromGithubFailure() {
    mockServerClient.when(
        request().withMethod("GET").withPath("/users/.*"))
        .respond(response().withStatusCode(404));
}
```

In **shouldGetDefaultUserProfileWhenFetchingFromGithubFails()** test, we are setting up the mockServer to respond
with a **404 error** to verify whether the **Hystrix fallback** method is working.

Similarly, in shouldGetDefaultUserProfileWhenFetchingFromGithubTimeout() test, we are setting up the mockServer to respond
with a **delay of 10 SECONDS** to verify whether the **Hystrix timeout** is working.

Make sure to reset **mockServerClient** using **mockServerClient.reset()** for every test in the **@BeforeEach** method to reset any expectations set in the previous test run.

> Sample application Code for this article can be found at https://github.com/sivaprasadreddy/spring-boot-tutorials/tree/master/testing/springboot-testing-demo

I hope we have covered many common testing scenarios in Spring Boot applications.

Thank you for reading the article; your feedback is welcome.
If you find this article useful, please share it on Twitter.




# Kafka Tutorial - Producer and Consumer using Spring Boot

In the previous post [Kafka Tutorial - Java Producer and Consumer]({{< relref "2019-06-21-kafka-tutorial-java-producer-consumer.md" >}}), we learned how to implement a Producer and Consumer for a Kafka topic using the plain Java Client API.

In this post, we are going to look at how to use Spring for Kafka, which provides a high-level abstraction over the Kafka Java Client API to make it easier to work with Kafka.

<!--more-->


> You can find the source code for this article at https://github.com/sivaprasadreddy/kafka-tutorial

## Spring for Kafka without Spring Boot
First, let's look at how to use Spring for Kafka without Spring Boot magic so that we will understand how to configure the necessary components.

Create a maven based project and configure the dependencies as follows:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
         http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.sivalabs</groupId>
    <artifactId>spring-kafka-sample</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka</artifactId>
            <version>2.2.6.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-test</artifactId>
            <version>5.1.7.RELEASE</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-classic</artifactId>
            <version>1.2.3</version>
        </dependency>
        <dependency>
            <groupId>org.assertj</groupId>
            <artifactId>assertj-core</artifactId>
            <version>3.12.2</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka-test</artifactId>
            <version>2.2.6.RELEASE</version>
            <scope>test</scope>
        </dependency>

    </dependencies>
</project>
```

Next, let us configure the beans for Producer and Consumer using Spring's JavaConfig class as follows:

```java
package com.sivalabs.sample;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.apache.kafka.common.serialization.StringSerializer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.annotation.EnableKafka;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.core.*;

import java.util.HashMap;
import java.util.Map;

@Configuration
@ComponentScan
@EnableKafka
public class KafkaConfig {

    public static final String TOPIC = "test-1";
    private String bootstrapServers = "localhost:9092";

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory =
                new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        return factory;
    }

    @Bean
    public ConsumerFactory<String, String> consumerFactory() {
        return new DefaultKafkaConsumerFactory<>(consumerConfigs());
    }

    @Bean
    public Map<String, Object> consumerConfigs() {
        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "group");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        return props;
    }

    @Bean
    public KafkaTemplate<String, String> kafkaTemplate() {
        return new KafkaTemplate<>(producerFactory());
    }

    @Bean
    public ProducerFactory<String, String> producerFactory() {
        return new DefaultKafkaProducerFactory<>(producerConfigs());
    }

    @Bean
    public Map<String, Object> producerConfigs() {
        Map<String, Object> props = new HashMap<>();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 33554432);
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        return props;
    }

}
```

We have configured the **producerConfigs, ProducerFactory, KafkaTemplate** beans for Producer and **consumerConfigs, ConsumerFactory, ConcurrentKafkaListenerContainerFactory** beans for Consumer.

Now, we can send message to Kafka topic using **KafkaTemplate** using **kafkaTemplate.send(topicName, key, value)**.


```java
package com.sivalabs.sample;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Component;

@Component
public class MessageSender {

    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    public void send(String key, String value) {
        kafkaTemplate.send(KafkaConfig.TOPIC, key, value);
    }
}
```

We can implement a Kafka topic listener using **@KafkaListener** annotation as follows:

```java
package com.sivalabs.sample;

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;

@Component
public class MessageListener {

    private static final Logger LOGGER = LoggerFactory.getLogger(MessageListener.class);

    @KafkaListener(topics = KafkaConfig.TOPIC)
    public void handle(ConsumerRecord<?, ?> cr) {
        LOGGER.info("Message: "+cr.key()+"="+cr.value());
    }
}
```

Well, this is how we can configure a Kafka Producer and Consumer using Spring JavaConfiguration without using Spring Boot.
As you might have guessed, with Spring Boot auto-configuration, the same application can be implemented with much less code.

## Spring for Kafka with Spring Boot

Let us create a Spring Boot application with the Kafka starter. 

```xml
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka-test</artifactId>
    <scope>test</scope>
</dependency>
```

Spring Boot provides **org.springframework.boot.autoconfigure.kafka.KafkaAutoConfiguration**, which auto-configures **ProducerFactory, KafkaTemplate, ConsumerFactory, ConcurrentKafkaListenerContainerFactory**, etc., beans. We just need to configure the following properties in **application.properties** as follows:


```properties
spring.kafka.bootstrap-servers=localhost:9092

spring.kafka.consumer.group-id=demo-group
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer

spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer
```

That's it. Now we can send messages to a topic using **KafkaTemplate** and implement listeners using **@KafkaListener** as explained in the earlier section.

> The Spring Boot auto-configuration for Kafka only works for a single **ProducerFactory** and **ConsumerFactory**. If you want to configure multiple **ProducerFactory** and **ConsumerFactory** beans, you can disable **KafkaAutoConfiguration** and configure the beans by yourself as shown in [this github repo](https://github.com/sivaprasadreddy/kafka-tutorial/tree/master/spring-boot-multiple-producers-consumers).

## Summary
In this post, we have learned how to implement a Kafka Producer and Consumer using **Spring for Kafka** with and without Spring Boot. In the next post, we will learn about working with Kafka using **Spring Cloud Streams with Kafka binder**.
# Kafka Tutorial - Java Producer and Consumer


[Kafka](https://kafka.apache.org/), depending on how you use it, can be seen as a Message Broker, Event Store, or a Streaming Platform, etc. Kafka became a preferred technology for many modern applications because of various reasons, like:

*   Kafka can be used as an Event Store if you are using an event-driven microservices architecture.
*   Kafka can be used as a Message Broker to enable communication across multiple applications.
*   Kafka can be used as a streaming platform for processing events in real-time.
    etc. etc.

<!--more-->

You can learn more about Kafka from its official documentation: https://kafka.apache.org/documentation/.

> If you like to learn from video tutorials, I would suggest watching https://www.learningjournal.guru/courses/kafka/kafka-foundation-training/.

In this post, we are going to use Kafka as a Message Broker and learn how to send and receive messages to and from Kafka topics using the Java Client API.

## Kafka Installation
You can download Kafka distribution from https://kafka.apache.org/downloads and start the Kafka server as follows.

[Kafka uses ZooKeeper](https://www.cloudkarafka.com/blog/2018-07-04-cloudkarafka_what_is_zookeeper.html) for Controller election, Configuration of Topics, Membership of the cluster, etc., and is bundled with the Kafka distribution. So, first we need to start ZooKeeper as follows:

```bash
kafka_2.12-2.2.0> bin/zookeeper-server-start.sh config/zookeeper.properties
```

Once ZooKeeper has started successfully, start the Kafka server as follows:

```bash
kafka_2.12-2.2.0> bin/kafka-server-start.sh config/server.properties
```

Instead of installing Kafka as described above, if you prefer using Docker to spin up a Kafka server, you can use the following docker-compose configuration. You can also find various docker-compose configurations at https://github.com/confluentinc/cp-docker-images/tree/5.2.2-post/examples.

**docker-compose.yml**

```yml
version: '3.1'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:5.2.2
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-enterprise-kafka:5.2.2
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "29092:29092"
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:29092
      CONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT: zookeeper:2181
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'true'
      CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'
```

You can start the Zookeeper and Kafka docker containers using **docker-compose up**.

## Working with Kafka Java Client API

With the Kafka setup in place, let us create a Maven/Gradle project and add the following dependency.

**Maven:**
```xml
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>2.2.0</version>
</dependency>
```

**Gradle:**
```xml
compile group: 'org.apache.kafka', name: 'kafka-clients', version: '2.2.0'
```

### Kafka Topic Producer
 A message to a Kafka topic typically contains a key, a value, and optionally a set of headers.
 While creating a producer, we need to specify Key and Value Serializers so that the API knows how to serialize those values.
 
 Let us create the **MessageProducer** class as follows:
```java
package com.sivalabs.kafkasample;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.header.Headers;
import org.apache.kafka.common.header.internals.RecordHeader;
import org.apache.kafka.common.header.internals.RecordHeaders;
import org.apache.kafka.common.serialization.LongSerializer;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.Properties;
import java.util.concurrent.TimeUnit;

public class MessageProducer {

    public static final String TOPIC_NAME = "topic-1";

    public static void main(String[] args) throws Exception {

        Properties producerProperties = new Properties();
        producerProperties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        producerProperties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, LongSerializer.class.getName());
        producerProperties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        producerProperties.put(ProducerConfig.ACKS_CONFIG, "all");

        Producer<Long, String> producer = new KafkaProducer<>(producerProperties);

        try {
            Long counter = 1L;
            while (true) {
                Headers headers = new RecordHeaders();
                headers.add(new RecordHeader("header-1", "header-value-1".getBytes()));
                headers.add(new RecordHeader("header-2", "header-value-2".getBytes()));
                ProducerRecord<Long, String> record = new ProducerRecord<Long, String>(TOPIC_NAME, null, counter, "A sample message", headers);

                producer.send(record);
                System.out.println("Send record#"+counter);
                counter++;
                TimeUnit.SECONDS.sleep(3);
            }
        } finally {
            producer.close();
        }
    }
}
```

We have created an instance of Properties to populate producer configuration properties like bootstrap servers and key and value serializer types. We are planning to send a **Long/String** key/value pair, so we used **LongSerializer** and **StringSerializer** respectively. Then we created a **KafkaProducer** instance using **producerProperties**. A message to be sent to a topic is represented as an instance of **ProducerRecord**. Finally, we send the message using **KafkaProducer.send()** method.

We have configured **ProducerConfig.ACKS_CONFIG** to **all** so that the message is considered successfully written to the Kafka topic only when the full set of in-sync replicas has acknowledged it.

### Kafka Topic Consumer
Let us create a Consumer to read from Kafka **topic-1** as follows:

```java
package com.sivalabs.kafkasample;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.errors.WakeupException;
import org.apache.kafka.common.serialization.LongDeserializer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.time.Duration;
import java.util.Collections;
import java.util.Properties;
import java.util.concurrent.TimeUnit;

public class MessageConsumer {

    public static final String TOPIC_NAME = "topic-1";

    public static void main(String[] args) {

        Properties consumerProperties = new Properties();
        consumerProperties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        consumerProperties.put(ConsumerConfig.GROUP_ID_CONFIG, "demo-group");
        consumerProperties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, LongDeserializer.class.getName());
        consumerProperties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        consumerProperties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        consumerProperties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
        consumerProperties.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 10);

        KafkaConsumer<Long, String> kafkaConsumer = new KafkaConsumer<>(consumerProperties);
        kafkaConsumer.subscribe(Collections.singletonList(TOPIC_NAME));

        try {
            while (true) {
                ConsumerRecords<Long, String> records = kafkaConsumer.poll(Duration.ofMillis(3000));
                System.out.println("Fetched " + records.count() + " records");
                for (ConsumerRecord<Long, String> record : records) {
                    System.out.println("Received: " + record.key() + ":" + record.value());
                    record.headers().forEach(header -> {
                        System.out.println("Header key: " + header.key() + ", value:" + header.value());
                    });
                }
                kafkaConsumer.commitSync();
                TimeUnit.SECONDS.sleep(5);
            }
        }catch (Exception e) {
            e.printStackTrace();
        } finally {
            kafkaConsumer.close();
        }
    }
}
```

Similar to the Producer, we have created a `Properties` instance with consumer configuration properties. We have created a **KafkaConsumer** and subscribed to a single topic, **test-topic**.

We are polling for new messages with a timeout value of 1000 milliseconds and printing each message key and value. Along with it, we are also printing the headers of each message. When we poll, if there are no new messages, it will return an empty list.

We have configured the **ConsumerConfig.AUTO_OFFSET_RESET_CONFIG** property to **earliest** so when we start the consumer, it will start reading messages from the last acknowledged offset.

We have turned off the default automatic acknowledgment mechanism by setting **ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG** to **false**, and as we read the message, we are acknowledging them with **kafkaConsumer.commitSync()**.

Also, we are limiting the maximum number of records to read per poll to 10 by configuring **ConsumerConfig.MAX_POLL_RECORDS_CONFIG** property.

## Summary
In this post, we have learned how to create a simple Producer and Consumer for a Kafka topic using the Java Client API. In the next article, we will learn how to implement a Kafka Producer and Consumer using **Spring for Kafka**.
# Spring Boot Best Practices


I have been working with [Spring Boot](https://spring.io/projects/spring-boot) for many years, and over time, I have worked with many Spring Boot-based codebases.
There are a few common mistakes that I observe in projects that use Spring Boot. So, I thought of writing down a few good practices that can be followed while using Spring Boot.

<!--more-->


## 1. Understand Spring Boot Core Concepts
I know this sounds very obvious, but I see many developers jumping into using Spring Boot without having any prior knowledge of Spring or Dependency Injection. So they just keep adding annotations and **@Bean** definitions here and there until it works.

Spring Boot is an opinionated framework with a powerful **auto-configuration** mechanism based on the **convention-over-configuration** philosophy. 
So, it is very important to understand the following things:

* **How to structure application code?**
A typical Spring Boot application has a main entry-point class that is annotated with the **@SpringBootApplication** annotation. Spring Boot automatically scans for Spring components from the entry-point class's package and its sub-packages.
Make sure to read and understand the section https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#using-boot-structuring-your-code.

* **How does AutoConfiguration work?** 
The most important aspect of Spring Boot is its AutoConfiguration mechanism. Without properly understanding how AutoConfiguration works, it all looks like magic. You can read my blog post [How Spring Boot AutoConfiguration magic works?](https://sivalabs.in/how-springboot-autoconfiguration-magic/) to learn about how AutoConfiguration works.

## 2. Leverage Spring Boot's built-in customization features
Spring Boot auto-configures many beans based on various criteria, like the existence of a class in the classpath or the absence of a property, and also provides the ability to customize the configuration using properties.

I sometimes see people configuring the entire bean definition using **@Bean** just to customize one property of it, whereas they can simply customize it using a property in the **application.properties** file.

For example, if you want to customize the **DataSource** bean default values you can use **spring.datasource.\*** properties instead of configuring the **DataSource** bean by yourself.

You can see all the supported customization properties here 
https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#common-application-properties

## 3. Don't add unnecessary starters and turn off unnecessary features
One of the primary reasons for the popularity of Spring Boot is its getting-started experience. You can simply go to https://start.spring.io/, select the starters you want to add, and run the application. Spring Boot creates a lot of beans behind the scenes based on the starters we add, and hence it may increase the memory footprint and also increase the startup time. So, just add the starters you absolutely need.

Spring Boot automatically registers beans using AutoConfiguration. Sometimes we may not need all those features, and we can turn them off.

For example, if we are not planning to use JMX feature we can turn it off using **spring.jmx.enabled=false**.

## 4. Prefer constructor based dependency injection
We can autowire dependencies using **constructor-injection**, **setter-injection**, or **@Autowired** on the property itself.
Personally, I think constructor injection is the better option. So, I try to always define the dependencies as **final** variables so that it will force initialization in the constructor.

```java
@Service
@Transactional
public class CustomerService {
    private final CustomerRepository customerRepository;

    public CustomerService(CustomerRepository customerRepository) {
        Assert.notNull(customerRepository, "customerRepository mustn't be null");
        this.customerRepository = customerRepository;
    }
}
```

This way, it will be easier to instantiate **CustomerService** with the necessary dependencies for testing, if required.

## 5. Use TestContainers to test with real dependencies
One common pattern that I have observed in Spring Boot applications is using in-memory databases like H2 for testing and using other RDBMS like Postgres, Oracle, etc., for production. I think this is not a good practice for several reasons. 

First, the production database's behavior might be different from an in-memory database, and hence some bugs might not be detected during testing. Second, some features (e.g., JSONB) might not be supported by in-memory databases, whereas the production database supports them.

These days, most applications use Docker in some way or another. So, we can use [TestContainers](https://www.testcontainers.org/), which is a Docker-based library for testing with a real database similar to the production database.

You can refer to the following to learn how to use TestContainers for Integration Testing with Spring Boot:

* https://github.com/testcontainers/testcontainers-java/tree/master/examples/spring-boot
* https://www.baeldung.com/spring-boot-testcontainers-integration-test

## 6. Use Test Slice annotations to speed up tests
We write Unit tests to test the behavior of a single class and Integration tests to test the behavior of a logical group of components. So, generally we don't need the Spring magic for unit tests. We should be able to create an instance of the class (probably with mock dependencies) and test its behavior.

However, while unit testing Spring-based components like **Controllers** or **Spring Data Repositories**, we need the Spring ApplicationContext to be created. Spring Boot provides **@SpringBootTest** to load the ApplicationContext, and we can test the components. But the **@SpringBootTest** annotation loads all components in the application, which might take more time. 

For unit tests, we generally want to load only the component we want to test with mock dependencies, without loading any other unnecessary dependencies.

Spring Boot provides various other test annotations like **@WebMvcTest**, **@DataJpaTest**, etc., to load only a slice of the ApplicationContext components. So, it is better to unit test the Spring components by using slice-specific annotations rather than **@SpringBootTest**.

You can learn more about using Spring Boot Testing annotations with the following articles:

* https://pivotal.io/application-transformation-recipes/testing/spring-boot-testing-best-practices
* https://spring.io/guides/gs/testing-web/
* https://reflectoring.io/spring-boot-web-controller-test/

## 7. Use database migration tools
While using a relational database with Spring Boot and JPA, it is easier to get started with the auto-generation of the database schema from entity definitions. However, for complex applications, it is better to use database migration tools like [Flyway](https://flywaydb.org/) or [Liquibase](https://www.liquibase.org/). These tools will help in maintaining versioned database migration scripts, keeping track of the current state of the database, and determining which migration scripts need to be run.

## 8. Prefer Off the Shelf security solutions instead of rolling your own
Spring Security makes it easy to implement security, but still, implementing security properly with OAuth, SAML, etc., integrations is still very complex. Security is a very complex subject and needs a lot of expertise to be implemented properly. If it is at all possible to use off-the-shelf security solutions like [Okta](https://www.okta.com/), then use them instead of rolling your own security implementation.

## 9. Use dedicated configuration classes
In our projects, we may need to create several Spring Configuration classes to define or customize beans. For example, we may want to configure **Swagger**, **Jackson**, **AWS**, **ElasticSearch**, **Redis**, etc. Instead of configuring all of them in one configuration class, it would be better to have separate, dedicated Configuration classes for each one of them.

## 10. Avoid boilerplate code with custom Spring Boot starter
In some organizations, there will be some homegrown libraries to achieve some common functionality like security, secrets management, caching, etc. We can create a custom Spring Boot starter to auto-configure the library features so that we don't need to repeat the boilerplate code for every application. 

You can refer to my post [Creating a Custom Spring Boot Starter for Twitter4j](https://sivalabs.in/creating-custom-springboot-starter-for/) on how to create a custom Spring Boot starter.

For more info on creating our own Spring Boot starter, see https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#boot-features-custom-starter


# MicroServices - Part 6 : Distributed Tracing with Spring Cloud Sleuth and Zipkin



One of the challenges in microservices architecture is the ability to debug issues. 
A simple user action might trigger a chain of downstream microservice calls. 
It would be tedious to trace the logs related to a particular user action across microservices. 
In addition to that, we might want to track down why a certain microservice call is taking so much time. 
We can use [Spring Cloud Sleuth](http://cloud.spring.io/spring-cloud-static/Finchley.M7/single/spring-cloud.html#_spring_cloud_sleuth) to handle these kinds of issues. 


<!--more-->


Spring Cloud Sleuth provides the distributed tracing capabilities and we can also export these trace information to [Zipkin](https://zipkin.io/) 
to visualize the call traces.

**MicroServices using Spring Boot & Spring Cloud**

* [Part 1 : MicroServices : Spring Boot & Spring Cloud Overview]({{< relref "2018-03-02-microservices-using-springboot-spring-cloud-part-1-overview.md" >}})
* [Part 2 : MicroServices : Configuration Management with Spring Cloud Config and Vault]({{< relref "2018-03-05-microservices-part-2-configuration-management-spring-cloud-config-vault.md" >}})
* [Part 3 : MicroServices : Spring Cloud Service Registry and Discovery]({{< relref "2018-03-08-microservices-springcloud-eureka.md" >}})
* [Part 4 : MicroServices : Spring Cloud Circuit Breaker using Netflix Hystrix]({{< relref "2018-03-12-spring-cloud-netflix-circuit-breaker.md" >}})
* [Part 5 : MicroServices : Spring Cloud Zuul Proxy as API Gateway]({{< relref "2018-03-15-microservices-part-5-spring-cloud-zuul-proxy-as-api-gateway.md" >}})
* [Part 6 : MicroServices : Distributed Tracing with Spring Cloud Sleuth and Zipkin]({{< relref "2018-03-20-microservices-part-6-distributed-tracing-with-spring-cloud-sleuth-and-zipkin.md" >}})

**In this post we are going to learn:**

* Tracing Distributed Service Calls
* Using Spring Cloud Sleuth for Distributed Tracing
* Distributed Tracing with Zipkin Server

# Tracing Distributed Service Calls

In the microservices world, a user action on UI may invoke one microservice API endpoint, which in turn invoke another microservice endpoint.

For example, when a user sees the catalog, **shoppingcart-ui** will invoke **catalog-service** REST API http://localhost:8181/api/products 
which in turn calls **inventory-service** REST API http://localhost:8282/api/inventory to check for inventory availability.

Suppose, an exception has occurred or the data returned is invalid and you want to investigate what is wrong by looking at logs. But as of now, there is no way to correlate the logs of that particular user across multiple services.

# Poor man's distributed tracing
One solution to this is at the beginning of the call chain we can create a CORRELATION_ID and add it to all log statements. 
Along with it, send CORRELATION_ID as a header to all the downstream services as well so that those downstream services 
also use CORRELATION_ID in logs. This way we can identify all the log statements related to a particular action across services.

We can implement this solution using [MDC](https://logback.qos.ch/manual/mdc.html) feature of Logging frameworks. 
Typically we will have a WebRequest Interceptor where you can check whether there is a CORRELATION_ID header. 
If there is no CORRELATION_ID in the header then create a new one and set it in MDC. 
The logging frameworks include the information set in MDC with all log statements.

But, instead of we doing all this work we can use Spring Cloud Sleuth which will do all this and much more for us.

# Using Spring Cloud Sleuth for Distributed Tracing
First, get familiarize with some terminology of Span, Trace, Annotations here http://cloud.spring.io/spring-cloud-static/Finchley.M7/single/spring-cloud.html#_terminology.

Let us add **Sleuth** starter to both **inventory-service** and **catalog-service**.

```xml
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-sleuth</artifactId>
</dependency>
```

Once you add **Sleuth** starter and start the services you can observe in logs something like this.

```shell
2018-03-20 10:19:15.512  INFO [inventory-service,,,] 53685 --- [trap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver ...
2018-03-20 10:24:15.507  INFO [inventory-service,,,] 53685 --- [trap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver ...
```

Now hit any inventory-service REST endpoint, say http://localhost:8282/api/inventory. Then you can observe **TraceID, SpanID** in the logs.

```shell
2018-03-20 10:15:38.466  INFO [inventory-service,683f8e4370413032,d8abe400c68a9a6b,false] 53685 --- [oryController-3] ... 
```

Sleuth includes the pattern **[appname,traceId,spanId,exportable]** in logs from the MDC.

Now invoke the catalog-service endpoint http://localhost:8181/api/products endpoint which internally invokes inventory-service endpoint http://localhost:8282/api/inventory.

In catalog-service logs you can find log statements something like:

```shell
2018-03-20 10:54:29.625  INFO [catalog-service,0335da07260d3d6f,0335da07260d3d6f,false] 53617 --- [io-8181-exec-10] ...
```

And, check logs in inventory-service, you can find log statements something like:

```shell
2018-03-20 10:54:29.662  INFO [inventory-service,0335da07260d3d6f,1af68249ac3a6902,false] 53685 --- [oryController-6] ...
```

Observe that TraceID **0335da07260d3d6f** is same in both catalog-service and inventory-service for the same REST API call. This way we can easily correlate the logs across services.

The **false** in [inventory-service,0335da07260d3d6f,1af68249ac3a6902,false] indicates that this trace is not exported to any Tracing Server like Zipkin. Let us see how we can export the tracing information to Zipkin.

# Distributed Tracing with Zipkin Server
We learned how to use Sleuth to add tracing information in logs. In addition to that, we can also export this information to Zipkin so that we can visualize this through UI.

## Zipkin Server using an executable jar
The quick and easiest way to start a Zipkin server is using zipkin executable jar provided by Zipkin team.

```shell
curl -sSL https://zipkin.io/quickstart.sh | bash -s
java -jar zipkin.jar
```

This will start Zipkin server on port 9411 and you can access the Zipkin UI Dashboard at http://localhost:9411/zipkin/.

## Zipkin Server as Docker container
As we are going to package our applications as docker images and run them in Docker containers anyway, we can use the official Docker image for OpenZipkin.

We can create a Zipkin server backed by in-memory datastore using the following **docker-compose-mem.yml** file.

```yml
version: '2'
 
services:
  # The zipkin process services the UI, and also exposes a POST endpoint that
  # instrumentation can send trace data to. Scribe is enabled by default.
  zipkin:
    image: openzipkin/zipkin
    container_name: zipkin
    # Environment settings are defined here https://github.com/openzipkin/zipkin/tree/1.19.0/zipkin-server#environment-variables
    environment:
      - STORAGE_TYPE=mem
      # Uncomment to disable scribe
      # - SCRIBE_ENABLED=false
      # Uncomment to enable self-tracing
      # - SELF_TRACING_ENABLED=true
      # Uncomment to enable debug logging
      # - JAVA_OPTS=-Dlogging.level.zipkin=DEBUG
    ports:
      # Port used for the Zipkin UI and HTTP Api
      - 9411:9411
```

You can also use docker image backed by MySQL datastore by using https://github.com/openzipkin/docker-zipkin/blob/master/docker-compose.yml.

## Create Zipkin Server as Spring Boot application (NOT RECOMMENDED)

> NOTE: Creating custom Zipkin server as a Spring Boot application is not a recommended approach anymore. 

We can start Zipkin server as a spring boot application or run in a docker container.

> As of now, Zipkin doesn't support Spring Boot 2. So we can create a Spring Boot application zipkin-server using 1.5.10.RELEASE version and add the zipkin-server, zipkin-autoconfigure-ui dependencies.

```xml
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
    http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
 
    <groupId>com.sivalabs</groupId>
    <artifactId>zipkin-server</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>
 
    <name>zipkin-server</name>
 
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.5.10.RELEASE</version>
    </parent>
 
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
        <spring-cloud.version>Edgware.RELEASE</spring-cloud.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-zipkin</artifactId>
        </dependency>
        <dependency>
            <groupId>io.zipkin.java</groupId>
            <artifactId>zipkin-server</artifactId>
        </dependency>
        <dependency>
            <groupId>io.zipkin.java</groupId>
            <artifactId>zipkin-autoconfigure-ui</artifactId>
            <scope>runtime</scope>
        </dependency>
    </dependencies>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
 
    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
 
    <repositories>
        <repository>
            <id>spring-milestones</id>
            <name>Spring Milestones</name>
            <url>https://repo.spring.io/milestone</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>
    </repositories>
</project>
```

Add **@EnableZipkinServer** annotation to main entry-point class.

```java
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import zipkin.server.EnableZipkinServer;
 
@SpringBootApplication
@EnableZipkinServer
public class ZipkinServerApplication {
 
    public static void main(String[] args) {
        SpringApplication.run(ZipkinServerApplication.class, args);
    }
}
```

Set port and application name in **application.properties**.

```properties
spring.application.name=zipkin-server
server.port=9411
```

Now you can start the Zipkin Server by running **ZipkinServerApplication**.

Once the Zipkin server is started you can go to http://localhost:9411/ to view the Zipkin Server UI Dashboard.

# Exporting tracing info to Zipkin Server
We observed that the tracing information is printed in logs but not exported. We can export them to Zipkin server so that we can visualize traces in Zipkin Server UI Dashboard.

Add **Zipkin Client** starter to both inventory-service and catalog-service.

```xml
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-zipkin</artifactId>
</dependency>
```

Configure Zipkin server URL in **bootstrap.properties** of both inventory-service and catalog-service.

```properties
spring.zipkin.base-url=http://localhost:9411/
spring.sleuth.sampler.probability=1
```

> NOTE: By default spring.sleuth.sampler.probability=0.1 which means only 10% of tracing information will be exported to Zipkin. Make it to your desired percentage.

Now restart both inventory-service and catalog-service and invoke http://localhost:8181/api/products endpoint. 
You can observe that **true** is printed in logs meaning it is exported.

```shell
2018-03-20 11:41:02.241  INFO [catalog-service,7d0d44fe314d7758,7d0d44fe314d7758,true] 53617 --- [nio-8181-exec-5] c.s.c.services.ProductService
```

Now go to Zipkin UI Dashboard, you can see the service names populated in the first dropdown. Select the service you want to check or select all and then click on Find Traces button.

![Zipkin](/images/zipkin-traces.webp "Zipkin")

Click on any trace you can see the call chain and latency details for each span.

![Zipkin Traces](/images/zipkin-trace-details.webp "Zipkin Traces")

You can also click on Dependencies link in the top nav bar to see the service dependencies.

![Zipkin Dependencies](/images/zipkin-deps.webp "Zipkin Dependencies")

> You can find the source code for this article at https://github.com/sivaprasadreddy/spring-boot-microservices-series


# MicroServices - Part 5 : Spring Cloud Zuul Proxy as API Gateway


In microservices architecture, there could be a number of API services and few UI components that are talking to APIs. As of now, many microservices based application still use monolithic front-ends where the entire UI is built as a single module. You may choose to go with micro-frontends where the UI is also decomposed into multiple microservice talking to APIs to get the relevant data. Instead of letting UI know about all our microservices details we can provide a unified proxy interface that will delegate the calls to various microservices based on URL pattern. In this post, we will learn how to create API Gateway using Spring Cloud Zuul Proxy.

<!--more-->


**MicroServices using Spring Boot & Spring Cloud**

* [Part 1 : MicroServices : Spring Boot & Spring Cloud Overview]({{< relref "2018-03-02-microservices-using-springboot-spring-cloud-part-1-overview.md" >}})
* [Part 2 : MicroServices : Configuration Management with Spring Cloud Config and Vault]({{< relref "2018-03-05-microservices-part-2-configuration-management-spring-cloud-config-vault.md" >}})
* [Part 3 : MicroServices : Spring Cloud Service Registry and Discovery]({{< relref "2018-03-08-microservices-springcloud-eureka.md" >}})
* [Part 4 : MicroServices : Spring Cloud Circuit Breaker using Netflix Hystrix]({{< relref "2018-03-12-spring-cloud-netflix-circuit-breaker.md" >}})
* [Part 5 : MicroServices : Spring Cloud Zuul Proxy as API Gateway]({{< relref "2018-03-15-microservices-part-5-spring-cloud-zuul-proxy-as-api-gateway.md" >}})
* [Part 6 : MicroServices : Distributed Tracing with Spring Cloud Sleuth and Zipkin]({{< relref "2018-03-20-microservices-part-6-distributed-tracing-with-spring-cloud-sleuth-and-zipkin.md" >}})

**In this post we are going to learn:**

* Why do we need API Gateway?
* Implementing API Gateway using Spring Cloud Zuul Proxy
* Using Zuul Filters for cross-cutting concerns

# Why do we need API Gateway?
API Gateway, aka Edge Service, provides a unified interface for a set of microservices so that clients no need to know about all the details of microservices internals. However, there are some pros and cons of using API Gateway pattern in microservices architecture.

## Pros:

* Provides easier interface to clients
* Can be used to prevent exposing the internal microservices structure to clients
* Allows to refactor microservices without forcing the clients to refactor consuming logic
* Can centralize cross-cutting concerns like security, monitoring, rate limiting etc

## Cons:

* It could become a single point of failure if proper measures are not taken to make it highly available
* Knowledge of various microservice API may creep into API Gateway

# Implementing API Gateway using Spring Cloud Zuul Proxy

Spring Cloud provides Zuul proxy, similar to **Nginx**, that can be used to create API Gateway.

Let us create a front-end UI module **shoppingcart-ui** as a Spring Boot application which also acts as Zuul proxy. 

Create a Spring Boot project with **Web, Config Client, Eureka Discovery, Zuul** starters and annotate the main entry-point class with 
**@EnableZuulProxy**.

```xml
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-config</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix-zuul</artifactId>
</dependency>
```

**ShoppingcartUiApplication.java**

```java
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.netflix.zuul.EnableZuulProxy;
 
@EnableZuulProxy
@SpringBootApplication
public class ShoppingcartUiApplication {
    public static void main(String[] args) {
        SpringApplication.run(ShoppingcartUiApplication.class, args);
    }
}
```

As we are using Eureka Discovery also, requests from the proxy with the URL patterns **/service-id/\*\*** will be routed to the service 
registered in Eureka Server with service id **service-id**.

For, ex: From UI application if we make a request to http://localhost:8080/catalog-service/products then it will lookup in 
Service Registry for ServiceID **catalog-service** and send the request with URL /products to one of the available catalog-service instances.

To make it happen we need to register "shoppingcart-ui" with Eureka Service Registry.

**bootstrap.properties**

```properties
spring.application.name=shoppingcart-ui
server.port=8080
eureka.client.service-url.defaultZone=http://localhost:8761/eureka/
```

With this configuration now we can fetch product information from catalog-service using jQuery as follows:

```js
$.ajax({
    url: '/catalog-service/products'
})
.done(function(data) {
    this.products = data;
}.bind(this));
```

Here from our UI application, we are making a call to http://localhost:8080/catalog-service/products. Assuming catalog-service is registered with ServiceID "catalog-service" and running on port 8181, this request will be forwarded to http://host:8181/products. But UI is completely unaware of where is the actual catalog-service running, its hostname port number etc.

We can also use a common prefix for URLs, like **/api**, for which we want Zuul to proxy by setting **zuul.prefix** property.

```shell
zuul.prefix=/api
```

Now from UI we can make a request to fetch products at http://localhost:8080/api/catalog-service/products. 
By default, Zuul will strip the prefix and forward the request.

You can also customize the path mappings of a service as follows:

```properties
zuul.routes.catalogservice.path=/catalog/**
zuul.routes.catalogservice.serviceId=catalog-service
```

With this configuration, you can use URL http://localhost:8080/api/catalog/products which will be forwarded to the service 
with serviceId catalog-service.

By default, all the services registered with Eureka Server will be exposed. You can use **zuul.ignored-services** property 
to disable this behavior and expose only the explicitly configured services.

```properties
zuul.ignored-services=*
 
zuul.routes.catalogservice.path=/catalog/**
zuul.routes.catalogservice.serviceId=catalog-service
 
zuul.routes.orderservice.path=/orders/**
zuul.routes.orderservice.serviceId=order-service
```

With this configuration only **catalog-service, order-service** is exposed through Zuul proxy but not **inventory-service**.

# Using Zuul Filters for cross-cutting concerns
As Zuul act as a proxy to all our microservices, we can use Zuul service to implement some cross-cutting concerns like security, rate limiting etc. One common use-case is forwarding the Authentication headers to all the downstream services.

Typically in microservices, we will use OAuth service for authentication and authorization. 
Once the client is authenticated OAuth service will generate a token which should be included in the requests making to other microservices so that client need not be authenticated for every service separately. We can use Zuul filter to implement features like this.

```java
import com.netflix.zuul.ZuulFilter;
import com.netflix.zuul.context.RequestContext;
import com.netflix.zuul.exception.ZuulException;
import javax.servlet.http.HttpServletRequest;
import java.util.UUID;
import static org.springframework.cloud.netflix.zuul.filters.support.FilterConstants.PRE_TYPE;
 
public class AuthHeaderFilter extends ZuulFilter {
    @Override
    public String filterType() {
        return PRE_TYPE;
    }
 
    @Override
    public int filterOrder() {
        return 0;
    }
 
    @Override
    public boolean shouldFilter() {
        return true;
    }
 
    @Override
    public Object run() throws ZuulException {
        RequestContext ctx = RequestContext.getCurrentContext();
        HttpServletRequest request = ctx.getRequest();
 
        if (request.getAttribute("AUTH_HEADER") == null) {
            //generate or get AUTH_TOKEN, ex from Spring Session repository
            String sessionId = UUID.randomUUID().toString();
             
            ctx.addZuulRequestHeader("AUTH_HEADER", sessionId);
        }
        return null;
    }
}
```

We are adding **AUTH_HEADER** as a request header using **RequestContext.addZuulRequestHeader()** which will be 
forwarded to downstream services. We need to register it as a Spring bean.

```java
@Bean
AuthHeaderFilter authHeaderFilter() {
    return new AuthHeaderFilter();
}
```

> You can find the source code for this article at https://github.com/sivaprasadreddy/spring-boot-microservices-series

# MicroServices - Part 4 : Spring Cloud Circuit Breaker using Netflix Hystrix


In the microservices world, to fulfill a client request one microservice may need to talk to other microservices. 
We should minimize this kind of direct dependencies on other microservices but in some cases it is unavoidable. 
If a microservice is down or not functioning properly then the issue may cascade up to the upstream services. 
Netflix created Hystrix library implementing [Circuit Breaker](https://martinfowler.com/bliki/CircuitBreaker.html) pattern to address these kinds of issues. 


<!--more-->


We can use **Spring Cloud Netflix Hystrix Circuit Breaker** to protect microservices from cascading failures.

**MicroServices using Spring Boot & Spring Cloud**

* [Part 1 : MicroServices : Spring Boot & Spring Cloud Overview]({{< relref "2018-03-02-microservices-using-springboot-spring-cloud-part-1-overview.md" >}})
* [Part 2 : MicroServices : Configuration Management with Spring Cloud Config and Vault]({{< relref "2018-03-05-microservices-part-2-configuration-management-spring-cloud-config-vault.md" >}})
* [Part 3 : MicroServices : Spring Cloud Service Registry and Discovery]({{< relref "2018-03-08-microservices-springcloud-eureka.md" >}})
* [Part 4 : MicroServices : Spring Cloud Circuit Breaker using Netflix Hystrix]({{< relref "2018-03-12-spring-cloud-netflix-circuit-breaker.md" >}})
* [Part 5 : MicroServices : Spring Cloud Zuul Proxy as API Gateway]({{< relref "2018-03-15-microservices-part-5-spring-cloud-zuul-proxy-as-api-gateway.md" >}})
* [Part 6 : MicroServices : Distributed Tracing with Spring Cloud Sleuth and Zipkin]({{< relref "2018-03-20-microservices-part-6-distributed-tracing-with-spring-cloud-sleuth-and-zipkin.md" >}})

**In this post we are going to learn:**

* Implementing Circuit Breaker pattern using @HystrixCommand
* How to propagate ThreadLocal variables
* Monitoring Circuit Breakers using Hystrix Dashboard

# Implementing Netflix Hystrix Circuit Breaker pattern
From **catalog-service** we are invoking REST endpoint on **inventory-service** to get the inventory level of a product. 
What if inventory-service is down? What if inventory-service is taking too long to respond thereby slowing down all the services depending on it? 
We would like to have some **timeouts** and implement some **fallback mechanism**.

Add **Hystrix** starter to catalog-service.

```xml
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
</dependency>
```

To enable Circuit Breaker add **@EnableCircuitBreaker** annotation on catalog-service entry-point class.

```java
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.circuitbreaker.EnableCircuitBreaker;
import org.springframework.cloud.client.loadbalancer.LoadBalanced;
import org.springframework.context.annotation.Bean;
import org.springframework.web.client.RestTemplate;
 
@EnableCircuitBreaker
@SpringBootApplication
public class CatalogServiceApplication {
 
    @Bean
    @LoadBalanced
    public RestTemplate restTemplate() {
        return new RestTemplate();
    }
 
    public static void main(String[] args) {
        SpringApplication.run(CatalogServiceApplication.class, args);
    }
}
```

Now we can use **@HystrixCommand** annotation on any method we want to apply timeout and fallback method.

Let us create **InventoryServiceClient.java** which will invoke inventory-service REST endpoint and 
apply **@HystrixCommand** with a fallback implementation.

```java
import com.netflix.hystrix.contrib.javanica.annotation.HystrixCommand;
import com.sivalabs.catalogservice.web.models.ProductInventoryResponse;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Service;
import org.springframework.web.client.RestTemplate;
 
import java.util.Optional;
 
@Service
@Slf4j
public class InventoryServiceClient {
    private final RestTemplate restTemplate;
 
    @Autowired
    public InventoryServiceClient(RestTemplate restTemplate) {
        this.restTemplate = restTemplate;
    }
 
    @HystrixCommand(fallbackMethod = "getDefaultProductInventoryByCode")
    public Optional<ProductInventoryResponse> getProductInventoryByCode(String productCode)
    {
        ResponseEntity<ProductInventoryResponse> itemResponseEntity =
                restTemplate.getForEntity("http://inventory-service/api/inventory/{code}",
                        ProductInventoryResponse.class,
                        productCode);
        if (itemResponseEntity.getStatusCode() == HttpStatus.OK) {
            return Optional.ofNullable(itemResponseEntity.getBody());
        } else {
            log.error("Unable to get inventory level for product_code: " + productCode + ", StatusCode: " + itemResponseEntity.getStatusCode());
            return Optional.empty();
        }
    }
 
    @SuppressWarnings("unused")
    Optional<ProductInventoryResponse> getDefaultProductInventoryByCode(String productCode) {
        log.info("Returning default ProductInventoryByCode for productCode: "+productCode);
        ProductInventoryResponse response = new ProductInventoryResponse();
        response.setProductCode(productCode);
        response.setAvailableQuantity(50);
        return Optional.ofNullable(response);
    }
}
```

```java
import lombok.Data;
 
@Data
public class ProductInventoryResponse {
    private String productCode;
    private int availableQuantity;
}
```

We have annotated the method from where we are making a REST call with **@HystrixCommand(fallbackMethod = "getDefaultProductInventoryByCode")** 
so that if it doesn't receive the response within the certain time limit the call gets timed out and invoke the configured fallback method. 
The fallback method should be defined in the same class and should have the same signature. 
In the fallback method **getDefaultProductInventoryByCode()** we are setting the **availableQuantity** to 50, obviously, 
this behavior depends on what business wants.

We can customize the **@HystrixCommand** default behavior by configuring properties using **@HystrixProperty** annotations.

```java
@HystrixCommand(fallbackMethod = "getDefaultProductInventoryByCode",
    commandProperties = {
       @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "3000"),
       @HystrixProperty(name = "circuitBreaker.errorThresholdPercentage", value="60")
    }
)
public Optional<ProductInventoryResponse> getProductInventoryByCode(String productCode)
{
    ....
}
```

Instead of configuring these parameter values in the code we can configure them in **bootstrap.properties/yml** files as follows.

```properties
hystrix.command.getProductInventoryByCode.execution.isolation.thread.timeoutInMilliseconds=2000
hystrix.command.getProductInventoryByCode.circuitBreaker.errorThresholdPercentage=60
```

Note that we are using the method name as the **commandKey** which is the default behavior. We can customize the **commandKey** name as follows:

```java
@HystrixCommand(commandKey = "inventory-by-productcode", fallbackMethod = "getDefaultProductInventoryByCode")
public Optional<ProductInventoryResponse> getProductInventoryByCode(String productCode)
{
    ...
}
```

```properties
hystrix.command.inventory-by-productcode.execution.isolation.thread.timeoutInMilliseconds=2000
hystrix.command.inventory-by-productcode.circuitBreaker.errorThresholdPercentage=60
```

You can find all the configuration options available here https://github.com/Netflix/Hystrix/wiki/Configuration.

# How to propagate ThreadLocal variables
By default, the methods with **@HystrixCommand** will be executed on a different thread because the default **execution.isolation.strategy** 
is **ExecutionIsolationStrategy.THREAD**. So, the **ThreadLocal** variables we set before invoking **@HystrixCommand** methods won't be 
available within **@HystrixCommand** methods.

One option to make the ThreadLocal variables available is using **execution.isolation.strategy=SEMAPHORE**.

```java
@HystrixCommand(fallbackMethod = "getDefaultProductInventoryByCode",
    commandProperties = {
        @HystrixProperty(name="execution.isolation.strategy", value="SEMAPHORE")
    }
)
public Optional<ProductInventoryResponse> getProductInventoryByCode(String productCode)
{
    ...
}
```

If you set the property **execution.isolation.strategy** to **SEMAPHORE** then Hystrix will use semaphores instead of threads to limit 
the number of concurrent parent threads that invoke the command. 
You can read more about How Isolation Works here https://github.com/Netflix/Hystrix/wiki/How-it-Works#isolation.

Another option to make ThreadLocal variables available within Hystrix command methods is implementing our own **HystrixConcurrencyStrategy**.

Suppose you want to propagate some **CorrelationId** set as a ThreadLocal variable.

```java
public class MyThreadLocalsHolder {
    private static final ThreadLocal<String> CORRELATION_ID = new ThreadLocal();
 
    public static void setCorrelationId(String correlationId) {
        CORRELATION_ID.set(correlationId);
    }
 
    public static String getCorrelationId() {
        return CORRELATION_ID.get();
    }
}
```

Let us implement our own **HystrixConcurrencyStrategy**.

```java
@Component
@Slf4j
public class ContextCopyHystrixConcurrencyStrategy extends HystrixConcurrencyStrategy {
 
    public ContextCopyHystrixConcurrencyStrategy() {
        HystrixPlugins.getInstance().registerConcurrencyStrategy(this);
    }
 
    @Override
    public <T> Callable<T> wrapCallable(Callable<T> callable) {
        return new MyCallable(callable, MyThreadLocalsHolder.getCorrelationId());
    }
 
    public static class MyCallable<T> implements Callable<T> {
 
        private final Callable<T> actual;
        private final String correlationId;
 
        public MyCallable(Callable<T> callable, String correlationId) {
            this.actual = callable;
            this.correlationId = correlationId;
        }
 
        @Override
        public T call() throws Exception {
            MyThreadLocalsHolder.setCorrelationId(correlationId);
            try {
                return actual.call();
            } finally {
                MyThreadLocalsHolder.setCorrelationId(null);
            }
        }
    }
}
```

Now you can set the **CorrelationId** before calling Hystrix command and access CorrelationId within Hystrix command.

**ProductService.java**

```java
public Optional<Product> findProductByCode(String code) 
{
    ....
    String correlationId = UUID.randomUUID().toString();
    MyThreadLocalsHolder.setCorrelationId(correlationId);
    log.info("Before CorrelationID: "+ MyThreadLocalsHolder.getCorrelationId());
    Optional<ProductInventoryResponse> responseEntity = inventoryServiceClient.getProductInventoryByCode(code);
    ...
    log.info("After CorrelationID: "+ MyThreadLocalsHolder.getCorrelationId());
    ....
     
}
```

**InventoryServiceClient.java**

```java
@HystrixCommand(fallbackMethod = "getDefaultProductInventoryByCode")
public Optional<ProductInventoryResponse> getProductInventoryByCode(String productCode)
{
    ...
    log.info("CorrelationID: "+ MyThreadLocalsHolder.getCorrelationId());
}
```

This is just one example of how to propagate the data to the Hystrix command. 
Similarly we can pass any data available in current HTTP Request, let's say by using Spring components like **RequestContextHolder** etc.

> Jakub Narloch wrote a nice article on how to propagate Request Context and even created a Spring Boot starter too. Please look at his blog https://jmnarloch.wordpress.com/2016/07/06/spring-boot-hystrix-and-threadlocals/ and GitHub Repo https://github.com/jmnarloch/hystrix-context-spring-boot-starter.

# Monitoring Circuit Breakers using Hystrix Dashboard
Once we add **Hystrix** starter to catalog-service we can get the circuits status as a stream of events using 
Actuator endpoint http://localhost:8181/actuator/hystrix.stream, assuming catalog-service is running on 8181 port.

Spring Cloud also provides a nice dashboard to monitor the status of Hystrix commands.
Create a Spring Boot application with **Hystrix Dashboard** starter and annotate the main entry-point class with **@EnableHystrixDashboard**.

```xml
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix-hystrix-dashboard</artifactId>
</dependency>
```

Let us say we are running Hystrix Dashboard on 8788 port, then go to http://localhost:8788/hystrix to view the dashboard.

![Hystrix Dashboard](/images/hystrix-dashboard.webp "Hystrix Dashboard")

Now in Hystrix Dashboard home page enter **http://localhost:8181/actuator/hystrix.stream** as stream URL and give Catalog Service as Title and click on Monitor Stream button.

Now invoke the catalog-service REST endpoint which internally invokes inventory-service REST endpoint and you can see the Circuit status along with how many calls succeed and how many failures occurred etc.


![Netflix Hystrix Circuit Breaker](/images/catalog-service-hystrix.webp "Netflix Hystrix Circuit Breaker")

Instead of having a separate dashboard for every service we can use Turbine to provide a unified view of all services in a single dashboard. 
For more details see http://cloud.spring.io/spring-cloud-static/Finchley.M7/single/spring-cloud.html#_turbine.

> You can find the source code for this article at https://github.com/sivaprasadreddy/spring-boot-microservices-series


# MicroServices - Part 3 : Spring Cloud Service Registry and Discovery




In the microservices world, **Service Registry and Discovery** plays an important role because we most likely run multiple instances of services, and
we need a mechanism to call other services without hardcoding their hostnames or port numbers.
In addition to that, in Cloud environments, service instances may come up and go down at any time.
So we need some automatic service registration and discovery mechanism. **Spring Cloud** provides **Service Registry and Discovery** features, as usual,
with multiple options. We can use **Netflix Eureka** or **Consul** for Service Registry and Discovery.

<!--more-->

In this post, we will learn how to use Spring Cloud Netflix Eureka for Service Registry and Discovery.

**MicroServices using Spring Boot & Spring Cloud**

*   [Part 1: MicroServices: Spring Boot & Spring Cloud Overview]({{< relref "2018-03-02-microservices-using-springboot-spring-cloud-part-1-overview.md" >}})
*   [Part 2: MicroServices: Configuration Management with Spring Cloud Config and Vault]({{< relref "2018-03-05-microservices-part-2-configuration-management-spring-cloud-config-vault.md" >}})
*   [Part 3: MicroServices: Spring Cloud Service Registry and Discovery]({{< relref "2018-03-08-microservices-springcloud-eureka.md" >}})
*   [Part 4: MicroServices: Spring Cloud Circuit Breaker using Netflix Hystrix]({{< relref "2018-03-12-spring-cloud-netflix-circuit-breaker.md" >}})
*   [Part 5: MicroServices: Spring Cloud Zuul Proxy as API Gateway]({{< relref "2018-03-15-microservices-part-5-spring-cloud-zuul-proxy-as-api-gateway.md" >}})
*   [Part 6: MicroServices: Distributed Tracing with Spring Cloud Sleuth and Zipkin]({{< relref "2018-03-20-microservices-part-6-distributed-tracing-with-spring-cloud-sleuth-and-zipkin.md" >}})

In my previous post, [Part 2: MicroServices: Configuration Management with Spring Cloud Config and Vault]({{< relref "2018-03-05-microservices-part-2-configuration-management-spring-cloud-config-vault.md" >}}), we learned how to store configuration parameters externally in a configuration server and how to securely store secrets in Vault.

**In this post, we are going to learn:**

*   What is Service Registry and Discovery?
*   A Spring Cloud Netflix Eureka-based Service Registry
*   Registering microservices as Eureka Clients
*   Discovering other services using a Eureka Client

# What is Service Registry and Discovery?
Suppose we have 2 microservices, **catalog-service** and **inventory-service**, and we are running 2 instances of `inventory-service` at http://localhost:8181/ and http://localhost:8282/. Now let's say we want to invoke some `inventory-service` REST endpoint from `catalog-service`. Which URL should we hit? Generally, in these scenarios, we use a load balancer, configuring these 2 URLs to be delegated to, and we will invoke the REST endpoint on the load balancer's URL. Fine.

But what if you want to spin up new instances dynamically based on load? Even if you are going to run only a few server nodes, manually updating the server node details in the load balancer configuration is error-prone and tedious. This is why we need an automatic Service Registration mechanism and to be able to invoke a service using some logical service ID instead of using a specific IP Address and port number.

We can use a Netflix Eureka Server to create a Service Registry and make our microservices Eureka Clients so that as soon as we start a microservice, it will get registered with the Eureka Server automatically with a logical Service ID. Then, the other microservices, which are also Eureka Clients, can use the Service ID to invoke REST endpoints.

Spring Cloud makes it very easy to create a Service Registry and discover other services using a Load-Balanced RestTemplate.

# Spring Cloud Netflix Eureka-based Service Registry
Let us create a Service Registry using Netflix Eureka, which is nothing but a Spring Boot application with the **Eureka Server** starter.

```xml
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix-eureka-server</artifactId>
</dependency>
```

We need to add the **@EnableEurekaServer** annotation to make our Spring Boot application a Eureka Server-based Service Registry.

```java
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.netflix.eureka.server.EnableEurekaServer;
 
@EnableEurekaServer
@SpringBootApplication
public class ServiceRegistryApplication {
 
    public static void main(String[] args) {
        SpringApplication.run(ServiceRegistryApplication.class, args);
    }
}
```

By default, each Eureka Server is also a Eureka client and needs at least one service URL to locate a peer. As we are going to have a single Eureka Server node (Standalone Mode), we are going to disable this client-side behavior by configuring the following properties in the `application.properties` file.

**application.properties**

```properties
spring.application.name=service-registry
server.port=8761
eureka.instance.hostname=localhost
eureka.client.registerWithEureka=false
eureka.client.fetchRegistry=false
eureka.client.serviceUrl.defaultZone=http://${eureka.instance.hostname}:${server.port}/eureka/
```

Netflix Eureka Service provides a UI where we can see all the details about registered services.

Now run **ServiceRegistryApplication** and access http://localhost:8761, which will display a UI similar to the screenshot below.

![Eureka Dashboard](/images/EurekaDashboard.webp "Eureka Dashboard")

# Registering microservices as Eureka Clients
In [Part 2: MicroServices: Configuration Management with Spring Cloud Config and Vault]({{< relref "2018-03-05-microservices-part-2-configuration-management-spring-cloud-config-vault.md" >}}),
we created a `catalog-service`. Let us make this service a Eureka Client and register it with the Eureka Server.

Add the **Eureka Discovery** starter to **catalog-service**, which will add the following dependency.

```xml
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>
```

With **spring-cloud-starter-netflix-eureka-client** on the classpath, we just need to configure the **eureka.client.service-url.defaultZone** property in **application.properties** to automatically register with the Eureka Server.

```properties
eureka.client.service-url.defaultZone=http://localhost:8761/eureka/
```

When a service is registered with the Eureka Server, it keeps sending heartbeats for a certain interval. If the Eureka server doesn't receive a heartbeat from any service instance, it will assume the service instance is down and take it out of the pool.

With this configuration in place, start `catalog-service` and visit http://localhost:8761.
You should see that `catalog-service` is registered with the **SERVICE ID** as **CATALOG-SERVICE**.
You can also notice the status as **UP(1)**, which means the services are up and running, and one instance of **catalog-service** is running.

Let us start another instance of `catalog-service` on a different port using the following command:

```shell
java -jar -Dserver.port=9797 target/catalog-service-0.0.1-SNAPSHOT-exec.jar
```

Now, if you go to http://localhost:8761, you will notice that 2 instances of `catalog-service` got registered, and you can see their hostname and port details as well.

# Discovering other services using a Eureka Client
In the previous section, we learned how to register a service as a Eureka client, and we also tried registering multiple instances of the same service.

Now we will create another microservice, **inventory-service**, which exposes a REST endpoint, **http://localhost:8282/api/inventory/{productCode}**, which will give the currently available quantity as a response.

```js
{
    productCode: "P001",
    availableQuantity: 250
}
```

Create the **inventory-service** Spring Boot application with **Web, JPA, H2/MySQL, Actuator, Config Client, and Eureka Discovery** starters.

Create a REST Controller to return Inventory details for a given product code.

```java
@RestController
@Slf4j
public class InventoryController {
    private final InventoryItemRepository inventoryItemRepository;
 
    @Autowired
    public InventoryController(InventoryItemRepository inventoryItemRepository) {
        this.inventoryItemRepository = inventoryItemRepository;
    }
 
    @GetMapping("/api/inventory/{productCode}")
    public ResponseEntity<InventoryItem> findInventoryByProductCode(@PathVariable("productCode") String productCode) {
        log.info("Finding inventory for product code :"+productCode);
        Optional<InventoryItem> inventoryItem = inventoryItemRepository.findByProductCode(productCode);
        if(inventoryItem.isPresent()) {
            return new ResponseEntity(inventoryItem, HttpStatus.OK);
        } else {
            return new ResponseEntity(HttpStatus.NOT_FOUND);
        }
    }
}
```

> Please look at the GitHub Repository for the `InventoryItem`, `InventoryItemRepository`, etc. code.

Register **inventory-service** with the Eureka server by configuring the Eureka `serviceUrl` in **src/main/resources/bootstrap.properties**.

```properties
spring.application.name=inventory-service
server.port=8282
eureka.client.service-url.defaultZone=http://localhost:8761/eureka/
```

Now build `inventory-service` and start 2 instances of it by running the following commands:

```shell
java -jar -Dserver.port=9898 target/inventory-service-0.0.1-SNAPSHOT-exec.jar
 
java -jar -Dserver.port=9999 target/inventory-service-0.0.1-SNAPSHOT-exec.jar
```

Now you can visit the Eureka Dashboard at http://localhost:8761/ and see 2 instances of `inventory-service` registered.

Suppose we want to invoke the `inventory-service` REST endpoint from `catalog-service`.
We can use **RestTemplate** to invoke the REST endpoint, but there are 2 instances running.

We can register **RestTemplate** as a Spring bean with the **@LoadBalanced** annotation.
The `RestTemplate` with the **@LoadBalanced** annotation will internally use the **Ribbon LoadBalancer** to resolve the **ServiceID**
and invoke the REST endpoint using one of the available servers.

```java
@SpringBootApplication
public class CatalogServiceApplication {
 
    @Bean
    @LoadBalanced
    public RestTemplate restTemplate() {
        return new RestTemplate();
    }
 
    public static void main(String[] args) {
        SpringApplication.run(CatalogServiceApplication.class, args);
    }
}
```

Now we can use `RestTemplate` to invoke the `inventory-service` endpoint at **http://inventory-service/api/inventory/{productCode}**.

```java
@Service
@Transactional
@Slf4j
public class ProductService {
    private final ProductRepository productRepository;
    private final RestTemplate restTemplate;
     
    @Autowired
    public ProductService(ProductRepository productRepository, RestTemplate restTemplate) {
        this.productRepository = productRepository;
        this.restTemplate = restTemplate;
    }
 
    public Optional<Product> findProductByCode(String code) {
        Optional<Product> productOptional = productRepository.findByCode(code);
        if(productOptional.isPresent()) {
            log.info("Fetching inventory level for product_code: "+code);
            ResponseEntity<ProductInventoryResponse> itemResponseEntity =
                    restTemplate.getForEntity("http://inventory-service/api/inventory/{code}",
                                                ProductInventoryResponse.class,
                                                code);
            if(itemResponseEntity.getStatusCode() == HttpStatus.OK) {
                Integer quantity = itemResponseEntity.getBody().getAvailableQuantity();
                log.info("Available quantity: "+quantity);
                productOptional.get().setInStock(quantity> 0);
            } else {
                log.error("Unable to get inventory level for product_code: "+code +
                ", StatusCode: "+itemResponseEntity.getStatusCode());
            }
        }
        return productOptional;
    }
}
```

```java
@Data
public class ProductInventoryResponse {
    private String productCode;
    private int availableQuantity;
}
```

Note that we have used http://inventory-service/api/inventory/{code} instead of http://localhost:9898/api/inventory/{code} or http://localhost:9999/api/inventory/{code} directly.

With this kind of automatic Service Registration and Discovery mechanism, we no longer need to worry about how many instances are running, what their hostnames and ports are, etc.

> You can find the source code for this article at https://github.com/sivaprasadreddy/spring-boot-microservices-series

# Summary
In this post, we learned how to use Spring Cloud **Netflix Eureka** for **Service Registry and Discovery**.
In the next post, we will look at implementing the **Circuit Breaker pattern using Netflix Hystrix**.

# References:
*   [Spring Cloud Netflix](http://cloud.spring.io/spring-cloud-static/Finchley.M7/single/spring-cloud.html#_spring_cloud_netflix)

# MicroServices - Part 2 : Configuration Management with Spring Cloud Config and Vault




In [MicroServices using Spring Boot & Spring Cloud – Part 1: Overview](http://sivalabs.in/microservices-using-springboot-spring-cloud-part-1-overview/),
we took a brief look at what microservices are and how we can use **Spring Boot** and **Spring Cloud** to build them.

In this post, we are going to learn:

*   What is the need for Spring Cloud Config and Vault?
*   How to create our first microservice: catalog-service
*   How to create a Spring Cloud Config Server
*   How to use Vault for storing sensitive data

<!--more-->

**MicroServices using Spring Boot & Spring Cloud**

*   [Part 1: MicroServices: Spring Boot & Spring Cloud Overview]({{< relref "2018-03-02-microservices-using-springboot-spring-cloud-part-1-overview.md" >}})
*   [Part 2: MicroServices: Configuration Management with Spring Cloud Config and Vault]({{< relref "2018-03-05-microservices-part-2-configuration-management-spring-cloud-config-vault.md" >}})
*   [Part 3: MicroServices: Spring Cloud Service Registry and Discovery]({{< relref "2018-03-08-microservices-springcloud-eureka.md" >}})
*   [Part 4: MicroServices: Spring Cloud Circuit Breaker using Netflix Hystrix]({{< relref "2018-03-12-spring-cloud-netflix-circuit-breaker.md" >}})
*   [Part 5: MicroServices: Spring Cloud Zuul Proxy as API Gateway]({{< relref "2018-03-15-microservices-part-5-spring-cloud-zuul-proxy-as-api-gateway.md" >}})
*   [Part 6: MicroServices: Distributed Tracing with Spring Cloud Sleuth and Zipkin]({{< relref "2018-03-20-microservices-part-6-distributed-tracing-with-spring-cloud-sleuth-and-zipkin.md" >}})

# What is the need for Spring Cloud Config and Vault?
Spring Boot already provides a lot of options to [externalize configuration properties](https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#boot-features-external-config).
However, once the application is started, you can't change those property values at runtime.
You need to update the properties and restart the application for those changes to take effect.

In the microservices world, there could be a large number of microservices, and multiple instances of those microservices are running.
Updating configuration properties and restarting all those instances manually, or even with automated scripts, may not be feasible.
Spring Cloud Config addresses this problem.

We can create a Spring Cloud Config Server that provides the configuration values for all of our microservices.
We can use **git**, **svn**, a **database**, or **Consul** as a backend to store the configuration parameters.
Then we can configure the location of the Spring Cloud Config server in our microservice so that it will load all the properties
when we start the application. In addition to that, whenever we update the properties, we can invoke the **/refresh** REST endpoint
in our microservice so that it will reload the configuration changes without requiring a restart of the application.

In our applications, we also need to configure various sensitive data like database credentials, keys, tokens, etc.
Obviously, we don't want to store them in plain text. A better approach would be to store them in an encrypted format,
and Spring Cloud Config Server provides the ability to encrypt and decrypt the data.
Even better, we should use secure data storage tools like [Vault](https://www.vaultproject.io/).
Spring Cloud also provides integration with Vault so that we can store any sensitive configuration properties in Vault.

I have already written a couple of tutorials on Spring Cloud Config Server which you can refer to:

*   [Introduction to Spring Cloud Config Server](http://sivalabs.in/spring-cloud-tutorials-introduction-to-spring-cloud-config-server/)
*   [Auto-Refresh Config Changes using Spring Cloud Bus](http://sivalabs.in/spring-cloud-tutorials-auto-refresh-config-changes-using-spring-cloud-bus/)

# Create our first microservice: catalog-service
Let us start with our first microservice, i.e., `catalog-service`.
Create a Spring Boot app with **Web, JPA, MySQL, Actuator, DevTools, and Lombok** starters.
Nothing fancy here so far; it's a typical Spring Boot application.

> You can find the source code for this article at https://github.com/sivaprasadreddy/spring-boot-microservices-series

First, let's implement a REST endpoint to provide product data and later refactor it to use the Cloud Config Server.

We are going to use Docker and run MySQL as a Docker container.

**docker-compose.yml**

```yml
version: '3'
services:
  mysqldb:
     image: mysql:5.7
     container_name: mysqldb
     ports:
       - "3306:3306"
     environment:
       MYSQL_ROOT_PASSWORD: admin
       MYSQL_DATABASE: catalog
```

Configure datasource properties in **application.properties** as follows:

```properties
server.port=8181
logging.level.com.sivalabs=debug
 
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://localhost:3306/catalog?useSSL=false
spring.datasource.username=root
spring.datasource.password=admin
 
spring.datasource.initialization-mode=always
spring.jpa.hibernate.ddl-auto=update
spring.jpa.show-sql=true
 
//expose all the Actuator endpoints
management.endpoints.web.exposure.include=*
```

Create the JPA entity **Product.java** as follows:

```java
import lombok.Data;
import javax.persistence.*;
 
@Data
@Entity
@Table(name = "products")
public class Product {
    @Id @GeneratedValue(strategy = GenerationType.AUTO)
    private Long id;
    @Column(nullable = false, unique = true)
    private String code;
    @Column(nullable = false)
    private String name;
    private String description;
    private double price;
}
```

Create the Spring Data JPA repository **ProductRepository.java** as follows:

```java
import com.sivalabs.catalogservice.entities.Product;
import org.springframework.data.jpa.repository.JpaRepository;
import java.util.Optional;
 
public interface ProductRepository extends JpaRepository<Product, Long> {
    Optional<Product> findByCode(String code);
}
```

Create **ProductService**, which just delegates to **ProductRepository** for now.
We can directly inject the Repository into our web layer components (Controllers), but going forward, there could be business logic that I don't like to put in either the Controller or the Repository.

```java
import com.sivalabs.catalogservice.entities.Product;
import com.sivalabs.catalogservice.repositories.ProductRepository;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.util.List;
import java.util.Optional;
 
@Service
@Transactional
@Slf4j
public class ProductService {
    private final ProductRepository productRepository;
 
    @Autowired
    public ProductService(ProductRepository productRepository) {
        this.productRepository = productRepository;
    }
 
    public List<Product> findAllProducts() {
        return productRepository.findAll();
    }
 
    public Optional<Product> findProductByCode(String code) {
        return productRepository.findByCode(code);
    }
}
```

Finally, create our REST controller, **ProductController.java**:

```java
import com.sivalabs.catalogservice.entities.Product;
import com.sivalabs.catalogservice.exceptions.ProductNotFoundException;
import com.sivalabs.catalogservice.services.ProductService;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
 
import java.util.List;
 
@RestController
@RequestMapping("/api/products")
@Slf4j
public class ProductController {
 
    private final ProductService productService;
 
    @Autowired
    public ProductController(ProductService productService) {
        this.productService = productService;
    }
 
    @GetMapping("")
    public List<Product> allProducts() {
        return productService.findAllProducts();
    }
 
    @GetMapping("/{code}")
    public Product productByCode(@PathVariable String code) {
        return productService.findProductByCode(code)
                .orElseThrow(() -> new ProductNotFoundException("Product with code ["+code+"] doesn't exist"));
    }
}
```

Create **ProductNotFoundException** extending **RuntimeException** and annotate it with **@ResponseStatus(HttpStatus.NOT_FOUND)**.

```java
import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.ResponseStatus;
 
@ResponseStatus(HttpStatus.NOT_FOUND)
public class ProductNotFoundException extends RuntimeException {
    public ProductNotFoundException() {
    }
 
    public ProductNotFoundException(String message) {
        super(message);
    }
 
    public ProductNotFoundException(String message, Throwable cause) {
        super(message, cause);
    }
 
    public ProductNotFoundException(Throwable cause) {
        super(cause);
    }
}
```

Let's insert some sample products into our database.

`src/main/resources/data.sql`

```sql
DELETE FROM products;
 
insert into products(id, code, name, description, price) VALUES
(1, 'P001', 'Product 1', 'Product 1 description', 25),
(2, 'P002', 'Product 2', 'Product 2 description', 32),
(3, 'P003', 'Product 3', 'Product 3 description', 50)
;
```

Okay, now we can start our Spring Boot application and hit http://localhost:8181/api/products. You should be able to see the JSON response with product info.

# Create a Spring Cloud Config Server

We are going to create a Spring Cloud Config Server using a Git backend. A Spring Cloud Config Server is nothing but a Spring Boot project. Create a Spring Boot project with the Config Server starter.

Configure the location of the Git repository where we are going to store all our configuration files in the **application.properties** file.

```properties
spring.config.name=configserver
server.port=8888
 
spring.cloud.config.server.git.uri=https://github.com/sivaprasadreddy/microservices-config-repo
spring.cloud.config.server.git.clone-on-start=true
 
management.endpoints.web.exposure.include=*
```

Now, annotate the entry point class with **@EnableConfigServer**.

```java
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.config.server.EnableConfigServer;
 
@EnableConfigServer
@SpringBootApplication
public class ConfigServerApplication {
 
  public static void main(String[] args) {
    SpringApplication.run(ConfigServerApplication.class, args);
  }
}
```

That's it. This is all you need to do to create a Spring Cloud Config Server, and you just need to add application-specific config files to the Git repository.

> If you have mentally prepared to write a bunch of code to create a Spring Cloud Config Server, sorry to disappoint you. :-)

# Refactor catalog-service to use the Config Server

Our `catalog-service` will become a client for the Config Server. So, let us add the **Config Client** starter to **catalog-service**, which will add the following dependency.

```xml
<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-config</artifactId>
</dependency>
```

Make sure you also add the **spring-cloud-dependencies** BOM in the `<properties>` section.

While using Spring Cloud Config Server, the properties loading process happens in multiple stages: first loading **bootstrap.properties/yml** and then from the config server.

So, let's rename **application.properties** to **bootstrap.properties** and update it to have the following properties.

```properties
spring.application.name=catalog-service
server.port=8181
management.endpoints.web.exposure.include=*
spring.cloud.config.uri=http://localhost:8888
```

Here, we have configured the location of our Config Server and gave the name **catalog-service** to our application using **spring.application.name**.

Now we need to add all the properties of our **catalog-service** to **catalog-service.properties** and commit/push it to our Git repo, `microservices-config-repo`.

**microservices-config-repo/catalog-service.properties**

```properties
logging.level.com.sivalabs=debug
 
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://localhost:3306/catalog?useSSL=false
spring.datasource.username=root
spring.datasource.password=admin
 
spring.datasource.initialization-mode=always
spring.jpa.hibernate.ddl-auto=update
spring.jpa.show-sql=true
```

You can also add separate config files for different profiles, like **catalog-service-qa.properties, catalog-service-prod.properties**, etc.

Now, first start the Config Server application and then the `catalog-service` application.
This should work fine. You can check the console logs to see that `catalog-service` is fetching the properties from the
config server at http://localhost:8888/ during startup.

Now we are getting a little close to our goal, but we are still storing the credentials in plain text.
Let's move the sensitive config properties to Vault.

# Using Vault for storing sensitive data

Vault is a tool for securely storing and accessing secrets. You can read more about Vault here: https://www.vaultproject.io/intro/index.html. Vault comes as a single binary which you can download from https://www.vaultproject.io/downloads.html.

Now start Vault in dev mode using the following command:

**$ vault server -dev**

In the console, you can see information about how to use Vault and the Root token.
Open a new terminal window and set the `VAULT_ADDR` environment variable.

**$ export VAULT_ADDR='http://127.0.0.1:8200′**

> NOTE: The Vault dev mode is only for development purposes and is not meant for production usage.

We can write secrets to Vault using **vault write secret/somename key1=value1 key2=value2**.
We can also put all our secrets in a JSON file and write from the file as well.
Let us create a JSON file with MySQL database credentials and write to Vault.

**catalog-service-credentials.json**

```json
{ 
    "spring.datasource.username": "root", 
    "spring.datasource.password": "admin"
}
```

**$ vault write secret/catalog-service @catalog-service-credentials.json**

You can verify the values by running `vault read secret/catalog-service`.

> We can automate this whole process of setting up Vault and initializing it with secrets using Docker. Please look at the source repository on GitHub to know how to do it—well, one way of doing it.

Now that Vault is configured and initialized with secrets, let us refactor **catalog-service** to use Vault.

Add the Vault Configuration starter to **catalog-service**, which will add the following dependency:

```xml
<dependency>
 <groupId>org.springframework.cloud</groupId>
 <artifactId>spring-cloud-starter-vault-config</artifactId>
</dependency>
```

Remove the following credentials from **microservices-config-repo/catalog-service.properties** and commit it.

```properties
spring.datasource.username=root
spring.datasource.password=admin
```

Add Vault configuration properties in **bootstrap.properties**.

```properties
spring.cloud.vault.host=localhost
spring.cloud.vault.port=8200
spring.cloud.vault.scheme=http
spring.cloud.vault.authentication=token
spring.cloud.vault.token=934f9eae-31ff-a8ef-e1ca-4bea9e07aa09
```

We have configured the Vault properties, using token-based authentication, and configured the Root Token that is printed in the console log
when you started the Vault server.

We are all set. We moved the service properties into an external config server and sensitive data into Vault.

Now start the Config Server and **catalog-service**, and it should work just fine.

# Summary
In this post, we learned how to use Spring Cloud Config to externalize configuration properties and Vault to store secrets.
You can use Spring Cloud Bus to auto-refresh config changes as described in
[Spring Cloud Tutorials – Auto-Refresh Config Changes using Spring Cloud Bus](http://sivalabs.in/spring-cloud-tutorials-auto-refresh-config-changes-using-spring-cloud-bus/).

You can find the source code for this article at https://github.com/sivaprasadreddy/spring-boot-microservices-series.

In the next article, we will take a look at how to use **Netflix Eureka for Service Registry and Service Discovery**.

# References:
*   [Spring Cloud Reference Documentation](http://cloud.spring.io/spring-cloud-static/Finchley.M7/single/spring-cloud.html)
*   [Vault Getting Started](https://www.vaultproject.io/intro/getting-started/install.html)
*   [Spring Blog: Managing Secrets with Vault](https://spring.io/blog/2016/06/24/managing-secrets-with-vault)

# MicroServices using Spring Boot & Spring Cloud – Part 1 : Overview



Nowadays, "MicroServices" is the hot buzzword in software development, and many organizations prefer building their enterprise applications
using a MicroServices architecture. In the Java community, Spring Boot is the most widely used framework for building both monoliths and microservices.
I am planning to write a series of articles covering how to build microservices using [Spring Boot](https://projects.spring.io/spring-boot/)
and [Spring Cloud](https://projects.spring.io/spring-cloud/).

<!--more-->

In this article, we are going to learn about the following:

*   Monoliths
*   What are MicroServices?
*   Advantages of MicroServices
*   Challenges with MicroServices
*   Why are Spring Boot & Spring Cloud a good choice for MicroServices?
*   Introducing the application

# Monoliths
Traditionally, we have been building large enterprise applications in a modularized fashion (??!!??) but finally deploying them together as a single deployment unit (EAR or WAR). These are called Monolithic applications.

There are some issues with the monolithic architecture, such as:

*   Large codebases become a mess over time.
*   Multiple teams working on a single codebase becomes tedious.
*   It is not possible to scale up only certain parts of the application.
*   Technology updates/rewrites become complex and expensive tasks.

However, IMHO, it is relatively easy to deploy and monitor Monoliths compared to MicroServices.

# MicroServices
A MicroService is a service built around a specific business capability that can be independently deployed. So, to build large enterprise applications, we can identify the sub-domains of our main business domain and build each sub-domain as a MicroService using Domain-Driven Design (DDD) techniques. But in the end, we need to make all these microservices work together to serve the end user as if it is a single application.

> You can read more about MicroServices on this famous Martin Fowler blog: https://martinfowler.com/articles/microservices.html

## Advantages of MicroServices

*   Comprehending a smaller codebase is easy.
*   Can independently scale up highly used services.
*   Each team can focus on one (or a few) MicroService(s).
*   Technology updates/rewrites become simpler.

## Challenges with MicroServices
*   Getting correct sub-domain boundaries at the beginning is hard.
*   Need more skilled developers to handle distributed application complexities.
*   Managing MicroServices-based applications without a proper DevOps culture is next to impossible.
*   Local developer environment setup might become complex to test cross-service communications. Though using Docker/Kubernetes, this can be mitigated to some extent.

# Why are Spring Boot and Spring Cloud a good choice for MicroServices?

**Spring Boot** is the most popular and widely used Java framework for building MicroServices. These days, many organizations prefer to deploy their applications in a Cloud environment instead of taking on all the headaches of maintaining a datacenter themselves. But we need to take good care of the various aspects to make our applications Cloud-Native. Therein lies the beauty of Spring Cloud.

**Spring Cloud** is essentially an implementation of various design patterns to be followed while building Cloud-Native applications. Instead of reinventing the wheel, we can simply take advantage of various Spring Cloud modules and focus on our main business problem rather than worrying about infrastructural concerns.

The following are just a few of the Spring Cloud modules that can be used to address distributed application concerns:

**Spring Cloud Config Server:** To externalize the configuration of applications in a central config server with the ability to update the configuration values without needing to restart the applications. We can use Spring Cloud Config Server with Git, Consul, or ZooKeeper as a config repository.

**Service Registry and Discovery:** As there could be many services and we need the ability to scale up or down dynamically, we need a Service Registry and Discovery mechanism so that service-to-service communication does not depend on hard-coded hostnames and port numbers. Spring Cloud provides Netflix Eureka-based Service Registry and Discovery support with just minimal configuration. We can also use Consul or ZooKeeper for Service Registry and Discovery.

**Circuit Breaker:** In a microservices-based architecture, one service might depend on another service, and if one service goes down, then failures may cascade to other services as well. Spring Cloud provides a Netflix Hystrix-based Circuit Breaker to handle these kinds of issues.

**Spring Cloud Data Streams:** These days, we may need to work with huge volumes of data streams using Kafka, Spark, etc. Spring Cloud Data Streams provides higher-level abstractions to use those frameworks in an easier manner.

**Spring Cloud Security:** Some of the microservices need to be accessible to authenticated users only, and most likely we might want a Single Sign-On feature to propagate the authentication context across services. Spring Cloud Security provides authentication services using OAuth2.

**Distributed Tracing:** One of the pain points with microservices is the ability to debug issues. One simple end-user action might trigger a chain of microservice calls; there should be a mechanism to trace the related call chains. We can use Spring Cloud Sleuth with Zipkin to trace the cross-service invocations.

**Spring Cloud Contract:** There is a high chance that separate teams work on different microservices. There should be a mechanism for teams to agree upon API endpoint contracts so that each team can develop their APIs independently. Spring Cloud Contract helps to create such contracts and validate them by both the service provider and consumer.

These are just a few of the Spring Cloud features. To explore more, visit https://projects.spring.io/spring-cloud/.

> You may also like to read about [Why Spring Boot is so popular and how to learn Spring Boot effectively?](http://sivalabs.in/why-springboot-so-popular-how-to-learn-springboot/)

# Our Sample application
I strongly believe in learning by example. So, let us learn how to build MicroServices using Spring Boot and Spring Cloud with a sample application. I will deliberately keep the application's business logic very simple so that we can focus on understanding the Spring Boot and Spring Cloud features.

We are going to build a simple shopping cart application and assume we are going to start with the following microservices:

*   **catalog-service:** It provides a REST API to provide catalog information like products.
*   **inventory-service:** It provides a REST API to manage product inventory.
*   **cart-service:** It provides a REST API to hold the customer's cart details.
*   **order-service:** It provides a REST API to manage orders.
*   **customer-service:** It provides a REST API to manage customer information.
*   **shoppingcart-ui:** It is a customer-facing front-end web application.

We are going to build various services and REST endpoints as we go through various microservice concepts.

> Stay tuned for the next post, where we are going to create a catalog-service and use a spring-cloud-config server to have a centralized configuration for all our microservices.

# Why Spring Boot is so popular and how to learn Spring Boot effectively?



Spring Boot is the most popular and widely used Java framework.
This discussion of **"Why is Spring Boot so popular?"** occasionally comes up between me and
my friends/colleagues. Also, I get emails from various people asking, **"Spring is huge; how can I learn it quickly?"**.
In this post, I will try to answer these two questions.

<!--more-->

# Why is Spring Boot so popular?
There could be many reasons why Spring and Spring Boot are very popular, but in my opinion, the following are the key reasons:

## 1. Drastic increase in developer productivity
Spring Boot's powerful auto-configuration mechanism makes it very easy to get started with a Spring-based application. More importantly, Spring Boot offers a wide array of Starters, which is more than sufficient for many applications.

You can create a REST API backed by a database by simply creating a project, selecting Web, Spring Data JPA/Mongo, etc., H2/EmbeddedMongo, and Spring Data REST starters, and then create your domain entity and Repository. That's it; you have a fully functional REST API.

This lowers the entry barrier for newbies. For those who are familiar with how to configure all of these manually, have probably done it hundreds of times, and are tired of writing this boilerplate, Spring Boot is a gift.

## 2. Simplified Higher-Level Abstractions
One of the primary goals of Spring and Spring Boot is to make things easier. The Spring portfolio has its own powerful Web MVC framework and Spring Security framework, but the majority of its other projects are to provide higher-level abstractions to make using them easier.

For example, Spring Data JPA makes using JPA very easy by providing APIs to perform CRUD operations, sorting, and pagination without requiring you to implement all of them by yourself.

Spring AMQP or Spring for Kafka provides higher-level abstractions so that you can easily work with RabbitMQ or Kafka without writing low-level boilerplate code.

## 3. Microservices and Cloud-Native friendly
The microservice architecture is the latest hot trend now, and many organizations prefer a microservices architecture and want to deploy their applications in cloud environments like AWS, CloudFoundry, etc.

Usually, Spring Boot applications are built as self-contained deployment units (FAT JARs), and using its Profiles concept, we can deploy the same application in multiple environments without any code changes. In addition to that, Spring Cloud modules provide a great set of features required for building cloud-native microservices.

## 4. Addresses modern business needs
Modern application needs have changed and are continuously changing rapidly. We can't wait for a 3 to 4-year release cycle to get new features. We need frameworks with rapid release cycles to support these business needs.

Spring at its core is just a Dependency Injection (DI) container. But the power of Spring comes from its rich set of portfolio projects. You want to use NoSQL databases, you want a robust Security framework, you want to integrate with Social platforms, you want to work with Big Data frameworks, you want to use Streaming platforms like Kafka… you are all covered.

![Spring Ecosystem](/images/spring-ecosystem.webp "Spring Ecosystem")

## 5. Spring Developer Community
This is the best part of the Spring framework. You can find a million blogs with fantastic blog posts on the Spring framework. You can almost find a solution to every problem on StackOverflow. You can reach out to the Spring Developer team via Twitter easily.

# Challenges with Spring and Spring Boot
When I talk to some people (both juniors and experienced developers), one thing they keep saying is, "Spring is huge and complex." But I have the exact opposite opinion on it. After talking to them more about why they feel Spring is complex, I understood why they are feeling that way. It is the same reason why I feel learning JavaScript frameworks is complex.

> Every now and then, I get tempted to learn NodeJS and one shiny new JavaScript framework like ReactJS or VueJS, etc. First, I try to make a list of things to learn so that I have a benchmark to say I have a decent knowledge of NodeJS/ReactJS, etc. After spending 4 to 6 hours and knowing there are 36,794 ways to do anything in JavaScript, I get lost in this ocean of JavaScript tools and options. Then I feel JavaScript is complex.
>
> What I should have done is stop chasing all frameworks/tools and learn bare-bones JS or ES6 first, and then pick one framework, say ReactJS, and one build tool, like WebPack, and just learn how to use them together properly.

In addition to this feeling that Spring is huge and complex, I also see the following reasons why people say Spring Boot is complex:

1.  **People directly jump on to Spring Boot without having any prior Spring knowledge**, and everything looks magical. It works great as long as you go with the defaults, but the moment you need to customize it, it feels complex.
2.  **Too many options**. When you look for help on some Spring-related issue, you will always find 100 options to solve it, and it is confusing which one to go with.
3.  IMHO, **some Spring modules are complex**. Some time back, I had to dig deeper into the Spring Security source code, and I feel it is very complex because it has customization hooks for almost everything.

## How to learn Spring and Spring Boot effectively?
I will give you some tips to learn Spring and Spring Boot, basically what I have followed while learning them.

### 1. Learn Iteratively
I don't think one can sit and learn everything about a framework in one go. I follow an iterative approach.

**Iteration 1:** Follow some Quick Start Guide, create a project with all the dependencies set, and build a Todo application (Todo is the new HelloWorld, right? :-)). It gives you confidence when you successfully run a sample application.

**Iteration 2:** Read the official documentation.
It is very important to understand what problem this particular framework is trying to solve and in what scenarios this framework is suitable and where it is not a good choice. In this regard, the Spring framework documentation is awesome and very long too (800+ pages) :-).

**Iteration 3:** Build a moderately complex application.

**Iteration 4:** Dig deep into the framework's source code and see how you can customize the framework.

### 2. First, learn the Spring Core framework
Please don't jump onto Spring Boot if you don't have prior knowledge of Spring. I would strongly suggest first learning the Spring framework without Spring Boot so that you know what magic Spring Boot is doing behind the scenes.

Get familiar with Spring DI concepts, Scopes, Annotations, Life Cycle callbacks, and bean configuration styles (annotations, @Beans, etc.). Know how Spring uses the Template pattern (JdbcTemplate, JmsTemplate, etc.) to avoid writing boilerplate code.

### 3. Learn how Spring Boot AutoConfiguration works
The power of Spring Boot comes from its AutoConfiguration mechanism, and it is not magic. Learn Spring's `@Conditional` feature and explore some of Spring Boot's AutoConfiguration classes like `DataSourceAutoConfiguration`, `JpaAutoConfiguration`, etc. You can easily understand how Spring Boot is automatically creating beans based on some criteria.

> You can read my blog post on "How Spring Boot AutoConfiguration magic works?" here: https://sivalabs.in/how-springboot-autoconfiguration-magic/

### 4. Follow the community and read blogs
This is the best part of the Spring framework: a huge community and thousands of blogs writing about Spring.
One simple way to find these awesome Spring blogs is to follow the "This Week in Spring" series by **Josh Long** ([@starbuxman](https://twitter.com/starbuxman)), posted on the
[Spring Blog](https://spring.io/blog) every Tuesday.

### 5. Practice, practice, and practice
Nothing can replace "practice." No matter how much you read or how many Spring videos you watch on YouTube, the best way to learn is to use it.

These days, I hardly see Java projects not using Spring and Spring Boot. You can learn while working on a project, but I strongly suggest creating a pet project and implementing it using various Spring portfolio projects.

After all this, it still takes some time to learn. Take the time to master it. Be patient. Reach out to people if you get stuck. Read books and blogs. Practice, practice, and practice.

I hope this is helpful, and please share it if you like it. :-)
# Spring Boot Messaging with RabbitMQ


[RabbitMQ](https://www.rabbitmq.com/) is one of the most popular message broker solutions and provides client libraries to be used from various programming languages, including
Java, Scala, .NET, Go, Python, Ruby, PHP, etc. In this tutorial, we will learn how to use the RabbitMQ message broker to send and receive messages
from a Spring Boot application. We will also look at how to send messages as JSON payloads and how to deal with errors using
Dead Letter Queues (DLQ).

<!--more-->

First, install the RabbitMQ server on your local machine as documented here: https://www.rabbitmq.com/download.html, or
run it as a Docker image with the following **docker-compose.yml**.

```yml
version: '3'
services:
 
  rabbitmq:
    container_name: rabbitmq
    image: 'rabbitmq:management'
    ports:
      - "5672:5672"
      - "15672:15672"
```

Now you can start RabbitMQ using `docker-compose up` and launch the Administration UI at http://localhost:15672/.

If you are familiar with other messaging brokers like ActiveMQ, we usually use Queues and Topics to send messages in a one-to-one and
publish-subscribe model of communication. In RabbitMQ, we send messages to an Exchange, and depending on the Routing Key, that
message will be forwarded to a Queue(s).

You can read more about RabbitMQ concepts here: https://www.rabbitmq.com/tutorials/amqp-concepts.html.

> You can find the source code for this article at https://github.com/sivaprasadreddy/sivalabs-blog-samples-code/tree/master/springboot-rabbitmq-demo

## Spring Boot application with RabbitMQ
Now, let us create a Spring Boot application from http://start.spring.io/ by selecting the **Web, Thymeleaf, and RabbitMQ** starters.

**pom.xml**

```xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
    http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
 
    <groupId>com.sivalabs</groupId>
    <artifactId>springboot-rabbitmq-demo</artifactId>
    <version>1.0-SNAPSHOT</version>
 
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.0.0.RC1</version>
        <relativePath/>
    </parent>
 
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
    </properties>
 
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-amqp</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-thymeleaf</artifactId>
        </dependency>
    </dependencies>
     
</project>
```

Let us start with the RabbitMQ configuration. Create a `RabbitConfig` configuration class and define **Queue, Exchange,
and Binding** beans as follows:

```java
import org.springframework.amqp.core.*;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
 
@Configuration
public class RabbitConfig  
{
    public static final String QUEUE_ORDERS = "orders-queue";
    public static final String EXCHANGE_ORDERS = "orders-exchange";
 
    @Bean
    Queue ordersQueue() {
        return QueueBuilder.durable(QUEUE_ORDERS).build();
    }
 
    @Bean
    Queue deadLetterQueue() {
        return QueueBuilder.durable(QUEUE_DEAD_ORDERS).build();
    }
 
    @Bean
    Exchange ordersExchange() {
        return ExchangeBuilder.topicExchange(EXCHANGE_ORDERS).build();
    }
 
    @Bean
    Binding binding(Queue ordersQueue, TopicExchange ordersExchange) {
        return BindingBuilder.bind(ordersQueue).to(ordersExchange).with(QUEUE_ORDERS);
    }
}
```

Here we are declaring a Queue with the name **orders-queue** and an Exchange with the name **orders-exchange**.
We also defined the binding between **orders-queue** and **orders-exchange** so that any message sent to **orders-exchange**
with a **routing-key** of **orders-queue** will be sent to **orders-queue**.

We can configure the RabbitMQ server details in **application.properties** as follows:

```properties
spring.rabbitmq.host=localhost
spring.rabbitmq.port=5672
spring.rabbitmq.username=guest
spring.rabbitmq.password=guest
```

Let us create a Spring bean, **OrderMessageSender**, to send a message to **orders-exchange**.

Spring Boot auto-configures the infrastructure beans required to send/receive messages to/from the RabbitMQ broker.
We can simply autowire **RabbitTemplate** and send a message by invoking the **rabbitTemplate.convertAndSend("routingKey", Object)** method.

```java
public class Order implements Serializable {
    private String orderNumber;
    private String productId;
    private double amount;
 
    //setters & getters
}    
```

```java
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
 
@Service
public class OrderMessageSender {
    private final RabbitTemplate rabbitTemplate;
 
    @Autowired
    public OrderMessageSender(RabbitTemplate rabbitTemplate) {
        this.rabbitTemplate = rabbitTemplate;
    }
 
    public void sendOrder(Order order) {
        this.rabbitTemplate.convertAndSend(RabbitConfig.QUEUE_ORDERS, order);
    }
}
```

By default, Spring Boot uses **org.springframework.amqp.support.converter.SimpleMessageConverter** and serializes the object into a **byte[]**.

Now, with this configuration in place, we can send a message to the RabbitMQ **orders-queue** by invoking the **OrderMessageSender.sendOrder(Order)**
method.

After sending a message, you can view the message from the Administration UI application by logging in with guest/guest credentials.
You can click on the **Exchanges/Queues** tabs to see that **orders-exchange** and **orders-queue** got created.
You can also check the bindings for **orders-exchange**, which look like the following:

![RabbitMQ Exchange Bindings](/images/RabbitMQ-exchange.webp "RabbitMQ Exchange Bindings")

Now go to the Queues tab and click on `orders-queue`. Scroll down to the **Get Messages** section, and by clicking on the **Get Message(s)** button,
you can view the contents of the message.

Now, let's create a Listener for the `orders-queue` using **@RabbitListener**.

Create a Spring bean, **OrderMessageListener**, as follows:

```java
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.stereotype.Component;
 
@Component
public class OrderMessageListener {
 
    static final Logger logger = LoggerFactory.getLogger(OrderMessageListener.class);
 
    @RabbitListener(queues = RabbitConfig.QUEUE_ORDERS)
    public void processOrder(Order order) {
        logger.info("Order Received: "+order);
    }
}
```

That's it! By simply adding **@RabbitListener** and defining which queue to listen to, we can create a Listener.

Now, if you send a message to `orders-queue`, it should be consumed by the `OrderMessageListener.processOrder()` method, and
you should see the log statement "Order Received: ".

## Sending and Receiving Messages as JSON Payloads
As we have seen, the default serialization mechanism converts the message object into a **byte[]** using **SimpleMessageConverter**, and
on the receiving end, it will deserialize the **byte[]** into the Object type (in our case, `Order`) using **GenericMessageConverter**.

In order to change this behavior, we need to customize the Spring Boot RabbitMQ auto-configured beans.

### To send a message as JSON
One quick way to send a message as a JSON payload is by using **ObjectMapper** to convert the **Order** object into JSON and send it.

```java
@Autowired
private ObjectMapper objectMapper;
 
public void sendOrder(Order order) {
    try {
        String orderJson = objectMapper.writeValueAsString(order);
        Message message = MessageBuilder
                            .withBody(orderJson.getBytes())
                            .setContentType(MessageProperties.CONTENT_TYPE_JSON)
                            .build();
        this.rabbitTemplate.convertAndSend(RabbitConfig.QUEUE_ORDERS, message);
    } catch (JsonProcessingException e) {
        e.printStackTrace();
    }
}
```

But converting objects into JSON like this is a kind of boilerplate. Instead, we can follow the approach below.

We can configure an **org.springframework.amqp.support.converter.Jackson2JsonMessageConverter** bean to be used by **RabbitTemplate** so that
the message will be serialized as JSON instead of a `byte[]`.

```java
@Configuration
public class RabbitConfig 
{
    ...
    ...
 
    @Bean
    public RabbitTemplate rabbitTemplate(final ConnectionFactory connectionFactory) {
        final RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
        rabbitTemplate.setMessageConverter(producerJackson2MessageConverter());
        return rabbitTemplate;
    }
 
    @Bean
    public Jackson2JsonMessageConverter producerJackson2MessageConverter() {
        return new Jackson2JsonMessageConverter();
    }
}
```

Now, when you send a message, it will be converted into JSON and sent to the Queue.

### To receive a message as JSON
In order to treat the message payload as JSON, we should customize the RabbitMQ configuration by implementing **RabbitListenerConfigurer**.

```java
@Configuration
public class RabbitConfig implements RabbitListenerConfigurer {
    ...
    ...
    @Override
    public void configureRabbitListeners(RabbitListenerEndpointRegistrar registrar) {
        registrar.setMessageHandlerMethodFactory(messageHandlerMethodFactory());
    }
 
    @Bean
    MessageHandlerMethodFactory messageHandlerMethodFactory() {
        DefaultMessageHandlerMethodFactory messageHandlerMethodFactory = new DefaultMessageHandlerMethodFactory();
        messageHandlerMethodFactory.setMessageConverter(consumerJackson2MessageConverter());
        return messageHandlerMethodFactory;
    }
 
    @Bean
    public MappingJackson2MessageConverter consumerJackson2MessageConverter() {
        return new MappingJackson2MessageConverter();
    }
}
```

## Handling Errors and Invalid Messages using Dead-Letter Queues (DLQ)
We may want to send invalid messages to a separate queue so that we can inspect and reprocess them later.
We can use the DLQ concept to do this automatically, instead of manually writing the code to handle such scenarios.

We can declare the **dead-letter-exchange** and **dead-letter-routing-key** for a Queue while defining the Queue bean as follows:

```java
@Configuration
public class RabbitConfig implements RabbitListenerConfigurer {
 
    public static final String QUEUE_ORDERS = "orders-queue";
    public static final String EXCHANGE_ORDERS = "orders-exchange";
    public static final String QUEUE_DEAD_ORDERS = "dead-orders-queue";
     
    @Bean
    Queue ordersQueue() {
 
        return QueueBuilder.durable(QUEUE_ORDERS)
                .withArgument("x-dead-letter-exchange", "")
                .withArgument("x-dead-letter-routing-key", QUEUE_DEAD_ORDERS)
                .withArgument("x-message-ttl", 15000) //if message is not consumed in 15 seconds, send to DLQ
                .build();
    }
 
    @Bean
    Queue deadLetterQueue() {
        return QueueBuilder.durable(QUEUE_DEAD_ORDERS).build();
    }
 
    ...
    ...
}
```

Now, if you try to send an invalid JSON message to **orders-queue**, it will be sent to **dead-orders-queue**.

> You can find the source code for this article at https://github.com/sivaprasadreddy/sivalabs-blog-samples-code/tree/master/springboot-rabbitmq-demo
# Session Management using Spring Session with JDBC DataStore



In web applications, user session management is very crucial for managing user state. In this article, we are going to learn about the approaches we have been following to manage user sessions in a clustered environment and how we can use Spring Session to implement it in a much simpler and more scalable way.

<!--more-->

Typically, in production environments, we will have multiple server nodes with a load balancer in front of them, and all the client traffic will be coming through the load balancer to one of the server nodes. So we need some mechanism to make the user session data available to each client in a clustered environment.

Traditionally, we have been using the following techniques to manage sessions:

1.  Single Node Server
2.  Multi-Node Server with Load Balancer and Sticky Sessions
3.  Multi-Node Server with Load Balancer and Session Replication
4.  Multi-Node Server with Load Balancer and Session Data in a Persistent DataStore

Let us briefly look at these approaches.

## 1. Single Node Server
If your application is not a critical service to your business, there won't be too many users concurrently, and some downtime is acceptable, then we can have a Single Node Server deployment as shown below:

![Single Node Sessions](/images/SingleNode-Sessions.webp "Single Node Sessions")

In this model, for each browser client, a session object is created on the server (HttpSession in the case of Java), and a SESSION_ID will be set as a cookie on the browser to identify the session object. But this Single Server Node deployment is not acceptable for most applications because if the server goes down, the service will be down altogether.

## 2. Multi-Node Server with Sticky Sessions

In order to make our application highly available and cater to a larger number of users, we can have multiple server nodes behind a load balancer. In the Sticky Sessions approach, we configure our load balancer to route all the requests from the same client to the same node.

![Multi-Node Server with Sticky Sessions](/images/MultiNode-StickySessions.webp "Multi-Node Server with Sticky Sessions")

In this model, the user session will be created on any one of the server nodes, and all further requests from that client will be sent to that same node. But the problem with this approach is that if a server node goes down, then all the user sessions on that server are gone.

## 3. Multi-Node Server with Session Replication
In this model, the user session data will be replicated on all the server nodes so that any request can be routed to any server node. Even if one node goes down, the client request can be served by another node.

![Multi-Node Server with SessionReplication](/images/MultiNode-SessionReplication.webp "Multi-Node Server with SessionReplication")

But Session Replication requires better hardware support and involves some server-specific configuration.

## 4. Multi-Node Server with Session Data in a Persistent DataStore

In this model, the user session data will not be held in the server's memory; instead, it will be persisted into a data store and associated with a SESSION_ID.

![Multi-Node Server with Spring Session](/images/MultiNode-SpringSession.webp "Multi-Node Server with Spring Session")

This solution will be server-independent, but we may need to write custom code to transparently store the session data in a persistent datastore whenever a user adds some information to his/her session.
This is where Spring Session comes into the picture.

## Spring Session

[Spring Session](https://projects.spring.io/spring-session/) is an implementation of approach 4, which is storing session data in a persistent datastore.
Spring Session supports multiple datastores like RDBMS, Redis, Hazelcast, MongoDB, etc., to transparently save user session data.
As usual, using Spring Session with Spring Boot is as simple as adding a dependency and configuring a few properties.
Let us see how we can use Spring Session with a JDBC backend store in a Spring Boot application.

> You can find the source code for this article at https://github.com/sivaprasadreddy/spring-session-samples

### Step 1: Create a Spring Boot application

Create a Spring Boot application using the latest version (it is 2.0.0.RC1 as of writing) with **Web, Thymeleaf, JPA, H2, and Session** starters.

By default, the Session starter will add the **org.springframework.session:spring-session-core** dependency.
Let us change it to **spring-session-jdbc** as we are going to use a JDBC backend.

```xml
<dependency>
    <groupId>org.springframework.session</groupId>
    <artifactId>spring-session-jdbc</artifactId>
</dependency>
```

### Step 2: Configure Spring Session properties
We can configure the type of Spring Session backend data store using the **spring.session.store-type** property in `application.properties`.

```properties
spring.session.store-type=jdbc
```

As we are using the H2 In-Memory database, Spring Session creates the following tables required to store session data automatically
from the script **spring-session-jdbc-2.0.1.RELEASE.jar!/org/springframework/session/jdbc/schema-h2.sql**.

```sql
CREATE TABLE SPRING_SESSION (
  PRIMARY_ID CHAR(36) NOT NULL,
  SESSION_ID CHAR(36) NOT NULL,
  CREATION_TIME BIGINT NOT NULL,
  LAST_ACCESS_TIME BIGINT NOT NULL,
  MAX_INACTIVE_INTERVAL INT NOT NULL,
  EXPIRY_TIME BIGINT NOT NULL,
  PRINCIPAL_NAME VARCHAR(100),
  CONSTRAINT SPRING_SESSION_PK PRIMARY KEY (PRIMARY_ID)
);
 
CREATE UNIQUE INDEX SPRING_SESSION_IX1 ON SPRING_SESSION (SESSION_ID);
CREATE INDEX SPRING_SESSION_IX2 ON SPRING_SESSION (EXPIRY_TIME);
CREATE INDEX SPRING_SESSION_IX3 ON SPRING_SESSION (PRINCIPAL_NAME);
 
CREATE TABLE SPRING_SESSION_ATTRIBUTES (
  SESSION_PRIMARY_ID CHAR(36) NOT NULL,
  ATTRIBUTE_NAME VARCHAR(200) NOT NULL,
  ATTRIBUTE_BYTES LONGVARBINARY NOT NULL,
  CONSTRAINT SPRING_SESSION_ATTRIBUTES_PK PRIMARY KEY (SESSION_PRIMARY_ID, ATTRIBUTE_NAME),
  CONSTRAINT SPRING_SESSION_ATTRIBUTES_FK FOREIGN KEY (SESSION_PRIMARY_ID) REFERENCES SPRING_SESSION(PRIMARY_ID) ON DELETE CASCADE
);
 
CREATE INDEX SPRING_SESSION_ATTRIBUTES_IX1 ON SPRING_SESSION_ATTRIBUTES (SESSION_PRIMARY_ID);
```

But if we are going to use another RDBMS like MySQL, we can configure it as follows:

Add the MySQL Maven dependency.

```xml
<dependency>
    <groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
</dependency>
```

Configure datasource properties for MySQL:

```properties
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://localhost:3306/demo
spring.datasource.username=root
spring.datasource.password=admin
```

With this property, Spring Session will try to create tables using the script
**classpath:org/springframework/session/jdbc/schema-@@platform@@.sql**; so in our case, it will use **schema-mysql.sql**.

### Step 3: Add data to HttpSession
Now create a simple form in `src/main/resources/templates/index.html`.

```xml
<!DOCTYPE html>
<html lang="en" xmlns:th="http://www.thymeleaf.org">
<head>
    <meta charset="UTF-8">
    <title>Spring Session + JDBC Demo</title>
</head>
<body>
    <div>
        <form th:action="@{/messages}" method="post">
            <textarea name="msg" cols="40" rows="4"></textarea>
            <input type="submit" value="Save"/>
        </form>    
    </div>
     
    <div>         
        <h2>Messages</h2>             
        <ul th:each="m : ${messages}">                
          <li th:text="${m}">msg</li>        
        </ul>
    </div>
</body>
</html>
```

Let us implement a Controller to add messages to the HttpSession and display them.

```java
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestParam;
 
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpSession;
import java.util.*;
 
@Controller
public class MessagesController 
{
 
    @GetMapping("/")
    public String index(Model model, HttpSession session) {
        List<String> msgs = (List<String>) session.getAttribute("MY_MESSAGES");
        if(msgs == null) {
            msgs = new ArrayList<>();
        }
        model.addAttribute("messages", msgs);
        return "index";
    }
 
    @PostMapping("/messages")
    public String saveMessage(@RequestParam("msg") String msg, HttpServletRequest request) 
    {
        List<String> msgs = (List<String>) request.getSession().getAttribute("MY_MESSAGES");
        if(msgs == null) {
            msgs = new ArrayList<>();
            request.getSession().setAttribute("MY_MESSAGES", msgs);
        }
        msgs.add(msg);
        return "redirect:/";
    }
}
```

Now you can start the application, add some messages to the HttpSession, and you can see the rows in the
**SPRING_SESSION** and **SPRING_SESSION_ATTRIBUTES** tables.
By default, Spring Session converts the objects that we are trying to add to the HttpSession into a **ByteArray** and stores it in the table.

## Spring Session with Spring Security
Spring Session seamlessly integrates with Spring Security because of Spring Boot's auto-configuration.
Let us add Spring Security to our application.

```xml
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-security</artifactId>
</dependency>
```

Add default user credentials in `application.properties` as follows:

```properties
spring.security.user.name=admin
spring.security.user.password=secret
```

Now, if you try to access http://localhost:8080/, you will be redirected to an auto-generated Login page.
Once you log in and see the data in the **SPRING_SESSION** table, you can see that the login username is stored in the **PRINCIPAL_NAME** column.

### How does Spring Session work?

Spring Session provides implementations for **HttpServletRequest** and **HttpSession**, which are **SessionRepositoryRequestWrapper**
and **HttpSessionWrapper**. Spring Session provides **SessionRepositoryFilter** to intercept all requests and wrap the **HttpServletRequest**
in **SessionRepositoryRequestWrapper**.

In **SessionRepositoryRequestWrapper.getSession(boolean)** is overridden to return an **HttpSessionWrapper** object instead of the default server implementation of HttpSession.
The **HttpSessionWrapper** uses **SessionRepository** to persist session information in a datastore.

The `SessionRepository` interface has various methods to manage sessions.

```java
public interface SessionRepository<S extends Session> 
{
  S createSession();
 
  void save(S session);
 
  S findById(String id);
 
  void deleteById(String id);
}
```

This `SessionRepository` interface is implemented by various classes based on the type of backend we are using.
In our case, we are using **JdbcOperationsSessionRepository**, provided by **spring-session-jdbc**.

## Conclusion

As you might have already observed, we can manage user sessions effectively by using Spring Session with very minimal configuration
because of Spring Boot's auto-configuration. If for any reason we want to change the backend from JDBC to Redis or Hazelcast, etc.,
it is only a simple configuration change, as we are not directly depending on any Spring Session classes.

> You can find the source code for this article at https://github.com/sivaprasadreddy/spring-session-samples

# CI/CD for Spring Boot applications using Travis-CI


In this article, we are going to learn how we can use **Travis CI** for Continuous Integration and Continuous Deployment (CI/CD)
of a **Spring Boot** application. We will learn how to run Maven build goals, perform test coverage validation using the JaCoCo plugin,
run Code Quality checks using **SonarCloud**, build a Docker image and push it to **DockerHub**, and finally deploy it to **Heroku**.

<!--more-->

> The source code for this article is at https://github.com/sivaprasadreddy/jblogger

Last week, I was talking to my friend about how easy it has become to build and deploy a Java application using Spring Boot. During the discussion, one point came up about how much it costs to build a Java application and deploy it somewhere (in the cloud). So, I thought of exploring more about the free services that we can use to automate all project development activities with minimal or no cost at all.

A few years ago, I used **CloudBees** and **CloudFoundry** to build and deploy my pet projects, which were offering free hosting services, but they are not providing free services anymore.

In the past, I have used **Travis CI** for my Java projects just for preliminary testing purposes, but looking at their documentation, I realized they provide a lot more features.

So I thought of checking if I can use Travis CI for my projects to do all the usual tasks, such as:

*   Checkout the latest code
*   Compile and run Unit and Integration Tests
*   Run JaCoCo code coverage and fail the build if the desired percentage is not met
*   Run SonarQube code quality checks
*   Optionally, build a Docker image and publish it to Docker Hub
*   Deploy the application on some free cloud hosting service like Heroku or OpenShift

After going through their documentation, I realized that we can do all these tasks by using some of the free online services and Travis CI's integration with them.

*   **GitHub** for the code repository
*   **SonarCloud** for a free SonarQube service
*   **Docker Hub** for publishing Docker images
*   **Heroku** for deploying the application

Let us see how we can do all the above-mentioned tasks using Travis CI for a Spring Boot project.

## Step 1: Create a Spring Boot project

Create a Spring Boot project either using http://start.spring.io or from your IDE. I am using the Maven build tool; you can use Gradle also if you prefer. Now commit the project into your GitHub repository.

## Step 2: Create a .travis.yml file

In order to enable Travis CI integration, we need to create a **.travis.yml** file in the project's root folder.
As we are creating a Maven-based Java project, create a `.travis.yml` file with the following content:

**.travis.yml**

```yml
language: java
jdk: oraclejdk8
```

This minimal configuration is sufficient for Travis CI to recognize our Maven-based Java project and build it.
If there is a `build.gradle` file in our project's root folder, Travis will treat it as a Gradle project, or if there is a `pom.xml`, it will treat it as a Maven project. If both `build.gradle` and `pom.xml` are there, then the Gradle build script will take priority.

By default, Travis will run **mvn test -B** for building the project.
If Travis finds the `mvnw` wrapper, then it will be used like **./mvnw test -B**.

But if you want to run a different command or want to run multiple commands, we can use a script block to customize it.

Now commit and push the `.travis.yml` file to GitHub.

## Step 3: Enable Travis CI for the GitHub repository

Go to https://travis-ci.org/ and **Sign in with GitHub**.

Now click on **Add New Repository** (+ symbol).

Enable Travis for the repository. After enabling Travis, click on that repository, and you can trigger a build by selecting
**More Options -> Trigger build**.

Now you can see that the build is running, tests are being executed, and an email notification will be sent to your email regarding the build status.

## Step 4: Add JaCoCo Code Coverage check

Add the Maven JaCoCo plugin to `pom.xml` with options like what is the desired code coverage percentage, packages/classes to ignore, etc.

```xml
<plugin>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <version>0.7.9</version>
    <configuration>
        <excludes>
            <exclude>com/sivalabs/jblogger/entities/*</exclude>
            <exclude>com/sivalabs/jblogger/*Application</exclude>
        </excludes>
    </configuration>
    <executions>
        <execution>
            <id>default-prepare-agent</id>
            <goals>
                <goal>prepare-agent</goal>
            </goals>
        </execution>
        <execution>
            <id>default-prepare-agent-integration</id>
            <goals>
                <goal>prepare-agent-integration</goal>
            </goals>
        </execution>
        <execution>
            <id>default-report</id>
            <phase>verify</phase>
            <goals>
                <goal>report</goal>
            </goals>
        </execution>
        <execution>
            <id>default-report-integration</id>
            <goals>
                <goal>report-integration</goal>
            </goals>
        </execution>
        <execution>
            <id>default-check</id>
            <goals>
                <goal>check</goal>
            </goals>
            <configuration>
                <rules>
                    <!-- implementation is needed only for Maven 2 -->
                    <rule implementation="org.jacoco.maven.RuleConfiguration">
                        <element>BUNDLE</element>
                        <limits>
                            <!-- implementation is needed only for Maven 2 -->
                            <limit implementation="org.jacoco.report.check.Limit">
                                <counter>COMPLEXITY</counter>
                                <value>COVEREDRATIO</value>
                                <minimum>0.60</minimum>
                            </limit>
                        </limits>
                    </rule>
                </rules>
            </configuration>
        </execution>
    </executions>
</plugin>
```

## Step 5: Run Unit and Integration Tests

As I mentioned earlier, by default, Travis runs **mvn test -B**, which will **only run Unit tests**.

We want to run Unit tests and Integration tests separately by using the **maven-failsafe-plugin**.
We will follow the convention by naming **Unit tests** as **\*Test.java/\*Tests.java** and **Integration tests** as **\*IT.java**.

Add the **maven-failsafe-plugin** as mentioned below:

```xml
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <configuration>
        <includes>
            <include>**/*IT.java</include>
        </includes>
    </configuration>
    <executions>
        <execution>
            <id>failsafe-integration-tests</id>
            <phase>integration-test</phase>
            <goals>
                <goal>integration-test</goal>
            </goals>
        </execution>
    </executions>
</plugin>
```

While configuring the `maven-failsafe-plugin` for a Spring Boot project, I hit this issue: https://github.com/spring-projects/spring-boot/issues/6254.
To fix this issue, I have added the **classifier** configuration to the `spring-boot-maven-plugin` as follows:

```xml
<plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
    <configuration>
        <classifier>exec</classifier>
    </configuration>
</plugin>
```

Now we are going to use the **script** block to specify our custom Maven goal to run instead of the default goal.

**.travis.yml**

```yml
language: java
jdk: oraclejdk8
 
script:
- ./mvnw clean install -B
```

## Step 6: SonarQube code quality checks using SonarCloud

SonarCloud, which is built on **SonarQube**, offers free code quality checks for Open Source projects.

**Login with GitHub**, go to **My Account -> Security**, and generate a new token for your project and save it somewhere.
Now click on the **Organizations** tab and create an Organization with a unique key.

Travis CI provides the ability to encrypt sensitive data (https://docs.travis-ci.com/user/encryption-keys/) so that we can encrypt any keys or passwords and configure them in the `.travis.yml` file.

`> sudo gem install travis`

From the project root folder, run the following command to encrypt data:

**travis encrypt SOMEVAR="secretvalue"**

This will generate output like:

**secure: "…. encrypted data …."**

We can add all the secrets as global environment variables as follows:

```yml
env:
  global:
  - secure: "....encrypted data....."
```

Now let us encrypt the SonarCloud Token as follows:

**travis encrypt SONAR_TOKEN="my-sonar-token-here"**

Finally, let us add SonarCloud support as an AddOn (https://docs.travis-ci.com/user/sonarcloud/) as follows:

```yml
language: java
jdk: oraclejdk8
 
env:
  global:
  - secure: "....encrypted sonar token here....."
 
addons:
  sonarcloud:
    organization: "sivaprasadreddy-github"
    token:
      secure: $SONAR_TOKEN
 
script:
- ./mvnw clean install -B
- ./mvnw clean org.jacoco:jacoco-maven-plugin:prepare-agent package sonar:sonar
```

Note that we used **$SONAR_TOKEN** to refer to the encrypted token variable and added one more command to run in the **script** block to run the **sonar:sonar** goal.

## Step 7: Build a Docker image and publish it to DockerHub

Travis CI builds can run and build Docker images, and can also push images to Docker repositories.
For more information, read https://docs.travis-ci.com/user/docker/.

Create a **Dockerfile** in the project's root folder for our Spring Boot application as follows:

```Dockerfile
FROM frolvlad/alpine-oraclejdk8:slim
VOLUME /tmp
ADD target/jblogger-0.0.1-SNAPSHOT.jar app.jar
RUN sh -c 'touch /app.jar'
ENV JAVA_OPTS="-Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=8787,suspend=n"
EXPOSE 8080 8787
ENTRYPOINT [ "sh", "-c", "java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -Dspring.profiles.active=docker -jar /app.jar" ]
```

To use Docker, add the following settings to **.travis.yml**:

```yml
sudo: required
 
services:
  - docker
```

Now we can run Docker commands in our build.

Once the build is successful, we may want to build the Docker image and push it to Docker Hub.
We can leverage the **after_success** section to perform this action.

We need to log in to DockerHub before pushing the image; we are going to configure the DockerHub credentials by encrypting them.

```shell
travis encrypt DOCKER_USER="dockerhub-username"
travis encrypt DOCKER_PASS="dockerhub-password"
```

Add these 2 secrets to the **env.global** section of `.travis.yml`.

Now we can add our Docker commands to build the image and publish it to Docker Hub in the **after_success** section as follows:

```yml
after_success:
- docker login -u $DOCKER_USER -p $DOCKER_PASS
- export TAG=`if [ "$TRAVIS_BRANCH" == "master" ]; then echo "latest"; else echo $TRAVIS_BRANCH; fi`
- export IMAGE_NAME=sivaprasadreddy/jblogger
- docker build -t $IMAGE_NAME:$COMMIT .
- docker tag $IMAGE_NAME:$COMMIT $IMAGE_NAME:$TAG
- docker push $IMAGE_NAME

```

## Step 8: Deploy to Heroku

Travis CI provides options to deploy on a wide range of platforms, including Heroku, OpenShift, AWS, Azure, etc.
Travis CI can automatically deploy your Heroku application after a successful build.

We are going to deploy our Spring Boot application on Heroku using Travis: https://docs.travis-ci.com/user/deployment/heroku/. Before deploying our application to Heroku, first we need to log in to https://www.heroku.com/ and create an application from the Dashboard.

Now create a **Procfile** in the root folder of the project as follows:

```shell
web java -Dserver.port=$PORT -Dspring.profiles.active=heroku $JAVA_OPTS -jar target/jblogger-0.0.1-SNAPSHOT-exec.jar
```

First, we need to get the Heroku API Key and add it as an encrypted secret.

**travis encrypt HEROKU_API_KEY="your-heroku-api-key-here"**

We can deploy to Heroku from Travis by adding a **deploy** section as follows:

```yml
deploy:
  provider: heroku
  api_key: $HEROKU_API_KEY
  app: jblogger
```

Now the complete `.travis.yml` file will look as follows:

```yml
sudo: required
language: java
jdk: oraclejdk8
 
services:
- docker
 
env:
  global:
  - secure: "encrypted-sonar-token"
  - secure: "encrypted-dockerhub-username"
  - secure: "encrypted-dockerhub-password"
  - secure: "encrypted-heroku-api-key"
  - COMMIT=${TRAVIS_COMMIT::7}
 
addons:
  sonarcloud:
    organization: "sivaprasadreddy-github"
    token:
      secure: $SONAR_TOKEN
 
script:
- ./mvnw clean install -B
- ./mvnw clean org.jacoco:jacoco-maven-plugin:prepare-agent package sonar:sonar
 
after_success:
- docker login -u $DOCKER_USER -p $DOCKER_PASS
- export TAG=`if [ "$TRAVIS_BRANCH" == "master" ]; then echo "latest"; else echo $TRAVIS_BRANCH; fi`
- export IMAGE_NAME=sivaprasadreddy/jblogger
- docker build -t $IMAGE_NAME:$COMMIT .
- docker tag $IMAGE_NAME:$COMMIT $IMAGE_NAME:$TAG
- docker push $IMAGE_NAME
 
deploy:
  provider: heroku
  api_key: $HEROKU_API_KEY
  app: jblogger
```

Once the build is successful and deployed on Heroku, you should be able to access the application at `https://<app>.herokuapp.com/`.

I have just covered the most commonly performed tasks in Java applications, but Travis CI can do a lot more.
Check out the Travis CI documentation at https://docs.travis-ci.com/.

# How was my year 2017 and my plans for 2018

I don't really believe in New Year's resolutions or making big 5-year plans!
But I do realize that making a list of things I want to do keeps me focused on those things and minimizes distractions.

<!--more-->

![Happy new year](/images/2018-year.webp "Happy new year")

Before talking about my grand plans for 2018, let me take a moment and see how my 2017 was.

## What I did in 2017

*   I wrote a new book, **Beginning Spring Boot 2**, with Apress publishers.
*   I learned a lot of good practices at ThoughtWorks.
*   I gave a conference talk on **MicroServices** at the **DevConHyd 2017 Conference**.
*   I gave a **GeekNight** talk on **Cloud-Native Applications using Spring Boot and Spring Cloud**.
*   I quit my job to start my own consulting gig.
*   **Learned to Say NO**

The year 2017 was a wonderful year for me in many ways.

### Public speaking
I am a very shy person and have a great fear of public speaking, especially in front of a large audience.
But I gave a talk on **MicroServices** at the **DevConHyd 2017 Conference**, which went fine (I felt it was okayish).

I learned from my mistakes, and after a month, I gave a GeekNight talk on **Cloud-Native Applications using Spring Boot and Spring Cloud**,
which went much better (I felt it was much better) and received very good feedback from the audience.

### Started saying NO more often
Like many people, I too had this problem of NOT saying NO to things that I don't really like to do.
It is not so easy to say NO, especially to people close to you. But I am getting good at it now.
At least I learned to say, **"I will think about it and let you know,"** instead of saying YES on the spot.

### Realized health is more important (one more time)
I had gone through a minor surgery and realized that **nothing is more important than health**.

I should have taken care of my health by doing simple things like:

*   Drinking more water
*   Going for a morning walk for 30 minutes daily
*   Taking short breaks during work hours
*   Setting up a proper workstation (table, chair, etc.) instead of putting a laptop on my lap and working long hours

### My new shiny MacBook
Recently, I realized that it is getting more and more painful to work on the Windows OS because many tools (Docker, Kubernetes, etc.) are not working well on Windows.

I gave enough trials to make Linux my desktop OS, but most of the time I spent time fixing Wi-Fi disconnect issues and battery draining problems instead of doing what I wanted to do.

So finally, I bought a MacBook Pro 15-inch, which Amazon delivered on Dec 31st.

### Switched to IntelliJ IDEA
After using Eclipse (STS, actually) for almost 10+ years, I finally switched to IntelliJ IDEA because that's what everyone uses at my company, and we do pair programming all the time.
Now I am liking IntelliJ IDEA more and more.

## My plans for 2018

*   Become a Corporate Trainer for Java and Spring
*   Write blog posts more often (2 per week?)
*   Make small video tutorials (2 per month?)
*   Planning to write a new book on **Building an e-commerce application using Spring Boot and Spring Cloud**
*   Make Kotlin my primary programming language
*   Learn the Go programming language (probably in the 3rd quarter of 2018)
*   Learn a new language/framework/library every month (inspired by Shekar Gulati's weekly series)
    *   Kafka
    *   Kubernetes
    *   Redis
*   Learn NodeJS and VueJS a little more in-depth
*   Take a 1-week vacation every 3 months
*   Do exercise regularly

I hope 2018 will be more challenging and more rewarding too. :-)
# My Review of Vlad Mihalcea's High-Performance Java Persistence book

In the Java world, JPA/Hibernate is the most popular and widely used framework.
When it comes to the overall performance of a software system, the database persistence layer plays a crucial role.
The tricky part with JPA/Hibernate is that it is very easy to get started but very, very hard to master.
This is where **Vlad Mihalcea**'s **High-Performance Java Persistence** book helps you a lot.

<!--more-->

![High-Performance Java Persistence](/images/VladBook.webp "High-Performance Java Persistence")

The "High-Performance Java Persistence" book is not just another Hibernate book saying,
"Put this annotation here and call that method there, and you are done."

Vlad went all the way to explain how things work in databases and JDBC, such as
database connection management, response time and throughput, batch updates, caching,
transaction management, etc., in a very detailed way.

For many of us, it is a bit overwhelming to understand it in one go, so I read it multiple times,
and every time I read it, I gain more and more insights about how things work under the hood.

Vlad Mihalcea is a master of Hibernate, and we all know it. He covered almost everything about
JPA/Hibernate that you can imagine... in a very detailed way… with a lot of code examples.
Be it JPA mappings, Identifier generation strategies, relationship configurations, batch operations,
fetching/projections, lazy loading, Caching, etc., you are covered.

Just because we are using an ORM in our application, we should not use it for all database operations.
For some things, like reporting or very complex queries, it is better to go with plain JDBC or
use some SQL-focused libraries like jOOQ or MyBatis. In this book, jOOQ is covered with
sufficient details on how to perform complex queries, work with stored procedures, etc.

To put it in one sentence:

> The **High-Performance Java Persistence** book is the one single book you need to know everything about the Java Persistence Layer.

# Spring Cloud Tutorials – Auto Refresh Config Changes using Spring Cloud Bus

In the previous article, [Introduction to Spring Cloud Config Server]({{< relref "2017-08-14-spring-cloud-tutorials-introduction-to-spring-cloud-config-server.md" >}}), we have seen how to use Spring Cloud Config Server.

But the problem is that to reload the config changes in Config Client applications, we need to trigger the **/refresh** endpoint manually. This is not practical or viable if you have a large number of applications.

<!--more-->

# Solution

The **Spring Cloud Bus** module can be used to link multiple applications with a message broker, and we can broadcast configuration changes.

Let us see how we can use RabbitMQ as a message broker and connect multiple applications to receive configuration change events and refresh the bound property values.

In the previous post, we created **catalog-service** as a Spring Boot application that acts as a Config Client.

Let us add the **Cloud Bus AMQP** starter to **catalog-service/pom.xml**.

```xml
<dependency>
	<groupId>org.springframework.cloud</groupId>
	<artifactId>spring-cloud-starter-bus-amqp</artifactId>
</dependency>
```

We are going to use RabbitMQ as a message broker to broadcast config changes. We can install RabbitMQ on our local machine or run it in a Docker container. I am going to run RabbitMQ in a Docker container using the following **docker-compose.yml** configuration.

```yml
version: '2'

services:
  rabbitmq:
      container_name: rabbitmq-server
      image: 'rabbitmq:management'
      environment:
        - RABBITMQ_DEFAULT_USER=guest
        - RABBITMQ_DEFAULT_PASS=guest
      ports:
        - "5672:5672"
        - "15672:15672"
```

Now run **docker-compose up** to start the RabbitMQ container.

Next, we need to configure the RabbitMQ server details in the `catalog-service` properties files.

**config-repo/catalogservice.properties**

```properties
spring.rabbitmq.host=localhost
spring.rabbitmq.port=5672
spring.rabbitmq.username=guest
spring.rabbitmq.password=guest
```

Now we can run **catalog-service**, and to reload the configuration changes, we can trigger POST – **http://localhost:8181/bus/refresh** instead of **http://localhost:8181/refresh**.

Next, let us create another Spring Boot application, **order-service**, which runs on port **8282** and configure the Cloud Config Client and Cloud Bus AMQP the same as `catalog-service`. The `order-service` is also connected to the same RabbitMQ message broker as `catalog-service`.

Now run the `order-service` application, which should be running at http://localhost:8282.

**Now, if you update the properties of both `catalog-service` and `order-service` and commit the changes, you just need to trigger `/bus/refresh` on any one application. This will automatically broadcast the config changes to all the services that are subscribed to RabbitMQ and refresh the properties.**

Not only different applications, but you may also be running multiple instances of the same application on different ports. The same process works in these cases also.

So, with Spring Cloud Bus AMQP, it is easy to reload configuration changes for any number of applications with one single `/bus/refresh` request.

The source code for this article is at https://github.com/sivaprasadreddy/spring-cloud-tutorial.

# Spring Cloud Tutorials – Introduction to Spring Cloud Config Server

Spring Boot provides a lot of flexibility in externalizing configuration properties via properties or YAML files. We can also configure properties for each environment (dev, qa, prod, etc.) separately using profile-specific configuration files such as **application.properties**, **application-dev.properties**, **application-prod.properties**, etc. But once the application is started, we cannot update the properties at runtime. If we change the properties, we need to restart the application to use the updated configuration properties.

<!--more-->

Also, in the context of a large number of MicroService-based applications, we want the ability to configure and manage the configuration properties of all MicroServices from a centralized place.

# Solution

We can use **Spring Cloud Config Server** ([http://cloud.spring.io/spring-cloud-static/Dalston.SR2/#\_spring\_cloud_config][1]) to centralize all the application's configuration and use the **Spring Cloud Config Client** module from the applications to consume configuration properties from the Config Server. We can also update the configuration properties at runtime without needing to restart the application.

> Many of the Spring Cloud modules can be used in Spring Boot applications even though you are not going to deploy your application on any Cloud platforms such as AWS, Pivotal CloudFoundry, etc.

## Spring Cloud Config Server

Spring Cloud Config Server is nothing but a Spring Boot application with a configured configuration properties source. The configuration source can be a **git** repository, an **svn** repository, or a Consul service (<https://www.consul.io/>).

{{< figure src="/images/config.webp" >}}

In this post, we are going to use a git repository as the configuration properties source.

### Git Config Repository

Create a git repository to store properties files. I have created a repository **config-repo** on GitHub, i.e., <https://github.com/sivaprasadreddy/config-repo.git>.

Suppose we are going to develop two Spring Boot applications, **catalog-service** and **order-service**.
Let us create configuration files **catalogservice.properties** and **orderservice.properties** for **catalog-service** and **order-service**, respectively.

**config-repo/catalogservice.properties**

```properties
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://localhost:3306/catalog
spring.datasource.username=root
spring.datasource.password=admin
```

**config-repo/orderservice.properties**

```properties
spring.rabbitmq.host=localhost
spring.rabbitmq.port=5672
spring.rabbitmq.username=guest
spring.rabbitmq.password=guest
```

We can also create profile-specific configuration files such as **catalogservice-dev.properties**, **catalogservice-prod.properties**, **orderservice-dev.properties**, and **orderservice-prod.properties**.

**config-repo/catalogservice-prod.properties**

```properties
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://appsrv1:3306/catalog
spring.datasource.username=appuser46
spring.datasource.password=T(iV&#)X84@1!
```

**config-repo/orderservice-prod.properties**

```properties
spring.rabbitmq.host=srv245.ind.com
spring.rabbitmq.port=5672
spring.rabbitmq.username=admin23
spring.rabbitmq.password=uY7&%we@1!
```

Now commit all the configuration properties files in the **config-repo** git repository.

#### Spring Cloud Config Server Application

Let us create a Spring Boot application, **spring-cloud-config-server**, from <http://start.spring.io> or from your favorite IDE by selecting the starters **Config Server** and **Actuator**.

This will generate the Maven project with the following **pom.xml**.

```xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
  http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.sivalabs</groupId>
	<artifactId>spring-cloud-config-server</artifactId>
	<version>1.0.0-SNAPSHOT</version>
	<packaging>jar</packaging>
	<name>spring-cloud-config-server</name>

	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>1.5.6.RELEASE</version>
		<relativePath/>
	</parent>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
		<java.version>1.8</java.version>
		<spring-cloud.version>Dalston.SR2</spring-cloud.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-actuator</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-config-server</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
		</dependency>
	</dependencies>

	<dependencyManagement>
		<dependencies>
			<dependency>
				<groupId>org.springframework.cloud</groupId>
				<artifactId>spring-cloud-dependencies</artifactId>
				<version>${spring-cloud.version}</version>
				<type>pom</type>
				<scope>import</scope>
			</dependency>
		</dependencies>
	</dependencyManagement>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>
</project>
```

To make our Spring Boot application a Spring Cloud Config Server, we just need to add the **@EnableConfigServer** annotation to the main entry point class and configure the **spring.cloud.config.server.git.uri** property pointing to the git repository.

```java
package com.sivalabs.configserver;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.config.server.EnableConfigServer;

@EnableConfigServer
@SpringBootApplication
public class ConfigServerApplication {

	public static void main(String[] args) {
		SpringApplication.run(ConfigServerApplication.class, args);
	}
}
```

**spring-cloud-config-server/src/main/resources/application.properties**

```properties
server.port=8888
spring.cloud.config.server.git.uri=https://github.com/sivaprasadreddy/config-repo.git
management.security.enabled=false
```

In addition to configuring the git repo URI, we configured the **server.port** to 8888 and **disabled actuator security**. Now you can start the application, which will start on port 8888.

Spring Cloud Config Server exposes the following REST endpoints to get application-specific configuration properties:

```properties
/{application}/{profile}[/{label}]
/{application}-{profile}.yml
/{label}/{application}-{profile}.yml
/{application}-{profile}.properties
/{label}/{application}-{profile}.properties
```

Here, **{application}** refers to the value of the **spring.config.name** property, **{profile}** is an active profile, and **{label}** is an optional git label (defaults to "master").

Now, if you access the URL http://localhost:8888/catalogservice/default, you will get the following response with the `catalogservice` **default** configuration details:

```json
{
    "name": "catalogservice",
    "profiles": [
        "default"
    ],
    "label": null,
    "version": "8a06f25aeb3f28a8f06b5634eae01858b2c6465d",
    "state": null,
    "propertySources": [
        {
            "name": "https://github.com/sivaprasadreddy/config-repo.git/catalogservice.properties",
            "source": {
                "spring.datasource.username": "root",
                "spring.datasource.driver-class-name": "com.mysql.jdbc.Driver",
                "spring.datasource.password": "admin",
                "spring.datasource.url": "jdbc:mysql://localhost:3306/catalog"
            }
        }
    ]
}
```

If you access the URL http://localhost:8888/catalogservice/prod, you will get the following response with the `catalogservice` **prod** configuration details.

```json
{
    "name": "catalogservice",
    "profiles": [
        "prod"
    ],
    "label": null,
    "version": "8a06f25aeb3f28a8f06b5634eae01858b2c6465d",
    "state": null,
    "propertySources": [
        {
            "name": "https://github.com/sivaprasadreddy/config-repo.git/catalogservice-prod.properties",
            "source": {
                "spring.datasource.username": "appuser46",
                "spring.datasource.driver-class-name": "com.mysql.jdbc.Driver",
                "spring.datasource.password": "T(iV&#)X84@1!",
                "spring.datasource.url": "jdbc:mysql://appsrv1:3306/catalog"
            }
        },
        {
            "name": "https://github.com/sivaprasadreddy/config-repo.git/catalogservice.properties",
            "source": {
                "spring.datasource.username": "root",
                "spring.datasource.driver-class-name": "com.mysql.jdbc.Driver",
                "spring.datasource.password": "admin",
                "spring.datasource.url": "jdbc:mysql://localhost:3306/catalog"
            }
        }
    ]
}
```

In addition to the application-specific configuration files like **catalogservice.properties** and **orderservice.properties**, you can create an **application.properties** file to contain common configuration properties for all applications. As you might have guessed, you can have profile-specific files like **application-dev.properties, application-prod.properties**.

Suppose you have an **application.properties** file in **config-repo** with the following properties:

```properties
message=helloworld
jdbc.datasource.url=jdbc:mysql://localhost:3306/defapp
```

Now, if you access http://localhost:8888/catalogservice/prod, you will get the following response:

```json
{
    "name": "catalogservice",
    "profiles": [
        "prod"
    ],
    "label": null,
    "version": "8a06f25aeb3f28a8f06b5634eae01858b2c6465d",
    "state": null,
    "propertySources": [
        {
            "name": "https://github.com/sivaprasadreddy/config-repo.git/catalogservice-prod.properties",
            "source": {
              "spring.datasource.username": "appuser46",
              "spring.datasource.driver-class-name": "com.mysql.jdbc.Driver",
              "spring.datasource.password": "T(iV&#)X84@1!",
              "spring.datasource.url": "jdbc:mysql://appsrv1:3306/catalog"
            }
        },
        {
            "name": "https://github.com/sivaprasadreddy/config-repo.git/catalogservice.properties",
            "source": {
                "spring.datasource.username": "root",
                "spring.datasource.driver-class-name": "com.mysql.jdbc.Driver",
                "spring.datasource.password": "admin",
                "spring.datasource.url": "jdbc:mysql://localhost:3306/catalog"
            }
        },
        {
            "name": "https://github.com/sivaprasadreddy/config-repo.git/application.properties",
            "source": {
                "message": "helloworld",
                "jdbc.datasource.url": "jdbc:mysql://localhost:3306/defapp"
            }
        }
    ]
}
```

Similarly, you can access http://localhost:8888/orderservice/default to get the `orderservice` configuration details.

```json
{
    "name": "orderservice",
    "profiles": [
        "default"
    ],
    "label": null,
    "version": "8a06f25aeb3f28a8f06b5634eae01858b2c6465d",
    "state": null,
    "propertySources": [
        {
            "name": "https://github.com/sivaprasadreddy/config-repo.git/orderservice.properties",
            "source": {
              "spring.rabbitmq.host": "localhost"
              "spring.rabbitmq.port": "5672"
              "spring.rabbitmq.username": "guest"
              "spring.rabbitmq.password": "guest"
            }
        },
        {
            "name": "https://github.com/sivaprasadreddy/config-repo.git/application.properties",
            "source": {
                "message": "helloworld",
                "jdbc.datasource.url": "jdbc:mysql://localhost:3306/defapp"
            }
        }
    ]
}
```

Now that we have seen how to create a configuration server using Spring Cloud Config Server and how to fetch application-specific configuration properties using a REST API.

Let us see how we can create a Spring Boot application and use configuration properties from the Config Server instead of putting them inside the application.

#### Spring Cloud Config Client (catalog-service)

Create a Spring Boot application, **catalog-service**, with **Config Client**, **Web**, and **Actuator** starters.

```xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
  http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.sivalabs</groupId>
	<artifactId>catalog-service</artifactId>
	<version>1.0.0-SNAPSHOT</version>
	<packaging>jar</packaging>

	<name>spring-cloud-config-client</name>

	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>1.5.6.RELEASE</version>
		<relativePath/>
	</parent>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
		<java.version>1.8</java.version>
		<spring-cloud.version>Dalston.SR2</spring-cloud.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-starter-config</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-actuator</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
		</dependency>
	</dependencies>

	<dependencyManagement>
		<dependencies>
			<dependency>
				<groupId>org.springframework.cloud</groupId>
				<artifactId>spring-cloud-dependencies</artifactId>
				<version>${spring-cloud.version}</version>
				<type>pom</type>
				<scope>import</scope>
			</dependency>
		</dependencies>
	</dependencyManagement>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>

</project>
```

Usually, in a Spring Boot application, we configure properties in **_application.properties_**. But while using Spring Cloud Config Server, we use **_bootstrap.properties_** or **_bootstrap.yml_** to configure the URL of the Config Server, and the Spring Cloud Config Client module will take care of starting the application by fetching the application properties from the Config Server.

Configure the following properties in **src/main/resources/bootstrap.properties**:

```properties
server.port=8181
spring.application.name=catalogservice
spring.cloud.config.uri=http://localhost:8888
management.security.enabled=false
```

We have configured the URL of the configuration server using the **spring.cloud.config.uri** property. We have also specified the application name using the **spring.application.name** property.

> Note that the value of the **spring.application.name** property should match the base filename (`catalogservice`) in `config-repo`.

Now run the following `catalog-service` main entry point class:

```java
package com.sivalabs.catalogservice;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class CatalogServiceApplication {

	public static void main(String[] args) {
		SpringApplication.run(CatalogServiceApplication.class, args);
	}
}
```

We can access the actuator endpoint http://localhost:8181/env to see all the configuration properties.

```json
{
    "profiles": [],
    "server.ports": {
        "local.server.port": 8080
    },
    "configService:configClient": {
        "config.client.version": "8a06f25aeb3f28a8f06b5634eae01858b2c6465d"
    },
    "configService:https://github.com/sivaprasadreddy/config-repo.git/catalogservice.properties": {
        "spring.datasource.username": "root",
        "spring.datasource.driver-class-name": "com.mysql.jdbc.Driver",
        "spring.datasource.password": "******",
        "spring.datasource.url": "jdbc:mysql://localhost:3306/catalog"
    },
    "configService:https://github.com/sivaprasadreddy/config-repo.git/application.properties": {
        "message": "helloworld",
        "jdbc.datasource.url": "jdbc:mysql://localhost:3306/defapp"
    },
    "servletContextInitParams": {},
    "systemProperties": {
        ...
        ...
    },
    "systemEnvironment": {
        ...
        ...
    },
    "springCloudClientHostInfo": {
        "spring.cloud.client.hostname": "192.168.0.101",
        "spring.cloud.client.ipAddress": "192.168.0.101"
    },
    "applicationConfig: [classpath:/bootstrap.properties]": {
        "management.security.enabled": "false",
        "spring.cloud.config.uri": "http://localhost:8888",
        "spring.application.name": "catalogservice"
    },
    "defaultProperties": {}
}
```

You can see that the `catalog-service` application fetches the `catalogservice` properties from the Config Server during bootstrap time. You can bind these properties using **@Value** or **@EnableConfigurationProperties** just the way you would if they were defined within the application itself.

#### Precedence of properties

Now that we know there are many ways to provide configuration properties in many files, such as **application.properties, bootstrap.properties**, and their profile variants inside the application's **src/main/resources** and **{application-name}-{profile}.properties, application-{profile}.properties** in `config-repo`.

The following diagram shows the precedence of configuration properties from various properties locations.

{{< figure src="/images/precedence.webp" >}}

#### Refresh properties at runtime

Let us see how we can update the configuration properties of `catalog-service` at runtime without needing to restart the application.

Update the **catalogservice.properties** in the `config-repo` git repository and commit the changes. Now, if you access http://localhost:8181/env, you will still see the old properties.

In order to reload the configuration properties, we need to do the following:

*   Mark Spring beans that you want to reload on config changes with `@RefreshScope`.
*   Issue an http://localhost:8181/refresh request using the **POST** method.

To test the reloading behavior, let's add a property **name=Siva** in **config-repo/catalogservice.properties** and commit it.

Create a simple RestController to display the **name** value as follows:

```java

@RestController
@RefreshScope
class HomeController
{
	@Value("${name}")
	String name;

	@GetMapping("/name")
	public String name()
	{
		return name;
	}
}
```

Now access http://localhost:8181/name, which will display **Siva**. Now change the property value to **name=Prasad** in **config-repo/catalogservice.properties** and commit it.

In order to reload the config changes, trigger an http://localhost:8181/refresh request using the **POST** method and again access http://localhost:8181/name, which should display **Prasad**.

But issuing **/refresh** requests manually is tedious and impractical in the case of a large number of applications and multiple instances of the same application. We will cover how to handle this problem using **Spring Cloud Bus** in the next post.

[Spring Cloud Tutorials – Auto-Refresh Config Changes using Spring Cloud Bus]({{< relref "2017-08-14-spring-cloud-tutorials-auto-refresh-config-changes-using-spring-cloud-bus.md" >}}).

The source code for this article is at https://github.com/sivaprasadreddy/spring-cloud-tutorial.

&nbsp;

 [1]: http://cloud.spring.io/spring-cloud-static/Dalston.SR2/#_spring_cloud_config

# Getting Started with Spring Boot in Intellij IDEA Community Edition

We can use **IntelliJ IDEA Community Edition** for working with **Spring Boot** applications, as we don't need support for configuring servers like Tomcat, Wildfly, etc., and can simply run the applications by running the **main()** method.

However, there is no provision in IntelliJ IDEA Community Edition to create a Spring Boot application directly, the way it is supported in the Ultimate Edition.

<!--more-->

We can go to http://start.spring.io/ and generate the project, and then import it into our IDE. But it would be nice to be able to create the Spring Boot project from the IDE itself.

One simple alternative I found to circumvent this problem is by using the **Project Template support in IntelliJ IDEA**.

For the first time, you can create the Spring Boot project from http://start.spring.io/ and import it into the IntelliJ IDE.

If you wish, you can add any other starters that you commonly use, configure properties like JDBC parameters, etc.

Now click on the menu **Tools** -> **Save Project as Template**. Enter a **Name** and **Description** for the template.

{{< figure src="/images/SBTemplate.webp" >}}

Once the template is created, we can use that template when creating new projects.

Select **File** -> **New** -> **Project**.

{{< figure src="/images/NewProjDlg.webp" >}}

In the **New Project** Dialog, you can see a **User-defined** section. When you click on **User-defined**, you can see all the project templates we have created.

Select the **Spring BootBasic** template we created earlier, provide the **project name**, and click **Finish**.

Well, we won't get any fancy Spring support features like the Ultimate Edition provides, but it will help us to get started quickly without requiring us to manually create a Maven/Gradle project, configure `<parent>`, etc.

Of course, this is not specific to Spring Boot only; we can create templates for any of our favorite tech stacks.

Happy coding. 🙂

# Update on Spring Boot : Learn By Example book

I would like to let you know that I have updated/added the following sections to my [Spring Boot: Learn By Example][1] book.

{{< figure src="/images/sblbe.webp" alt="Spring Boot : Learn By Example"  width="250" height="200" >}}

## Additions to existing chapters:

*   **Working with Multiple Databases**
*   **Exposing JPA entities with bi-directional references through RESTful services**

<!--more-->

In some of our applications, we need to work with multiple databases. For example, we may have a primary database and a reporting database, where most of the application uses the primary database and the application reports will be generated out of the reporting database's data.

The section **Working with Multiple Databases** explains how to configure multiple databases while using JPA in a Spring Boot application.

As Spring Boot can't auto-configure components such as **TransactionManagers**, **EntityManagerFactoryBeans**, and **DataSourceInitializers** for multiple databases automatically, this section will show how to turn off the auto-configuration mechanism for a specific **AutoConfiguration** class and provide the required configuration explicitly.

## Added new chapter: Chapter 16: Deploying Spring Boot Applications.

*   **Running Spring Boot applications in production mode**
*   **Deploying a Spring Boot application on Heroku**
*   **Running a Spring Boot application on Docker**

In this new chapter, I have explained how to run the Spring Boot self-contained jar in production and how to override the configuration properties configured in **src/main/resources/application-\*.properties**.

This chapter also covers how we can deploy a **Spring Boot + JPA + Postgres** application on the [**Heroku**][2] platform by linking to our GitHub repository. We can enable automatic deployment so that whenever we push our changes to our GitHub repository, the updated application will be automatically deployed on Heroku.

Finally, we will learn how to run a **Spring Boot + JPA + Postgres** application on a [Docker][3] container. This chapter explains how to run both the **Postgres** server and the application in separate containers and link them. Then, we will also look into running the Docker containers using **Docker Compose** instead of starting them individually.

I hope you will enjoy these new additions. 🙂

 [1]: https://leanpub.com/springboot-learn-by-example/
 [2]: https://www.heroku.com/
 [3]: https://www.docker.com/

# My New Book Spring Boot : Learn By Example Published Today

I am happy to announce that my new book, [Spring Boot: Learn By Example](https://leanpub.com/springboot-learn-by-example), was published today on **Leanpub**.

{{< figure src="/images/sblbe.webp" alt="Spring Boot : Learn By Example"  width="250" height="200" >}}

<!--more-->

### What is Spring Boot?

Spring is one of the most popular Java frameworks out there for building web and enterprise applications.

Spring supports a variety of configuration approaches (XML, Annotations, JavaConfig, etc.), and properly configuring Spring applications can become a bit tedious and repetitive. To avoid these problems, the Spring team introduced Spring Boot to address the complexity of configuring a Spring application.

Spring Boot is an opinionated approach to configuring Spring-based applications. Spring Boot automatically configures the application components based on various criteria, like the jars you have on your classpath, the beans you have already configured, the properties configured, etc., which makes it very easy to get up and running quickly.

I strongly believe that **the best way to learn or teach is by using examples.** This book is filled with plenty of examples explaining the concepts in a step-by-step manner.

### Who is this book for?

If you are looking for a fast-paced guide to learn how to build Spring-based applications using Spring Boot, this book is for you!

If you are a Spring developer but haven't gotten your hands on Spring Boot and you feel like, "I know the drill, just tell me how to do it the Spring Boot way," this book is what you are looking for!

If you are just starting to use Spring Boot and it is working great, but it all looks magical and you want to understand how Spring Boot works behind the scenes, this book will definitely be helpful to you!

If you are a seasoned Spring developer with a good grasp of Spring framework concepts and you just want a fast-paced guide to learn Spring Boot quickly, sure, this book is definitely for you!

I see a lot of people asking, **"What is the difference between Spring MVC and Spring Boot?"**, **"Should I use Spring Boot or Spring MVC for my next web application?"**, etc. See [Here](http://stackoverflow.com/questions/29479814/spring-mvc-or-spring-boot), [Here](https://www.quora.com/What-is-the-difference-between-Spring-Boot-and-the-Spring-framework), and [Here](https://www.reddit.com/r/java/comments/4mw91b/spring_mvc_vs_spring_boot/). The short answer is, **Spring Boot is nothing but Spring, but it helps you a lot to build Spring-based applications quickly.** The next question would be, **How?**. The best way to explain it is by creating an application **WITHOUT Spring Boot** and showing how quickly you can develop the same application **WITH Spring Boot**.

This book explains how we normally build a Spring application without Spring Boot and shows how we can build the same application using Spring Boot in a much easier way so that you can correlate the concepts and see how Spring Boot increases developer productivity.

Spring Boot does so many things behind the scenes to automatically configure the application. This book explains how Spring Boot AutoConfiguration works behind the scenes in a very detailed manner and shows how we can create our own AutoConfiguration as well.

### What does this book cover?

The **Spring Boot: Learn By Example** book covers the concepts that we commonly use in our Spring applications.

*   What is Spring Boot and how to get started with it?
*   How Spring Boot **AutoConfiguration Works behind the scenes using @Conditional**?
*   Spring Boot goodies like **Logging, ConfigurationProperties, DeveloperTools**
*   Working with Databases using **JdbcTemplate, MyBatis, jOOQ, Spring Data JPA**
*   Working with the **MongoDB** NoSQL database
*   Developing **Web Applications using Spring MVC**
    *   Spring Boot **Web Starter** awesomeness
    *   Using **Thymeleaf** Views
    *   **File Uploading**
    *   **Validation, I18N**
*   Developing **RESTful Services** using **Spring MVC** and **Spring Data REST**
*   Consuming a **Spring Boot REST API** from an **AngularJS** application
*   **Securing** **Web** applications and **REST APIs**
*   Monitoring Spring Boot applications with **Spring Boot Actuator**
*   Testing Spring Boot applications

To get a complete list of topics covered in this book, please see the below **Table of Contents**.

**Table of Contents:**

**1. Introduction to Spring Boot**

*   Overview of the Spring framework
*   Developing a Web Application using Spring MVC and JPA
*   A Quick Taste of Spring Boot

**2. Getting Started with Spring Boot**

*   What is Spring Boot?
*   Our First Spring Boot Application

**3. Spring Boot Behind the scenes**

*   Exploring the Power of @Conditional
*   How Spring Boot AutoConfiguration Works

**4. Creating a Custom Spring Boot Starter**

*   Create a `twitter4j-spring-boot-starter`

**5. Spring Boot Essentials**

*   Logging
*   Externalizing Configuration Properties
*   Developer Tools

**6. Working with JdbcTemplate**

*   Using JdbcTemplate without Spring Boot
*   Using JdbcTemplate with Spring Boot
*   Initializing a Database
*   How to use other Connection Pooling libraries?

**7. Working with MyBatis**

*   Using the Spring Boot MyBatis Starter

**8. Working with jOOQ**

*   Using the Spring Boot jOOQ Starter

**9. Working with JPA**

*   Introducing Spring Data JPA
*   Using Spring Data JPA with Spring Boot

**10. Working with MongoDB**

*   Introducing MongoDB
*   Getting Started with MongoDB using the Mongo Shell
*   Talking to MongoDB using Spring Data Mongo

**11. Web Applications with Spring Boot**

*   Developing a Web Application using Spring Boot
*   Using Tomcat, Jetty, and Undertow Embedded Servlet Containers
*   Customizing Spring MVC configuration
*   A Spring Boot web application as a deployable war
*   Spring Boot Supporting View Templates
*   Working with Thymeleaf Forms
*   Data Validation
*   File Uploading
*   Using ResourceBundles for Internationalization (I18N)

**12. Building a REST API using Spring Boot**

*   Introduction to RESTful Web Services
*   A REST API using Spring MVC
*   A REST API using Spring Data REST
*   Consuming a REST API from AngularJS

**13. Securing Web Applications using Spring Boot**

*   Spring Security in a Spring Boot Web Application
*   Method-Level Security
*   Securing a REST API using Spring Security

**14. Spring Boot Actuator**

*   Exploring Actuator Endpoints
*   Customizing Actuator Endpoints
*   Securing Actuator Endpoints
*   CORS support for Actuator Endpoints
*   Monitoring and management over JMX
*   Monitoring and management using a remote shell

**15. Testing Spring Boot Applications**

*   Testing Support in Spring Boot
*   Testing RESTful Services using RestTemplate
*   Testing Secured Controller/Service Methods
*   Testing with Mockito

**Thank You, Manuel Jordan**

I would like to thank **Manuel Jordan** (**@dr_pompeii**) for reviewing the book and providing feedback, which helped me a lot to improve the quality of the book's content.

If you wish, please do me a favor by spreading the word on **Twitter, Facebook, LinkedIn**, etc.

I hope you will enjoy the book. 🙂

# Videos

## [My YouTube Channel](https://www.youtube.com/c/sivalabs)

## Spring Boot MicroServices Series - Project Based Learning

{{< youtubepl id="PLuNxlOYbv61g_ytin-wgkecfWDKVCEDmB" >}}

## Spring Boot + Kubernetes Tutorial Series

{{< youtubepl id="PLuNxlOYbv61h66_QlcjCEkVAj6RdeplJJ" >}}

## Spring Boot - The Missing Guide

{{< youtubepl id="PLuNxlOYbv61jZL1IiciTgWezZoqEp4WXh" >}}

## Spring Boot Tips Series

{{< youtubepl id="PLuNxlOYbv61jFFX2ARQKnBgkMF6DvEEic" >}}

## Java Testing Made Easy: Learn to write Unit, Integration, E2E & Performance Tests

{{< youtubepl id="PLuNxlOYbv61jtHHFHBOc9N7Dg5jn013ix" >}}


# Creating Custom Spring Boot Starter for Twitter4j

**Spring Boot** provides a lot of starter modules to get up and running quickly. Spring Boot's auto-configure mechanism takes care of configuring Spring Beans on our behalf based on various criteria.

In addition to the Spring Boot starters that come out-of-the-box provided by the Core Spring Team, we can also create our own starter modules.

In this post, we will look into how to create a custom Spring Boot starter. To demonstrate, we are going to create a **twitter4j-spring-boot-starter** which will auto-configure Twitter4J beans.

<!--more-->

To achieve this, we are going to create:

1.  A **twitter4j-spring-boot-autoconfigure** module which contains Twitter4J AutoConfiguration bean definitions.
2.  A **twitter4j-spring-boot-starter** module which pulls in **twitter4j-spring-boot-autoconfigure** and **twitter4j-core** dependencies.
3.  A sample application which uses the **twitter4j-spring-boot-starter**.

### Create Parent Module spring-boot-starter-twitter4j

First, we are going to create a parent `pom` type module to define dependency versions and sub-modules.

```xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                        http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.sivalabs</groupId>
    <artifactId>spring-boot-starter-twitter4j</artifactId>
    <packaging>pom</packaging>
    <version>1.0-SNAPSHOT</version>
    <name>spring-boot-starter-twitter4j</name>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <twitter4j.version>4.0.3</twitter4j.version>
        <spring-boot.version>1.3.2.RELEASE</spring-boot.version>
    </properties>

    <modules>
        <module>twitter4j-spring-boot-autoconfigure</module>
        <module>twitter4j-spring-boot-starter</module>
        <module>twitter4j-spring-boot-sample</module>
    </modules>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-dependencies</artifactId>
                <version>${spring-boot.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>

            <dependency>
                <groupId>org.twitter4j</groupId>
                <artifactId>twitter4j-core</artifactId>
                <version>${twitter4j.version}</version>
            </dependency>
        </dependencies>
    </dependencyManagement>

</project>
```

In this **pom.xml**, we are defining the Spring Boot and Twitter4j versions in the `<dependencyManagement>` section so that we don't need to specify versions all over the place.

### Create twitter4j-spring-boot-autoconfigure module

Create a child module with the name **twitter4j-spring-boot-autoconfigure** in our parent Maven module **spring-boot-starter-twitter4j**.

Add the Maven dependencies such as `spring-boot`, **spring-boot-autoconfigure**, **twitter4j-core**, and **junit** as follows:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
               http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.sivalabs</groupId>
    <artifactId>twitter4j-spring-boot-autoconfigure</artifactId>
    <packaging>jar</packaging>
    <version>1.0-SNAPSHOT</version>

    <parent>
        <groupId>com.sivalabs</groupId>
        <artifactId>spring-boot-starter-twitter4j</artifactId>
        <version>1.0-SNAPSHOT</version>
    </parent>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-autoconfigure</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-configuration-processor</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.twitter4j</groupId>
            <artifactId>twitter4j-core</artifactId>
            <optional>true</optional>
        </dependency>
    </dependencies>
</project>
```

Note that we have specified **twitter4j-core** as an **optional** dependency because **twitter4j-core** should be added to the project only when **twitter4j-spring-boot-starter** is added to the project.

### Create Twitter4jProperties to hold the Twitter4J config parameters

Create `Twitter4jProperties.java` to hold the Twitter4J OAuth config parameters.

```java
package com.sivalabs.spring.boot.autoconfigure;

import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.context.properties.NestedConfigurationProperty;

@ConfigurationProperties(prefix= Twitter4jProperties.TWITTER4J_PREFIX)
public class Twitter4jProperties {

    public static final String TWITTER4J_PREFIX = "twitter4j";

    private Boolean debug = false;

    @NestedConfigurationProperty
    private OAuth oauth = new OAuth();

    public Boolean getDebug() {
        return debug;
    }

    public void setDebug(Boolean debug) {
        this.debug = debug;
    }

    public OAuth getOauth() {
        return oauth;
    }

    public void setOauth(OAuth oauth) {
        this.oauth = oauth;
    }

    public static class OAuth {

        private String consumerKey;
        private String consumerSecret;
        private String accessToken;
        private String accessTokenSecret;

        public String getConsumerKey() {
            return consumerKey;
        }
        public void setConsumerKey(String consumerKey) {
            this.consumerKey = consumerKey;
        }
        public String getConsumerSecret() {
            return consumerSecret;
        }
        public void setConsumerSecret(String consumerSecret) {
            this.consumerSecret = consumerSecret;
        }
        public String getAccessToken() {
            return accessToken;
        }
        public void setAccessToken(String accessToken) {
            this.accessToken = accessToken;
        }
        public String getAccessTokenSecret() {
            return accessTokenSecret;
        }
        public void setAccessTokenSecret(String accessTokenSecret) {
            this.accessTokenSecret = accessTokenSecret;
        }
    }
}
```

With this configuration object, we can configure the `twitter4j` properties in `application.properties` as follows:

```java
twitter4j.debug=true
twitter4j.oauth.consumer-key=your-consumer-key-here
twitter4j.oauth.consumer-secret=your-consumer-secret-here
twitter4j.oauth.access-token=your-access-token-here
twitter4j.oauth.access-token-secret=your-access-token-secret-here
```

## Create Twitter4jAutoConfiguration to auto-configure Twitter4J

Here comes the key part of our starter. The `Twitter4jAutoConfiguration` configuration class contains the bean definitions that will be automatically configured based on some criteria.

What is that criteria?

*   If `twitter4j.TwitterFactory.class` is on the classpath.
*   If a `TwitterFactory` bean is not already defined explicitly.

So, the `Twitter4jAutoConfiguration` goes like this:

```java
package com.sivalabs.spring.boot.autoconfigure;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.condition.ConditionalOnClass;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import twitter4j.Twitter;
import twitter4j.TwitterFactory;
import twitter4j.conf.ConfigurationBuilder;

@Configuration
@ConditionalOnClass({ TwitterFactory.class, Twitter.class })
@EnableConfigurationProperties(Twitter4jProperties.class)
public class Twitter4jAutoConfiguration {

    private static Log log = LogFactory.getLog(Twitter4jAutoConfiguration.class);

    @Autowired
    private Twitter4jProperties properties;

    @Bean
    @ConditionalOnMissingBean
    public TwitterFactory twitterFactory(){

        if (this.properties.getOauth().getConsumerKey() == null
            || this.properties.getOauth().getConsumerSecret() == null
            || this.properties.getOauth().getAccessToken() == null
            || this.properties.getOauth().getAccessTokenSecret() == null)
        {
            String msg = "Twitter4j properties not configured properly." + 
                         " Please check twitter4j.* properties settings in configuration file.";
            log.error(msg);
            throw new RuntimeException(msg);
        }

        ConfigurationBuilder cb = new ConfigurationBuilder();
        cb.setDebugEnabled(properties.getDebug())
          .setOAuthConsumerKey(properties.getOauth().getConsumerKey())
          .setOAuthConsumerSecret(properties.getOauth().getConsumerSecret())
          .setOAuthAccessToken(properties.getOauth().getAccessToken())
          .setOAuthAccessTokenSecret(properties.getOauth().getAccessTokenSecret());
        TwitterFactory tf = new TwitterFactory(cb.build());
        return tf;
    }

    @Bean
    @ConditionalOnMissingBean
    public Twitter twitter(TwitterFactory twitterFactory){
        return twitterFactory.getInstance();
    }

}
```

We have used `@ConditionalOnClass({ TwitterFactory.class, Twitter.class })` to specify that this auto-configuration should take place only when the `TwitterFactory.class` and `Twitter.class` classes are present.

We have also used `@ConditionalOnMissingBean` on the bean definition methods to specify that this bean definition should be considered only if `TwitterFactory`/`Twitter` beans are not already defined explicitly.

Also, note that we have annotated with `@EnableConfigurationProperties(Twitter4jProperties.class)` to enable support for `ConfigurationProperties` and injected the `Twitter4jProperties` bean.

Now we need to configure our custom `Twitter4jAutoConfiguration` in the `src/main/resources/META-INF/spring.factories` file as follows:

`org.springframework.boot.autoconfigure.EnableAutoConfiguration=com.sivalabs.spring.boot.autoconfigure.Twitter4jAutoConfiguration`

## Create the twitter4j-spring-boot-starter module

Create a child module with the name `twitter4j-spring-boot-starter` in our parent Maven module `spring-boot-starter-twitter4j`.

```xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                        http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.sivalabs</groupId>
    <artifactId>twitter4j-spring-boot-starter</artifactId>
    <packaging>jar</packaging>
    <version>1.0-SNAPSHOT</version>

    <parent>
        <groupId>com.sivalabs</groupId>
        <artifactId>spring-boot-starter-twitter4j</artifactId>
        <version>1.0-SNAPSHOT</version>
    </parent>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>


    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>com.sivalabs</groupId>
            <artifactId>twitter4j-spring-boot-autoconfigure</artifactId>
            <version>${project.version}</version>
        </dependency>

        <dependency>
            <groupId>org.twitter4j</groupId>
            <artifactId>twitter4j-core</artifactId>
        </dependency>

    </dependencies>

</project>
```

Note that in this Maven module, we are actually pulling in the `twitter4j-core` dependency.

We don't need to add any code in this module, but optionally we can specify what dependencies we are going to provide through this starter in the `src/main/resources/META-INF/spring.provides` file as follows:

`provides: twitter4j-core`

That's all for our starter.

Let us create a sample using our brand-new starter, `twitter4j-spring-boot-starter`.

## Create a twitter4j-spring-boot-sample application

Let us create a simple Spring Boot application and add our `twitter4j-spring-boot-starter` dependency.

```xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                        http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.sivalabs</groupId>
    <artifactId>twitter4j-spring-boot-sample</artifactId>
    <packaging>jar</packaging>
    <version>1.0-SNAPSHOT</version>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.3.2.RELEASE</version>
    </parent>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <java.version>1.8</java.version>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

    <dependencies>

        <dependency>
            <groupId>com.sivalabs</groupId>
            <artifactId>twitter4j-spring-boot-starter</artifactId>
            <version>1.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

</project>
```

Create the entry-point class `SpringbootTwitter4jDemoApplication` as follows:

```java
package com.sivalabs.demo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class SpringbootTwitter4jDemoApplication {

    public static void main(String[] args) {
        SpringApplication.run(SpringbootTwitter4jDemoApplication.class, args);
    }
}
```

Create `TweetService` as follows:

```java
package com.sivalabs.demo;

import java.util.ArrayList;
import java.util.List;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import twitter4j.ResponseList;
import twitter4j.Status;
import twitter4j.Twitter;
import twitter4j.TwitterException;

@Service
public class TweetService {

    @Autowired
    private Twitter twitter;

    public List<String> getLatestTweets(){
        List<String> tweets = new ArrayList<>();
        try {
            ResponseList<Status> homeTimeline = twitter.getHomeTimeline();
            for (Status status : homeTimeline) {
                tweets.add(status.getText());
            }
        } catch (TwitterException e) {
            throw new RuntimeException(e);
        }
        return tweets;
    }
}
```

Now create a Test to verify our Twitter4j AutoConfiguration.

**Before that, make sure you have set your `twitter4j` OAuth configuration parameters to your actual values.
You can get them from https://apps.twitter.com/**

```java
package com.sivalabs.demo;

import java.util.List;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.SpringApplicationConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

import twitter4j.TwitterException;

@RunWith(SpringJUnit4ClassRunner.class)
@SpringApplicationConfiguration(SpringbootTwitter4jDemoApplication.class)
public class SpringbootTwitter4jDemoApplicationTest  {


    @Autowired
    private TweetService tweetService;

    @Test
    public void testGetTweets() throws TwitterException {
        List<String> tweets = tweetService.getLatestTweets();
        for (String tweet : tweets) {
            System.err.println(tweet);
        }
    }

}
```

Now you should be able to see the latest tweets in your console output.

You can find the code on GitHub: https://github.com/sivaprasadreddy/twitter4j-spring-boot-starter

# Spring Boot : Working with JOOQ

In my previous article, [Spring Boot: Working with MyBatis]({{< relref "2016-03-14-springboot-working-with-mybatis.md" >}}), we learned how to use the Spring Boot MyBatis Starter to quickly get up and running with Spring and MyBatis. In this article, we are going to learn about how to use the Spring Boot jOOQ Starter.

jOOQ (jOOQ Object Oriented Querying) is a persistence framework that embraces SQL.

<!--more-->

jOOQ provides the following features:

*   Building typesafe SQL using a DSL API
*   Typesafe database object referencing using Code Generation
*   An easy-to-use API for Querying and Data fetching
*   SQL logging and debugging
*   etc.

Spring Boot provides a starter, **spring-boot-starter-jooq**, to be able to quickly integrate with jOOQ.

In this article, we will see how to use **spring-boot-starter-jooq** using a step-by-step approach.

### Step 1: Create a Spring Boot Maven Project

Create a Spring Boot Maven-based project and configure the **spring-boot-starter-jooq** dependency.

```xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
                        http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.sivalabs</groupId>
    <artifactId>springboot-jooq-demo</artifactId>
    <packaging>jar</packaging>
    <version>1.0-SNAPSHOT</version>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.3.3.RELEASE</version>
    </parent>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <java.version>1.8</java.version>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-jooq</artifactId>
        </dependency>
        <dependency>
            <groupId>com.h2database</groupId>
            <artifactId>h2</artifactId>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
        </dependency>
    </dependencies>
</project>
```

We are going to use the **H2** in-memory database first; later, we will see how to use MySQL.

### Step 2: Create the database initialization scripts

We are going to create a simple database with 2 tables.

**src/main/resources/schema.sql**

```sql
DROP TABLE IF EXISTS POSTS;

CREATE TABLE POSTS (
  ID int(11) NOT NULL AUTO_INCREMENT,
  TITLE varchar(200) NOT NULL,
  CONTENT LONGTEXT DEFAULT NULL,
  CREATED_ON datetime DEFAULT NULL,
  PRIMARY KEY (ID)
);

DROP TABLE IF EXISTS COMMENTS;

CREATE TABLE COMMENTS (
  ID int(11) NOT NULL AUTO_INCREMENT,
  POST_ID int(11) NOT NULL, 
  NAME varchar(200) NOT NULL,
  EMAIL varchar(200) NOT NULL,
  CONTENT LONGTEXT DEFAULT NULL,
  CREATED_ON datetime DEFAULT NULL,
  PRIMARY KEY (ID),
  FOREIGN KEY (POST_ID) REFERENCES POSTS(ID)
);
```

We will populate some sample data using a **data.sql** script.

**src/main/resources/data.sql**

```sql
insert into posts(id, title, content, created_on) 
values(1, 'Post 1', 'This is post 1', '2016-01-03');

insert into posts(id, title, content, created_on) 
values(2, 'Post 2', 'This is post 2', '2016-01-05');

insert into posts(id, title, content, created_on) 
values(3, 'Post 3', 'This is post 3', '2016-01-07');

insert into comments(id, post_id, name, email, content, created_on) 
values(1, 1, 'User1', 'user1@gmail.com', 'This is comment 1 on post 1', '2016-01-07');

insert into comments(id, post_id, name, email, content, created_on) 
values(2, 1, 'User2', 'user2@gmail.com', 'This is comment 2 on post 1', '2016-01-07');

insert into comments(id, post_id, name, email, content, created_on) 
values(3, 2, 'User1', 'user1@gmail.com', 'This is comment 1 on post 2', '2016-01-07');
```

### Step 3: Configure the jOOQ Maven Codegen Plugin to generate database artifacts

We will use Maven profiles to configure the **jooq-codegen-maven** configuration properties based on the database type.

```xml
<profiles>
    <profile>
        <id>h2</id>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.jooq</groupId>
                    <artifactId>jooq-codegen-maven</artifactId>
                    <executions>
                        <execution>
                            <goals>
                                <goal>generate</goal>
                            </goals>
                        </execution>
                    </executions>
                    <dependencies>
                        <dependency>
                            <groupId>com.h2database</groupId>
                            <artifactId>h2</artifactId>
                            <version>${h2.version}</version>
                        </dependency>
                    </dependencies>
                    <configuration>
                        <jdbc>
                            <driver>org.h2.Driver</driver>
                            <url>jdbc:h2:~/springbootjooq</url>
                        </jdbc>
                        <generator>
                            <name>org.jooq.util.DefaultGenerator</name>
                            <database>
                                <name>org.jooq.util.h2.H2Database</name>
                                <includes>.*</includes>
                                <excludes />
                                <inputSchema>PUBLIC</inputSchema>
                            </database>
                            <target>
                                <packageName>com.sivalabs.demo.jooq.domain</packageName>
                                <directory>gensrc/main/java</directory>
                            </target>
                        </generator>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>
    <profile>
        <id>mysql</id>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.jooq</groupId>
                    <artifactId>jooq-codegen-maven</artifactId>
                    <executions>
                        <execution>
                            <goals>
                                <goal>generate</goal>
                            </goals>
                        </execution>
                    </executions>
                    <dependencies>
                        <dependency>
                            <groupId>mysql</groupId>
                            <artifactId>mysql-connector-java</artifactId>
                            <version>${mysql.version}</version>
                        </dependency>
                    </dependencies>
                    <configuration>
                        <jdbc>
                            <driver>com.mysql.jdbc.Driver</driver>
                            <url>jdbc:mysql://localhost:3306/test</url>
                            <user>root</user>
                            <password>admin</password>
                        </jdbc>
                        <generator>
                            <name>org.jooq.util.DefaultGenerator</name>
                            <database>
                                <name>org.jooq.util.mysql.MySQLDatabase</name>
                                <includes>.*</includes>
                                <excludes />
                                <inputSchema>test</inputSchema>
                            </database>
                            <target>
                                <packageName>com.sivalabs.demo.jooq.domain</packageName>
                                <directory>gensrc/main/java</directory>
                            </target>
                        </generator>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>
</profiles>
```

We have configured two profiles (**h2** and **mysql**) with appropriate JDBC configuration parameters.

We have specified to generate the code artifacts and place them in the **com.sivalabs.demo.jooq.domain** package within the **gensrc/main/java directory**.

We can run the Maven build, activating the **h2** or **mysql** profile as follows:

**mvn clean install -P h2** (or) **mvn clean install -P mysql**

### Step 4: Configure the Maven build-helper-maven-plugin to add the generated source as a sources folder

We will configure the **build-helper-maven-plugin** so that Maven will add the jOOQ-generated code that resides in the **gensrc/main/java** directory as a source folder.

```xml
<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>build-helper-maven-plugin</artifactId>
    <executions>
        <execution>
            <phase>generate-sources</phase>
            <goals>
                <goal>add-source</goal>
            </goals>
            <configuration>
                <sources>
                    <source>gensrc/main/java</source>
                </sources>
            </configuration>
        </execution>
    </executions>
</plugin>
```

### Step 5: Create domain objects.

We can use these domain objects to pass data across the layers and the jOOQ-generated database artifacts to talk to the database.

```java
public class Post
{
    private Integer id;
    private String title;
    private String content;
    private Timestamp createdOn;
    private List<Comment> comments = new ArrayList<>();
    //setters & getters

}
```

```java
public class Comment
{
    private Integer id;
    private Post post;
    private String name;
    private String email;
    private String content;
    private Timestamp createdOn;
    //setters & getters
}
```

### Step 6: Implement the data persistence methods using jOOQ as follows:

```java
package com.sivalabs.demo;

import static com.sivalabs.demo.jooq.domain.tables.Posts.POSTS;
import static com.sivalabs.demo.jooq.domain.tables.Comments.COMMENTS;

import java.sql.Timestamp;
import java.util.ArrayList;
import java.util.List;

import org.jooq.DSLContext;
import org.jooq.Record;
import org.jooq.Result;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

import com.sivalabs.demo.entities.Comment;
import com.sivalabs.demo.entities.Post;
import com.sivalabs.demo.jooq.domain.tables.records.CommentsRecord;
import com.sivalabs.demo.jooq.domain.tables.records.PostsRecord;

@Service
@Transactional
public class BlogService
{
    @Autowired
    private DSLContext dsl;
    
    public Post createPost(Post post){
        PostsRecord postsRecord = dsl.insertInto(POSTS)
                .set(POSTS.TITLE, post.getTitle())
                .set(POSTS.CONTENT, post.getContent())
                .set(POSTS.CREATED_ON, post.getCreatedOn())
                .returning(POSTS.ID)
                .fetchOne();
            
        post.setId(postsRecord.getId());
        return post;
    }
    
    public List<Post> getAllPosts(){        
        List<Post> posts = new ArrayList<>();       
        Result<Record> result = dsl.select().from(POSTS).fetch();
        for (Record r : result) {
            posts.add(getPostEntity(r));
        }
        return posts ;
    }

    public Post getPost(Integer postId){
        Record record = dsl.select().
                                from(POSTS)
                                .where(POSTS.ID.eq(postId))
                                .fetchOne();
        if(record != null)
        {
            Post post = getPostEntity(record);
            
            Result<Record> commentRecords = dsl.select().
                                        from(COMMENTS)
                                        .where(COMMENTS.POST_ID.eq(postId))
                                        .fetch();
            
            for (Record r : commentRecords) {
                post.addComment(getCommentEntity(r));
            }
            return post;
        }
        return null;
    }
    
    
    public Comment createComment(Comment comment){
        CommentsRecord commentsRecord = dsl.insertInto(COMMENTS)
                .set(COMMENTS.POST_ID, comment.getPost().getId())
                .set(COMMENTS.NAME, comment.getName())
                .set(COMMENTS.EMAIL, comment.getEmail())
                .set(COMMENTS.CONTENT, comment.getContent())
                .set(COMMENTS.CREATED_ON, comment.getCreatedOn())
                .returning(COMMENTS.ID)
                .fetchOne();
            
        comment.setId(commentsRecord.getId());
        return comment;
    }
    
    public void deleteComment(Integer commentId){
        dsl.deleteFrom(COMMENTS)
                .where(COMMENTS.ID.equal(commentId))
                .execute();
    }
    
    private Post getPostEntity(Record r){
        Integer id = r.getValue(POSTS.ID, Integer.class);
        String title = r.getValue(POSTS.TITLE, String.class);
        String content = r.getValue(POSTS.CONTENT, String.class);
        Timestamp createdOn = r.getValue(POSTS.CREATED_ON, Timestamp.class);
        return new Post(id, title, content, createdOn);
    }
    
    private Comment getCommentEntity(Record r) {
        Integer id = r.getValue(COMMENTS.ID, Integer.class);
        Integer postId = r.getValue(COMMENTS.POST_ID, Integer.class);
        String name = r.getValue(COMMENTS.NAME, String.class);
        String email = r.getValue(COMMENTS.EMAIL, String.class);
        String content = r.getValue(COMMENTS.CONTENT, String.class);
        Timestamp createdOn = r.getValue(COMMENTS.CREATED_ON, Timestamp.class);
        return new Comment(id, postId, name, email, content, createdOn);
    }
}
```

Observe that we are autowiring the **DSLContext** instance into our Spring Bean and using it to build the typesafe queries.

### Step 7: Create an entry point class and a JUnit test

```java
@SpringBootApplication
public class SpringbootJooqDemoApplication
{
    public static void main(String[] args) {
        SpringApplication.run(SpringbootJooqDemoApplication.class, args);
    }
}
```

```java
@RunWith(SpringJUnit4ClassRunner.class)
@SpringApplicationConfiguration(SpringbootJooqDemoApplication.class)
public class SpringbootJooqDemoApplicationTests
{

    @Autowired
    private BlogService blogService;
    
    @Test
    public void findAllPosts()  {
        List<Post> posts = blogService.getAllPosts();
        assertNotNull(posts);
        assertTrue(!posts.isEmpty());
        for (Post post : posts)
        {
            System.err.println(post);
        }
    }
    
    @Test
    public void findPostById()  {
        Post post = blogService.getPost(1);
        assertNotNull(post);
        System.out.println(post);
        List<Comment> comments = post.getComments();
        System.out.println(comments);
        
    }
    
    @Test
    public void createPost() {
        Post post = new Post(0, "My new Post", 
                            "This is my new test post", 
                            new Timestamp(System.currentTimeMillis()));
        Post savedPost = blogService.createPost(post);
        Post newPost = blogService.getPost(savedPost.getId());
        assertEquals("My new Post", newPost.getTitle());
        assertEquals("This is my new test post", newPost.getContent());
    }
    
    @Test
    public void createComment() {
        Integer postId = 1;
        Comment comment = new Comment(0, postId, "User4", 
                                "user4@gmail.com", "This is my new comment on post1", 
                                new Timestamp(System.currentTimeMillis()));
        Comment savedComment = blogService.createComment(comment);
        Post post = blogService.getPost(postId);
        List<Comment> comments = post.getComments();
        assertNotNull(comments);
        for (Comment comm : comments)
        {
            if(savedComment.getId() == comm.getId()){
                assertEquals("User4", comm.getName());
                assertEquals("user4@gmail.com", comm.getEmail());
                assertEquals("This is my new comment on post1", comm.getContent());
            }
        }
        
    }
    
}
```

Assuming you have generated code using the **H2** profile, we can run the JUnit test without any further configuration.

But if you have generated code using the **mysql** profile, then you will have to configure the following properties in **application.properties**:

```properties
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://localhost:3306/test
spring.datasource.username=root
spring.datasource.password=admin

spring.jooq.sql-dialect=MYSQL
```

> Note that we should use the correct **SqlDialect** for the database; otherwise, you may get SQL syntax errors at runtime.

You can find the source code for this article at my GitHub repository: https://github.com/sivaprasadreddy/springboot-tutorials/tree/master/database/springboot-jooq-demo

For more info on **jOOQ**, you can look at http://www.jooq.org/learn/.

# Spring Boot : Working with MyBatis

**MyBatis** is an SQL Mapping framework with support for custom SQL, stored procedures, and advanced mappings.

> Spring Boot doesn't provide official support for MyBatis integration, but the MyBatis community built a Spring Boot starter for MyBatis.

<!--more-->

You can read about the Spring Boot MyBatis Starter release announcement at http://blog.mybatis.org/2015/11/mybatis-spring-boot-released.html, and you can explore the source code on GitHub at https://github.com/mybatis/mybatis-spring-boot.

### Create a Spring Boot Maven project and add the following MyBatis Starter dependency.

```xml
<dependency>
    <groupId>org.mybatis.spring.boot</groupId>
    <artifactId>mybatis-spring-boot-starter</artifactId>
    <version>1.0.0</version>
</dependency>
```

We will be reusing the **User.java, schema.sql, and data.sql** files created in my previous article, [Spring Boot: Working with JdbcTemplate]({{< relref "2016-03-14-springboot-working-with-jdbctemplate.md" >}}).

Create a MyBatis SQL Mapper interface, **UserMapper.java**, with a few database operations as follows:

```java
package com.sivalabs.demo.domain;

public interface UserMapper
{
    void insertUser(User user);
    User findUserById(Integer id);
    List<User> findAllUsers();
}
```

We need to create Mapper XML files to define the queries for the mapped SQL statements for the corresponding Mapper interface methods.

Create the **UserMapper.xml** file in the **src/main/resources/com/sivalabs/demo/mappers/** directory as follows:

```xml
<!DOCTYPE mapper
    PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN"
    "http://mybatis.org/dtd/mybatis-3-mapper.dtd">

<mapper namespace="com.sivalabs.demo.mappers.UserMapper">

    <resultMap id="UserResultMap" type="User">
        <id column="id" property="id" />
        <result column="name" property="name" />
        <result column="email" property="email" />
    </resultMap>

    <select id="findAllUsers" resultMap="UserResultMap">
        select id, name, email from users
    </select>

    <select id="findUserById" resultMap="UserResultMap">
        select id, name, email from users WHERE id=#{id}
    </select>

    <insert id="insertUser" parameterType="User" useGeneratedKeys="true" keyProperty="id">
        insert into users(name,email) values(#{name},#{email})
    </insert>
</mapper>
```

A few things to observe here are:

*   The namespace in the Mapper XML should be the same as the Fully Qualified Name (FQN) for the Mapper Interface.
*   Statement `id` values should be the same as the Mapper Interface method names.
*   If the query result column names are different from the bean property names, we can use the `<resultMap>` configuration to provide a mapping between the column names and their corresponding bean property names.

MyBatis also provides annotation-based query configurations without requiring Mapper XMLs.

We can create the **UserMapper.java** interface and configure the mapped SQLs using annotations as follows:

```java
public interface UserMapper
{
    @Insert("insert into users(name,email) values(#{name},#{email})")
    @SelectKey(statement="call identity()", keyProperty="id",
    before=false, resultType=Integer.class)
    void insertUser(User user);

    @Select("select id, name, email from users WHERE id=#{id}")
    User findUserById(Integer id);

    @Select("select id, name, email from users")
    List<User> findAllUsers();

}
```

The Spring Boot MyBatis starter provides the following MyBatis configuration parameters, which we can use to customize MyBatis settings:

```java
mybatis.config=mybatis config file name
mybatis.mapperLocations=mappers file locations
mybatis.typeAliasesPackage=domain object's package
mybatis.typeHandlersPackage=handler's package
mybatis.check-config-location=check that the mybatis configuration exists
mybatis.executorType=mode of execution. Default is SIMPLE
```

Configure the **typeAliasesPackage** and **mapperLocations** in **application.properties**.

```java
mybatis.typeAliasesPackage=com.sivalabs.demo.domain
mybatis.mapperLocations=classpath*:**/mappers/*.xml
```

Create the entry point class **SpringbootMyBatisDemoApplication.java**.

```java
@SpringBootApplication
@MapperScan("com.sivalabs.demo.mappers")
public class SpringbootMyBatisDemoApplication
{
    public static void main(String[] args)
    {
        SpringApplication.run(SpringbootMyBatisDemoApplication.class, args);
    }
}
```

Observe that we have used the **@MapperScan("com.sivalabs.demo.mappers")** annotation to specify where to look for Mapper interfaces.

Now create a JUnit test class and test our `UserMapper` methods.

```java
@RunWith(SpringJUnit4ClassRunner.class)
@SpringApplicationConfiguration(SpringbootMyBatisDemoApplication.class)
public class SpringbootMyBatisDemoApplicationTests
{
    @Autowired
    private UserMapper userMapper;

    @Test
    public void findAllUsers() {
        List<User> users = userMapper.findAllUsers();
        assertNotNull(users);
        assertTrue(!users.isEmpty());
    }

    @Test
    public void findUserById() {
        User user = userMapper.findUserById(1);
        assertNotNull(user);
    }

    @Test
    public void createUser() {
        User user = new User(0, "Siva", "siva@gmail.com");
        userMapper.insertUser(user);
        User newUser = userMapper.findUserById(user.getId());
        assertEquals("Siva", newUser.getName());
        assertEquals("siva@gmail.com", newUser.getEmail());
    }
}
```

You can find the source code of the article at my GitHub repo: https://github.com/sivaprasadreddy/springboot-tutorials

You can read more about MyBatis and Spring integration at http://blog.mybatis.org/p/products.html and http://www.mybatis.org/spring/.

# Spring Boot : Working with JdbcTemplate

Spring provides a nice abstraction on top of the JDBC API using **JdbcTemplate** and also provides great transaction management capabilities
using an annotation-based approach.

<!--more-->

First, let's take a quick look at how we generally use Spring's **JdbcTemplate** (**without Spring Boot**) by registering **DataSource**,
**TransactionManager**, and **JdbcTemplate** beans. Optionally, we can register a **DataSourceInitializer** bean to initialize our database.

```java
@Configuration
@ComponentScan
@EnableTransactionManagement
@PropertySource(value = { "classpath:application.properties" })
public class AppConfig 
{
    @Autowired
    private Environment env;

    @Value("${init-db:false}")
    private String initDatabase;
    
    @Bean
    public static PropertySourcesPlaceholderConfigurer placeHolderConfigurer()
    {
        return new PropertySourcesPlaceholderConfigurer();
    }    

    @Bean
    public JdbcTemplate jdbcTemplate(DataSource dataSource)
    {
        return new JdbcTemplate(dataSource);
    }

    @Bean
    public PlatformTransactionManager transactionManager(DataSource dataSource)
    {
        return new DataSourceTransactionManager(dataSource);
    }

    @Bean
    public DataSource dataSource()
    {
        BasicDataSource dataSource = new BasicDataSource();
        dataSource.setDriverClassName(env.getProperty("jdbc.driverClassName"));
        dataSource.setUrl(env.getProperty("jdbc.url"));
        dataSource.setUsername(env.getProperty("jdbc.username"));
        dataSource.setPassword(env.getProperty("jdbc.password"));
        return dataSource;
    }

    @Bean
    public DataSourceInitializer dataSourceInitializer(DataSource dataSource)
    {
        DataSourceInitializer dataSourceInitializer = new DataSourceInitializer();    
        dataSourceInitializer.setDataSource(dataSource);
        ResourceDatabasePopulator databasePopulator = new ResourceDatabasePopulator();
        databasePopulator.addScript(new ClassPathResource("data.sql"));
        dataSourceInitializer.setDatabasePopulator(databasePopulator);
        dataSourceInitializer.setEnabled(Boolean.parseBoolean(initDatabase));
        return dataSourceInitializer;
    }
}
```

With this configuration in place, we can inject **JdbcTemplate** into Data Access components to interact with databases.

```java
public class User
{
    private Integer id;
    private String name;
    private String email;

    // setters & getters
}
```

```java
@Repository
public class UserRepository
{
    @Autowired
    private JdbcTemplate jdbcTemplate;

    @Transactional(readOnly=true)
    public List<User> findAll() {
        return jdbcTemplate.query("select * from users", new UserRowMapper());
    }
}
```

```java
class UserRowMapper implements RowMapper<User>
{
    @Override
    public User mapRow(ResultSet rs, int rowNum) throws SQLException 
    {
        User user = new User();
        user.setId(rs.getInt("id"));
        user.setName(rs.getString("name"));
        user.setEmail(rs.getString("email"));

        return user;
    }
}
```

You might have observed that most of the time, we use this similar kind of configuration in our applications.

Now let us see how to use **JdbcTemplate** without needing to configure all these beans manually by using **Spring Boot**.

## Using JdbcTemplate with Spring Boot

By using Spring Boot, we can take advantage of the auto-configuration feature and eliminate the need to configure beans by ourselves.

**Create a Spring Boot Maven-based project and add the spring-boot-starter-jdbc module.**

```xml
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
```

By adding the **spring-boot-starter-jdbc** module, we get the following auto-configuration:

*   The `spring-boot-starter-jdbc` module transitively pulls `tomcat-jdbc-{version}.jar`, which is used to configure the `DataSource` bean.
*   If you have not defined any `DataSource` bean explicitly and if you have any embedded database driver in the classpath, such as H2, HSQL, or Derby, then Spring Boot will automatically register a `DataSource` bean using in-memory database settings.
*   If you haven't registered any of the following types of beans, then Spring Boot will register them automatically:
    *   **PlatformTransactionManager (DataSourceTransactionManager)**
    *   **JdbcTemplate**
    *   **NamedParameterJdbcTemplate**
*   We can have **schema.sql** and **data.sql** files in the root classpath, which Spring Boot will automatically use to initialize the database. In addition to `schema.sql` and `data.sql`, Spring Boot will load **schema-${platform}.sql** and **data-${platform}.sql** files if they are available in the root classpath.

Here, the `platform` value is the value of the property `spring.datasource.platform`, which can be **hsqldb, h2, oracle, mysql, postgresql**, etc.
You can customize the default names of the scripts using the following properties:

*   **spring.datasource.schema=create-db.sql**
*   **spring.datasource.data=seed-data.sql**

Spring Boot uses the **spring.datasource.initialize** property value, which is **true** by default, to determine whether to initialize the database or not. If you want to turn off database initialization, you can set **spring.datasource.initialize=false**.

If there are any errors in executing the scripts, the application will fail to start. If you want to continue, then you can set **spring.datasource.continueOnError=true**.

Let us add the **H2** database driver to our **pom.xml**.

```xml
<dependency>
    <groupId>com.h2database</groupId>
    <artifactId>h2</artifactId>
</dependency>
```

Create **schema.sql** in **src/main/resources** as follows:

```sql
CREATE TABLE users
(
    id int(11) NOT NULL AUTO_INCREMENT,
    name varchar(100) NOT NULL,
    email varchar(100) DEFAULT NULL,
    PRIMARY KEY (id)
);
```

Create **data.sql** in **src/main/resources** as follows:

```sql
insert into users(id, name, email) values(1,'Siva','siva@gmail.com');
insert into users(id, name, email) values(2,'Prasad','prasad@gmail.com');
insert into users(id, name, email) values(3,'Reddy','reddy@gmail.com');
```

Now you can inject **JdbcTemplate** into **UserRepository** as follows:

```java
@Repository
public class UserRepository
{
    @Autowired
    private JdbcTemplate jdbcTemplate;

    @Transactional(readOnly=true)
    public List<User> findAll() {
        return jdbcTemplate.query("select * from users", 
                new UserRowMapper());
    }

    @Transactional(readOnly=true)
    public User findUserById(int id) {
        return jdbcTemplate.queryForObject(
            "select * from users where id=?",
            new Object[]{id}, new UserRowMapper());
    }

    public User create(final User user) 
    {
        final String sql = "insert into users(name,email) values(?,?)";

        KeyHolder holder = new GeneratedKeyHolder();
        jdbcTemplate.update(new PreparedStatementCreator() {
            @Override
            public PreparedStatement createPreparedStatement(Connection connection) throws SQLException {
                PreparedStatement ps = connection.prepareStatement(sql, Statement.RETURN_GENERATED_KEYS);
                ps.setString(1, user.getName());
                ps.setString(2, user.getEmail());
                return ps;
            }
        }, holder);

        int newUserId = holder.getKey().intValue();
        user.setId(newUserId);
        return user;
    }
}

class UserRowMapper implements RowMapper<User>
{
    @Override
    public User mapRow(ResultSet rs, int rowNum) throws SQLException {
        User user = new User();
        user.setId(rs.getInt("id"));
        user.setName(rs.getString("name"));
        user.setEmail(rs.getString("email"));
        return user;
    }
}
```

Create the entry point **SpringbootJdbcDemoApplication.java**.

```java
@SpringBootApplication
public class SpringbootJdbcDemoApplication
{
    public static void main(String[] args)
    {
        SpringApplication.run(SpringbootJdbcDemoApplication.class, args);
    }
}
```

Let us create a JUnit Test class to test our `UserRepository` methods.

```java
@RunWith(SpringJUnit4ClassRunner.class)
@SpringApplicationConfiguration(SpringbootJdbcDemoApplication.class)
public class SpringbootJdbcDemoApplicationTests
{
    @Autowired
    private UserRepository userRepository;

    @Test
    public void findAllUsers() {
        List<User> users = userRepository.findAll();
        assertNotNull(users);
        assertTrue(!users.isEmpty());
    }

    @Test
    public void findUserById() {
        User user = userRepository.findUserById(1);
        assertNotNull(user);
    }

    @Test
    public void createUser() {
        User user = new User(0, "John", "john@gmail.com");
        User savedUser = userRepository.create(user);
        User newUser = userRepository.findUserById(savedUser.getId());
        assertNotNull(newUser);
        assertEquals("John", newUser.getName());
        assertEquals("john@gmail.com", newUser.getEmail());
    }
}
```

> By default, Spring Boot features such as external properties, logging, etc., are available in
the ApplicationContext only if you use **SpringApplication**. So, Spring Boot provides the **@SpringApplicationConfiguration** annotation
to configure the ApplicationContext for tests, which uses **SpringApplication** behind the scenes.

We have learned how to get started quickly with an embedded database.

**What if we want to use non-embedded databases like MySQL, Oracle, or PostgreSQL, etc.?**

We can configure the database properties in the `application.properties` file so that Spring Boot will use those JDBC parameters to configure the `DataSource` bean.

```properties
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://localhost:3306/test
spring.datasource.username=root
spring.datasource.password=admin
```

For any reason, if you want to have more control and configure the **DataSource** bean by yourself, then you can configure the `DataSource` bean in a Configuration class. If you register a `DataSource` bean, then Spring Boot will not configure `DataSource` automatically using AutoConfiguration.

**What if you want to use another Connection Pooling library?**

Spring Boot by default pulls in **tomcat-jdbc-{version}.jar** and uses **org.apache.tomcat.jdbc.pool.DataSource** to configure the **DataSource** bean.

Spring Boot checks for the availability of the following classes and uses the first one that is available in the classpath:

*   `org.apache.tomcat.jdbc.pool.DataSource`
*   `com.zaxxer.hikari.HikariDataSource`
*   `org.apache.commons.dbcp.BasicDataSource`
*   `org.apache.commons.dbcp2.BasicDataSource`

For example, if you want to use **HikariDataSource**, then you can exclude **tomcat-jdbc** and add the **HikariCP** dependency as follows:

```xml
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-jdbc</artifactId>
    <exclusions>
        <exclusion>
        <groupId>org.apache.tomcat</groupId>
        <artifactId>tomcat-jdbc</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<dependency>
    <groupId>com.zaxxer</groupId>
    <artifactId>HikariCP</artifactId>
</dependency>
```

With this dependency configuration, Spring Boot will use **HikariCP** to configure the **DataSource** bean.

You can find the source code of the article at my GitHub repo: https://github.com/sivaprasadreddy/springboot-tutorials

# How Spring Boot AutoConfiguration magic works?

In my previous post, [Why Spring Boot?]({{< relref "2016-03-13-why-springboot.md" >}}), we looked at how to create a Spring Boot application. But you may or may not understand what is going on behind the scenes. You may want to understand the magic behind Spring Boot's AutoConfiguration.

<!--more-->

But before that, you should know about Spring's **@Conditional** feature, on which all of Spring Boot's AutoConfiguration magic depends.

## Exploring the power of @Conditional

While developing Spring-based applications, we may come across the need to register beans conditionally.

For example, you may want to register a DataSource bean pointing to a DEV Database while running the application locally and point to a different PRODUCTION Database while running in production.

You can externalize the database connection parameters into properties files and use the file appropriate for the environment. But you need to change the configuration whenever you need to point to a different environment and build the application.

To address this problem, Spring 3.1 introduced the concept of **Profiles**. You can register multiple beans of the same type and associate them with one or more profiles. When you run the application, you can activate the desired profiles, and only the beans associated with the activated profiles will be registered.

```java
@Configuration
public class AppConfig
{
    @Bean
    @Profile("DEV")
    public DataSource devDataSource() {
        ...
    }

    @Bean
    @Profile("PROD")
    public DataSource prodDataSource() {
        ...
    }
}
```

Then you can specify the active profile using the System Property **-Dspring.profiles.active=DEV**.

This approach works for simple cases like enabling or disabling bean registrations based on activated profiles. But if you want to register beans based on some conditional logic, then the Profiles approach itself is not sufficient.

To provide much more flexibility for registering Spring beans conditionally, Spring 4 introduced the concept of **@Conditional**. By using the **@Conditional** approach, you can register a bean conditionally based on any arbitrary condition.

For example, you may want to register a bean when:

*   A specific class is present in the classpath.
*   A Spring bean of a certain type is not already registered in the ApplicationContext.
*   A specific file exists in a location.
*   A specific property value is configured in a configuration file.
*   A specific system property is present/absent.

These are just a few examples, and you can have any condition you want.

Let us take a look at how Spring's @Conditional works.

Suppose we have a **UserDAO** interface with methods to get data from a data store. We have two implementations of the **UserDAO** interface, namely **JdbcUserDAO**, which talks to a **MySQL** database, and **MongoUserDAO**, which talks to **MongoDB**.

We may want to enable only one of **JdbcUserDAO** and **MongoUserDAO** based on a System Property, say **dbType**.

If the application is started using **java -jar myapp.jar -DdbType=MySQL**, then we want to enable **JdbcUserDAO**; otherwise, if the application is started using **java -jar myapp.jar -DdbType=MONGO**, we want to enable **MongoUserDAO**.

Suppose we have the **UserDAO** interface and **JdbcUserDAO**, **MongoUserDAO** implementations as follows:

```java
public interface UserDAO
{
    List<String> getAllUserNames();
}

public class JdbcUserDAO implements UserDAO
{
    @Override
    public List<String> getAllUserNames()
    {
        System.out.println("**** Getting usernames from RDBMS *****");
        return Arrays.asList("Siva","Prasad","Reddy");
    }
}

public class MongoUserDAO implements UserDAO
{
    @Override
    public List<String> getAllUserNames()
    {
        System.out.println("**** Getting usernames from MongoDB *****");
        return Arrays.asList("Bond","James","Bond");
    }
}
```

We can implement the Condition **MySQLDatabaseTypeCondition** to check whether the System Property **dbType** is **"MYSQL"** as follows:

```java
public class MySQLDatabaseTypeCondition implements Condition
{
    @Override
    public boolean matches(ConditionContext conditionContext, AnnotatedTypeMetadata metadata)
    {
        String enabledDBType = System.getProperty("dbType");
        return (enabledDBType != null && enabledDBType.equalsIgnoreCase("MYSQL"));
    }
}
```

We can implement the Condition **MongoDBDatabaseTypeCondition** to check whether the System Property **dbType** is "**MONGODB**" as follows:

```java
public class MongoDBDatabaseTypeCondition implements Condition
{
    @Override
    public boolean matches(ConditionContext conditionContext, AnnotatedTypeMetadata metadata)
    {
        String enabledDBType = System.getProperty("dbType");
        return (enabledDBType != null && enabledDBType.equalsIgnoreCase("MONGODB"));
    }
}
```

Now we can configure both **JdbcUserDAO** and **MongoUserDAO** beans conditionally using **@Conditional** as follows:

```java
@Configuration
public class AppConfig
{
    @Bean
    @Conditional(MySQLDatabaseTypeCondition.class)
    public UserDAO jdbcUserDAO(){
        return new JdbcUserDAO();
    }

    @Bean
    @Conditional(MongoDBDatabaseTypeCondition.class)
    public UserDAO mongoUserDAO(){
        return new MongoUserDAO();
    }
}
```

If we run the application like **java -jar myapp.jar -DdbType=MYSQL**, then only the **JdbcUserDAO** bean will be registered.

But if you set the System property like **-DdbType=MONGODB**, then only the **MongoUserDAO** bean will be registered.

Now that we have seen how to conditionally register a bean based on a System Property.

Suppose we want to register the **MongoUserDAO** bean only when the **MongoDB** java driver class **"com.mongodb.Server"** is available on the classpath; if not, we want to register the **JdbcUserDAO** bean.

To accomplish that, we can create Conditions to check for the presence or absence of the MongoDB driver class **"com.mongodb.Server"** as follows:

```java
public class MongoDriverPresentsCondition implements Condition
{
    @Override
    public boolean matches(ConditionContext conditionContext,AnnotatedTypeMetadata metadata)
    {
        try {
            Class.forName("com.mongodb.Server");
            return true;
        } catch (ClassNotFoundException e) {
            return false;
        }
    }
}

public class MongoDriverNotPresentsCondition implements Condition
{
    @Override
    public boolean matches(ConditionContext conditionContext, AnnotatedTypeMetadata metadata)
    {
        try {
            Class.forName("com.mongodb.Server");
            return false;
        } catch (ClassNotFoundException e) {
            return true;
        }
    }
}
```

We have just seen how to register beans conditionally based on the presence/absence of a class in the classpath.

What if we want to register the **MongoUserDAO** bean only if no other Spring bean of type **UserDAO** is already registered?

We can create a Condition to check if there is any existing bean of a certain type as follows:

```java
public class UserDAOBeanNotPresentsCondition implements Condition
{
    @Override
    public boolean matches(ConditionContext conditionContext, AnnotatedTypeMetadata metadata)
    {
        UserDAO userDAO = conditionContext.getBeanFactory().getBean(UserDAO.class);
        return (userDAO == null);
    }
}
```

What if we want to register the **MongoUserDAO** bean only if the property **app.dbType=MONGO** is set in a properties placeholder configuration file?

We can implement that Condition as follows:

```java
public class MongoDbTypePropertyCondition implements Condition
{
    @Override
    public boolean matches(ConditionContext conditionContext,
    AnnotatedTypeMetadata metadata)
    {
        String dbType = conditionContext.getEnvironment()
                            .getProperty("app.dbType");
        return "MONGO".equalsIgnoreCase(dbType);
    }
}
```

We have just seen how to implement various types of Conditions.

But there is an even more elegant way to implement Conditions using Annotations. Instead of creating a Condition implementation for both MYSQL and MongoDB, we can create a `DatabaseType` annotation as follows:

```java
@Target({ ElementType.TYPE, ElementType.METHOD })
@Retention(RetentionPolicy.RUNTIME)
@Conditional(DatabaseTypeCondition.class)
public @interface DatabaseType
{
    String value();
}
```

Then we can implement **DatabaseTypeCondition** to use the **DatabaseType** value to determine whether to enable or disable bean registration as follows:

```java
public class DatabaseTypeCondition implements Condition
{
    @Override
    public boolean matches(ConditionContext conditionContext,
    AnnotatedTypeMetadata metadata)
    {
        Map<String, Object> attributes = metadata.getAnnotationAttributes(DatabaseType.class.getName());
        String type = (String) attributes.get("value");
        String enabledDBType = System.getProperty("dbType","MYSQL");
        return (enabledDBType != null && type != null && enabledDBType.equalsIgnoreCase(type));
    }
}
```

Now we can use the **@DatabaseType** annotation on our bean definitions as follows:

```java
@Configuration
@ComponentScan
public class AppConfig
{
    @DatabaseType("MYSQL")
    public UserDAO jdbcUserDAO(){
        return new JdbcUserDAO();
    }

    @Bean
    @DatabaseType("MONGO")
    public UserDAO mongoUserDAO(){
        return new MongoUserDAO();
    }
}
```

Here we are getting the metadata from the **DatabaseType** annotation and checking it against the System Property **dbType** value to determine whether to enable or disable the bean registration.

We have seen a good number of examples to understand how we can register beans conditionally using the **@Conditional** annotation.

Spring Boot extensively uses the **@Conditional** feature to register beans conditionally based on various criteria.

You can find various Condition implementations that Spring Boot uses in the **org.springframework.boot.autoconfigure** package of **spring-boot-autoconfigure-{version}.jar**.

Now that we know how Spring Boot uses the **@Conditional** feature to conditionally check whether to register a bean or not.

But what exactly triggers the auto-configuration mechanism?

This is what we are going to look at in the next section.

### Spring Boot AutoConfiguration

The key to Spring Boot's auto-configuration magic is the **@EnableAutoConfiguration** annotation.
Typically, we annotate our Application entry point class with either **@Spring BootApplication** or,
if we want to customize the defaults, we can use the following annotations:

```java
@Configuration
@EnableAutoConfiguration
@ComponentScan
public class Application
{

}
```

The **@EnableAutoConfiguration** annotation enables the auto-configuration of the Spring **ApplicationContext** by scanning the classpath components and registering the beans that match various Conditions.

Spring Boot provides various **AutoConfiguration** classes in **spring-boot-autoconfigure-{version}.jar** which are responsible for registering various components.

Typically, **AutoConfiguration** classes are annotated with **@Configuration** to mark them as Spring configuration classes and annotated with **@EnableConfigurationProperties** to bind the customization properties and one or more Conditional bean registration methods.

For example, consider the **org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration** class.

```java
@Configuration
@ConditionalOnClass({ DataSource.class, EmbeddedDatabaseType.class })
@EnableConfigurationProperties(DataSourceProperties.class)
@Import({ Registrar.class, DataSourcePoolMetadataProvidersConfiguration.class })
public class DataSourceAutoConfiguration 
{
    ...
    ...
    @Conditional(DataSourceAutoConfiguration.EmbeddedDataSourceCondition.class)
    @ConditionalOnMissingBean({ DataSource.class, XADataSource.class })
    @Import(EmbeddedDataSourceConfiguration.class)
    protected static class EmbeddedConfiguration {

    }

    @Configuration
    @ConditionalOnMissingBean(DataSourceInitializer.class)
    protected static class DataSourceInitializerConfiguration {
        @Bean
        public DataSourceInitializer dataSourceInitializer() {
        return new DataSourceInitializer();
        }
    }

    @Conditional(DataSourceAutoConfiguration.NonEmbeddedDataSourceCondition.class)
    @ConditionalOnMissingBean({ DataSource.class, XADataSource.class })
    protected static class NonEmbeddedConfiguration {
        @Autowired
        private DataSourceProperties properties;

        @Bean
        @ConfigurationProperties(prefix = DataSourceProperties.PREFIX)
        public DataSource dataSource() {
            DataSourceBuilder factory = DataSourceBuilder
                    .create(this.properties.getClassLoader())
                    .driverClassName(this.properties.getDriverClassName())
                    .url(this.properties.getUrl()).username(this.properties.getUsername())
                    .password(this.properties.getPassword());
            if (this.properties.getType() != null) {
                factory.type(this.properties.getType());
            }
            return factory.build();
        }
    }
    ...
    ...
    @Configuration
    @ConditionalOnProperty(prefix = "spring.datasource", name = "jmx-enabled")
    @ConditionalOnClass(name = "org.apache.tomcat.jdbc.pool.DataSourceProxy")
    @Conditional(DataSourceAutoConfiguration.DataSourceAvailableCondition.class)
    @ConditionalOnMissingBean(name = "dataSourceMBean")
    protected static class TomcatDataSourceJmxConfiguration {
        @Bean
        public Object dataSourceMBean(DataSource dataSource) {
        ....
        ....
        }
    }
    ...
    ...
}
```

Here, **DataSourceAutoConfiguration** is annotated with **@ConditionalOnClass({ DataSource.class, EmbeddedDatabaseType.class })**,
which means that the Auto-Configuration of beans within **DataSourceAutoConfiguration** will be considered only if **DataSource.class**
and **EmbeddedDatabaseType.class** classes are available on the classpath.

The class is also annotated with **@EnableConfigurationProperties(DataSourceProperties.class)**, which enables binding the properties
in **application.properties** to the properties of the **DataSourceProperties** class automatically.

```java
@ConfigurationProperties(prefix = DataSourceProperties.PREFIX)
public class DataSourceProperties implements BeanClassLoaderAware, EnvironmentAware, InitializingBean {

    public static final String PREFIX = "spring.datasource";
    ...
    ...
    private String driverClassName;
    private String url;
    private String username;
    private String password;
    ...
    //setters and getters
}
```

With this configuration, all the properties that start with **spring.datasource.*** will be automatically bound to the **DataSourceProperties** object.

```properties
spring.datasource.url=jdbc:mysql://localhost:3306/test
spring.datasource.username=root
spring.datasource.password=secret
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
```

You can also see some inner classes and bean definition methods that are annotated with Spring Boot's Conditional annotations,
such as **@ConditionalOnMissingBean, @ConditionalOnClass, and @ConditionalOnProperty**, etc.

These bean definitions will be registered in the **ApplicationContext** only if those conditions are matched.

You can also explore many other AutoConfiguration classes in **spring-boot-autoconfigure-{version}.jar**, such as:

*   `org.springframework.boot.autoconfigure.web.DispatcherServletAutoConfiguration`
*   `org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaAutoConfiguration`
*   `org.springframework.boot.autoconfigure.data.jpa.JpaRepositoriesAutoConfiguration`
*   `org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration`

    etc.

I hope you now have an understanding of how Spring Boot auto-configuration works by using various AutoConfiguration classes
along with **@Conditional** features.

# Why Spring Boot?

Spring is a very popular Java-based framework for building web and enterprise applications. Unlike many other frameworks which focus on only one area, the Spring framework provides a wide variety of features addressing modern business needs via its portfolio of projects.

Spring framework provides the flexibility to configure beans in multiple ways, such as **XML**, **Annotations**, and **JavaConfig**. As the number of features increased, the complexity also increased, and configuring Spring applications became tedious and error-prone.

<!--more-->

The Spring team created Spring Boot to address the complexity of configuration.

But before diving into Spring Boot, we will take a quick look at the Spring framework and see what kind of problems Spring Boot is trying to address.

**In this article, we will cover:**

*   An overview of the Spring framework
*   A web application using Spring MVC and JPA (Hibernate)
*   A quick taste of Spring Boot

## Overview of the Spring framework

If you are a Java developer, then there is a high chance that you might have heard about the Spring framework and probably have used it in your projects. The Spring framework was created primarily as a Dependency Injection container, but it is much more than that.

**Spring is very popular for several reasons:**

*   Spring's dependency injection approach encourages writing testable code.
*   Easy-to-use but powerful database transaction management capabilities.
*   Spring simplifies integration with other Java frameworks like JPA/Hibernate ORM, Struts/JSF, etc., web frameworks.
*   A state-of-the-art Web MVC framework for building web applications.

Along with the Spring framework, there are many other Spring sister projects which help to build applications addressing modern business needs:

*   Spring Data: Simplifies data access from relational and NoSQL data stores.
*   Spring Batch: Provides a powerful batch processing framework.
*   Spring Security: A robust security framework to secure applications.
*   Spring Social: Supports integration with social networking sites like Facebook, Twitter, LinkedIn, GitHub, etc.
*   Spring Integration: An implementation of Enterprise Integration Patterns to facilitate integration with other enterprise applications using lightweight messaging and declarative adapters.

There are many other interesting projects addressing various other modern application development needs. For more information, take a look at http://spring.io/projects.

In the initial days, the Spring framework provided an XML-based approach for configuring beans. Later, Spring introduced XML-based DSLs, Annotations, and JavaConfig-based approaches for configuring beans.

Let us take a quick look at how each of those configuration styles looks.

### XML based configuration

```xml
<bean id="userService" class="com.sivalabs.myapp.service.UserService">
    <property name="userDao" ref="userDao"/>
</bean>

<bean id="userDao" class="com.sivalabs.myapp.dao.JdbcUserDao">
    <property name="dataSource" ref="dataSource"/>
</bean>

<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
    <property name="driverClassName" value="com.mysql.jdbc.Driver"/>
    <property name="url" value="jdbc:mysql://localhost:3306/test"/>
    <property name="username" value="root"/>
    <property name="password" value="secret"/>
</bean>
```

### Annotation based configuration

```java
@Service
public class UserService
{
    private UserDao userDao;

    @Autowired
    public UserService(UserDao dao){
        this.userDao = dao;
    }
    ...
    ...
}
```

```java
@Repository
public class JdbcUserDao
{
    private DataSource dataSource;

    @Autowired
    public JdbcUserDao(DataSource dataSource){
        this.dataSource = dataSource;
    }
    ...
    ...
}
```

### JavaConfig based configuration

```java
@Configuration
public class AppConfig
{
    @Bean
    public UserService userService(UserDao dao){
        return new UserService(dao);
    }

    @Bean
    public UserDao userDao(DataSource dataSource){
        return new JdbcUserDao(dataSource);
    }

    @Bean
    public DataSource dataSource(){
        BasicDataSource dataSource = new BasicDataSource();
        dataSource.setDriverClassName("com.mysql.jdbc.Driver");
        dataSource.setUrl("jdbc:mysql://localhost:3306/test");
        dataSource.setUsername("root");
        dataSource.setPassword("secret");
        return dataSource;
    }
}
```

Wow… Spring provides many approaches for doing the same thing, and we can even mix the approaches as well, like you can use both JavaConfig and Annotation-based configuration styles in the same application.

> That is a lot of flexibility, and it is good in one way and bad in another. People new to the Spring framework may get confused about which approach to follow. As of now, the Spring team is suggesting following the JavaConfig-based approach as it gives more flexibility.

But there is no one-size-fits-all kind of solution. One has to choose the approach based on their own application needs.

OK, now that you have had a glimpse of how various styles of Spring bean configurations look.

**You can find the source code of this article on GitHub at https://github.com/sivaprasadreddy/springboot-tutorials**

Let us take a quick look at what the configuration of a typical Spring MVC + JPA/Hibernate web application looks like.

## A web application using Spring MVC and JPA(Hibernate)

Before getting to know what Spring Boot is and what kind of features it provides, let us take a look at how a typical Spring web application configuration looks, what the pain points are, and then we will discuss how Spring Boot addresses those problems.

**Step 1: Configure Maven Dependencies**

The first thing we need to do is configure all the dependencies required in our **pom.xml**.

```xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
                        http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.sivalabs</groupId>
    <artifactId>springmvc-jpa-demo</artifactId>
    <packaging>war</packaging>
    <version>1.0-SNAPSHOT</version>
    <name>springmvc-jpa-demo</name>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>      
        <failOnMissingWebXml>false</failOnMissingWebXml>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-webmvc</artifactId>
            <version>4.2.4.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.data</groupId>
            <artifactId>spring-data-jpa</artifactId>
            <version>1.9.2.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>jcl-over-slf4j</artifactId>
            <version>1.7.13</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>1.7.13</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
            <version>1.7.13</version>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>1.2.17</version>
        </dependency>
        <dependency>
            <groupId>com.h2database</groupId>
            <artifactId>h2</artifactId>
            <version>1.4.190</version>
        </dependency>
        <dependency>
            <groupId>commons-dbcp</groupId>
            <artifactId>commons-dbcp</artifactId>
            <version>1.4</version>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.38</version>
        </dependency>
        <dependency>
            <groupId>org.hibernate</groupId>
            <artifactId>hibernate-entitymanager</artifactId>
            <version>4.3.11.Final</version>
        </dependency>
        <dependency>
            <groupId>javax.servlet</groupId>
            <artifactId>javax.servlet-api</artifactId>
            <version>3.1.0</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.thymeleaf</groupId>
            <artifactId>thymeleaf-spring4</artifactId>
            <version>2.1.4.RELEASE</version>
        </dependency>
    </dependencies>
</project>
```

We have configured all our Maven jar dependencies to include Spring MVC, Spring Data JPA, JPA/Hibernate, Thymeleaf, and Log4j.

**Step 2: Configure Service/DAO layer beans using JavaConfig.**

```java
@Configuration
@EnableTransactionManagement
@EnableJpaRepositories(basePackages="com.sivalabs.demo")
@PropertySource(value = { "classpath:application.properties" })
public class AppConfig 
{
    @Autowired
    private Environment env;

    @Bean
    public static PropertySourcesPlaceholderConfigurer placeHolderConfigurer()
    {
        return new PropertySourcesPlaceholderConfigurer();
    }
    
    @Value("${init-db:false}")
    private String initDatabase;
    
    @Bean
    public PlatformTransactionManager transactionManager()
    {
        EntityManagerFactory factory = entityManagerFactory().getObject();
        return new JpaTransactionManager(factory);
    }

    @Bean
    public LocalContainerEntityManagerFactoryBean entityManagerFactory()
    {
        LocalContainerEntityManagerFactoryBean factory = new LocalContainerEntityManagerFactoryBean();

        HibernateJpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
        vendorAdapter.setGenerateDdl(Boolean.TRUE);
        vendorAdapter.setShowSql(Boolean.TRUE);

        factory.setDataSource(dataSource());
        factory.setJpaVendorAdapter(vendorAdapter);
        factory.setPackagesToScan("com.sivalabs.demo");

        Properties jpaProperties = new Properties();
        jpaProperties.put("hibernate.hbm2ddl.auto", env.getProperty("hibernate.hbm2ddl.auto"));
        factory.setJpaProperties(jpaProperties);

        factory.afterPropertiesSet();
        factory.setLoadTimeWeaver(new InstrumentationLoadTimeWeaver());
        return factory;
    }

    @Bean
    public HibernateExceptionTranslator hibernateExceptionTranslator()
    {
        return new HibernateExceptionTranslator();
    }
    
    @Bean
    public DataSource dataSource()
    {
        BasicDataSource dataSource = new BasicDataSource();
        dataSource.setDriverClassName(env.getProperty("jdbc.driverClassName"));
        dataSource.setUrl(env.getProperty("jdbc.url"));
        dataSource.setUsername(env.getProperty("jdbc.username"));
        dataSource.setPassword(env.getProperty("jdbc.password"));
        return dataSource;
    }
    
    @Bean
    public DataSourceInitializer dataSourceInitializer(DataSource dataSource) 
    {
        DataSourceInitializer dataSourceInitializer = new DataSourceInitializer();
        dataSourceInitializer.setDataSource(dataSource);
        ResourceDatabasePopulator databasePopulator = new ResourceDatabasePopulator();
        databasePopulator.addScript(new ClassPathResource("data.sql"));
        dataSourceInitializer.setDatabasePopulator(databasePopulator);
        dataSourceInitializer.setEnabled(Boolean.parseBoolean(initDatabase));
        return dataSourceInitializer;
    }   
}
```

In our `AppConfig.java` configuration class, we have done the following:

*   Marked it as a Spring Configuration class using the **@Configuration** annotation.
*   Enabled Annotation-based transaction management using **@EnableTransactionManagement**.
*   Configured **@EnableJpaRepositories** to indicate where to look for Spring Data JPA repositories.
*   Configured a PropertyPlaceHolder bean using the **@PropertySource** annotation and a **PropertySourcesPlaceholderConfigurer** bean definition, which loads properties from the **application.properties** file.
*   Defined beans for **DataSource**, JPA **EntityManagerFactory**, and **JpaTransactionManager**.
*   Configured a **DataSourceInitializer** bean to initialize the database by executing the **data.sql** script on application startup.

We need to configure property placeholder values in **application.properties** in `src/main/resources` as follows:

```properties
jdbc.driverClassName=com.mysql.jdbc.Driver
jdbc.url=jdbc:mysql://localhost:3306/test
jdbc.username=root
jdbc.password=admin
init-db=true
hibernate.dialect=org.hibernate.dialect.MySQLDialect
hibernate.show_sql=true
hibernate.hbm2ddl.auto=update
```

We can create a simple SQL script, **data.sql**, to populate sample data into the **USER** table.

```sql
delete from user;
insert into user(id, name) values(1,'Siva');
insert into user(id, name) values(2,'Prasad');
insert into user(id, name) values(3,'Reddy');
```

We can create a **log4j.properties** file with a basic configuration as follows:

```properties
log4j.rootCategory=INFO, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%5p %t %c{2}:%L - %m%n

log4j.category.org.springframework=INFO
log4j.category.com.sivalabs=DEBUG
```

**Step 3: Configure Spring MVC web layer beans**

We will have to configure the Thymeleaf **ViewResolver**, static **ResourceHandlers**, **MessageSource** for i18n, etc.

```java
@Configuration
@ComponentScan(basePackages = { "com.sivalabs.demo"}) 
@EnableWebMvc
public class WebMvcConfig extends WebMvcConfigurerAdapter
{
    @Bean
    public TemplateResolver templateResolver() {
        TemplateResolver templateResolver = new ServletContextTemplateResolver();
        templateResolver.setPrefix("/WEB-INF/views/");
        templateResolver.setSuffix(".html");
        templateResolver.setTemplateMode("HTML5");
        templateResolver.setCacheable(false);
        return templateResolver;
    }

    @Bean
    public SpringTemplateEngine templateEngine() {
        SpringTemplateEngine templateEngine = new SpringTemplateEngine();
        templateEngine.setTemplateResolver(templateResolver());
        return templateEngine;
    }

    @Bean
    public ThymeleafViewResolver viewResolver() {
        ThymeleafViewResolver thymeleafViewResolver = new ThymeleafViewResolver();
        thymeleafViewResolver.setTemplateEngine(templateEngine());
        thymeleafViewResolver.setCharacterEncoding("UTF-8");
        return thymeleafViewResolver;
    }

    @Override
    public void addResourceHandlers(ResourceHandlerRegistry registry)
    {
        registry.addResourceHandler("/resources/**").addResourceLocations("/resources/");
    }

    @Override
    public void configureDefaultServletHandling(DefaultServletHandlerConfigurer configurer)
    {
        configurer.enable();
    }

    @Bean(name = "messageSource")
    public MessageSource configureMessageSource()
    {
        ReloadableResourceBundleMessageSource messageSource = new ReloadableResourceBundleMessageSource();
        messageSource.setBasename("classpath:messages");
        messageSource.setCacheSeconds(5);
        messageSource.setDefaultEncoding("UTF-8");
        return messageSource;
    }
}
```

In our **WebMvcConfig.java** configuration class, we have done the following:

*   Marked it as a Spring Configuration class using the **@Configuration** annotation.
*   Enabled Annotation-based Spring MVC configuration using **@EnableWebMvc**.
*   Configured the Thymeleaf ViewResolver by registering **TemplateResolver**, **SpringTemplateEngine**, and **ThymeleafViewResolver** beans.
*   Registered a ResourceHandlers bean to indicate that requests for static resources with the URI **/resources/\*\*** will be served from the **/resources/** directory.
*   Configured a **MessageSource** bean to load i18n messages from the ResourceBundle **messages-{country-code}.properties** from the classpath.

For now, we do not have any messages to be configured, so create an empty **messages.properties** file in the **src/main/resources** folder.

**Step 4: Register the Spring MVC FrontController servlet, DispatcherServlet.**

Prior to the Servlet 3.x specification, we had to register Servlets/Filters in **web.xml**. Since the Servlet 3.x specification, we can register Servlets/Filters programmatically using **ServletContainerInitializer**.

Spring MVC provides a convenient class, **AbstractAnnotationConfigDispatcherServletInitializer**, to register **DispatcherServlet**.

```java
public class SpringWebAppInitializer extends AbstractAnnotationConfigDispatcherServletInitializer
{

    @Override
    protected Class<?>[] getRootConfigClasses()
    {
        return new Class<?>[] { AppConfig.class};
    }

    @Override
    protected Class<?>[] getServletConfigClasses()
    {
        return new Class<?>[] { WebMvcConfig.class };
    }

    @Override
    protected String[] getServletMappings()
    {
        return new String[] { "/" };
    }

    @Override
    protected Filter[] getServletFilters() {
       return new Filter[]{ new OpenEntityManagerInViewFilter() };
    }
}
```

In our **SpringWebAppInitializer.java** configuration class, we have done the following:

*   We have configured **AppConfig.class** as **RootConfirationClasses**, which will become the parent **ApplicationContext** that contains bean definitions shared by all child (**DispatcherServlet**) contexts.
*   We have configured **WebMvcConfig.class** as **ServletConfigClasses**, which is a child **ApplicationContext** that contains WebMvc bean definitions.
*   We have configured **"/"** as a **ServletMapping**, which means all requests will be handled by **DispatcherServlet**.
*   We have registered **OpenEntityManagerInViewFilter** as a Servlet Filter so that we can lazy-load the JPA Entity lazy collections while rendering the view.

**Step 5: Create a JPA Entity and a Spring Data JPA Repository**

Create a JPA entity, **User.java**, and a Spring Data JPA repository for the User entity.

```java
@Entity
public class User
{
    @Id @GeneratedValue(strategy=GenerationType.AUTO)
    private Integer id;
    private String name;

    //setters and getters
}
```

```java
public interface UserRepository extends JpaRepository<User, Integer>
{
}
```

**Step 6: Create a Spring MVC Controller**

Create a Spring MVC controller to handle the URL **"/"** and render a list of users.

```java
@Controller
public class HomeController
{
    @Autowired UserRepository userRepo;

    @RequestMapping("/")
    public String home(Model model)
    {
        model.addAttribute("users", userRepo.findAll());
        return "index";
    }
}
```

**Step 7: Create a Thymeleaf view, /WEB-INF/views/index.html, to render a list of Users.**

```xml
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" 
      xmlns:th="http://www.thymeleaf.org">
<head>
<meta charset="utf-8"/>
<title>Home</title>
</head>
<body>
    <table>
        <thead>
            <tr>
                <th>Id</th>
                <th>Name</th>
            </tr>
        </thead>
        <tbody>
            <tr th:each="user : ${users}">
                <td th:text="${user.id}">Id</td>
                <td th:text="${user.name}">Name</td>
            </tr>
        </tbody>
    </table>
</body>
</html>
```

We are all set now to run the application. But before that, we need to download and configure a server like
**Tomcat**, **Jetty**, or **Wildfly**, etc., in your IDE.

You can download Tomcat 8 and configure it in your favorite IDE, run the application, and point your browser to
**http://localhost:8080/springmvcjpa-demo**. You should see the list of user details in a table.

Yay… We did it.

**But wait... Isn't it too much work to just show a list of user details pulled from a database table?
Let us be honest and fair. All this configuration is not just for this one use case.
This configuration is the basis for the rest of the application also.**

**But again, this is too much work to do if you want to get up and running quickly.
Another problem with it is, assume you want to develop another Spring MVC application with a similar technical stack.**

**Well, you copy-paste the configuration and tweak it. Right? But remember one thing: if you have to do the same thing again and again,
you should find an automated way to do it.**

Apart from writing the same configuration again and again, do you see any other problems here?

**Well, let me list what problems I am seeing here.**

*   You need to hunt for all the **compatible libraries** for the specific Spring version and configure them.
*   95% of the time, we configure the **DataSource**, **EntityManagerFactory**, **TransactionManager**, etc., beans in the same way. Wouldn't it be great if Spring could do it for me automatically?
*   Similarly, we configure Spring MVC beans like **ViewResolver**, **MessageSource**, etc., in the same way most of the time.

> If Spring can automatically do it for me, that would be awesome!

Imagine, what if Spring is capable of configuring beans automatically?
What if you can customize the automatic configuration using simple, customizable properties?
For example, instead of mapping the DispatcherServlet URL pattern to "/", you want to map it to "/app/".
Instead of putting Thymeleaf views in the "/WEB-INF/views" folder, you may want to place them in the "/WEB-INF/templates/" folder.
So basically, you want Spring to do things automatically but provide the flexibility to override the default configuration in a simpler way?
Well, you are about to enter the world of Spring Boot, where your dreams come true!

## A quick taste of Spring Boot

Welcome to Spring Boot! Spring Boot does exactly what you are looking for. It will do things automatically for you but allows you to override the defaults if you want to.

Instead of explaining in theory, I prefer to explain by example.

So let us implement the same application that we built earlier, but this time using Spring Boot.

**Step 1: Create a Maven-based Spring Boot Project**

Create a Maven project and configure the dependencies as follows:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" 
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
                        http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.sivalabs</groupId>
    <artifactId>hello-springboot</artifactId>
    <packaging>jar</packaging>
    <version>1.0-SNAPSHOT</version>
    <name>hello-springboot</name>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.3.2.RELEASE</version>
    </parent>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <java.version>1.8</java.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-jpa</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-thymeleaf</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
        </dependency>
    </dependencies>
</project>
```

Wow, our **pom.xml** suddenly became so small!

**Step 2: Configure datasource/JPA properties in application.properties in src/main/resources as follows.**

```properties
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://localhost:3306/test
spring.datasource.username=root
spring.datasource.password=admin
spring.datasource.initialize=true
spring.jpa.hibernate.ddl-auto=update
spring.jpa.show-sql=true
```

You can copy the same **data.sql** file into the **src/main/resources** folder.

**Step 3: Create a JPA Entity and a Spring Data JPA Repository Interface for the entity.**

Create **User.java, UserRepository.java**, and **HomeController.java** the same as in the **springmvc-jpa-demo** application.

**Step 4: Create a Thymeleaf view to show a list of users.**

Copy **/WEB-INF/views/index.html** that we created in the **springmvc-jpa-demo** application into the **src/main/resources/templates** folder in our new project.

**Step 5: Create a Spring Boot EntryPoint Class.**

Create a Java class, **Application.java**, with a `main` method as follows:

```java
@SpringBootApplication
public class Application
{
    public static void main(String[] args)
    {
        SpringApplication.run(Application.class, args);
    }
}
```

Now run **Application.java** as a Java Application and point your browser to **http://localhost:8080/**.

You should see the list of users in table format. Cool!

OK, OK, I hear you shouting, "What is going on???"

Let me explain what just happened.

**1. Easy Dependency Management**

*   The first thing to observe is that we are using some dependencies named like **spring-boot-starter-\***.

    Remember I said, "95% of the time, I use the same configuration." So when you add the **springboot-starter-web** dependency, by default, it will pull all the commonly used libraries while developing Spring MVC applications, such as **spring-webmvc, jackson-json, validation-api**, and **tomcat**.
*   We have added the **spring-boot-starter-data-jpa** dependency. This pulls all the **spring-data-jpa** dependencies and also adds **Hibernate** libraries because the majority of applications use Hibernate as the JPA implementation.

**2. Auto-Configuration**

*   Not only does **spring-boot-starter-web** add all these libraries, but it also configures the commonly registered beans like **DispatcherServlet, ResourceHandlers, MessageSource**, etc., with sensible defaults.
*   We also added **spring-boot-starter-thymeleaf**, which not only adds the Thymeleaf library dependencies but also configures **ThymeleafViewResolver** beans automatically.
*   We haven't defined any of the **DataSource, EntityManagerFactory, TransactionManager**, etc., beans, but they are automatically created. **How?**

    If we have any in-memory database drivers like **H2** or **HSQL** in our classpath, then Spring Boot will automatically create an in-memory **DataSource** and then register **EntityManagerFactory, TransactionManager** beans automatically with sensible defaults. But we are using MySQL, so we need to explicitly provide MySQL connection details. We have configured those MySQL connection details in the **application.properties** file, and Spring Boot creates a **DataSource** using these properties.

**3. Embedded Servlet Container Support**

The most important and surprising thing is that we have created a simple Java class annotated with a magical annotation, **@SpringBootApplication**, having a `main` method, and by running that `main`, we are able to run the application and access it at **http://localhost:8080/**.

**Where does the servlet container come from?**

We have added **spring-boot-starter-web**, which pulls **spring-boot-starter-tomcat** automatically, and when we run the `main()` method, it starts Tomcat as an **embedded container** so that we don't have to deploy our application on any externally installed Tomcat server.

By the way, have you observed that our packaging type in **pom.xml** is **‘jar'**, not **‘war'**? Wonderful!

**OK, but what if I want to use the Jetty server instead of Tomcat?**

Simple, exclude **spring-boot-starter-tomcat** from **spring-boot-starter-web** and include **spring-boot-starter-jetty**.

That's it.

But this looks all magical!

I can imagine what you are thinking. You are thinking that Spring Boot looks cool and it is doing a lot of things automatically for me. But I still don't fully understand how it is all really working behind the scenes. Right?

I can understand. Watching a magic show is fun normally, but not in Software Development. Don't worry, we will be looking at each of those things and explaining in detail how things are happening behind the scenes in future articles. But I don't want to overwhelm you by dumping everything on your head right now in this article.

### Summary

In this article, we had a quick overview of various Spring configuration styles and understood the complexity of configuring Spring applications. Also, we had a quick look at Spring Boot by creating a simple web application.

In the next article, we will deep dive into Spring Boot and understand how it works.

[How Spring Boot AutoConfiguration Works?]({{< relref "2016-03-13-how-springboot-autoconfiguration-magic.md" >}})

# Retrying Method Execution using Spring AOP

One of my blog followers sent an email asking me to show an example of **"Real-World Usage of Spring AOP"**. He mentioned that in most examples, the usage of **Spring AOP** is demonstrated for **logging method entry/exit**, **Transaction management**, or **Security checks**. He wanted to know how Spring AOP is being used in **"Real Projects for Real Problems"**.

So I would like to show how I have used Spring AOP in one of my projects to handle a real problem.

<!--more-->

**We won't face some kinds of problems in development phases and will only come to know about them during Load Testing or in production environments.**

**For example:**

*   Remote WebService invocation failures due to network latency issues
*   Database query failures because of Lock exceptions

    etc.

In most cases, just retrying the same operation is sufficient to solve these kinds of failures.

Let us see how we can use Spring AOP to automatically retry method execution if any exception occurs.

We can use Spring AOP's **@Around** advice to create a proxy for those objects whose methods need to be retried and implement the retry logic in an **Aspect**.

Before jumping on to implementing these Spring Advice and Aspect, first, let us write a simple utility to execute a **"Task"** which automatically retries for N times, ignoring the given set of Exceptions.

```java
public interface Task<T> {
	T execute();
}
```

```java
import java.util.HashSet;
import java.util.Set;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class TaskExecutionUtil 
{
	
	private static Logger logger = LoggerFactory.getLogger(TaskExecutionUtil.class);

	@SafeVarargs
	public static <T> T execute(Task<T> task, 
								int noOfRetryAttempts, 
								long sleepInterval, 
								Class<? extends Throwable>... ignoreExceptions) 
	{
		
		if (noOfRetryAttempts < 1) {
			noOfRetryAttempts = 1;
		}
		Set<Class<? extends Throwable>> ignoreExceptionsSet = new HashSet<Class<? extends Throwable>>();
		if (ignoreExceptions != null && ignoreExceptions.length > 0) {
			for (Class<? extends Throwable> ignoreException : ignoreExceptions) {
				ignoreExceptionsSet.add(ignoreException);
			}
		}
		
		logger.debug("noOfRetryAttempts = "+noOfRetryAttempts);
		logger.debug("ignoreExceptionsSet = "+ignoreExceptionsSet);
		
		T result = null;
		for (int retryCount = 1; retryCount <= noOfRetryAttempts; retryCount++) {
			logger.debug("Executing the task. Attemp#"+retryCount);
			try {
				result = task.execute();
				break;
			} catch (RuntimeException t) {
				Throwable e = t.getCause();
				logger.error(" Caught Exception class"+e.getClass());
				for (Class<? extends Throwable> ignoreExceptionClazz : ignoreExceptionsSet) {
					logger.error(" Comparing with Ignorable Exception : "+ignoreExceptionClazz.getName());
					
					if (!ignoreExceptionClazz.isAssignableFrom(e.getClass())) {
						logger.error("Encountered exception which is not ignorable: "+e.getClass());
						logger.error("Throwing exception to the caller");
						
						throw t;
					}
				}
				logger.error("Failed at Retry attempt :" + retryCount + " of : " + noOfRetryAttempts);
				if (retryCount >= noOfRetryAttempts) {
					logger.error("Maximum retrial attempts exceeded.");
					logger.error("Throwing exception to the caller");
					throw t;
				}
				try {
					Thread.sleep(sleepInterval);
				} catch (InterruptedException e1) {
					//Intentionally left blank
				}
			}
		}
		return result;
	}

}
```

I hope this method is self-explanatory. It takes a **Task** and retries **noOfRetryAttempts** times in case the method **task.execute()** throws any Exception, and **ignoreExceptions** indicates what type of exceptions to be ignored while retrying.

Now let us create a Retry annotation as follows:

```java
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public  @interface Retry {
	
	public int retryAttempts() default 3;
	
	public long sleepInterval() default 1000L; //milliseconds
	
	Class<? extends Throwable>[] ignoreExceptions() default { RuntimeException.class };
	
}
```

We will use this **@Retry** annotation to demarcate which methods need to be retried.

Now let us implement the Aspect which applies to methods with the **@Retry** annotation.

```java
import java.lang.reflect.Method;

import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.reflect.MethodSignature;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;

@Component
@Aspect
public class MethodRetryHandlerAspect {
	
	private static Logger logger = LoggerFactory.getLogger(MethodRetryHandlerAspect.class);
	
	@Around("@annotation(com.sivalabs.springretrydemo.Retry)")
	public Object audit(ProceedingJoinPoint pjp) 
	{
		Object result = null;
		result = retryableExecute(pjp);
	    return result;
	}
	
	protected Object retryableExecute(final ProceedingJoinPoint pjp)
	{
		MethodSignature signature = (MethodSignature) pjp.getSignature();
		Method method = signature.getMethod();
		logger.debug("-----Retry Aspect---------");
		logger.debug("Method: "+signature.toString());

		Retry retry = method.getDeclaredAnnotation(Retry.class);
		int retryAttempts = retry.retryAttempts();
		long sleepInterval = retry.sleepInterval();
		Class<? extends Throwable>[] ignoreExceptions = retry.ignoreExceptions();
		
		Task<Object> task = new Task<Object>() {
			@Override
			public Object execute() {
				try {
					return pjp.proceed();
				} catch (Throwable e) {
					throw new RuntimeException(e);
				}
			}
		};
		return TaskExecutionUtil.execute(task, retryAttempts, sleepInterval, ignoreExceptions);
	}
}
```

That's it. We just need some test cases to actually test it.

First, create the **AppConfig.java** configuration class as follows:

```java
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.EnableAspectJAutoProxy;

@Configuration
@ComponentScan
@EnableAspectJAutoProxy
public class AppConfig {

}
```

And a couple of dummy Service beans.

```java
import org.springframework.stereotype.Service;

@Service
public class ServiceA {
	
	private int counter = 1;
	
	public void method1() {
		System.err.println("----method1----");
	}
	
	@Retry(retryAttempts=5, ignoreExceptions={NullPointerException.class})
	public void method2() {
		System.err.println("----method2 begin----");
		if(counter != 3){
			counter++;
			throw new NullPointerException();
		}
		System.err.println("----method2 end----");		
	}
}
```

```java
import java.io.IOException;
import org.springframework.stereotype.Service;

@Service
public class ServiceB {
	
	@Retry(retryAttempts = 2, ignoreExceptions={IOException.class})
	public void method3() {
		System.err.println("----method3----");
		if(1 == 1){
			throw new ArrayIndexOutOfBoundsException();
		}
	}
	
	@Retry
	public void method4() {
		System.err.println("----method4----");
	}
}
```

Finally, write a simple JUnit test to invoke these methods.

```java
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(classes=AppConfig.class)
public class RetryTest 
{
	@Autowired ServiceA svcA;
	@Autowired ServiceB svcB;
	
	@Test
	public void testA()
	{
		svcA.method1();
	}
	
	@Test
	public void testB()
	{
		svcA.method2();
	}
	
	@Test(expected=RuntimeException.class)
	public void testC()
	{
		svcB.method3();
	}
	
	@Test
	public void testD()
	{
		svcB.method4();
	}
}
```

Yeah, I know I could have written these test methods a bit better, but I hope you get the idea.

Run the JUnit tests and observe the log statements to verify whether the method retry is happening in case of an Exception or not.

**Case#1:** When `ServiceA.method1()` is invoked, `MethodRetryHandlerAspect` won't be applied at all.

**Case#2:** When `ServiceA.method2()` is invoked, we are maintaining a counter and throwing a `NullPointerException` two times. But we have marked that method to ignore `NullPointerException`. So it will continue to retry for 5 times. But the 3rd time, the method will be executed normally and exit.

**Case#3:** When `ServiceB.method3()` is invoked, we are throwing an `ArrayIndexOutOfBoundsException`, but that method is marked to ignore only `IOException`.

So this method execution won't be retried and will throw the Exception immediately.

**Case#4:** When `ServiceB.method4()` is invoked, everything is fine, so it should exit in the first attempt itself.

I hope this example demonstrates a good enough real-world usage of Spring AOP. 🙂

# My Life as a Software Developer in India

I am Siva, working as a Software Developer in Hyderabad, India. Working as a Software Developer is very exciting as you see new things coming in every day. If there are no new things, at least we put a new fancy name to some old concept and celebrate (#microservices) :-). But in my opinion, working as a Software Developer in India is a little bit different and hard too. A few things are good, and a few things are bad.

The opinions of Software Developers from other countries on Indian Software Developers have surprised me a lot.

Here, I would like to share how I feel as a Software Developer in the Indian IT Industry.

> These are all purely my opinions; you may or may not agree. If you feel I am generalizing something, I mean the MAJORITY, not ALL.

## My First Job
I got a job in a small organization as a Java developer. Along with me, a couple of my close friends also joined the same organization. I am fortunate that I got a chance to work with very talented people at the very beginning of my career.

My first manager used to force us to use only Notepad++/EditPlus for coding for the first 2 to 3 months so that we would get familiar with how to compile and run programs and be familiar with compile-time and runtime classpaths, etc. On the other hand, our Architect thought another way. One day, he saw me coding in EditPlus, came to me, and asked, "Do you go to war without any weapons?" I had no idea what he was talking about, and he understood that based on my blank face expression. He pulled the keyboard from me, downloaded Lomboz Eclipse (one variant of Eclipse, not active anymore), and explained how to use it. Also, every now and then, he taught me some interesting keyboard shortcuts, and I was amazed at how quickly we can code in an IDE instead of Notepad/EditPlus.

I learned very important things from my manager and architect, like how important it is to understand how things work and how to effectively use tools to speed up the work.

On the other hand, my team lead was a very hands-on, experienced developer. He literally remembered most of the core Java API methods and had a good grasp of design patterns. He is a kind of perfectionist. He never let us commit the code, even if the code indentation was not correct. One day, we were working late at night (12:00 AM) and had to deliver some urgent deliverable. At that time also, he insisted on correcting the indentation, cleaning up imports, removing unused variables, and strictly following TABs or spaces consistently, etc.

But later, we realized how important it is while merging all the code changes from different developers. He always says, "If you are going to do something, do it well. Otherwise, don't do it."

As my other team members were also my close friends, we didn't have any conflicts and no politics at all between us. We used to have Friday night parties from 9 PM to 2 AM, cracking jokes on all the people in the office. So everything was wonderful.

## The Real Journey Starts
After 3 years at my first company, I resigned and joined a BIG company, assuming **a big company means better work, better pay, and a better work culture**. I quickly realized that it is not at all the case! There, I was working on a legacy application that had the most horrible codebase I have ever worked on.

In addition to that, I ended up working with developers who had been with that company for a long time, had stopped learning anything new, had no interest in making the application better, and wanted to do things in the same familiar way, even though it was painful.

As the Indian IT industry is mostly based on the offshore development model, most of the time we end up maintaining legacy codebases. Rarely do we get a chance to work on greenfield projects. Even for those greenfield projects, the key architecture/design aspects will be done by the onsite team.

Fortunately, I got a chance to work on a few greenfield projects and was involved in designing the application from scratch.

In many of the organizations, there is no clear career path for those who want to continue in the Technology Stream and don't want to go into People Management. Even worse, some companies force developers to step into people management after some years of experience.

Different people will have different interests in their lives, and not all developers must be passionate about technology, and that's fine. Some people are 9 AM to 6 PM types, and some just get it done by whatever means and don't care about code quality or maintainability, etc. It is very difficult to find passionate developers in India because most people start programming as part of their job only, not for fun or as a hobby.

At times when I learn something new or do some cool stuff, I look around to share my "wow" moment and find no one who can understand my excitement.

## The New Blogger On The Floor
I was working on a project using the Struts 1.x framework, and I got into some tricky issues. I couldn't find any solution after struggling for 3 days. I kept on debugging and searching all over the books and forums. Finally, someone wrote an article on the exact problem and provided a solution. I followed his solution, and it worked. I gave a million thanks to the blog author. But the thought that some unknown developer's blog helped me a lot stuck in my mind. I too wanted to start a blog. Maybe I too can give something back to the community. So I started my blog, **www.sivalabs.in**, writing about the things that I learn.

Soon my articles got published on community websites like DZone and JavaCodeGeeks. Though my English writing skills are not good enough, I keep on writing articles, and I hope now I am getting a little bit better.

When someone posts a comment saying, "Your article helped me to learn something," I feel very happy. I take a lot of help from the community, like StackOverflow, blogs, forums, etc. If possible, I try to give something back to the community, my 2 cents. 🙂

## Becoming A Book Author

After I started blogging and my articles got published on DZone and JavaCodeGeeks, my blog started getting more and more visitors. One day, I received an email from Packt Publishers asking me whether I was interested in writing a book on **MyBatis**, and I was very excited about it.

Once I agreed to write the book, they explained the writing process and the payment details. They might not have known that I would have written the book even without any pay. I agreed to write the book because of my passion for technology, not to earn money.

When I had almost completed writing my first book, I was asked to write another book on **PrimeFaces**, and I agreed. It took approximately 18 months to write those two books. During all these 18 months, I didn't have any breaks, no weekends, no outings, nothing. Whenever I found time, I kept on writing or reading what I had already written to see if I could make it better.

That was the time I realized how important it is to have some free time in life. Writing a book is incredibly time-consuming. If you are planning to write a book with the motivation of earning money, I would suggest you look for other options!

Finally, one day I got a courier, and I saw the printed copies of my MyBatis book. It feels great to hold the book that you wrote in your hand. I got very emotional that day.

## Challenges for passionate developers in India
There are many issues that Indian Software Developers face.

### Good developers are like war heroes

As most of the projects we get are legacy projects or may be in maintenance mode, the management likes those people who are loyal and stick with the company (even though they don't pay well or treat them like one type of RESOURCE) rather than strong technical people.

Good developers are like war heroes; they are required while starting a new project or creating a core design. Once the core design is ready, other developers can follow the design and add methods to the flow. Just like once the war is over, no one cares where those heroes are, once the core design of the system is ready, many companies won't care about those good developers.

### Many Non-Indian Techies hate Indian developers

Every now and then, I read articles about Indian Software Developers by someone bashing all the Indian software developers, concluding that the whole Indian IT is crap. Well, that is one individual's opinion, and different people have different opinions, and that's perfectly OK.

See this article: https://avinashsingh.wordpress.com/2007/04/05/a-myth-called-the-indian-programmer/ and read the comments. Someone even wants to sponsor Pakistan to bomb India. Very mature people!

The main problem with Indian IT companies is that in order to get a project, they convince the client that they can deliver the project in almost impossible timelines. To meet these crazy timelines, they push managers to get it done, managers force team leads to get it done, and the team leads put deadlines on developers. And here is the biggest problem with Indian developers. Many of the Indian Developers don't know how to say "NO." They simply accept the crazy timelines and try to finish it. And the outcome of this whole process is "unmaintainable, crappy code."

But I always have a few questions in my mind. How does the on-site, non-Indian client believe that the "Indian off-shore team can build it in 6 months when the on-site team estimated 18 months for the same project"? Why don't they ask for an End-To-End Test Suite? Why don't they ask for Code Quality reports?

**Greediness to get it done for the cheapest possible cost! You get what you paid for!**

Those who generalize and say, "The whole of Indian Software Developers are bad," should think before saying things like this. I don't think all the people living in one geographical location will automatically become good or bad. There will be good developers as well as bad developers everywhere. I worked with some onsite developers who are very arrogant. So is it correct to say all the on-site developers are arrogant? That would be ABSOLUTELY WRONG.

I strongly believe that there are great developers and architects also in India, and I have worked with some of them.

There are people like me who want to learn something every day and become a better developer than yesterday. Over the years, I learned "How to learn new things quickly?" So I learn as many things as possible and am ready to work on anything. You want me to work on **Spring** or **JavaEE**; I am fine. You want me to work on **Python** or **RoR**; I am ready. You want me to use **Eclipse** or **NetBeans**; no problem.

## Following the Community
As I said, it is very difficult to find passionate developers in India, so I try to make contacts with passionate developers around the world using Social Networks like Twitter.

I like to attend technical conferences and meet other great developers. But unfortunately, there are very few conferences that happen in India compared to the USA. So I try to watch those conference talks on InfoQ, Parleys, or YouTube, and I enjoy it a lot.

I love the talks by **Venkat Subramaniam** (@venkat_s), which are very informative and funny. I can't stop laughing after reading a few jokes in his book, "Programming Groovy 2: Dynamic Productivity for the Java Developer" (https://pragprog.com/book/vslg2/programming-groovy-2). He is really an amazing man.

**Talks by Venkat Subramaniam on Parleys:** https://www.parleys.com/author/venkat-subramaniam

*   How To Approach Refactoring by Venkat Subramaniam: https://www.youtube.com/watch?v=iGsPeR-SYYo
*   Ten Cool Things We Can Do with Popular JVM Languages: https://www.youtube.com/watch?v=Ulcl2TjHktA
*   33rd Degree 2012 - Pointy-haired bosses and pragmatic programmers - Venkat Subramaniam: https://www.youtube.com/watch?v=lfmKvRaNnUs
*   Scala for the Intrigued: https://www.youtube.com/watch?v=grvvKURwGNg

I am a big fan of **Rod Johnson**, and I love his talks on **Entrepreneurialism** (<http://www.infoq.com/interviews/rod-johnson-entrepreneurialism>) and "**Things I wish I'd known**" (<http://www.infoq.com/presentations/Things-I-Wish-I-d-Known>), in which he shared his journey of building the Spring framework and creating an ecosystem around it. What a great presentation.

I also watched his "**Scala in 2018**" talk (<https://www.youtube.com/watch?v=DBu6zmrZ_50>) and a discussion regarding the same Scala talk on YouTube (<https://www.youtube.com/watch?v=hZlxBRnxzDc>). His way of taking constructive criticism amazed me. He is an awesome man.

I also love the talks by **Uncle Bob** (<https://blog.8thlight.com/uncle-bob/archive.html>), **Martin Fowler** (<http://martinfowler.com/>), and **David Heinemeier Hansson (DHH)**. We can learn a lot of things from their decades of experience.

I follow the talks and blogs of the awesome Java community leaders like:
*   **Antonio Goncalves** @agoncal <http://antoniogoncalves.org/>
*   **Adam Bien** @AdamBien <http://www.adam-bien.com/roller/abien/>
*   **Reza Rahman** @reza_rahman <https://blogs.oracle.com/reza/>
*   **Arun Gupta** @arungupta <http://blog.arungupta.me/>
*   **Josh Long** @starbuxman
*   **Phil Webb** @phillip_webb
*   **Cagatay Civici** @cagataycivici <http://www.primefaces.org/>
*   **Oliver Gierke** @olivergierke <http://olivergierke.de/>
*   **David Blevins** @dblevins <http://www.tomitribe.com/blog/>
*   and many more.

I don't miss even a single blog post or tweet from the rock star bloggers like:
*   **Baeldung** @baeldung <http://www.baeldung.com/>
*   **Petri Kainulainen** @petrikainulaine <http://www.petrikainulainen.net/>
*   **Vlad Mihalcea** @vlad_mihalcea <http://vladmihalcea.com/>
*   **Abhishek Gupta** @abhi_tweeter <https://abhirockzz.wordpress.com/>
*   **Manuel Jordan** @dr_pompeii <http://manueljordanelera.blogspot.in/2014/06/manuel-jordan.html>
*   **Lukas Eder** @lukaseder [http://blog.jooq.org][1]
*   **Trisha Gee** @trisha_gee [http://trishagee.github.io][2]
*   **Thorben Janssen** @thjanssen123 [http://www.thoughts-on-java.org][3]
*   **Nicolas Frankel** @nicolas_frankel [http://blog.frankel.ch][4]
*   and many more.

I love this whole awesome Java community!

## Looking towards a better future

In recent years, things have been changing. Now, the Indian IT industry is not completely dependent on USA-based projects. There is a lot of growth in technology adoption in Indian businesses. But again, unless the thought process changes, nothing is going to get better. Companies should stop expecting **"9 developers to deliver a baby in 1 month"**.

A developer should become more professional and have the guts to say "**NO**" when they are asked to code something they can't implement with good quality.

Anyway, over the years, I have learned that Software Development is not all about technology. The key part is understanding the business domain and communicating with other people.
Hoping things will get better. 🙂

 [1]: http://blog.jooq.org/
 [2]: http://trishagee.github.io/
 [3]: http://www.thoughts-on-java.org/
 [4]: http://blog.frankel.ch/

# A Developers Perspective on Spring vs JavaEE

In the Java community, Spring vs. JavaEE is a never-ending debate. In such debates, people form two groups consisting of evangelists, architects, and hardcore fans of one platform and debate endlessly. Those who participate in the debates may be architects who are responsible for platform selection. But what would developers think about this Spring vs. JavaEE debate?

I am a Java developer who uses both Spring and JavaEE, and I am not part of the Spring or JavaEE fan club. Here, I would like to share my own thoughts on this epic Spring vs. JavaEE debate.

## 1. Business (sometimes political) Aspects
In many organizations, technology selection may not completely depend on the developers' choice. More specifically, if you are working in so-called giant enterprise organizations, there are high chances that there is an Architecture Team that will decide what platform/language/framework/libraries to use in the projects.

In addition to that, large enterprises also consider the following aspects while choosing the technology platform:

*   Maturity of the platform/language/framework/libraries
*   Commercial support
*   Licensing cost, etc.

As a developer, I can hardly influence the decision-making process for any of the above aspects, especially when I am a developer in an offshore development center. So I don't worry too much about these things.

## 2. If you are really good at Spring/JavaEE, then learning the other one shouldn't be difficult
I am always surprised when someone says, "I am a JavaEE expert, but I can't understand Spring," or vice-versa. Both JavaEE and Spring work on the same core APIs (Servlet, JPA, JMS, BeanValidation, etc.); the difference is who is gluing the things together, Spring or the AppServer.

Even though there are some different APIs for things like dependency injection (Spring DI, CDI), REST (JAX-RS, SpringMVC), etc., they look and behave pretty similarly to each other.

Maybe someone can say CDI is more typesafe than Spring DI. Doesn't Spring and CDI behave similarly when:

*   Injection using `@Autowired` or `@Inject` works fine if there is only one Spring/CDI Bean.
*   Injection fails when there are more than one Spring or CDI bean implementations by throwing errors saying, "Found more than one eligible bean that can be injected."
*   You use `@Produces` or `@Bean` annotated methods to provide custom-made objects as bean providers.

As long as they are behaving similarly, I don't care whether they are implemented in a more typesafe manner or use String-based mappings in their internal implementations.

How can one be an expert in Spring and not understand JavaEE, and vice-versa? How much time can it take for a Spring expert to learn JavaEE?

## 3. Which is more "Average Joe developer" friendly
I think by now many people should have realized that the success of a technology may not completely depend on its merits, but also on developers' adoption. The most important thing to realize is, "Not every software developer is a rock star developer. There are more average joe developers than passionate tech ninjas." So, in order for people to adapt to any framework, it should be "Average Joe Developer" friendly.

I think Spring is doing a pretty good job at it by providing more tools like Spring Boot, User Guides, etc. Spring Security, Spring Integration, Spring XD, and Spring Social address modern business needs very well. Also, think about the various templates provided by Spring that make it easy to do things without worrying about boilerplate coding.

JavaEE is also doing very well by introducing JBoss Forge, Wildfly Swarm, etc., to quickly get started. I came across a few JavaEE-based frameworks like PicketLink, which addresses Security requirements, but I felt it is much more complex than it should be.

The point I am trying to convey is, "You can do pretty much everything in JavaEE that you can do with Spring." The difference is which one is giving more out-of-the-box to the average joe developer.

## 4. Lame arguments without context
Whenever the Spring vs. JavaEE debate arises, people form two groups and debate endlessly. Unfortunately, the debates focus on some useless or outdated points.

**XML heavy:**
JavaEE fans first start by saying Spring is XML-heavy, and "I hate XML, blah blah blah." If you are still using a version of Spring older than 2.5 and assuming it is still the same XML-based framework, then my friend, you should wake up and head to http://spring.io.

**EJBs are bad (or) JSF is bad**
Spring fans jump on to bashing EJB and JSF as if they are the same as EJB 2.x or JSF 1.x. If they really look at EJB 3.x and JSF 2.x, then they wouldn't argue about this at all. Don't judge EJB 3.x with your 6-year-old EJB 2.x experience.

**Heavyweight or lightweight**
My interpretation of this 'weight' thing is based on runtime footprint. To my knowledge, when you deploy your managed beans into a JavaEE container, the container will proxy it and inject all enterprise services (Transactions, Security, etc.), and in the case of Spring, it will be done by Spring AOP.
I don't have any metrics to say which is more heavyweight, the Container Proxy or the SpringAOP Proxy, but I guess there may not be a significant difference.

Some people consider the size of the WAR file as its 'weight'. In that case, compare the (JavaEE AppServer + war) size with a (SpringApp with 126 jars) and see which is lightweight. 🙂

**JavaEE is standards-based**
Come on, guys!

**Vendor lock-in**
I think choosing a platform that doesn't make you stick with one particular vendor is good. But going with an option purely based on the ability to move to a different implementation is not correct. How many times in a year do you switch from one server to another? Choosing a platform that doesn't lock you with a vendor is a 'nice to have,' but it should not be a major factor in choosing your platform.

**We don't need external libraries**
This is called "Arguing for the sake of arguing." Show me any real application without any dependencies. If you say, "I will develop my own logging library, I will write my own HTTP client, I will develop my own common-utilities," then you need to look for a slightly more lazy architect/developer who doesn't have the "Re-invent all the wheels" sickness.

## 5. Don't look at the crowd and say, "You are all idiots because you are using X; you should migrate to Y."
This is a common pattern that I observe on many community sites, especially on Reddit. Just post anything related to the JavaEE vs. Spring thing, and there will be two groups who bash the other group like anything because the other group is not using their favorite platform.

Think for a minute. If Spring is not any good, why do so many people use it and love it? If JavaEE is not good, why do so many people switch from Spring to JavaEE? There are so many good things in each platform. Respect others for choosing whatever option they choose. If possible, ask them the reasons why they went with one over the other and learn if you missed anything.

Just saying, "You all are idiots for not using my favorite option," doesn't make them use your favorite technology. In fact, it triggers the thought to come up with a list of points why your favorite platform sucks.

If you really want them to switch to your favorite platform, then show the reasons with code examples. Show them how easy it is to develop applications using your favorite platform with sample applications. Write more articles on commonly faced issues and how to resolve them. Get the "Average Joe Developer" on board with your favorite platform.

As an enthusiastic Java developer, I read the Spring vs. JavaEE discussions, hoping there might be a few things I don't know, such as "in which areas one is better than the other." But I find 70% of the discussions go on lame arguments, which is not very interesting to me.

I wish the Spring and JavaEE camps would fight more and more and make their platforms superior to the other. At the end of the day, no matter who wins the debate, developers will ultimately have more powerful platforms.  
 

# Core Java Training

This Core Java training aims at the beginners who want to learn Java programming starting from scratch.

Please email me at sivalabs.in@gmail.com for further details.

### Core Java Course Topics:

  1. **Java Programming Fundamentals** 
      * Introduction to Java Programming
      * OOPS concepts
      * What are JDK, JVM, JRE?
  2. **Java Basics** 
      * DataTypes, Operators, Statements
      * Classes, Interfaces, Objects
      * Abstract, Static, final etc
      * public, private, protected modifiers, Access specifiers
      * Inheritance
      * package, import
      * Enum, Annotations
  3. **Control Structures** 
      * if-else, while, for, switch etc
  4. **Exception Handling** 
      * try-catch-finally
      * checked-vs-unchecked exceptions
  5. **Java Collections framework** 
      * List, Set, Map etc
      * Sorting, Filtering
      * Searching
  6. **Java Generics** 
      * Parameterized Collections
  7. **Java lang/util packages, File I/O** 
      * String, StringTokenizer, Date
      * Regular Expressions
      * File Reader/Writer, Input/Output Streams etc
  8. **Multi Threading** 
      * Thread, Runnable
      * Synchronization
  9. **Database Programming with JDBC** 
      * Connecting to MySQL Database
      * Performing Read, Write, Update, Delete operations
      * Transaction Management
 10. **Introduction to Java 8 features** 
      * Lambdas
      * Streams
      * DateTime API
 11. **Servlets** 
      * Introduction to Web application development and Servlets
      * Servlet Life Cycle
      * HttpServlet Request, Response, Session, ServletContext
      * Deployment descriptor web.xml
      * Servlet Listeners and Filters
 12. **JSP** 
      * Introduction to MVC and JSP
      * JSP implicit Objects
      * Scriptlets
      * JSTL
      * Custom Tags

Please email me at sivalabs.in@gmail.com for further details.

# Spring Boot Training

This Spring Boot training aims at Java developers who want to learn Spring and Spring Boot starting from scratch.

Please email me at sivalabs.in@gmail.com for further details.

### Spring Boot Course Topics:

1. **Introduction to Spring Framework**
    * Spring Framework Overview
    * Introducing IOC/DI, AOP
    * Spring Portfolio Projects (Spring Security, Spring Data, Spring Integration, Spring Batch etc)

2. **Introduction to Spring Boot**
    * Auto Configuration
    * DevTools

3. **Dependency Injection using Spring**
    * Configuring beans using XML
    * Using Setter, Constructor injection
    * Injecting Primitives, Enums, Collections, Properties
    * AutoWiring Dependencies
    * Bean Scopes
    * Life Cycle of Spring Bean
    * Configuring Beans using Annotations
    * Classpath Scanning for Spring Beans
    * Configuring Beans using JavaConfig

4. **Advanced Dependency Injection**
    * Defining beans using multiple XMLs/JavaConfig Classes
    * Externalizing configuration
    * Environment and Profiles
    * Internationalization using MessageSource
    
5. **Introducing Spring AOP**
    * What is AOP?
    * Practical usages of AOP
    * Introducing AOP terms: Pointcut, JoinPoint, Aspect, Advice, Weaving
    * Types of Advices
    * Implementing Spring Aspects

6. **Data Access using Spring JDBC**
    * Pain points of using JDBC directly
    * Spring JDBC Template
    * Performing CRUD operations using JdbcTemplate
    * Transaction Management

7. **Data Access using Spring ORM using JPA/Hibernate**
    * Introducing JPA/Hibernate
    * How Spring ORM simplifies working with ORMs
    * Performing CRUD operations
    * Transaction Management

8. **Introducing Spring Data JPA**

9. **Developing Web applications using SpringMVC**
    * Introducing SpringMVC features
    * Configuring DispatcherServlet using web.xml and Java based configuration
    * Web Request Life Cycle
    * Creating Controllers using Annotations
    * RequestMapping and Parameter Binding
    * Customizing WebDataBinder initialization
    * Assisting Controllers with the @ControllerAdvice annotation
    * Multipart FileUploading
    * Intercepting requests with a HandlerInterceptor
    * Handling exceptions HandlerExceptionResolver, @ExceptionHandler
    * Creating JSP views using Spring Form Taglibs

10. **Building RESTful Web Services using SpringMVC**
    * Introducing RESTful WebServices
    * SpringMVC support for building RESTful WebServices
    * Mapping the request body with the @RequestBody annotation
    * Mapping the response body with the @ResponseBody annotation
    * Creating REST Controllers with the @RestController annotation
    * Using HttpEntity
    * Consuming RESTful services using RestTemplate

11. **Spring Security**
    * Introduction to Spring Security features
    * Configuring SpringSecurity
    * Custom Login form and Logout
    * Jdbc based Authentication Provider
    * Role-based View rendering using Spring Security Taglib

12. **Exploring Actuator Endpoints**
    * Customizing Actuator Endpoints
    * Securing Actuator Endpoints

13. **Testing Spring Boot Applications**
    * Testing Support in Spring Boot
    * Testing RESTful Services using RestTemplate
    * Testing Secured Controller/Service Methods
    * Testing with Mockito
    

Please email me at sivalabs.in@gmail.com for further details.

# Can MicroServices Architecture Solve All Your Problems?

IT is one field where you can find new things coming every day. These days, the whole developer community websites are flooded with MicroServices and Docker-related stuff. Among them, the idea of MicroServices is very exciting and encourages a better way of building software systems. But as with any architectural style, there will be pros and cons to every approach.

Before discussing the good and bad sides of the MicroServices approach, first, let me say what I understood about MicroServices.

> The MicroServices architecture encourages building small, focused subsystems that can be integrated into the whole system, preferably using the REST protocol.

Now let's discuss various aspects of the MicroServices architecture.

## The dream of every software architect/developer
First of all, the idea of MicroServices is not new at all. From the very beginning, our elders suggested writing classes focusing on a single responsibility and writing methods to do one particular thing and do it well. Also, we were encouraged to build separate modules that can perform some functionally related tasks. Then we bundle all these separate modules together and build an application, delegating the appropriate tasks to their respective modules. This is what we have been trying to do for many years.

But the idea of MicroServices took this approach to the next level, where you can deploy each module as an individual deployable unit, and each service can communicate with any other service based on some agreed-upon protocol (preferably REST, another trendy cool thing :-)).

### So what are the advantages of this MicroServices architecture?

There are plenty.

*   You will have many small services with manageable codebases that are easy to read and understand.
*   You can confidently refactor or rewrite an entire service because there won't be any impact on other services.
*   Each microservice can be deployed independently so that adding new features or upgrading any existing software/hardware platform won't affect other services.
*   You can easily adopt the next cool technology. If one of the microservices is a very critical service and performance is the highest priority, then we can write that particular service using Scala in order to leverage your multi-core hardware support.
*   If you are a service provider company, you can sell each service separately, possibly making better money compared to selling a whole monolithic product.
*   And the most important factor is, the term "MicroService" is cool. 🙂

## What is the other side of the MicroServices architecture?

As with any approach, MicroServices also has some downsides and an associated cost.

> Great power comes with great responsibility. -Uncle Ben

Let us see what the challenges are to implementing a system using a MicroServices architecture.

**The idea of MicroServices is very simple but very complex to implement in reality.**
In a monolithic system, the communication between various subsystems is mostly direct object communication. But in a MicroServices-based system, in order to communicate with other services, you may use REST services, which means additional HTTP call overhead and its inherent issues like network latency, possible communication failures, etc. So we need to consider various aspects while implementing inter-service communication logic, such as retry, fail-over, and service-down scenarios, etc.

**How good is your DevOps infrastructure?**
In order to go with a MicroServices architecture, an organization should have a good DevOps team to properly maintain the dozens of MicroService applications.
Does your organization have a DevOps culture? Or does your organization have the problem of a blame game between Devs and Ops? If your organization doesn't have a good DevOps culture and software/hardware resources, then a MicroServices architecture will be much more difficult to adopt.

## Are we fixing the actual problem at all?
Now many people are saying a MicroServices architecture is better than a Monolithic architecture.
**But is a Monolithic architecture the actual reason why many projects are failing? Will a MicroServices architecture save projects from failing? I guess NO.**

Think, what were the reasons for your previously failed projects. Did those projects fail because of technology issues or people issues?
I have never seen a project that failed because of the wrong technology selection or the wrong architectural approach. But I have seen many projects failing just because of problems with people.

_I feel there are more severe issues than architectural issues that are causing projects to fail, such as:_

*   Having developers without sufficient skills
*   Having developers who don't want to learn anything new
*   Having developers who don't have the courage to say, "NO, we can't do that in that time."
*   Having Architects who abandoned coding years ago
*   Having Architects who think they know everything and don't need to listen to their developers' pain
*   Having Managers who just blame the developers for not meeting the imposed deadlines without ever asking the developers for timelines

These are the real problems that are causing project failures.

**Now, do you think just moving to a MicroServices architecture will save IT without fixing these problems?**

Continuously innovating new ways of doing things is awesome and is required to move ahead. At the same time, assuming "the next cool methodology/technology will fix all the problems" is also wrong.

> So those of you who are just about to jump on the MicroServices boat... THINK. FIX THE REAL PROBLEMS FIRST. You can't fill a bottle that has a hole at its bottom.

# Clean Code: Don't mix different levels of abstractions

We spend more time reading code than writing it. So if the code is more readable, it will obviously increase developer productivity.

Many people associate the readability of code with coding conventions like following standard naming conventions, closing files, DB resources, etc. When it comes to code reviews, most people focus on these trivial things only, like checking for naming convention violations or whether resources are properly released in a finally block.

Do we need "Senior Resources" on the team (I hate to call a human being a "Resource") to do these things? Tools like Findbugs, PMD, Checkstyle, and Sonar can do that for you. I agree that following a standard naming convention by all team members is a good thing. But that doesn't necessarily increase the readability of the code.

Let us take a simple example. I would like to implement a Fund Transfer use case, and the following are the rules to implement:

*   Source and target accounts should be valid accounts.
*   Check whether the source account has a sufficient amount.
*   Check whether the source account has a provision for Overdraft and check whether this transaction exceeds the overdraft limit.
*   Check for a duplicate transaction with the last transaction. If the source, target accounts, and amount are the same as the last transaction, consider it a duplicate transaction.
*   If everything is fine, then transfer the amount to the target account.

Assume we have the following implementation for the above use case:

```java
public class UglyMoneyTransferService 
{
	public void transferFunds(Account source, 
	                          Account target, 
	                          BigDecimal amount, 
	                          boolean allowDuplicateTxn) 
	                     throws IllegalArgumentException, RuntimeException 
	{	
	Connection conn = null;
	try {
		conn = DBUtils.getConnection();
		PreparedStatement pstmt = 
		    conn.prepareStatement("Select * from accounts where acno = ?");
		pstmt.setString(1, source.getAcno());
		ResultSet rs = pstmt.executeQuery();
		Account sourceAccount = null;
		if(rs.next()) {
			sourceAccount = new Account();
			//populate account properties from ResultSet
		}
		if(sourceAccount == null){
			throw new IllegalArgumentException("Invalid Source ACNO");
		}
		Account targetAccount = null;
		pstmt.setString(1, target.getAcno());
		rs = pstmt.executeQuery();
		if(rs.next()) {
			targetAccount = new Account();
			//populate account properties from ResultSet
		}
		if(targetAccount == null){
			throw new IllegalArgumentException("Invalid Target ACNO");
		}
		if(!sourceAccount.isOverdraftAllowed()) {
			if((sourceAccount.getBalance() - amount) < 0) {
				throw new RuntimeException("Insufficient Balance");
			}
		}
		else {
			if(((sourceAccount.getBalance()+sourceAccount.getOverdraftLimit()) - amount) < 0) {
				throw new RuntimeException("Insufficient Balance, Exceeding Overdraft Limit");
			}
		}
		AccountTransaction lastTxn = .. ; //JDBC code to obtain last transaction of sourceAccount
		if(lastTxn != null) {
			if(lastTxn.getTargetAcno().equals(targetAccount.getAcno()) && lastTxn.getAmount() == amount && !allowDuplicateTxn) {
			throw new RuntimeException("Duplicate transaction exception");//ask for confirmation and proceed
			}
		}
		sourceAccount.debit(amount);
		targetAccount.credit(amount);
		TransactionService.saveTransaction(source, target,  amount);
	}
	catch(Exception e){
		logger.error("",e);
	}
	finally {
		try { 
			conn.close(); 
		} 
		catch(Exception e){ 
			//Not everything is in your control..sometimes we have to believe in GOD/JamesGosling and proceed
		}
	}
}	
}
```

The above code is readable, right?…because:

1.  We have followed naming conventions like camel-casing variable names.
2.  We have all the open braces ({) on the method definition line.
3.  We have closed the DB Connection in a `finally` block.
4.  We have logged the exception instead of using `System.err.println()`.

And most importantly, it is working as expected.

_**So is it readable and clean code? In my opinion, absolutely not. There are many issues in the above code from a readability perspective.**_

**1. Mixing DB interaction code with business logic.**

**2. Throwing `IllegalArgumentException`, `RuntimeException`, etc., from business methods instead of business-specific exceptions.**

**3. Most importantly, the code is mixed with different levels of abstractions.**

Let me explain what I mean by "different levels of abstractions."

Firstly, from a business perspective, a fund transfer means validating source/target accounts, checking for sufficient balance, checking for an overdraft limit, checking for a duplicate transaction, and making the fund transfer.

From a technical point of view, there are various tasks like fetching Account details from the DB, performing all the business-related checks, throwing Exceptions if there are any violations, properly closing the resources, etc.

But in the above code, everything is mixed together.

_While reading the code, you start looking at JDBC code, and your brain is working in Technical mode. After getting the Account object from the ResultSet, you are checking for null and throwing an Exception if it is null, which is a business requirement. So immediately, you need to switch your mind to Business mode and think, "OK, if the account is invalid, we want to abort the operation immediately."_

_Though you managed to switch between Technical and Business modes, what about making an enhancement to one particular subtask, like "A fund transfer is considered a duplicate only if it matches the last transaction that happened within an hour"? To make that enhancement, you have to go through the entire method because you haven't modularized your sub-tasks, and there is no separation of concerns._

Let's rewrite the above method as follows:

```java
class FundTransferTxn
{
	private Account sourceAccount; 
	private Account targetAccount;
	private BigDecimal amount;
	private boolean allowDuplicateTxn;
	//setters & getters
}

public class CleanMoneyTransferService 
{
	public void transferFunds(FundTransferTxn txn) {
		Account sourceAccount = validateAndGetAccount(txn.getSourceAccount().getAcno());
		Account targetAccount = validateAndGetAccount(txn.getTargetAccount().getAcno());
		checkForOverdraft(sourceAccount, txn.getAmount());
		checkForDuplicateTransaction(txn);
		makeTransfer(sourceAccount, targetAccount, txn.getAmount());
	}
	
	private Account validateAndGetAccount(String acno){
		Account account = AccountDAO.getAccount(acno);
		if(account == null){
			throw new InvalidAccountException("Invalid ACNO :"+acno);
		}
		return account;
	}
	
	private void checkForOverdraft(Account account, BigDecimal amount){
		if(!account.isOverdraftAllowed()){
			if((account.getBalance() - amount) < 0)	{
				throw new InsufficientBalanceException("Insufficient Balance");
			}
		}
		else{
			if(((account.getBalance()+account.getOverdraftLimit()) - amount) < 0){
				throw new ExceedingOverdraftLimitException("Insufficient Balance, Exceeding Overdraft Limit");
			}
		}
	}
	
	private void checkForDuplicateTransaction(FundTransferTxn txn){
		AccountTransaction lastTxn = TransactionDAO.getLastTransaction(txn.getSourceAccount().getAcno());
		if(lastTxn != null)	{
			if(lastTxn.getTargetAcno().equals(txn.getTargetAccount().getAcno()) 
					&& lastTxn.getAmount() == txn.getAmount() 
					&& !txn.isAllowDuplicateTxn())	{
				throw new DuplicateTransactionException("Duplicate transaction exception");
			}
		}
	}
	
	private void makeTransfer(Account source, Account target, BigDecimal amount){
		sourceAccount.debit(amount);
		targetAccount.credit(amount);
		TransactionService.saveTransaction(source, target,  amount);
	}	
}
```

The above improved method does exactly what the initial version is doing, but now it looks a lot better than the earlier version.

*   We have divided the entire task into sub-tasks and implemented each sub-task in a separate method.
*   We have delegated DB interactions to DAOs.
*   We are throwing business-specific Exceptions instead of Java language Exceptions.
*   All in all, we have separated the levels of abstractions.

_At the first level, we have the highest level of abstraction in the `transferFunds(FundTransferTxn txn)` method. By looking at this method, we can understand what we are doing as part of the Fund Transfer operation without worrying much about implementation or technical details._

_At the second level, we have business logic implementation methods `checkForOverdraft`, `checkForDuplicateTransaction`, etc., which perform business logic, again without worrying much about technical details._

_At the lowest level, we have technical implementation details in `AccountDAO` and `TransactionDAO`, which contain DB interaction logic._

So the reader (future developer/maintainer) of your code can easily understand what you are doing at a high level and can dig into the method in which he is interested.

As I said earlier, if we have to make the change to consider the transaction as a duplicate only if it happened within an hour, we can easily understand that `checkForDuplicateTransaction()` is the one we have to look into and make the change.

Happy coding!!

# PrimeFaces Beginner's Guide book published

I am glad to announce that my second book, "PrimeFaces Beginner's Guide," has been published.

As many of us know, PrimeFaces is the leading JSF component library for JSF-based web applications. This "PrimeFaces Beginner's Guide" book targets Java developers with a basic knowledge of JSF and jQuery and covers most of the commonly used PrimeFaces components.

> The good news is that the "PrimeFaces Beginner's Guide" book covers the latest PrimeFaces 4.0 version features, such as the Client-Side Validations (CSV) framework, Dialog Framework, Search Expressions, Sticky Component, and many other enhancements as well.

[![Primefaces Beginners Guide](/images/primefaces-beginners-guide-cover.webp)](https://www.packtpub.com/web-development/primefaces-beginners-guide)

The book contains the following 13 chapters covering various PrimeFaces components:

*   **Chapter 1: Introduction to PrimeFaces**, introduces PrimeFaces, along with details on how to install and configure it by creating a simple "Hello World" application. This chapter also provides an overview of various PrimeFaces features, such as AJAX support, Partial Page Rendering (PPR), and explains the Poll and Remote Command components, which come in very handy at times.
*   **Chapter 2: Introducing Sample Application TechBuzz**, provides an overview of the sample application TechBuzz, which we are going to build incrementally throughout the book, and explains what PrimeFaces components will be used for various requirements. This chapter also provides an overall picture of setting up the development environment with all the required software, such as JDK, IDE, database, and so on.
*   **Chapter 3: Using PrimeFaces Common Utility Components**, introduces various commonly used utility components provided by PrimeFaces, such as Growl, Tooltip, BlockUI, NotificationBar, and so on. This chapter also explains new features introduced in PrimeFaces 4.0, such as Fragment, Sticky, the Search Expression Framework, and the Dialog Framework.
*   **Chapter 4: Introducing PrimeFaces Client-Side Validation Framework**, covers the PrimeFaces Client-Side Validations (CSV) Framework in detail, explaining how to configure and enable client-side validations, integrating the CSV framework with the Bean Validation API, and extending the CSV framework to support custom JSF Validators/Converters and custom Bean Validation Annotations.
*   **Chapter 5: Introducing Text Input Components**, covers various input text components such as InputText, InputTextArea, Password, InputMask, Inplace Editor, and Rich Text Editor. This chapter also explains the AutoComplete component in detail, explaining how to use it with POJO support, multiple item selection, and so on.
*   **Chapter 6: Working with Selection Input Components**, covers several PrimeFaces selection components, including SelectOneMenu, SelectManyMenu, SelectOneRadio, SelectManyCheckbox, PickList, MultiSelectListbox, and so on.
*   **Chapter 7: Introducing Advanced Input Components**, explains how to work with advanced input components, such as Calendar, Rating, Spinner, and Slider. Readers will also learn about uploading and downloading files using the FileUpload and FileDownload components. This chapter also explains how to prevent spam and bots using CAPTCHA validation.
*   **Chapter 8: Working with Data Components**, covers various data components, such as DataList, DataGrid, and explains DataTable features in depth, including pagination, sorting, filtering, row/cell editing, and so on. Readers will also learn about exporting DataTable data to Excel, PDF, and XML using the DataExporter component.
*   **Chapter 9: Introducing Advanced Data Visualization Components**, introduces advanced data visualization components, such as Carousel, TagCloud, and Schedule. Readers will also learn about displaying data in a Tree structure using the Tree and TreeTable components.
*   **Chapter 10: Working with Layout Components**, explains how to create simple and complex layouts using Panel, PanelGrid, AccordionPanel, TabView, and Layout components, and describes various customization options. This chapter also covers creating workflow-style forms using the Wizard component and Portal-like Layouts using the Dashboard component.
*   **Chapter 11: Introducing Navigation Components**, explains how to create different styles of menus using various navigation components, such as Menu, SlideMenu, TieredMenu, Menubar, MegaMenu, TabMenu, and Breadcrumb.
*   **Chapter 12: Drawing Charts**, covers how to visualize data in a graphical format using a variety of chart formats, such as Line, Bar, Pie, and Donut charts. This chapter also introduces how to render charts created by the JFreeChart API.
*   **Chapter 13: Using PrimeFaces Themes**, introduces how to use the built-in themes provided by PrimeFaces and how to create custom themes using the online ThemeRoller tool. Readers will also learn how to dynamically change themes using the ThemeSwitcher component. This chapter also gives a brief overview of various PrimeFaces sibling projects, such as PrimeFaces Extensions, PrimeUI, PrimeFaces Mobile, and PrimeFaces Push.

The book contains loads of code examples along with a decent explanation so that the reader will get hands-on experience with most of the commonly used components.

I would like to thank Packt Publishers for giving me an opportunity to write the "PrimeFaces Beginner's Guide" book. I would especially like to thank Joanne Fitzpatrick, Kranti Berde, and Ritika Dewani for their continuous support throughout the writing process.

I would also like to thank the reviewers Thomas Andraschko, Mauricio Fenoglio, Sudheer Jonna, and Nilesh Namdeo Mali for their great support in catching mistakes/typos and for providing a lot of ideas for improving the quality of the content. In fact, I learned a lot from the reviewers' comments. 🙂

I would like to say a special thanks to Çağatay Çivici (I consider his name a synonym for PASSION) for providing such a great framework to the Java community.

I hope readers will enjoy reading this book. 🙂

# Packt Publishing "Java Persistence With MyBatis3" published

Hurray… My first book, **Java Persistence with MyBatis3**, is published. I would like to thank Packt Publishers for giving me this opportunity to write about my favorite framework, MyBatis.

For most software applications, data persistence is a key and important aspect. In the Java world, we have many ways of implementing the persistence layer, starting from low-level JDBC to fancy ORM frameworks.
JDBC is too low-level an API and requires writing a lot of boilerplate code. On the other hand, we have full-fledged ORM frameworks like JPA (Hibernate, EclipseLink, etc.) which hide the complexity of working with SQL directly by letting developers work with objects and generate SQL based on the RDBMS (Dialect) being used. But each approach has its own set of pros and cons; there is no one-size-fits-all solution. Many large applications are using Hibernate successfully, and many other applications got screwed up by using Hibernate/JPA incorrectly. It is not a problem with JPA/Hibernate; it is simply because JPA/Hibernate may not be the best fit for those applications, or developers don't understand them properly.

**So what if I don't want to use a low-level JDBC API and can't go for full-fledged ORMs? MyBatis to the rescue.**

MyBatis is an SQL Mapper framework that simplifies data persistence logic by hiding all the low-level JDBC code and provides an easy-to-use API. Nothing more, nothing less... No magic… You need to write SQL by yourself. (Whenever I hear someone saying, "By using ORM frameworks, we don't need to know SQL; we can use HQL, and the ORM will take care of it," I just LOL.)

So what exactly does MyBatis offer you?
**1. Manage Resources:** Based on the provided configuration, MyBatis takes care of creating a Connection (pool), Statement/PreparedStatement/CallableStatement, and ResultSet, and closing them once the task is done.

**2. SQL Results <-> Java Beans:** For a SELECT statement, MyBatis will take care of looping through the ResultSet and populating Java Objects. For DML statements, we can pass Java Objects as inputs, and MyBatis will unwrap the data and put it in placeholders based on the provided mapping.

**3. Supports Mapping of One-To-Many/One-To-One mapping:** MyBatis supports mapping of an SQL ResultSet into a Java Object graph based on One-To-Many/One-To-One mapping.

**4. Caching Support:** Caching SQL results is a very common requirement for any non-trivial application. MyBatis provides in-built support for Caching. In addition to that, MyBatis provides support for popular cache libraries like EHCache, HazelCast, etc.

**5. Integration with other frameworks:** MyBatis is very lightweight and works well with popular IoC frameworks like Spring and Guice.

**6. Annotation-based Mappers:** For those XML haters, MyBatis also provides annotation-based mapping.

> **I feel that just as Gradle is the sweet spot between Ant and Maven, MyBatis is the sweet spot between JDBC and ORMs (JPA/Hibernate/EclipseLink).**

{{< figure src="/images/mybatis-3-cover.webp" width="250" height="200" >}}

The **Java Persistence With MyBatis3** book covers:

**Chapter 1 – Getting Started with MyBatis:**
Introduces the MyBatis persistence framework and explains the advantages of using MyBatis instead of plain JDBC. We will also look at how to create a project, install MyBatis framework dependencies with and without the Maven build tool, and configure and use MyBatis.

**Chapter 2 – Bootstrapping MyBatis:**
Covers how to bootstrap MyBatis using XML and Java API-based configuration. We will also learn various MyBatis configuration options, such as type aliases, type handlers, global settings, and so on.

**Chapter 3 – SQL Mappers Using XML:**
This chapter goes in-depth into writing SQL-mapped statements using Mapper XML files. We will learn how to configure simple statements, statements with one-to-one and one-to-many relationships, and map results using ResultMaps. We will also learn how to build dynamic queries, paginated results, and custom ResultSet handling.

**Chapter 4 – SQL Mappers Using Annotations:**
This chapter covers writing SQL-mapped statements using annotations. We will learn how to configure simple statements and statements with one-to-one and one-to-many relationships. We will also look into building dynamic queries using SqlProvider annotations.

**Chapter 5 – Integration with Spring:**
This chapter covers how to integrate MyBatis with the Spring framework. We will learn how to install Spring libraries, register MyBatis beans in the Spring ApplicationContext, inject SqlSession and Mapper beans, and use Spring's annotation-based transaction handling mechanism with MyBatis.

Please let me know if anyone is interested in reviewing the book.

# A bunch of Maven Archetypes for Spring based Projects

Maven is a good project management tool that greatly reduces the amount of time we spend creating Java projects with a proper structure. With so many predefined Maven archetypes, it is even easier to create projects by simply selecting the archetype based on the technologies we need and the type (jar/war/ear) of project we want to create.

However, sometimes those predefined archetypes' structures may not suit our needs well, or we may need some more additions to the pre-configured dependencies/frameworks, etc.

Also, in Eclipse, the default Java compiler level is 1.5, and configuring it to 1.6 every time is frustrating.
So I thought of creating some custom archetypes for the most common combination of technologies/frameworks that I frequently use.

Yes, I am aware of **AppFuse**, which provides a lot more Maven archetypes. But the structure I follow is a bit different, so I thought of creating archetypes that suit my needs/style. 🙂

Most of my projects are Spring-based applications, so I have created the following templates so far:

**1. Quickstart Java App:**
A Simple Java Application with JDK 1.6, SLF4J (Logback/Log4J), and failsafe plugin configuration.

**2. Quickstart Web App:**
A Simple Web Application with JDK 1.6, Servlet 2.5, SLF4J (Logback/Log4J), and failsafe plugin configuration.

**3. Quickstart Spring App:**
A Java project with Spring 3.1.x, MySQL, JdbcTemplate, and Log4j/Logback configuration.

**4. Quickstart SpringMVC App:**
A Web project with SpringMVC 3.1.x, MySQL, JdbcTemplate, jQuery, and Log4j/Logback configuration.

**5. Quickstart SpringMVC-Tiles-SpringSecurity App:**
A Web project with SpringMVC 3.1.x, Apache Tiles, SpringSecurity 3.1.x, jQuery, and Log4j/Logback configuration.

**6. Quickstart SpringMVC-SiteMesh-SpringSecurity App:**
A Web project with SpringMVC 3.1.x, SiteMesh 2.x, SpringSecurity 3.1.x, jQuery, and Log4j/Logback configuration.

**7. Quickstart SpringMVC-JPA2(Hibernate) App:**
A Web project with SpringMVC 3.1.x, JPA2 (Hibernate 4.x), SpringDataJPA, Apache Tiles, jQuery, SpringSecurity, and Log4j/Logback configuration.

**8. Quickstart SpringMVC-MyBatis App:**
A Web project with SpringMVC 3.1.x, MyBatis, jQuery, and Log4j/Logback configuration.

**9. Quickstart Spring-JSF2(PrimeFaces)-JPA2(Hibernate) App:**
A Web project with Spring 3.1.x, JSF2 (PrimeFaces), JPA2 (Hibernate), SpringDataJPA, and Log4j/Logback configuration.

These are the archetypes I have completed so far and uploaded to my GitHub repository: <https://github.com/sivaprasadreddy/maven-archetype-templates>

**I have mentioned how to install them in your local repository in the README file.**

I am planning on writing some more template archetypes, including Spring RESTful Services, Spring-ApacheCXF App, Spring Integration, Spring-JavaEE6, etc.

Stay tuned 🙂

# MyBatis Tutorial : Part4 – Spring Integration

[**MyBatis Tutorial: Part1 &#8211; CRUD Operations**]({{< relref "2012-10-21-mybatis-tutorial-part1-crud-operations.md" >}}) 
  
[**MyBatis Tutorial: Part-2: CRUD operations Using Annotations**]({{< relref "2012-10-21-mybatis-tutorial-part-2-crud-operations-using-annotations.md" >}}) 
  
[**MyBatis Tutorial: Part 3 &#8211; Mapping Relationships**]({{< relref "2012-10-21-mybatis-tutorial-part-3-mapping-relationships.md" >}}) 
  
[**MyBatis Tutorial : Part4 &#8211; Spring Integration**]({{< relref "2012-10-24-mybatis-tutorial-part4-spring-integration.md" >}}) 

MyBatis-Spring is a sub-project of MyBatis that provides Spring integration support, which drastically simplifies MyBatis usage. For those who are familiar with Spring's Dependency Injection process, using MyBatis-Spring is very simple.

First, let us see the process of using MyBatis without Spring.

1.  Create SqlSessionFactory using SqlSessionFactoryBuilder by passing mybatis-config.xml, which contains DataSource properties, a list of Mapper XMLs, TypeAliases, etc.
2.  Create an SqlSession object from SqlSessionFactory.
3.  Get a Mapper instance from SqlSession and execute queries.
4.  Commit or roll back the transaction using the SqlSession object.

With MyBatis-Spring, most of the above steps can be configured in the Spring ApplicationContext, and SqlSession or Mapper instances can be injected into Spring Beans. Then we can use Spring's TransactionManagement features without writing transaction commit/rollback code all over the place.

Now, let us see how we can configure MyBatis+Spring integration.

**Step#1:** Configure MyBatis-Spring dependencies in pom.xml

```xml
<dependency>
  <groupId>junit</groupId>
  <artifactId>junit</artifactId>
  <version>4.10</version>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.mybatis</groupId>
  <artifactId>mybatis</artifactId>
  <version>3.1.1</version>
</dependency>
<dependency>
  <groupId>org.mybatis</groupId>
  <artifactId>mybatis-spring</artifactId>
  <version>1.1.1</version>
</dependency>
<dependency>
   <groupId>org.springframework</groupId>
   <artifactId>spring-context-support</artifactId>
   <version>3.1.1.RELEASE</version>
</dependency>
<dependency>
   <groupId>org.springframework</groupId>
   <artifactId>spring-test</artifactId>
   <version>3.1.1.RELEASE</version>
   <scope>test</scope>
</dependency>
<dependency>
 <groupId>mysql</groupId>
 <artifactId>mysql-connector-java</artifactId>
 <version>5.1.21</version>
 <scope>runtime</scope>
</dependency>
<dependency>
   <groupId>cglib</groupId>
   <artifactId>cglib-nodep</artifactId>
   <version>2.2.2</version>
</dependency>
```

**Step#2:** You don't need to configure Database properties in mybatis-config.xml.

We can configure DataSource in Spring Container and use it to build MyBatis SqlSessionFactory.

Instead of SqlSessionFactoryBuilder, MyBatis-Spring uses org.mybatis.spring.SqlSessionFactoryBean to build SqlSessionFactory.

We can pass dataSource, Mapper XML files locations, typeAliases etc to SqlSessionFactoryBean.

```xml
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource">
  <property name="driverClassName" value="${jdbc.driverClassName}"/>
  <property name="url" value="${jdbc.url}"/>
  <property name="username" value="${jdbc.username}"/>
  <property name="password" value="${jdbc.password}"/>
</bean>
 
<bean id="sqlSessionFactory" class="org.mybatis.spring.SqlSessionFactoryBean">
    <property name="dataSource" ref="dataSource" />
    <property name="typeAliasesPackage" 
              value="com.sivalabs.mybatisdemo.domain"/>
    <property name="mapperLocations" 
              value="classpath*:com/sivalabs/mybatisdemo/mappers/**/*.xml" />
</bean>
```

**Step#3:** Configure SqlSessionTemplate which provides ThreadSafe SqlSession object.

```xml
<bean id="sqlSession" class="org.mybatis.spring.SqlSessionTemplate">
   <constructor-arg index="0" ref="sqlSessionFactory" />
</bean>
```

**Step#4:** To be able to inject Mappers directly we should register org.mybatis.spring.mapper.MapperScannerConfigurer and configure the package name where to find Mapper Interfaces.

```xml
<bean class="org.mybatis.spring.mapper.MapperScannerConfigurer">
   <property name="basePackage" value="com.sivalabs.mybatisdemo.mappers"/>
</bean>
```

**Step#5:** Configure TransactionManager to support Annotation based Transaction support.

```xml
<tx:annotation-driven transaction-manager="transactionManager"/>
 
<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
    <property name="dataSource" ref="dataSource" />
</bean>
```

**Step#6:** Update the Service classes and register them in Spring container.

```java
package com.sivalabs.mybatisdemo.service;

import java.util.List;
import org.apache.ibatis.session.SqlSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

import com.sivalabs.mybatisdemo.domain.User;
import com.sivalabs.mybatisdemo.mappers.UserMapper;

@Service
@Transactional
public class UserService
{
    @Autowired
    private SqlSession sqlSession; //This is to demonstrate injecting SqlSession object
 
    public void insertUser(User user) 
    {
        UserMapper userMapper = sqlSession.getMapper(UserMapper.class);
        userMapper.insertUser(user);
    }

    public User getUserById(Integer userId) 
    {
        UserMapper userMapper = sqlSession.getMapper(UserMapper.class);
        return userMapper.getUserById(userId);
    }
}
```

```java
package com.sivalabs.mybatisdemo.service;

import java.util.List;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import com.sivalabs.mybatisdemo.domain.Blog;
import com.sivalabs.mybatisdemo.mappers.BlogMapper;

@Service
@Transactional
public class BlogService
{
    // This is to demonstrate how to inject Mappers directly
    @Autowired
    private BlogMapper blogMapper; 
 
    public void insertBlog(Blog blog) {
        blogMapper.insertBlog(blog);
    }
 
    public Blog getBlogById(Integer blogId) {
        return blogMapper.getBlogById(blogId);
    }
 
    public List<Blog> getAllBlogs() {
        return blogMapper.getAllBlogs();
    }
}
```

**Note:** When we can directly inject Mappers then why do we need to inject SqlSession objects? Because SqlSession object contains more fine grained method which comes handy at times.

For Example: If we want to get count of how many records got updated by an Update query we can use SqlSession as follows:


`int updatedRowCount = sqlSession.update("com.sivalabs.mybatisdemo.mappers.UserMapper.updateUser", user);`

~~So far I didn't find a way to get the row update count without using SqlSession object.~~

**PS: You can have your interface insert/update/delete methods returning int, then MyBatis returns the number of records updated as an integer.**

**Step#7** Write JUnit Tests to test UserService and BlogService.

```java
package com.sivalabs.mybatisdemo;

import java.util.List;

import org.junit.Assert;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

import com.sivalabs.mybatisdemo.domain.User;
import com.sivalabs.mybatisdemo.service.UserService;

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations="classpath:applicationContext.xml")
public class SpringUserServiceTest 
{
    @Autowired
    private UserService userService;
 
    @Test
    public void testGetUserById() 
    {
      User user = userService.getUserById(1);
      Assert.assertNotNull(user);
      System.out.println(user);
      System.out.println(user.getBlog());
    }
        
    @Test
    public void testUpdateUser() 
    {
        long timestamp = System.currentTimeMillis();
        User user = userService.getUserById(2);
        user.setFirstName("TestFirstName"+timestamp);
        user.setLastName("TestLastName"+timestamp);
        userService.updateUser(user);
        User updatedUser = userService.getUserById(2);
        Assert.assertEquals(user.getFirstName(), updatedUser.getFirstName());
        Assert.assertEquals(user.getLastName(), updatedUser.getLastName());
    }
}
```

```java
package com.sivalabs.mybatisdemo;

import java.util.Date;
import java.util.List;

import org.junit.Assert;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

import com.sivalabs.mybatisdemo.domain.Blog;
import com.sivalabs.mybatisdemo.domain.Post;
import com.sivalabs.mybatisdemo.service.BlogService;

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations="classpath:applicationContext.xml")
public class SpringBlogServiceTest 
{
    @Autowired
    private BlogService blogService;
 
    @Test
    public void testGetBlogById() 
    {
        Blog blog = blogService.getBlogById(1);
        Assert.assertNotNull(blog);
        System.out.println(blog);
        List<Post> posts = blog.getPosts();
        for (Post post : posts) {
            System.out.println(post);
        }
    }
    
    @Test
    public void testInsertBlog() 
    {
         Blog blog = new Blog();
         blog.setBlogName("test_blog_"+System.currentTimeMillis());
         blog.setCreatedOn(new Date());
         
         blogService.insertBlog(blog);
         Assert.assertTrue(blog.getBlogId() != 0);
         Blog createdBlog = blogService.getBlogById(blog.getBlogId());
         Assert.assertNotNull(createdBlog);
         Assert.assertEquals(blog.getBlogName(), createdBlog.getBlogName());
    }
}
```

# MyBatis Tutorial: Part 3 – Mapping Relationships

In this post, let us see how to use MyBatis ResultMap configuration to map relationships.

[**MyBatis Tutorial: Part1 &#8211; CRUD Operations**]({{< relref "2012-10-21-mybatis-tutorial-part1-crud-operations.md" >}}) 
  
[**MyBatis Tutorial: Part-2: CRUD operations Using Annotations**]({{< relref "2012-10-21-mybatis-tutorial-part-2-crud-operations-using-annotations.md" >}}) 
  
[**MyBatis Tutorial: Part 3 &#8211; Mapping Relationships**]({{< relref "2012-10-21-mybatis-tutorial-part-3-mapping-relationships.md" >}}) 
  
[**MyBatis Tutorial : Part4 &#8211; Spring Integration**]({{< relref "2012-10-24-mybatis-tutorial-part4-spring-integration.md" >}}) 

To illustrate, we are considering the following sample domain model:
  
There will be Users and each User may have a Blog and each Blog can contain zero or more posts.

The database structure of the three tables is as follows:

```sql
CREATE TABLE user (
  user_id int(10) unsigned NOT NULL auto_increment,
  email_id varchar(45) NOT NULL,
  password varchar(45) NOT NULL,
  first_name varchar(45) NOT NULL,
  last_name varchar(45) default NULL,
  blog_id int(10) unsigned default NULL,
  PRIMARY KEY  (user_id),
  UNIQUE KEY Index_2_email_uniq (email_id),
  KEY FK_user_blog (blog_id),
  CONSTRAINT FK_user_blog FOREIGN KEY (blog_id) REFERENCES blog (blog_id)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

CREATE TABLE blog (
  blog_id int(10) unsigned NOT NULL auto_increment,
  blog_name varchar(45) NOT NULL,
  created_on datetime NOT NULL,
  PRIMARY KEY  (blog_id)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

CREATE TABLE post (
  post_id int(10) unsigned NOT NULL auto_increment,
  title varchar(45) NOT NULL,
  content varchar(1024) NOT NULL,
  created_on varchar(45) NOT NULL,
  blog_id int(10) unsigned NOT NULL,
  PRIMARY KEY  (post_id),
  KEY FK_post_blog (blog_id),
  CONSTRAINT FK_post_blog FOREIGN KEY (blog_id) REFERENCES blog (blog_id)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

```

Here I am going to explain how to fetch and map *-has-One and One-To-Many result mappings.

```java
package com.sivalabs.mybatisdemo.domain;

public class User 
{
     private Integer userId;
     private String emailId;
     private String password;
     private String firstName;
     private String lastName;
     private Blog blog;
     //setters and getters
}

```

```java
package com.sivalabs.mybatisdemo.domain;

import java.util.ArrayList;
import java.util.Date;
import java.util.List;

public class Blog 
{
     private Integer blogId;
     private String blogName;
     private Date createdOn;
     private List<Post> posts = new ArrayList<Post>();
     //setters and getters
}
```

```java
package com.sivalabs.mybatisdemo.domain;

import java.util.Date;

public class Post 
{
     private Integer postId;
     private String title;
     private String content;
     private Date createdOn;
     //setters and getters
}
```

In mybatis-config.xml, configure type aliases for beans.

```xml
<typeAliases>
  <typeAlias type="com.sivalabs.mybatisdemo.domain.User" alias="User"/>
  <typeAlias type="com.sivalabs.mybatisdemo.domain.Blog" alias="Blog"/>
  <typeAlias type="com.sivalabs.mybatisdemo.domain.Post" alias="Post"/>  
</typeAliases>
```

  
### ***-has-One Result Mapping

In UserMapper.xml, configure sql queries and result maps as follows:

```xml
<mapper namespace="com.sivalabs.mybatisdemo.mappers.UserMapper">

 <resultMap type="User" id="UserResult">
    <id property="userId" column="user_id"/>
    <result property="emailId" column="email_id"/>
    <result property="password" column="password"/>
    <result property="firstName" column="first_name"/>
    <result property="lastName" column="last_name"/>
    <association property="blog" resultMap="BlogResult"/>
 </resultMap>
   
 <resultMap type="Blog" id="BlogResult">
    <id property="blogId" column="blog_id"/>
    <result property="blogName" column="BLOG_NAME"/>
    <result property="createdOn" column="CREATED_ON"/>    
 </resultMap>
 
 <select id="getUserById" parameterType="int" resultMap="UserResult">
     SELECT 
          U.USER_ID, U.EMAIL_ID, U.PASSWORD, U.FIRST_NAME, U.LAST_NAME, 
          B.BLOG_ID, B.BLOG_NAME, B.CREATED_ON
     FROM USER U LEFT OUTER JOIN BLOG B ON U.BLOG_ID=B.BLOG_ID
     WHERE U.USER_ID = #{userId}
 </select>
  
 <select id="getAllUsers" resultMap="UserResult">
    SELECT 
         U.USER_ID, U.EMAIL_ID, U.PASSWORD, U.FIRST_NAME, U.LAST_NAME, 
         B.BLOG_ID, B.BLOG_NAME, B.CREATED_ON
    FROM USER U LEFT OUTER JOIN BLOG B ON U.BLOG_ID=B.BLOG_ID
 </select>
  
</mapper>
```

In JUnit Test, write a method to test the association loading.

```java
public void getUserById() 
{
     SqlSession sqlSession = MyBatisUtil.getSqlSessionFactory().openSession();
     
     try{
          UserMapper userMapper = sqlSession.getMapper(UserMapper.class);
          User user = userMapper.getUserById(1);
          System.out.println(user.getBlog());
     } 
     finally {
      sqlSession.close();
     }
}
```

### One-To-Many Results Mapping

In BlogMapper.xml configure Blog to Posts relationship as follows:

```xml
<mapper namespace="com.sivalabs.mybatisdemo.mappers.BlogMapper">

 <resultMap type="Blog" id="BlogResult">
    <id property="blogId" column="blog_id"/>
    <result property="blogName" column="BLOG_NAME"/>
    <result property="createdOn" column="CREATED_ON"/>
    <collection property="posts" ofType="Post" 
        resultMap="PostResult" columnPrefix="post_"/>
 </resultMap>
   
 <resultMap type="Post" id="PostResult">
    <id property="postId" column="post_id"/>
    <result property="title" column="title"/>
    <result property="content" column="content"/>
    <result property="createdOn" column="created_on"/>
 </resultMap>
   
 <select id="getBlogById" parameterType="int" resultMap="BlogResult">
     SELECT 
          b.blog_id, b.blog_name, b.created_on, 
          p.post_id as post_post_id, p.title as post_title, 
          p.content as post_content, p.created_on as post_created_on
    FROM blog b left outer join post p on b.blog_id=p.blog_id
    WHERE b.BLOG_ID=#{blogId}
 </select>
  
 <select id="getAllBlogs" resultMap="BlogResult">
   SELECT 
        b.blog_id, b.blog_name, b.created_on as blog_created_on, 
        p.post_id as post_post_id, p.title as post_title, 
        p.content as post_content, p.created_on as post_created_on
    FROM blog b left outer join post p on b.blog_id=p.blog_id
 </select>
  
</mapper>
```

In JUnit Test, write a test method to test blog-to-posts relationship mapping.

```java
public void getBlogById() 
{
     SqlSession sqlSession = MyBatisUtil.getSqlSessionFactory().openSession();
     
     try{
         BlogMapper blogMapper = sqlSession.getMapper(BlogMapper.class);
         Blog blog = blogMapper.getBlogById(1);
         System.out.println(blog);
         
         List<Post> posts = blog.getPosts();
         for (Post post : posts) {
            System.out.println(post);
         }
     }
     finally {
        sqlSession.close();
     }
}
```


# MyBatis Tutorial: Part-2: CRUD operations Using Annotations

In this post, I will explain how to perform CRUD operations using MyBatis's annotation support without needing to configure queries in XML mapper files.

[**MyBatis Tutorial: Part1 &#8211; CRUD Operations**]({{< relref "2012-10-21-mybatis-tutorial-part1-crud-operations.md" >}}) 
  
[**MyBatis Tutorial: Part-2: CRUD operations Using Annotations**]({{< relref "2012-10-21-mybatis-tutorial-part-2-crud-operations-using-annotations.md" >}}) 
  
[**MyBatis Tutorial: Part 3 &#8211; Mapping Relationships**]({{< relref "2012-10-21-mybatis-tutorial-part-3-mapping-relationships.md" >}}) 
  
[**MyBatis Tutorial : Part4 &#8211; Spring Integration**]({{< relref "2012-10-24-mybatis-tutorial-part4-spring-integration.md" >}}) 

Step#1: Create a table BLOG and a java domain Object Blog.

```sql
CREATE TABLE  blog 
(
      blog_id int(10) unsigned NOT NULL auto_increment,
      blog_name varchar(45) NOT NULL,
      created_on datetime NOT NULL,
      PRIMARY KEY  (blog_id)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
```

```java
package com.sivalabs.mybatisdemo.domain;

import java.util.Date;

public class Blog 
{
     private Integer blogId;
     private String blogName;
     private Date createdOn;
     
     //Setters and getters
     
     @Override
     public String toString() {
      return "Blog [blogId=" + blogId + ", blogName=" + blogName
        + ", createdOn=" + createdOn + "]";
     }
}
```

Step#2: Create UserMapper.java interface with SQL queries in Annotations.

```java
package com.sivalabs.mybatisdemo.mappers;

import java.util.List;

import org.apache.ibatis.annotations.Delete;
import org.apache.ibatis.annotations.Insert;
import org.apache.ibatis.annotations.Options;
import org.apache.ibatis.annotations.Result;
import org.apache.ibatis.annotations.Results;
import org.apache.ibatis.annotations.Select;
import org.apache.ibatis.annotations.Update;

import com.sivalabs.mybatisdemo.domain.Blog;

public interface BlogMapper 
{
    @Insert("INSERT INTO BLOG(BLOG_NAME, CREATED_ON) 
             VALUES(#{blogName}, #{createdOn})")
    @Options(useGeneratedKeys=true, keyProperty="blogId")
    public void insertBlog(Blog blog);
 
    @Select("SELECT BLOG_ID AS blogId, BLOG_NAME as blogName, 
             CREATED_ON as createdOn FROM BLOG WHERE BLOG_ID=#{blogId}")
    public Blog getBlogById(Integer blogId);

    @Select("SELECT * FROM BLOG ")
    @Results({
      @Result(id=true, property="blogId", column="BLOG_ID"),
      @Result(property="blogName", column="BLOG_NAME"),
      @Result(property="createdOn", column="CREATED_ON")  
    })
    public List<Blog> getAllBlogs();
 
    @Update("UPDATE BLOG SET BLOG_NAME=#{blogName}, CREATED_ON=#{createdOn} 
             WHERE BLOG_ID=#{blogId}")
    public void updateBlog(Blog blog);

    @Delete("DELETE FROM BLOG WHERE BLOG_ID=#{blogId}")
    public void deleteBlog(Integer blogId);
}

```

Step#3: Configure BlogMapper in mybatis-config.xml

```xml
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE configuration
  PUBLIC "-//mybatis.org//DTD Config 3.0//EN"
  "http://mybatis.org/dtd/mybatis-3-config.dtd">
<configuration>
 
 <properties resource="jdbc.properties"/>
 
 <environments default="development">
      <environment id="development">
        <transactionManager type="JDBC"/>
        <dataSource type="POOLED">
           <property name="driver" value="${jdbc.driverClassName}"/>
           <property name="url" value="${jdbc.url}"/>
           <property name="username" value="${jdbc.username}"/>
           <property name="password" value="${jdbc.password}"/>
        </dataSource>
      </environment>
  </environments>
  <mappers>
      <mapper class="com.sivalabs.mybatisdemo.mappers.BlogMapper"/>
  </mappers>
</configuration>
```

Step#4: Create BlogService.java

```java
package com.sivalabs.mybatisdemo.service;

import java.util.List;
import org.apache.ibatis.session.SqlSession;
import com.sivalabs.mybatisdemo.domain.Blog;
import com.sivalabs.mybatisdemo.mappers.BlogMapper;

public class BlogService
{
     public void insertBlog(Blog blog) 
     {
          SqlSession sqlSession = MyBatisUtil
                .getSqlSessionFactory().openSession();
          try{
              BlogMapper blogMapper = sqlSession.getMapper(BlogMapper.class);
              blogMapper.insertBlog(blog);
              sqlSession.commit();
          }
          finally {
              sqlSession.close();
          }
     }

     public Blog getBlogById(Integer blogId) 
     {
          SqlSession sqlSession = MyBatisUtil
                .getSqlSessionFactory().openSession();
          try{
              BlogMapper blogMapper = sqlSession.getMapper(BlogMapper.class);
              return blogMapper.getBlogById(blogId);
          }
          finally {
            sqlSession.close();
          }
     }

     public List<Blog> getAllBlogs() 
     {
          SqlSession sqlSession = MyBatisUtil
                .getSqlSessionFactory().openSession();
          try {
              BlogMapper blogMapper = sqlSession.getMapper(BlogMapper.class);
              return blogMapper.getAllBlogs();
          } 
          finally {
            sqlSession.close();
          }
     }
 
     public void updateBlog(Blog blog) {
          SqlSession sqlSession = MyBatisUtil
                .getSqlSessionFactory().openSession();
          try{
              BlogMapper blogMapper = sqlSession.getMapper(BlogMapper.class);
              blogMapper.updateBlog(blog);
              sqlSession.commit();
          }
          finally {
              sqlSession.close();
          }  
     }

     public void deleteBlog(Integer blogId) 
     {
      SqlSession sqlSession = MyBatisUtil
            .getSqlSessionFactory().openSession();
      try{
          BlogMapper blogMapper = sqlSession.getMapper(BlogMapper.class);
          blogMapper.deleteBlog(blogId);
          sqlSession.commit();
      }
      finally {
          sqlSession.close();
      } 
     }

}
```

Step#5: Create JUnit Test for BlogService methods

```java
package com.sivalabs.mybatisdemo;

import java.util.Date;
import java.util.List;

import org.junit.AfterClass;
import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.Test;

import com.sivalabs.mybatisdemo.domain.Blog;
import com.sivalabs.mybatisdemo.service.BlogService;

public class BlogServiceTest 
{
    private static BlogService blogService;
 
    @BeforeClass
    public static void setup() 
    {
      blogService = new BlogService();
    }
 
    @AfterClass
    public static void teardown() 
    {
        blogService = null;
    }
 
    @Test
    public void testGetBlogById() 
    {
        Blog blog = blogService.getBlogById(1);
        Assert.assertNotNull(blog);
        System.out.println(blog);
    }
    
    @Test
    public void testGetAllBlogs() 
    {
         List<Blog> blogs = blogService.getAllBlogs();
         Assert.assertNotNull(blogs);
         for (Blog blog : blogs) 
         {
           System.out.println(blog);
         } 
    }
    
    @Test
    public void testInsertBlog() 
    {
         Blog blog = new Blog();
         blog.setBlogName("test_blog_"+System.currentTimeMillis());
         blog.setCreatedOn(new Date());
         
         blogService.insertBlog(blog);
         Assert.assertTrue(blog.getBlogId() != 0);
         Blog createdBlog = blogService.getBlogById(blog.getBlogId());
         Assert.assertNotNull(createdBlog);
         Assert.assertEquals(blog.getBlogName(), createdBlog.getBlogName());      
    }
    
    @Test
    public void testUpdateBlog() 
    {
        long timestamp = System.currentTimeMillis();
        Blog blog = blogService.getBlogById(2);
        blog.setBlogName("TestBlogName"+timestamp);
        blogService.updateBlog(blog);
        Blog updatedBlog = blogService.getBlogById(2);
        Assert.assertEquals(blog.getBlogName(), updatedBlog.getBlogName());
    }
    
    @Test
    public void testDeleteBlog() 
    {
        Blog blog = blogService.getBlogById(4);
        blogService.deleteBlog(blog.getBlogId());
        Blog deletedBlog = blogService.getBlogById(4);
        Assert.assertNull(deletedBlog);
    }
}
```

# MyBatis Tutorial: Part1 – CRUD Operations

MyBatis is an SQL Mapper tool which greatly simplifies database programming when compared to using JDBC directly.

[**MyBatis Tutorial: Part1 &#8211; CRUD Operations**]({{< relref "2012-10-21-mybatis-tutorial-part1-crud-operations.md" >}}) 
  
[**MyBatis Tutorial: Part-2: CRUD operations Using Annotations**]({{< relref "2012-10-21-mybatis-tutorial-part-2-crud-operations-using-annotations.md" >}}) 
  
[**MyBatis Tutorial: Part 3 &#8211; Mapping Relationships**]({{< relref "2012-10-21-mybatis-tutorial-part-3-mapping-relationships.md" >}}) 
  
[**MyBatis Tutorial : Part4 &#8211; Spring Integration**]({{< relref "2012-10-24-mybatis-tutorial-part4-spring-integration.md" >}}) 

**Step1:** Create a Maven project and configure MyBatis dependencies.

```xml
<project xmlns="http://maven.apache.org/POM/4.0.0" 
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 <modelVersion>4.0.0</modelVersion>

 <groupId>com.sivalabs</groupId>
 <artifactId>mybatis-demo</artifactId>
 <version>0.0.1-SNAPSHOT</version>
 <packaging>jar</packaging>

 <name>mybatis-demo</name>
 <url>http://maven.apache.org</url>

 <properties>
  <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
 </properties>

 <build>
  <plugins>
   <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>2.3.2</version>
    <configuration>
     <source>1.6</source>
     <target>1.6</target>
     <encoding>${project.build.sourceEncoding}</encoding>
    </configuration>
   </plugin>
  </plugins>
 </build>

 <dependencies>
  <dependency>
   <groupId>junit</groupId>
   <artifactId>junit</artifactId>
   <version>4.10</version>
   <scope>test</scope>
  </dependency>
  <dependency>
      <groupId>org.mybatis</groupId>
      <artifactId>mybatis</artifactId>
      <version>3.1.1</version>
  </dependency>
  <dependency>
     <groupId>mysql</groupId>
     <artifactId>mysql-connector-java</artifactId>
     <version>5.1.21</version>
     <scope>runtime</scope>
  </dependency>
 </dependencies>
</project>
```

**Step#2:** Create the table USER and a Java domain Object User as follows:

```sql
CREATE TABLE  user (
  user_id int(10) unsigned NOT NULL auto_increment,
  email_id varchar(45) NOT NULL,
  password varchar(45) NOT NULL,
  first_name varchar(45) NOT NULL,
  last_name varchar(45) default NULL,
  PRIMARY KEY  (user_id),
  UNIQUE KEY Index_2_email_uniq (email_id)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
```


```java
package com.sivalabs.mybatisdemo.domain;
public class User 
{
    private Integer userId;
    private String emailId;
    private String password;
    private String firstName;
    private String lastName;
    //setters and getters 
    
    @Override
    public String toString() {
        return "User [userId=" + userId + ", emailId=" + emailId
        + ", password=" + password + ", firstName=" + firstName
        + ", lastName=" + lastName + "]";
    }
    
}
```

**Step#3:** Create MyBatis configuration files.

**a) Create jdbc.properties file in src/main/resources folder**

```properties
jdbc.driverClassName=com.mysql.jdbc.Driver
jdbc.url=jdbc:mysql://localhost:3306/mybatis-demo
jdbc.username=root
jdbc.password=admin
```

**b) Create mybatis-config.xml file in src/main/resources folder**

```xml
<?xml version="1.0" encoding="UTF-8" ?>
  <!DOCTYPE configuration
    PUBLIC "-//mybatis.org//DTD Config 3.0//EN"
    "http://mybatis.org/dtd/mybatis-3-config.dtd">
  <configuration>
  
   <properties resource="jdbc.properties"/>
   
   <typeAliases>
    <typeAlias type="com.sivalabs.mybatisdemo.domain.User" alias="User"/>
   </typeAliases>
   
   <environments default="development">
    <environment id="development">
      <transactionManager type="JDBC"/>
      <dataSource type="POOLED">    
         <property name="driver" value="${jdbc.driverClassName}"/>
         <property name="url" value="${jdbc.url}"/>
         <property name="username" value="${jdbc.username}"/>
         <property name="password" value="${jdbc.password}"/>
      </dataSource>
    </environment>
   </environments>
   <mappers>
        <mapper resource="com/sivalabs/mybatisdemo/mappers/UserMapper.xml"/>
    </mappers>
  </configuration>
```

**Step#4:** Create an interface UserMapper.java in src/main/java folder in com.sivalabs.mybatisdemo.mappers package.

```java
package com.sivalabs.mybatisdemo.mappers;

import java.util.List;
import com.sivalabs.mybatisdemo.domain.User;

public interface UserMapper 
{

    public void insertUser(User user);
    
    public User getUserById(Integer userId);
    
    public List<User> getAllUsers();
    
    public void updateUser(User user);
    
    public void deleteUser(Integer userId);

}

```

**Step#5:** Create UserMapper.xml file in src/main/resources folder in com.sivalabs.mybatisdemo.mappers package.

```xml
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN"
  "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
  
<mapper namespace="com.sivalabs.mybatisdemo.mappers.UserMapper">

  <select id="getUserById" parameterType="int" 
            resultType="com.sivalabs.mybatisdemo.domain.User">
     SELECT 
          user_id as userId, 
          email_id as emailId , 
          password, 
          first_name as firstName, 
          last_name as lastName
     FROM USER 
     WHERE USER_ID = #{userId}
  </select>
  
  <!-- Instead of referencing Fully Qualified Class Names 
        we can register Aliases in mybatis-config.xml and use Alias names. -->
  <resultMap type="User" id="UserResult">
        <id property="userId" column="user_id"/>
        <result property="emailId" column="email_id"/>
        <result property="password" column="password"/>
        <result property="firstName" column="first_name"/>
        <result property="lastName" column="last_name"/>   
  </resultMap>
  
  <select id="getAllUsers" resultMap="UserResult">
        SELECT * FROM USER
  </select>
  
  <insert id="insertUser" parameterType="User" useGeneratedKeys="true" 
            keyProperty="userId">
        INSERT INTO USER(email_id, password, first_name, last_name)
        VALUES(#{emailId}, #{password}, #{firstName}, #{lastName})
  </insert>
  
  <update id="updateUser" parameterType="User">
      UPDATE USER 
      SET
         PASSWORD= #{password},
         FIRST_NAME = #{firstName},
         LAST_NAME = #{lastName}
      WHERE USER_ID = #{userId}
  </update>
  
  <delete id="deleteUser" parameterType="int">
      DELETE FROM USER WHERE USER_ID = #{userId}
  </delete>
  
</mapper>
```

**Step#6:** Create MyBatisUtil.java to instantiate SqlSessionFactory.

```java
package com.sivalabs.mybatisdemo.service;

import java.io.IOException;
import java.io.Reader;
import org.apache.ibatis.io.Resources;
import org.apache.ibatis.session.SqlSessionFactory;
import org.apache.ibatis.session.SqlSessionFactoryBuilder;

public class MyBatisUtil 
{
    private static SqlSessionFactory factory;

    private MyBatisUtil() {}
 
    static
    {
          Reader reader = null;
          try {
             reader = Resources.getResourceAsReader("mybatis-config.xml");
          } 
          catch (IOException e) {
            throw new RuntimeException(e.getMessage());
          }
          factory = new SqlSessionFactoryBuilder().build(reader);
    }
 
    public static SqlSessionFactory getSqlSessionFactory() 
    {
        return factory;
    }
}
```

**Step#7:** Create UserService.java in src/main/java folder.

```java
package com.sivalabs.mybatisdemo.service;

import java.util.List;
import org.apache.ibatis.session.SqlSession;
import com.sivalabs.mybatisdemo.domain.User;
import com.sivalabs.mybatisdemo.mappers.UserMapper;

public class UserService
{

	public void insertUser(User user)
	{
		SqlSession sqlSession = MyBatisUtil.getSqlSessionFactory().openSession();
		try
		{
			UserMapper userMapper = sqlSession.getMapper(UserMapper.class);
			userMapper.insertUser(user);
			sqlSession.commit();
		} finally
		{
			sqlSession.close();
		}
	}

	public User getUserById(Integer userId)
	{
		SqlSession sqlSession = MyBatisUtil.getSqlSessionFactory().openSession();
		try
		{
			UserMapper userMapper = sqlSession.getMapper(UserMapper.class);
			return userMapper.getUserById(userId);
		} finally
		{
			sqlSession.close();
		}
	}

	public List<User> getAllUsers()
	{
		SqlSession sqlSession = MyBatisUtil.getSqlSessionFactory().openSession();
		try
		{
			UserMapper userMapper = sqlSession.getMapper(UserMapper.class);
			return userMapper.getAllUsers();
		} finally
		{
			sqlSession.close();
		}
	}

	public void updateUser(User user)
	{
		SqlSession sqlSession = MyBatisUtil.getSqlSessionFactory().openSession();
		try
		{
			UserMapper userMapper = sqlSession.getMapper(UserMapper.class);
			userMapper.updateUser(user);
			sqlSession.commit();
		} finally
		{
			sqlSession.close();
		}
	}

	public void deleteUser(Integer userId)
	{
		SqlSession sqlSession = MyBatisUtil.getSqlSessionFactory().openSession();
		try
		{
			UserMapper userMapper = sqlSession.getMapper(UserMapper.class);
			userMapper.deleteUser(userId);
			sqlSession.commit();
		} finally
		{
			sqlSession.close();
		}
	}

}
```

**Step#8:** Create a JUnit Test class to test UserService methods.

```java
package com.sivalabs.mybatisdemo;

import java.util.List;

import org.junit.AfterClass;
import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.Test;

import com.sivalabs.mybatisdemo.domain.User;
import com.sivalabs.mybatisdemo.service.UserService;

public class UserServiceTest
{
	private static UserService userService;

	@BeforeClass
	public static void setup()
	{
		userService = new UserService();
	}

	@AfterClass
	public static void teardown()
	{
		userService = null;
	}

	@Test
	public void testGetUserById()
	{
		User user = userService.getUserById(1);
		Assert.assertNotNull(user);
		System.out.println(user);
	}

	@Test
	public void testGetAllUsers()
	{
		List<User> users = userService.getAllUsers();
		Assert.assertNotNull(users);
		for (User user : users)
		{
			System.out.println(user);
		}
	}

	@Test
	public void testInsertUser()
	{
		User user = new User();
		user.setEmailId("test_email_" + System.currentTimeMillis() + "@gmail.com");
		user.setPassword("secret");
		user.setFirstName("TestFirstName");
		user.setLastName("TestLastName");

		userService.insertUser(user);
		Assert.assertTrue(user.getUserId() != 0);
		User createdUser = userService.getUserById(user.getUserId());
		Assert.assertNotNull(createdUser);
		Assert.assertEquals(user.getEmailId(), createdUser.getEmailId());
		Assert.assertEquals(user.getPassword(), createdUser.getPassword());
		Assert.assertEquals(user.getFirstName(), createdUser.getFirstName());
		Assert.assertEquals(user.getLastName(), createdUser.getLastName());
	}

	@Test
	public void testUpdateUser()
	{
		long timestamp = System.currentTimeMillis();
		User user = userService.getUserById(2);
		user.setFirstName("TestFirstName" + timestamp);
		user.setLastName("TestLastName" + timestamp);
		userService.updateUser(user);
		User updatedUser = userService.getUserById(2);
		Assert.assertEquals(user.getFirstName(), updatedUser.getFirstName());
		Assert.assertEquals(user.getLastName(), updatedUser.getLastName());
	}

	@Test
	public void testDeleteUser()
	{
		User user = userService.getUserById(4);
		userService.deleteUser(user.getUserId());
		User deletedUser = userService.getUserById(4);
		Assert.assertNull(deletedUser);
	}
}
```

# Keep The Code Clean: WatchDog & SpotTheBug Approach

Before discussing the **WatchDog & SpotTheBug Approach**, let me give a brief context on the need for this.

Three months back, I was asked to write core infrastructure code for our new application, which uses all the latest and greatest technologies.
I have written the infrastructure code and implemented two use cases to demonstrate which logic should go into which layer, and the code looks good (at least to me :-)). Then I moved on to my main project, and I was hearing that the project that I designed (from now onwards, I will refer to this as ProjectA) is going well.

After 3 months, last week, one of the developers of ProjectA came to me to help him in resolving a JAXB Marshalling issue. Then I imported the latest code into Eclipse and started looking into the issue, and I was literally shocked by looking at the messy code. First, I resolved that issue and started looking into the whole code, and I was speechless. How did the code become such a mess in this short span of time? It has only been 3 months.

*   There are Date Formatting methods in almost every Service class (Copy & Paste with different names).
*   There are Domain classes with 58 String properties and setters/getters. The Customer class contains homeAddressLine1, homeAddressLine2, homeCity..., officeAddrLine1, officeAddrLine2, officeCity… There is no Address class.
*   In some classes, XML to Java marshaling is done using JAXB, in some other classes using XStream, and in some other places, constructing XML strings manually even though there is a core utilities module with lots of XML marshaling utility methods.
*   In some classes, SLF4J Logger is used, and in some places, Log4J Logger is being used.

and the list goes on…

So what just happened? Where is the problem?

We started this project by pledging to keep the code clean and highly maintainable/enhanceable. But now it is in the worst possible state.

> **_Somehow it is understandable if the code is legacy and messy because today's latest way of doing things becomes tomorrow's legacy and bad approach. For example, externalizing the application configuration into XML was the way to go sometime back, and now it has become an XML hell with shiny new Annotations. I am pretty sure that in a couple of years, we will see "Get Rid of Annotation Hell by Using Some New Great Way"._**

But, in my case, it is just a 3-month-old project.

When I think about the causes of why that code became such a mess, I end up with a never-ending list of reasons:

*   Tight deadlines
*   Incompetent developers
*   Not using code quality checking tools
*   No code reviews
*   No time to clean the messy code

etc., etc.

So whatever the reason, your code will become messy after some time, especially when more people are working on the project.

The worst part is you can't blame anyone. A developer will say, "I have no time to clean up the code as I have been assigned high-priority tasks." The Tech Lead is busy analyzing and assigning new tasks to developers.
The manager is busy aggregating the team's task status reports to satisfy his boss. The architect is busy designing new modules for new third-party integration services. QA people are busy preparing/executing their test cases for upcoming releases.

So whose responsibility is it to clean the code? Or, in other words, how can we keep the code clean even with all the above-said busy circumstances?

Before explaining how the "WatchDog & SpotTheBug Approach" works, let me tell you another story.

3 years back, I worked on a banking project which was well-designed, well-organized, and had the best-written code I have ever seen so far. That project started almost 10 years back, but still, the code quality is very good. How is it possible?

> **_The only reason is that if any developer checks in the code with some bad code, like adding duplicate utility methods, then within 4 hours, that developer will receive an email from a GUY asking for an explanation of what the need was to add that method when that utility method was already available in the core-utilities module. In case there is no valid reason, that developer has to open a new defect with "Cleaning Bad Code" in the defect title, assign the defect to himself, change the code, and check in the files ASAP._**

With this process, every team member in our team used to triple-check the code before checking it into the repository.

I think this is the best possible way to keep the code clean. By now, you may have a clue on what I mean by "WatchDog". Yes, I called the GUY a WatchDog. First of all, I'm sorry for calling such an important role a Dog, but it better describes what that guy will do. It will bark as soon as it sees some bad code.

**Need for WatchDog:**

As I mentioned above, everyone in the team might be busy with their high-priority tasks. They might not be able to spend time cleaning the code. Also, from the business perspective, adding new customer-requested features might be a higher priority than cleaning the code. Sometimes, even though Business knows that in the long run, there is a chance that the entire application will become un-maintainable if they don't clean up the mess, they will have to satisfy their customer first with some quick new features and will opt for short-term benefits.

We have plenty of Quality Checking tools like PMD, FindBugs, and Sonar. But do these tools suggest creating an Address class instead of repeating all address properties for different types of addresses, as I mentioned above? Do these tools suggest you use the same XML marshaling library across the project? As far as I know, they won't.

So if you really want your software/product to sustain over time, I would suggest hiring a dedicated WatchDog (a Human Being).

**The WatchDog's primary responsibilities would be:**

*   Continuously checking for code smells, duplicate methods, and coding standards violations and sending the report to the entire team.
*   If possible, point out the existing utility to use instead of creating duplicate methods.
*   Checking for design violations like establishing a Database Connection or Transaction management code in the wrong places (the web layer, for example).
*   Checking for cyclic dependencies between modules.
*   Exploring and suggesting well-established, tested generic libraries like Apache Commons-\*.jars, Google Guava instead of writing homegrown solutions (I feel like instead of writing homegrown Cache Management, it's better to use Guava Cache, but YMMV).

So far, so good if the WatchDog does its job well. What if the WatchDog itself is inefficient? What if the WatchDog is not skilled enough to perform its job? Who is going to check whether the WatchDog is doing a good job or not? Here, the **SpotTheBug** program comes into the picture.

**SpotTheBug**
I strongly believe in having a friendly culture to encourage developers to come up with thoughts to better the software.

Every week, each team member should come up with 3 points to better/clean the code. They can be: Bad code Identification, Better Design, New Features, etc.

Instead of just saying that the code is bad, he has to specify why he is feeling that the code is bad, how to rewrite it in a better way, and what the impact would be.

Based on the effectiveness of the points, value points should be given to the developer, and those points should definitely be considered in the performance review (there should be some motivation, right? :-)).

With WatchDog and SpotTheBug programs in place, if the team can identify the bad code before the WatchDog catches it, then it is going to be a negative point for the WatchDog. If the WatchDog continuously gets negative points, then it is time to evaluate the effectiveness of the WatchDog itself.

**_By using this WatchDog & SpotTheBug approach, combined with proper usage of Code Quality Checking Tools (FindBugs, PMD, Sonar), we can make sure the code is clean to the maximum extent._**

# When to use RequestDispatcher.forward() and response.sendRedirect()?

Many people know how RequestDispatcher.forward() and response.sendRedirect() work.

RequestDispatcher.forward() is generally called server-side redirection and is used to forward to a resource within the same application. That resource could be a JSP or another servlet.

response.sendRedirect() is generally called client-side redirection, as it issues a new request from the browser. This method is used to redirect to another resource within the same application, to a resource in some other application running in the same web container, or to any other resource running in some other web container.

There is one more important thing to consider when choosing between forward() and sendRedirect().

Suppose you are on a new customer creation form, you have filled in the data, and you submit it to CreateCustomerServlet. In CreateCustomerServlet, you get all the data entered in the form, insert a row into the database, and show status.jsp, saying "Customer Created successfully."

Assume you use requestDispatcher.forward("status.jsp") to display the status page.
Then, in the browser, the URL remains http://localhost:8080/App/CreateCustomerServlet.

**Now, if the user presses the Refresh (F5) button on the browser, the web container starts executing the request from CreateCustomerServlet. Again, it will insert another duplicate row in the database and show the status page.**

**But in CreateCustomerServlet, if you use response.sendRedirect("status.jsp") to show the status page, the browser URL will be changed to http://localhost:8080/App/status.jsp.
Now, if the user presses F5, the container will start processing status.jsp only. It won't invoke CreateCustomerServlet.**

So if you are doing any data modifications like insertion, updation, or deletion, always use response.sendRedirect().

# Are frameworks making developers dumb?

Last week, I had to conduct interviews to hire senior Java developers with around 5 years of experience. But after the interview process was over, I felt like frameworks make developers' lives easier but, at the same time, make them dumber.

Everyone puts almost all the new frameworks on their resume, claiming they have "Strong, working experience on Spring, Hibernate, Web Services, etc.".

Here is how the interviews went.

**Me:** You have used Spring in your latest project. What are the advantages of using Spring?
**Interviewee:** We can configure beans in XML, and it will take care of instantiating and giving them to us.

**Me:** If Spring is only for creating objects, why is it required at all? I can directly instantiate the dependencies using "new". Why should I configure the class names in XML and get the object from Spring?

**Interviewee:** If tomorrow we want to create another implementation of our interface, we can create a new implementation and update the XML configuration to use the new impl. We don't need to change the Java class and compile it.
**Me:** But you are writing a new Java class, so obviously you need to compile the project.
Regarding the XML change, 99% of the time your XML will be packaged in a WAR or EAR file.
So you will run an ANT script and create the WAR with all the new changes.
Then your point of "if it is XML, I don't need to compile" is not valid.

**Interviewee:** Hmmm, but the Dependency Injection design pattern suggests following this way.
**Me:** OK. I am done with the interview. Our HR will get back to you. 🙂

Interview with another guy:

**Me:** Can you explain your latest project and what technologies you have used?
**Interviewee:** It is some XYZ System, and we are using Spring, Hibernate, and REST WebServices.
**Me:** Ok. Can you explain something about RESTful architecture?
**Interviewee:** We can develop a RESTful application by using @RequestMapping(value="/url", method="POST"). And we can also use PUT and DELETE methods.
**Me:** That's OK, but what is the concept of RESTful architecture?
**Interviewee:** That's what I am explaining. If you use @RequestMapping(value="/url", method="POST"), you can develop a RESTful application.

**Me:** Ok, how good are you at Hibernate?
**Interviewee:** I have been using Hibernate for the last 2 years. I am very good at using Hibernate.
**Me:** What are the advantages of using Hibernate over JDBC?
**Interviewee:** By using it, we don't need to write anything to interact with the database; Hibernate will take care of it.
**Me:** How does Hibernate come to know about your project requirements?
**Interviewee:** If we use Hibernate, it will take care of saving, updating, and fetching data from the database.
**Me:** Uffffffuuuuu… OK.. In your free time, do you read any technology-related blogs?
**Interviewee:** Yeah, why not? That's how I learn Hibernate in-depth.
**Me:** Very good, nice talking to you. Our HR will get back to you. 🙂

The interview process went on like this…

I strongly believe frameworks increase developer productivity. But developers should try to understand how the framework is doing things. You need not learn all the internal working mechanisms of frameworks. If you are really good at Servlets and JSP, then it is very easy to understand any Java Web framework like Struts, Spring MVC, etc. If you aren't good at the basics, then obviously for every other question, the reply would be... "the framework's annotation/xml will execute this."

I strongly recommend that people who want to start their career as a Java developer work on Core Java, Servlets, and JSP for some time. Then, one can understand the frameworks in a proper way.

# 10 things to become an outstanding Java developer

If you are a Java developer and passionate about technology, you can follow the tips below, which will make you an outstanding Java developer.

#### 1. Have a strong foundation and understanding of OO Principles
  
For a Java developer, having a strong understanding of Object-Oriented Programming is a must. Without a strong foundation in OOPS, one can't realize the beauty of an object-oriented programming language like Java. If you don't have a good idea of what OOPS is, even though you are using an OOP language, you may still be coding in a procedural way. Just studying the definitions of OO principles won't help much. You should know how to apply those OO principles when designing a solution in an OO way. So, one should have strong knowledge of object modeling, inheritance, polymorphism, and design patterns.

#### 2. Master the core APIs
  
It doesn't matter how strong you are in terms of theoretical knowledge if you don't know the language constructs and core APIs. In the case of Java, one should have very strong hands-on experience with core APIs like java.lang.*, I/O, Exceptions, Collections, Generics, Threads, JDBC, etc. When it comes to Web application development, no matter which framework you are using, having strong knowledge of Servlets and JSPs is a must.

#### 3. Keep coding
  
Things look simpler when talking about them in theory. We can provide a solution to a problem very easily in theory. But we can realize the depth of the problem when we start implementing our approach. You will come to know the language's limitations or design best practices while coding. So keep coding.

#### 4. Subscribe to forums
  
We are not alone. There are a lot of people working on the same technologies that we are working on. Doing a simple proof of concept on a framework may not present real challenges. But when you start using it on real projects, you will face weird issues and you won't find any solution in their official documentation. When you start working on a new technology, the best and first thing to do is subscribe to that technology's forums. Whatever issue you are facing, someone else in this world might have already faced it and found a solution. And it would be really great if you could answer the questions asked by other forum users.

#### 5. Follow blogs and respond
  
As I already said, you are not alone. There are thousands of enthusiastic technology enthusiasts around the world blogging their insights on technology. You can see different perspectives on the same technology on blogs. Someone might find great features in a technology, while someone else feels it's a stupid framework, giving their own reasons for why they feel that way. So you can see both the good and bad of a technology on blogs. Follow good blogs and respond/comment on posts with your opinion.

#### 6. Read open source frameworks source code
  
A good developer will learn how to use a framework. But if you want to be an outstanding developer, you should study the source code of various successful and popular frameworks, where you can see the internal working mechanism of the framework and a lot of best practices. It will help a lot in using the frameworks in a very effective way.

#### 7. Know the technology trends
  
In open-source software development, technology trends are constantly changing. By the time you get a good handle on a framework, it might become obsolete, and a brand-new framework with a superset of features might have come into the picture. The problem you are trying to solve with your current framework may already be solved by the new framework with a single line of configuration. So keep an eye on what's coming in and what's going out.

#### 8. Keep repeatedly used code snippets/utilities handy
  
Over time, you may need to write/copy-paste the same piece of code/configuration again and again. Keeping those kinds of configuration snippets, like log4.properties, JDBC configuration, etc., and utilities like StringUtils, ReflectionUtils, and DBUtils, will be more helpful. I know this in itself won't make you an outstanding developer. But just imagine a co-developer comes and asks you to help fetch the list of values of a property from a collection of objects, and you just use your ReflectionUtil and provide the solution in a few minutes. That will make you stand out.

#### 9. Know different development methodologies
  
Be familiar with various kinds of methodologies like Agile, SCRUM, XP, Waterfall etc. Nowadays, the choice of development methodology depends on the client. Some clients prefer Agile, and some are happy with the waterfall model. So having an idea of various methodologies would be great.

#### 10. Document/blog your thoughts on technology
  
In your day-to-day job, you may learn new things, new and better ways of doing things, best practices, and architectural ideas. Keep documenting those thoughts or blog about them and share them with the community. Imagine you solved a weird problem that occurred while doing a simple POC and you blogged about it. Maybe a developer elsewhere in the world is facing the same issue on a production-deployed application. Think how important that solution is for that developer. So blog your thoughts; they might be helpful for others or for yourself.

# Welcome note

Hi all,
Welcome, folks.
This is a blog where you can find tutorials on various Java and JavaEE technologies and have discussions regarding technical problems and application development-related issues.

I used to blog at https://sivalabs.blogspot.com/, but now moved to https://sivalabs.in.