Storing data in Minio, making posts editable

Hi. This is the twenty-ninth part of the diary about developing the “Programmers’ diary” blog. Previous part: https://hashnode.programmersdiary.com/programmersdiary-security-upgrades. The open source code of this project is on: https://github.com/TheProgrammersDiary. The first diary entry explains why this project was done: https://medium.com/@vievaldas/developing-a-website-with-microservices-part-1-the-idea-fe6e0a7a96b5.

Next entry:

—————

2024-02-03

Currently we store post content inside post database.

If we have lots of large files in db, it becomes inefficient.

Let’s store post content in Minio instead.

We need to update Post microservice. We will only need changes in this microservice.

Let’s add Minio in pom.xml:

<dependency>
    <groupId>io.minio</groupId>
    <artifactId>minio</artifactId>
    <version>8.5.7</version>
</dependency>

We add MinioConfig:

@Configuration
public class MinioConfig {
    @Value("${minio.url}")
    private String url;
    @Value("${minio.bucket}")
    private String bucket;
    @Value("${minio.username}")
    private String username;
    @Value("${minio.password}")
    private String password;

    @Bean
    public MinioClient minioClient() throws Exception {
        MinioClient client = MinioClient.builder().endpoint(url).credentials(username, password).build();
        if (!client.bucketExists(BucketExistsArgs.builder().bucket(bucket).build())) {
            client.makeBucket(MakeBucketArgs.builder().bucket(bucket).build());
        }
        return client;
    }
}

This class will connect to Minio and create the bucket in which we will store post content.

Next, we need to update our properties file:

minio:
  url: <http://minio:9000>
  bucket: posts
  username: admin
  password: ${minio_password}

Since posts are visible to everyone, and we will store post content, we can use a http connection.

We also inject minio_password from env variables.

In PostRepository remove the content String - this repo is for PostgreSQL and we are migrating the content to Minio. In PostRepository we only leave id, author and title. The id will be used to find the post content in Minio. Also in liquibase file remove content String.

Let’s create a content storage interface:

public interface ContentStorage {
    void upload(String objectId, String content);
    String download(String objectId);
}

and implement it with Minio storage:

@Service
public class MinioStorage implements ContentStorage {
    @Autowired
    private MinioClient minioClient;
    @Value("${minio.bucket}")
    private String bucket;

    @Override
    public void upload(String objectId, String content) {
        try {
            InputStream stream = new ByteArrayInputStream(content.getBytes());
            minioClient.putObject(PutObjectArgs.builder()
                    .bucket(bucket)
                    .object(objectId)
                    .stream(stream, stream.available(), -1)
                    .build());
        } catch (MinioException | IOException | InvalidKeyException | NoSuchAlgorithmException e) {
            throw new RuntimeException("Failed to upload data to minio.");
        }
    }

    @Override
    public String download(String objectId) {
        try {
            InputStream stream = minioClient.getObject(GetObjectArgs.builder()
                    .bucket(bucket)
                    .object(objectId)
                    .build());
            return new BufferedReader(new InputStreamReader(stream))
                    .lines()
                    .collect(Collectors.joining("\\n"));
        } catch (MinioException | IOException | InvalidKeyException | NoSuchAlgorithmException e) {
            throw new RuntimeException("Failed to download data from minio.");
        }
    }
}

Let’s change the Post class to manage file upload/download:

public final class Post {
    private final String author;
    private final String title;
    private final String content;

    public static Post newlyCreated(String author, String title, String content) {
        return new Post(author, title, content);
    }

    public static Optional<Post> existing(String id, PostRepository postRepository, ContentStorage contentStorage) {
        return postRepository
                .findById(id)
                .map(entry -> new Post(entry.getAuthor(), entry.getTitle(), contentStorage.download(entry.getId())));
    }

    private Post(String author, String title, String content) {
        this.author = author;
        this.title = title;
        this.content = content;
    }

    public PostRepository.PostEntry save(PostRepository postRepository, ContentStorage contentStorage) {
        PostRepository.PostEntry entry = postRepository.save(new PostRepository.PostEntry(author, title));
        contentStorage.upload(entry.getId(), content);
        return entry;
    }

    public String getAuthor() {
        return author;
    }

    public String getTitle() {
        return title;
    }

    public String getContent() {
        return content;
    }
}

Let’s update the Controller to deal with the change:

@RestController
@RequestMapping("posts")
final class PostController {
    private final PostRepository postRepository;
    private final ContentStorage contentStorage;

    @Autowired
    PostController(PostRepository postRepository, ContentStorage contentStorage) {
        this.postRepository = postRepository;
        this.contentStorage = contentStorage;
    }

    @GetMapping
    ResponseEntity<Collection<PostRepository.PostEntry>> getAll()
    {
        return ResponseEntity.ok(postRepository.findAll());
    }

    @GetMapping(value = "/{id}")
    ResponseEntity<Post> getById(@PathVariable String id)
    {
        return Post
                .existing(id, postRepository, contentStorage)
                .map(ResponseEntity::ok)
                .orElseThrow(() -> new RestNotFoundException("Post with id: " + id + " not found."));
    }

    @PostMapping(value = "/create")
    ResponseEntity<PostRepository.PostEntry> create(@RequestBody Post post)
    {
        return ResponseEntity.ok(post.save(postRepository, contentStorage));
    }
}

There are also some changes to tests (like adding a Minio container for integration tests or implementing a fake Minio repository for unit tests). All changes are visible in commit: https://github.com/TheProgrammersDiary/Post/commit/3beafff481174776549de80f52b8e93717d16d37 [Note from the future: that’s a big commit. By the time I wrote this I have not yet read Kent Beck’s book Tidy First? Although it focuses on small refactorings its a good book to make your commits small and easy to understand].

With these changes our database will not suffer efficiency penalty, since are not storing post content.

You can also check the changes in GlobalTests: https://github.com/TheProgrammersDiary/Docker/commits/main/?since=2024-02-03&until=2024-02-03. You need to add Minio container to docker compose and provide Minio password where needed. Also, in GlobalTests Github settings Minio password secret is setup.

Another feature for today: post editing functionality. For this feature I will not comment on the code, however you can see it here: https://github.com/TheProgrammersDiary/Post/commit/7aca63d58222b1505523391f3f9c6e206b3d4ace . I will comment on the idea.

In short we need to ensure only owners can edit their posts. Our post database contained only author name which is not unique. In our bounded context email is unique. Since we receive a JWT token, which contains email, we can easily check user rights to edit the post. So I added author email in Post database. I also added version since now we can edit owned posts. In Minio, you can use this format: (postId + “_v” + version) and upload/download the versioned file. In FE dashboard and in post page I show only the most recent version. Here are changes in FE: https://github.com/TheProgrammersDiary/Frontend/commits/main/?since=2024-02-03&until=2024-02-03.

—————

Thanks for reading.

The project logged in this diary is open source, so if you would like to code or suggest changes, please visit https://github.com/TheProgrammersDiary.

Next part: https://hashnode.programmersdiary.com/making-posts-versions-visible.