Vitess: LFX experience blog

It all started in December 2023, when I wanted to learn how databases are scaled. So, I started contributing to Vitess with a theory called 'learning by doing' in my mind.

While I was contributing, I became aware of the fact that Vitess was accepting LFX 2024 (a mentorship program by The Linux Foundation Engineering) applications for one of their projects called "Improve unit test coverage." So, I applied and later in March 2024, received the confirmation that I was selected for the project. Since then, I've been gaining experience on a lot of database areas.

Since then I have worked on:

• Introducing new flag --unmanaged (vttablet), which indicates an unmanaged tablet, i.e. using an external mysql-compatible database. It will help add both better visibility and user experience while setting up unmanaged tablets. This is available starting vitess v20.0.0. Do checkout the release notes for the same here.

• Implementing new in-built MySQL functions directly in the vtgate/evalengine, so we don't have to pass certain queries to mysql-server and we can directly execute these in the vtgate

• Adding e2e and unit tests for a lot of vitess go packages, giving me an insight on how different components of the vitess work. And as we all know, robust unit testing is one of the main parts of developing any software!

This blog is a place of residence for my learning conclusions 😄

MySQL and precision.

It's fascinating how different mysql functions believe what precision should be correct for the result. Since, I've worked on implementing some mysql functions in evalengine, it was important to go through the current implementation of the functions done in mysql-server (it's open-sourced thankfully 😮‍💨). While working on functions that take datetime, time, float64 (double actually), float32, decimal or other precision-bearing typed variable as argument or return as an output, it was important to know how mysql actually processes the precisions and how it decides what should be correct precision of the output. You can have a look at my pull requests where I've implemented some of these functions:

• This PR adds implementation of SEC_TO_TIME func in evalengine. [For more info about SEC_TO_TIME function, visit mysql docs]

• This PR adds implementation of TO_SECONDS func in evalengine. [For more info about TO_SECONDS function, visit mysql docs]

MySQL and collation.

The main thing to keep in mind while implementing MySQL string related functions is how to handle the collation/charset of the arguments and the result. Most of these functions have their own way of handling collation of the arguments and the resultant string. Some of the functions I implemented in the vtgate/evalengine in the vitess are:

• This PR adds implementation of ELT and FIELD functions in evalengine. [For more info about ELT and FIELD functions, visit mysql docs - ELT and mysql docs - FIELD]

• This PR adds implementation of INSERT func in evalengine. [For more info about INSERT function, visit mysql docs]

• This PR adds implementation of SPACE and REVERSE functions in evalengine. [For more info about ELT and FIELD functions, visit mysql docs - REVERSE and mysql docs - SPACE]

Other MySQL functions I've implemented in VtGate.

Wanted to mention some other MySQL functions too, which made me realise why mysql docs are not sufficient for implementing in-built functions, you'll need to get through the mysql-server codebase too 😬. Here's a list of some other MySQL functions I've worked on:

• This PR adds implementation of FROM_DAYS func in evalengine. [For more info about FROM_DAYS function, visit mysql docs]

• This PR adds implementation of LAST_DAY func in evalengine. [For more info about LAST_DAY function, visit mysql docs]

• This PR adds implementation of TO_DAYS func in evalengine. [For more info about TO_DAYS function, visit mysql docs]

• This PR adds implementation of TIME_TO_SEC func in evalengine. [For more info about TIME_TO_SEC function, visit mysql docs]

New flag for Unmanaged VtTablet.

Worked on introducing new flag --unmanaged (vttablet), which indicates an unmanaged tablet, i.e. using an external mysql-compatible database. It will help add both better visibility and user experience while setting up unmanaged tablets. This is available starting vitess v20.0.0. Do checkout the release notes for the same here.

All the changes for implementing this flag can be viewed in this Pull Request: vitessio/14871

Robust Unit Tests.

Worked on adding e2e and unit tests for a lot of vitessgo packages, giving me an insight on how different components of the vitess work.

Food for thought: Tests which only increase the coverage, but doesn't provide any real testing. Should these tests be added for increasing coverage? (PS: Keep in mind CI/CD time increases with every test we add)

Here's a list of pull requests which add unit tests for various packages. (Beware: some of these PRs might even contain more than 1000 lines of code 🧑‍💻)

• test: Add required tests for go/tb

• test: Add required tests for go/unicode2

• test: Add missing tests for go/event/syslogger

• Add check to avoid runtime error and add tests for go/mysql/fastparse

• Remove AppendFloat from go/mysql/format and add required tests

• test: Add required tests for go/mysql/hex

• test: Add missing test for go/bucketpool

• test: Add required tests for go/acl

• test: Add missing tests for go/vt/vtorc/collection

• test: Add required tests for go/netutil

• test: Add required tests for go/stats/opentsdb

• test: Add required tests for go/mysql/collations/charset

• test: Add missing tests for go/mysql/datetime

• test: Add missing tests for go/vt/mysqlctl

• test: Replace t.fatalf with testify require in go/vt/schemamanager

• test: Add e2e tests for replication

• test: Use testify require/assert instead of t.Fatal/Error in go/vt/throttler

• test: Add missing tests and refactor existing tests for go/flagutil

• test: Add tests for go/ioutil and refactor existing

• test: Add required tests for go/logstats

• test: Add missing/required tests for sqltypes and mathstats

• test: Add required tests for vt/key, timer and cache/theine/bf

• test: Add required tests for internal/flag

Evalengine fixes.

Just wanted to mention some other evalengine fixes I pushed.

• This PR fixes evalengine functions for dates on/before 0000-02-29, along with fixes for zero time in evalToTime.
  Earlier, evalengine considered 0000 as a leap year 🙂, while MySQL doesn't. We were also missing the test case of 0000-02-29 in the functions where Date was expected as an argument.

  As a result, when we passed dates before 0000-02-29 we obtained different results from MySQL and evalengine.

• evalengine: Fix temporal cases in MAKETIME
  Precision for time, datetime and timestamp, is the number of decimals in fractional second. Earlier, we missed this in MAKETIME function.

  Previously, the tests failed for MAKETIME(1, 2, time'00:00:01.0000') as the function returned TIME("01:02:01.000000") instead of TIME("01:02:01.0000") which wasn't correct as there are 4 decimals in the fractional second on TIME'00:00:01.0000'.

Thank you so much for reading the blog! I hope you were not bored by all these Pull Requests. 🤭

Wrapping up the blog saying that working with Vitess was the best thing which happened with me in my entire college life.

Do connect with me on Linkedin, and follow me on Instagram. Here's my github handle: @beingnoble03

Drop me a message if you need any help. I'll respond asap. Thanks again.