將 k6 從設計測試到開發再到 CI pipeline 整個完整寫一篇。

2024年9月，Grafana 決定把 k6 的icon改色，並於 v0.54.0 發佈。連 result output 也改成紅色了，還加入 Grafana 的字眼。

---

OpenTelemetry Demo 專案

OTel Demo 專案中本來就有 Python Locust 框架撰寫的測試，我們能翻寫成 k6 的版本來練習。Grafana 有提供一個 OTel demo 電商的線上網站https://otel-demo.field-eng.grafana.net/，我們能以此網站來進行 k6 的測試。

該Locust測試腳本的主要測試對象與行為︰

API端點測試 ︰

基礎服務檢測

• GET / 首頁訪問

  • 洪水攻擊，根據 feature flagloadGeneratorFloodHomepage值，動態調整請求次數

• GET /api/products/{id} 商品詳情查詢

• GET /api/recommendations 商品推薦系統

• GET /api/data/ 廣告推薦系統

購物車流程

• GET /api/cart 查看購物車

• POST /api/cart 添加商品到購物車。購物車操作包含： 隨機商品選擇（10種預設商品） 數量隨機（1-10件） 用戶ID綁定

結帳系統

• POST /api/checkout 結帳操作

• 兩種結帳模式： 單商品結帳 多商品結帳（2-4件隨機組合）

• 使用預先載入的100組用戶資料

瀏覽器行為測試 ︰

UI操作驗證

• 切換貨幣為瑞士法郎(CHF)

• 點擊「Roof Binoculars」商品

• 執行加入購物車按鈕操作

流量識別機制

所有請求添加synthetic_request=true header，方便識別該請求是測試產生的還是其他使用者產生的。

為此我們能根據以上的測試案例，區分成壓力測試以及功能測試。

Test for Performance

k6 官方有一篇專門關於測試性能的測試入門文章，Test for performance 以及 API 測試指南。

主要有，使用 thresholds 確認性能標準（performance criteria）。使用 scenarios 來配置多種負載模型。最重要的是團隊已經定義出 SLO 了，例如 99% 請求應該成功，99%請求應該≦1000ms的延遲，只有定義出 SLO，多種負載模型場景才能測試出系統是否滿足這些 SLO。

因為這裡主要是測試客戶端的性能表現，能參考 R.E.D. 或是 4 Golden Signals 中的 Latency、Traffic、Errors。

這裡我們設定請求失敗率需要小於 1 %，也就是 1000 個請求最多只能有 9 個請求是失誤的。令一個則是 p95 的請求都要在 200ms 內取得回應，也包含了失敗請求的回應（Fail should fast return）。

設定 thresholds 之前我們要先知道 k6 提供了哪些內建指標。

常用內建指標

• checks︰rate類型，我們設定的斷言檢查通過比例，這是最重要的資料正確性檢查。

• http_req_duration︰trend類型，就所有請求加總時間，等於 http_req_sending + http_req_waiting + http_req_receiving（即遠端伺服器處理請求並回應所需的時間，不包括初始的 DNS 查詢/連接時間）。

• http_req_failed︰rate類型，http 回應 status >= 400的次數。

• http_reqs︰counter類型，k6 產生了多少 http 請求。

• browser_web_vital_fid，browser_web_vital_lcp︰k6 browser 產生的指標，用來測量瀏覽器多久能看到內容以及跟元件互動。

• 其他自己看，我幾乎沒在用，除了即時輸出至儀表板會用到，像是收到多少資料大小（data_received）、執行了幾次測試迭代（iterations）、測試持續時間（iteration_duration）、多少 VU 正在執行測試中（vus）。

• 也能自己定義 metrics 設定於 thresholds 中。

我們就能利用內建指標，設定這次測試對象的 thresholds 作為測試的驗收標準（Criteria）。

export const options = {
  thresholds: {
    checks: ['rate>0.95'], // the rate of successful checks should be higher than 95%
    http_req_failed: ['rate<0.01'], // http errors should be less than 1%
    http_req_duration: ['p(95)<150'], // 95% of requests should be below 150ms
  },
};

k6 小知識

系統有 RBAC 的授權功能，我們要檢查該使用者無法存取某些資源，並且搭配用 check 做檢查驗收，check 檢查沒問題，但最終的 http_req_failed 指標卻增加了。

before :

const response = http.get("https://xxx.com/v1/accounts/organizations/yyyy/accounts");

check(response, {
     'expect request accounts status 403': (r) => r.status === 403,
 });

http_req_failed................: 100.00% 1 out of 1

check 是過的, 但我是要檢查確保這使用者真的沒法存取這資源.

但指標結果就是判定 failed +1

原始程式，只要 http status > 400 , 該指標都會被計算。

after : 加入callback斷言（expectedStatuses）

const response = http.get("https://xxx.com/v1/accounts/organizations/yyyy/accounts", {
     responseCallback: http.expectedStatuses(403),
});

http_req_failed................: 0.00% 0 out of 0

k6 API performance testing implements

• api_test.js

這段初版程式碼有些重點︰

• options 中的 thresholds 與 scenarios

• Lift cycle 的 setup，teardown，default。

  

• k6 JS library，像我這裡用到 uuid，和 隨機產生。稍後能用另一個套件重構一下程式。

import http from "k6/http";
import { check, group, sleep } from "k6";
import {
  randomItem,
  randomIntBetween,
  uuidv4,
} from "https://jslib.k6.io/k6-utils/1.6.0/index.js";

const FLOOD_COUNT = __ENV || 5;

const products = [
  "0PUK6V6EV0",
  "1YMWWN1N4O",
  "2ZYFJ3GM2N",
  "66VCHSJNUP",
  "6E92ZMYYFZ",
  "9SIQT8TOJO",
  "L9ECAV7KIM",
  "LS4PSXUNUM",
  "OLJCESPC7Z",
  "HQTGWGPNH4",
];
const people = JSON.parse(open('./people.json'));

export const options = {
  thresholds: {
    checks: ["rate>0.95"], // the rate of successful checks should be higher than 95%
    http_req_failed: ["rate<0.01"], // http errors should be less than 1%
    http_req_duration: ["p(95)<150"], // 95% of requests should be below 150ms
  },
  scenarios: {
    checkout_flow: {
      executor: "ramping-vus",
      //exec: "checkout",
      stages: [
        { duration: "10s", target: 20 },
        { duration: "10s", target: 50 },
      ],
    },
  },
};

// init code
export function setup() {
  return {
    people,
    products,
  };
}

// teardown code
export function teardown(data) {
  console.log("Teardown phase completed.");
}

// VU code
export default function (data) {
  const { people, products } = data;

  group("Home Page", () => {
    floodHome();
  });

  group("Browse Products", () => {
    browseProduct(products);
  });

  group("Recommendations", () => {
    getRecommendations(products);
  });

  group("Advertisements", () => {
    getAds();
  });

  group("Cart Operations", () => {
    viewCart();
    const user = uuidv4();
    const product = randomItem(products);
    addToCart(product, user);
  });

  group("Checkout Flow", () => {
    const user = uuidv4();
    const checkoutPerson = randomItem(people);
    checkout(user, checkoutPerson);
  });

  sleep(randomIntBetween(1, 5));
}


// --- Helper Functions ---

function floodHome() {
  const floodCount = FLOOD_COUNT;
  for (let i = 0; i < floodCount; i++) {
    const res = http.get("https://otel-demo.field-eng.grafana.net/");
    check(res, { "home page success": (r) => r.status === 200 });
  }
}

function browseProduct(products) {
  const product = randomItem(products);
  const res = http.get(`https://otel-demo.field-eng.grafana.net/api/products/${product}`);
  check(res, { "browse product success": (r) => r.status === 200 });
}

function getRecommendations(products) {
  const params = { productIds: [randomItem(products)] };
  const res = http.get("https://otel-demo.field-eng.grafana.net/api/recommendations", { params });
  check(res, { "get recommendations success": (r) => r.status === 200 });
}

function getAds() {
  const categories = [
    "binoculars",
    "telescopes",
    "accessories",
    "assembly",
    "travel",
    "books",
    null,
  ];
  const params = { contextKeys: [randomItem(categories)] };
  const res = http.get("https://otel-demo.field-eng.grafana.net/api/data/", { params });
  check(res, { "get ads success": (r) => r.status === 200 });
}

function viewCart() {
  const res = http.get("https://otel-demo.field-eng.grafana.net/api/cart");
  check(res, { "view cart success": (r) => r.status === 200 });
}


function addToCart(product, user) {
  let res = http.get(`https://otel-demo.field-eng.grafana.net/api/products/${product}`);
  check(res, { "fetch product success": (r) => r.status === 200 });

  const cartItem = {
    item: {
      productId: product,
      quantity: randomIntBetween(1, 10),
    },
    userId: user,
  };
  res = http.post(
    "https://otel-demo.field-eng.grafana.net/api/cart",
    JSON.stringify(cartItem),
    {
      headers: { "Content-Type": "application/json" },
    }
  );
  check(res, { "add to cart success": (r) => r.status === 200 });
}

function checkout(user, checkoutPerson) {
  const checkoutPayload = {
    userId: user,
    email: checkoutPerson.email,
    address: checkoutPerson.address,
    userCurrency: checkoutPerson.userCurrency,
    creditCard: checkoutPerson.creditCard,
  };

  const res = http.post(
    "https://otel-demo.field-eng.grafana.net/api/checkout",
    JSON.stringify(checkoutPayload),
    {
      headers: { "Content-Type": "application/json" },
    }
  );
  check(res, { "checkout success": (r) => r.status === 200 });
}

function checkoutMulti(user, products, people) {
  for (let i = 0; i < randomIntBetween(2, 4); i++) {
    const product = randomItem(products);
    addToCart(product, user);
  }
  const checkoutPerson = randomItem(people);
  checkout(user, checkoutPerson);
}

然後執行測試腳本。

> k6 run api_test.js

         /\      Grafana   /‾‾/  
    /\  /  \     |\  __   /  /   
   /  \/    \    | |/ /  /   ‾‾\ 
  /          \   |   (  |  (‾)  |
 / __________ \  |_|\_\  \_____/ 

     execution: local
        script: api_test.js
        output: -

     scenarios: (100.00%) 1 scenario, 50 max VUs, 50s max duration (incl. graceful stop):
              * checkout_flow: Up to 50 looping VUs for 20s over 2 stages (gracefulRampDown: 30s, gracefulStop: 30s)

INFO[0026] Teardown phase completed.                     source=console

     █ Home Page

     █ Browse Products

       ✓ browse product success

     █ Recommendations

       ✓ get recommendations success

     █ Advertisements

       ✗ get ads success
        ↳  91% — ✓ 107 / ✗ 10

     █ Cart Operations

       ✓ view cart success
       ✓ fetch product success
       ✓ add to cart success

     █ Checkout Flow

       ✓ checkout success

   ✓ checks.........................: 98.77% 809 out of 819
     data_received..................: 889 kB 34 kB/s
     data_sent......................: 170 kB 6.5 kB/s
     group_duration.................: avg=308.7ms  min=10.04µs  med=234.38ms max=1.82s    p(90)=617.97ms p(95)=639.93ms
     http_req_blocked...............: avg=572.16µs min=190ns    med=510ns    max=39.94ms  p(90)=681ns    p(95)=9.19ms  
     http_req_connecting............: avg=162.08µs min=0s       med=0s       max=3.96ms   p(90)=0s       p(95)=2.42ms  
   ✗ http_req_duration..............: avg=230.82ms min=197.66ms med=205.43ms max=1.82s    p(90)=249.9ms  p(95)=309.68ms
       { expected_response:true }...: avg=231.07ms min=197.66ms med=205.36ms max=1.82s    p(90)=250.07ms p(95)=309.87ms
   ✗ http_req_failed................: 1.06%  10 out of 936
     http_req_receiving.............: avg=95.37µs  min=23.08µs  med=75.79µs  max=5.23ms   p(90)=156.16µs p(95)=178.56µs
     http_req_sending...............: avg=47.17µs  min=15.21µs  med=42.92µs  max=186.81µs p(90)=69.08µs  p(95)=80.15µs 
     http_req_tls_handshaking.......: avg=371.57µs min=0s       med=0s       max=9.02ms   p(90)=0s       p(95)=6.2ms   
     http_req_waiting...............: avg=230.67ms min=197.57ms med=205.29ms max=1.82s    p(90)=249.76ms p(95)=309.56ms
     http_reqs......................: 936    35.705284/s
     iteration_duration.............: avg=5.03s    min=2.65s    med=5.25s    max=7.6s     p(90)=6.74s    p(95)=7.05s   
     iterations.....................: 117    4.463161/s
     vus............................: 1      min=1          max=49
     vus_max........................: 50     min=50         max=50


running (26.2s), 00/50 VUs, 117 complete and 0 interrupted iterations
checkout_flow ✓ [======================================] 00/50 VUs  20s
ERRO[0026] thresholds on metrics 'http_req_duration, http_req_failed' have been crossed

來解讀分析這結果。

1. 測試概況：

   • 執行時間：測試總計執行了 26.2 秒。

   • VU：

     • 最大 VU 數量：50。

     • 測試過程中達到的最大 VU 數量：49。

   • 請求總數（http_reqs）：936 次請求。

   • 迭代次數（iterations）：117 次迭代完成。

2. 成功率與錯誤：

   • 總檢查成功率（checks）：98.77%（809 次成功，819 次檢查）。

   • HTTP 請求錯誤率（http_req_failed）：1.06%（10 次失敗，936 次請求）。

   • get_ads API 的成功率：91%（107 次成功，10 次失敗）。

3. 性能指標：

   • http_req_duration：230.82ms（目標是 95% 的請求小於 150ms，未達標）。

   • 請求延遲分佈：

     • 最短延遲：197.66ms。

     • 最長延遲：1.82 秒。

     • 第 90 百分位延遲：249.9ms（90% 的請求在此延遲以下）。

     • 第 95 百分位延遲：309.68ms（95% 的請求在此延遲以下）。

4. 功能測試結果：

   • Home Page：測試執行成功。

   • Browse Products：產品瀏覽成功。

   • Recommendations：推薦 API 成功。

   • Advertisements：失敗率較高（10 次失敗，成功率 91%）。

   • Cart Operations：購物車相關操作成功。

   • Checkout Flow：結帳流程成功。

我們就能根據結果針對這兩個情況深入分析了。

• Advertisements API 的成功率較低（91%），需要進一步排查問題。

• HTTP 請求延遲過高（95% 請求延遲為 309.68ms），需要調查後端系統的性能瓶頸。

用 Chai 重構

k6 chai.js 提供了熟悉的 describe 以及 expect ，但其實就是 group 和 check 的包裝而已。

然後把網址的 domain 改成能用變數替換。因為可能會同一個測試腳本去測試 Staging/preProduct/Product等的需求。

關於 k6 專案架構的布局與設計能參考 k6 Best practices and guidelines。畢竟測試專案其實也跟著產品專案一起開始到結束的，有良好的設計與布局，會讓後續開發與維護更為輕鬆。

import http from "k6/http";
import { sleep, group } from "k6";
import { describe, expect } from "https://jslib.k6.io/k6chaijs/4.5.0.1/index.js";
import {
  randomItem,
  randomIntBetween,
  uuidv4,
} from "https://jslib.k6.io/k6-utils/1.6.0/index.js";

const FLOOD_COUNT = __ENV.FLOOD_COUNT || 5;
const DOMAIN =  __ENV.DOMAIN || "https://otel-demo.field-eng.grafana.net"

const products = [
  "0PUK6V6EV0",
  "1YMWWN1N4O",
  "2ZYFJ3GM2N",
  "66VCHSJNUP",
  "6E92ZMYYFZ",
  "9SIQT8TOJO",
  "L9ECAV7KIM",
  "LS4PSXUNUM",
  "OLJCESPC7Z",
  "HQTGWGPNH4",
];
const people = JSON.parse(open("./people.json"));

export const options = {
  thresholds: {
    checks: ["rate>0.95"], // 成功率需超過 95%
    http_req_failed: ["rate<0.01"], // HTTP 錯誤率需低於 1%
    http_req_duration: ["p(95)<150"], // 95% 的請求需小於 150ms
  },
  scenarios: {
    checkout_flow: {
      executor: "ramping-vus",
      stages: [
        { duration: "10s", target: 20 },
        { duration: "10s", target: 50 },
      ],
    },
  },
};

export function setup() {
  return { people, products };
}

export function teardown(data) {
  console.log("Teardown phase completed.");
}

export default function (data) {
  const { people, products } = data;

  describe("[Home Page] Flood homepage requests", () => {
    floodHome();
  });

  describe("[Browse Products] Fetch product details", () => {
    browseProduct(products);
  });

  describe("[Recommendations] Get product recommendations", () => {
    getRecommendations(products);
  });

  describe("[Advertisements] Fetch advertisements", () => {
    getAds();
  });

  describe("[Cart Operations] Add to cart and view cart", () => {
    viewCart();
    const user = uuidv4();
    const product = randomItem(products);
    addToCart(product, user);
  });

  describe("[Checkout Flow] Perform checkout", () => {
    const user = uuidv4();
    const checkoutPerson = randomItem(people);
    checkout(user, checkoutPerson);
  });

  sleep(randomIntBetween(1, 5));
}

// --- Helper Functions ---

function floodHome() {
  for (let i = 0; i < FLOOD_COUNT; i++) {
    const res = http.get(DOMAIN);
    describe("Flood homepage", () => {
      expect(res.status, "response status").to.equal(200);
    });
  }
}

function browseProduct(products) {
  const product = randomItem(products);
  const res = http.get(`${DOMAIN}/api/products/${product}`);
  describe("Browse product", () => {
    expect(res.status, "response status").to.equal(200);
    expect(res).to.have.validJsonBody();
  });
}

function getRecommendations(products) {
  const params = { productIds: [randomItem(products)] };
  const res = http.get(`${DOMAIN}//api/recommendations`, { params });
  describe("Get recommendations", () => {
    expect(res.status, "response status").to.equal(200);
    expect(res).to.have.validJsonBody();
    expect(res.json().length, "Number of recommendations").to.be.above(0);
  });
}

function getAds() {
  const categories = [
    "binoculars",
    "telescopes",
    "accessories",
    "assembly",
    "travel",
    "books",
    null,
  ];
  const params = { contextKeys: [randomItem(categories)] };
  const res = http.get(`${DOMAIN}//api/data/`, { params });
  describe("Fetch advertisements", () => {
    expect(res.status, "response status").to.equal(200);
    expect(res).to.have.validJsonBody();
  });
}

function viewCart() {
  const res = http.get(`${DOMAIN}//api/cart`);
  describe("View cart", () => {
    expect(res.status, "response status").to.equal(200);
    expect(res).to.have.validJsonBody();
  });
}

function addToCart(product, user) {
  let res = http.get(`${DOMAIN}//api/products/${product}`);
  describe("Fetch product details for cart", () => {
    expect(res.status, "response status").to.equal(200);
    expect(res).to.have.validJsonBody();
  });

  const cartItem = {
    item: {
      productId: product,
      quantity: randomIntBetween(1, 10),
    },
    userId: user,
  };
  res = http.post(
    `${DOMAIN}//api/cart`,
    JSON.stringify(cartItem),
    {
      headers: { "Content-Type": "application/json" },
    }
  );
  describe("Add to cart", () => {
    expect(res.status, "response status").to.equal(200);
    expect(res).to.have.validJsonBody();
  });
}

function checkout(user, checkoutPerson) {
  const checkoutPayload = {
    userId: user,
    email: checkoutPerson.email,
    address: checkoutPerson.address,
    userCurrency: checkoutPerson.userCurrency,
    creditCard: checkoutPerson.creditCard,
  };

  const res = http.post(
    `${DOMAIN}//api/checkout`,
    JSON.stringify(checkoutPayload),
    {
      headers: { "Content-Type": "application/json" },
    }
  );
  describe("Checkout", () => {
    expect(res.status, "response status").to.equal(200);
    expect(res).to.have.validJsonBody();
  });
}

接著來加入 k6 browser 的介面流程與功能測試的部份。k6 browser 的初步介紹能參考 Grafana k6 Browser。

Test for UI Flow

能參考 k6 Browser Functional testing implements 以及網站測試指南。

程式碼實做的部份，能參考上一篇的 Example。

這裡我們先將上一篇的程式存在 functional_test.js 裡，此時專案目錄如下。

│   ├── api_test.js
│   ├── functional_test.js
│   └── people.json

然後執行一下 k6 run functional_test.js，確定能執行測試完成。

此時我們應該會發現兩隻 js 都有 options 。這時我們能抽取出來整合。使用 scenarios 透過 exec 指定測試對象名稱就能進行不同測試場景的測試。

export const options = {
    thresholds: {
      http_req_failed: ["rate<0.01"], // HTTP 錯誤率需低於 1%
      http_req_duration: ["p(95)<150"], // 95% 的請求需小於 150ms
      'browser_web_vital_lcp': ['p(90) < 1500'],
      'browser_web_vital_inp': ['p(90) < 1500'],
      //'browser_web_vital_inp{url:https://otel-demo.field-eng.grafana.net/}': ['p(90) < 1500'],
      'browser_http_req_failed': ['rate < 0.3'],
      checks: ['rate>0.95'], // 成功率需超過 95%
    },
    scenarios: {
      checkout_flow: {
        executor: "ramping-vus",
        exec: 'apiFlow',
        stages: [
          { duration: "10s", target: 20 },
          { duration: "10s", target: 50 },
        ],
      },
      ui_flow_test: {
        executor: "shared-iterations",
        exec: "ui",
        options: {
          browser: {
            type: "chromium",
          },
        },
      },
    },
  };

加上測試場景都有一些需要共享的設定與測試資料，為此在下沉整理在 config.js 中。此時我的專案目錄如下，options 以及 test lifecycle 的設定都在 main.js，作為執行腳本的進入點。共享的設定與測試資料則是能放在 config.js 中，而原本的測試單原則不動。

├── api_test.js
├── config.js
├── functional_test.js
├── main.js
├── people.json

main.js

import {apiFlow} from "./api_test.js";
import {ui} from "./functional_test.js"
import * as config from "./config.js";


export function setup() {
    const people = config.people;
    const products = config.products;
  return {  people, products };
}

export function teardown(data) {
  console.log("Teardown phase completed.");
}

export const options = {
    thresholds: {
      http_req_failed: ["rate<0.01"], // HTTP 錯誤率需低於 1%
      http_req_duration: ["p(95)<150"], // 95% 的請求需小於 150ms
      'browser_web_vital_lcp': ['p(90) < 1500'],
      'browser_web_vital_inp': ['p(90) < 1500'],
      //'browser_web_vital_inp{url:https://otel-demo.field-eng.grafana.net/}': ['p(90) < 1500'],
      'browser_http_req_failed': ['rate < 0.3'],
      checks: ['rate>0.95'], // 成功率需超過 95%
    },
    scenarios: {
      checkout_flow: {
        executor: "ramping-vus",
        exec: 'apiFlow',
        stages: [
          { duration: "10s", target: 20 },
          { duration: "10s", target: 50 },
        ],
      },
      ui_flow_test: {
        executor: "shared-iterations",
        exec: "ui",
        options: {
          browser: {
            type: "chromium",
          },
        },
      },
    },
  };

  export {apiFlow, ui };

config.js

const FLOOD_COUNT = __ENV.FLOOD_COUNT || 5;
const DOMAIN =  __ENV.DOMAIN || "https://otel-demo.field-eng.grafana.net";

const products = [
    "0PUK6V6EV0",
    "1YMWWN1N4O",
    "2ZYFJ3GM2N",
    "66VCHSJNUP",
    "6E92ZMYYFZ",
    "9SIQT8TOJO",
    "L9ECAV7KIM",
    "LS4PSXUNUM",
    "OLJCESPC7Z",
    "HQTGWGPNH4",
  ];
  const people = JSON.parse(open("./people.json"));

export {
    FLOOD_COUNT,
    DOMAIN,
    products,
    people
};

api_test.js

import http from "k6/http";
import { sleep, group } from "k6";
import { describe, expect } from "https://jslib.k6.io/k6chaijs/4.5.0.1/index.js";
import {
  randomItem,
  randomIntBetween,
  uuidv4,
} from "https://jslib.k6.io/k6-utils/1.6.0/index.js";
import * as config from "./config.js";

// 這裡從 default function 變成具名的 function
export async function apiFlow(data) {
  const { people, products } = data;

  describe("[Home Page] Flood homepage requests", () => {
    floodHome();
  });

  describe("[Browse Products] Fetch product details", () => {
    browseProduct(products);
  });

  describe("[Recommendations] Get product recommendations", () => {
    getRecommendations(products);
  });

  describe("[Advertisements] Fetch advertisements", () => {
    getAds();
  });

  describe("[Cart Operations] Add to cart and view cart", () => {
    viewCart();
    const user = uuidv4();
    const product = randomItem(products);
    addToCart(product, user);
  });

  describe("[Checkout Flow] Perform checkout", () => {
    const user = uuidv4();
    const checkoutPerson = randomItem(people);
    checkout(user, checkoutPerson);
  });

  sleep(randomIntBetween(1, 5));
}

functional_test.js

在 k6 browser 的測試中，使用 group 要注意 async 的調用，其實 group 是不能處理 promise的，所以這裡只能針對值做 check 用，而不能把 await 的 promise 調用包進去。

Avoid using group with async functions or asynchronous code. If you do, k6 might apply tags in an unreliable or unintuitive way.

為了方便區別 group 我會加上前綴 [UI] 便於區分。

import { browser } from "k6/browser";
import { check, group } from "k6";
import { sleep, fail } from "k6";
import * as config from "./config.js";

const LESS_IMPORTANT = `info`;
const TWO_SECONDS = 2000;

// 這裡從 default function 變成具名的 function
export async function ui() {
  // ... ignore
  // Homepage
  const title = await page.title();
  // group 我會加上前綴 [UI] 便於區分
  group('[UI] Homepage', function() {
    check(title, {
      "Homepage title is correct": (title) => title.includes("OTel demo"),
    });
  });
  // ... ignore
}

執行一下測試，可以正常執行，此時就看到 Summary 有 browser_ 開頭的指標以及http_ 開頭的指標，分別就是 k6 browser 以及 k6 http 產生的指標。

k6 run main.js

         /\      Grafana   /‾‾/  
    /\  /  \     |\  __   /  /   
   /  \/    \    | |/ /  /   ‾‾\ 
  /          \   |   (  |  (‾)  |
 / __________ \  |_|\_\  \_____/ 

     execution: local
        script: main.js
        output: -

     scenarios: (100.00%) 2 scenarios, 51 max VUs, 10m30s max duration (incl. graceful stop):
              * checkout_flow: Up to 50 looping VUs for 20s over 2 stages (gracefulRampDown: 30s, exec: apiFlow, gracefulStop: 30s)
              * ui_flow_test: 1 iterations shared among 1 VUs (maxDuration: 10m0s, exec: ui, gracefulStop: 30s)

INFO[0000] Result: 56                                    source=console
INFO[0004] Product page 4                                source=console
INFO[0006] Order confirmation 4                          source=console
INFO[0035] Teardown phase completed.                     source=console

     █ [Home Page] Flood homepage requests

       █ Flood homepage

         ✓ expected response status to equal 200

     █ [Browse Products] Fetch product details

       █ Browse product

         ✓ expected response status to equal 200
         ✓ has valid json body

     █ [Recommendations] Get product recommendations

       █ Get recommendations

         ✓ expected response status to equal 200
         ✓ has valid json body
         ✓ expected Number of recommendations to be above +0

     █ [Advertisements] Fetch advertisements

       █ Fetch advertisements

         ✗ expected response status to equal 200
          ↳  92% — ✓ 79 / ✗ 6
         ✓ has valid json body

     █ [Cart Operations] Add to cart and view cart

       █ View cart

         ✓ expected response status to equal 200
         ✓ has valid json body

       █ Fetch product details for cart

         ✓ expected response status to equal 200
         ✓ has valid json body

       █ Add to cart

         ✗ expected response status to equal 200
          ↳  98% — ✓ 84 / ✗ 1
         ✓ has valid json body

     █ [UI] Homepage

       ✓ Homepage title is correct

     █ [UI] Product Listing

       ✓ URL changed to products page

     █ [UI] Product Details

       ✓ URL changed to product detail page
       ✓ Product price is visible

     █ [UI] Recommended Products Check

       ✓ 4 recommended products are displayed

     █ [Checkout Flow] Perform checkout

       █ Checkout

         ✓ expected response status to equal 200
         ✓ has valid json body

     █ [UI] Shopping Cart

       ✓ URL changed to cart page

     █ [UI] Checkout

       ✓ URL changed to order checkout page

     browser_data_received..........: 12 MB  336 kB/s
     browser_data_sent..............: 163 kB 4.7 kB/s
     browser_http_req_duration......: avg=216.76ms min=53.88ms  med=209.09ms max=565.93ms p(90)=259.68ms p(95)=323.7ms 
   ✓ browser_http_req_failed........: 4.22%  3 out of 71
     browser_web_vital_cls..........: avg=0.024098 min=0.024098 med=0.024098 max=0.024098 p(90)=0.024098 p(95)=0.024098
     browser_web_vital_fcp..........: avg=532ms    min=532ms    med=532ms    max=532ms    p(90)=532ms    p(95)=532ms   
     browser_web_vital_fid..........: avg=2.7ms    min=2.7ms    med=2.7ms    max=2.7ms    p(90)=2.7ms    p(95)=2.7ms   
   ✓ browser_web_vital_inp..........: avg=48ms     min=48ms     med=48ms     max=48ms     p(90)=48ms     p(95)=48ms    
   ✓ browser_web_vital_lcp..........: avg=1.23s    min=1.23s    med=1.23s    max=1.23s    p(90)=1.23s    p(95)=1.23s   
     browser_web_vital_ttfb.........: avg=229.59ms min=229.59ms med=229.59ms max=229.59ms p(90)=229.59ms p(95)=229.59ms
   ✓ checks.........................: 99.58% 1694 out of 1701
     data_received..................: 34 MB  964 kB/s
     data_sent......................: 540 kB 15 kB/s
     group_duration.................: avg=313.78ms min=26.46µs  med=1.08ms   max=6.95s    p(90)=1.22s    p(95)=1.53s   
     http_req_blocked...............: avg=385.53µs min=130ns    med=521ns    max=99.18ms  p(90)=651ns    p(95)=766ns   
     http_req_connecting............: avg=102.33µs min=0s       med=0s       max=6.61ms   p(90)=0s       p(95)=0s      
   ✗ http_req_duration..............: avg=296.98ms min=196.71ms med=210.13ms max=4.42s    p(90)=424.98ms p(95)=496.01ms
       { expected_response:true }...: avg=296.55ms min=196.71ms med=210.13ms max=4.42s    p(90)=422.36ms p(95)=495.98ms
   ✓ http_req_failed................: 0.43%  7 out of 1615
     http_req_receiving.............: avg=29.86ms  min=16.98µs  med=47.71µs  max=293.35ms p(90)=100.55ms p(95)=246.22ms
     http_req_sending...............: avg=42.72µs  min=12.78µs  med=38.37µs  max=151.52µs p(90)=61.85µs  p(95)=73.34µs 
     http_req_tls_handshaking.......: avg=224.06µs min=0s       med=0s       max=10.6ms   p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=267.07ms min=196.63ms med=206.59ms max=4.42s    p(90)=243.28ms p(95)=292.06ms
     http_reqs......................: 1615   46.217757/s
     iteration_duration.............: avg=8.72s    min=5.02s    med=8.27s    max=14.97s   p(90)=12.66s   p(95)=13.12s  
     iterations.....................: 86     2.461131/s
     vus............................: 1      min=1            max=49
     vus_max........................: 51     min=51           max=51


running (00m34.9s), 00/51 VUs, 86 complete and 0 interrupted iterations
checkout_flow ✓ [======================================] 00/50 VUs  20s            
ui_flow_test  ✓ [======================================] 1 VUs      00m06.1s/10m0s  1/1 shared iters
ERRO[0035] thresholds on metrics 'http_req_duration' have been crossed

細化 duration threshold

本來我們的duration標準是這樣設定，這樣是全部的 API 調用都會用這標準來評判。但真實場景中不同業務的 API 其標準肯定不同。此時能用 group_duration 來分別設定。也能透過 tag 來區分，只是我覺得用 group 更加好閱讀程式。

 http_req_duration: ["p(95)<150"],

每個都能細緻的給對應的驗收標準，這樣其實才是合理的。

thresholds: {
      http_req_failed: ["rate<0.01"], // HTTP 錯誤率需低於 1%
      //http_req_duration: ["p(95)<150"], // 95% 的請求需小於 150ms
      'browser_web_vital_lcp': ['p(90) < 1500'],
      'browser_web_vital_inp': ['p(90) < 1500'],
      'browser_http_req_failed': ['rate < 0.3'],
      'group_duration{group:::[Recommendations] Get product recommendations}': ["p(95)<1000"], 
      'group_duration{group:::[Home Page] Flood homepage requests}': ["p(95)<1000"], 
      'group_duration{group:::[Browse Products] Fetch product details}': ["p(95)<500"], 
      'group_duration{group:::[Advertisements] Fetch advertisements}': ["p(95)<500"],
      'group_duration{group:::[Cart Operations] Add to cart and view cart}': ["p(95)<500"],
      'group_duration{group:::[Checkout Flow] Perform checkout}': ["p(95)<1000"],
      'group_duration{group:::[Checkout Flow] Perform checkout}': ["p(95)<1000"],
      checks: ['rate>0.95'], // 成功率需超過 95%
    }

這樣我們的測試結果也能很清楚的看見哪些細顆粒的測試場景沒滿足效能驗收標準然後討論 SLO。就能針對該場景的相關系統與調用鏈路去進行探索與優化。可觀測性驅動開發 ODD 就這樣潛移默化的在開發過程中開始了。

     group_duration................................................: avg=267.15ms min=26.81µs  med=1.08ms   max=2.48s    p(90)=1.21s    p(95)=1.38s   
     ✗ { group:::[Advertisements] Fetch advertisements }...........: avg=610.6ms  min=596.59ms med=606.99ms max=656.48ms p(90)=624.87ms p(95)=640.35ms
     ✓ { group:::[Browse Products] Fetch product details }.........: avg=209.12ms min=200.47ms med=204.51ms max=280.19ms p(90)=221.86ms p(95)=232.63ms
     ✗ { group:::[Cart Operations] Add to cart and view cart }.....: avg=1.26s    min=1.2s     med=1.23s    max=1.69s    p(90)=1.3s     p(95)=1.34s   
     ✓ { group:::[Checkout Flow] Perform checkout }................: avg=546.04ms min=430.86ms med=457.02ms max=1.79s    p(90)=791.98ms p(95)=921.49ms
     ✗ { group:::[Home Page] Flood homepage requests }.............: avg=1.76s    min=1.19s    med=1.72s    max=2.48s    p(90)=2.25s    p(95)=2.41s

k6 Group 的選擇與設計

在 k6 測試中，group 的顆粒度與設計是重要的。我們能根據業務邏輯的模組化、測試場景的獨立性、效能指標的分析需求等來規劃。因為顆粒度如果過細可能一個功能API 也是能一個 group，但這樣沒什麼意義，因為這除了能從 API 的監控上取得這樣的資訊外，過細的 group 其實無法對應到是哪個業務的測試。但過粗又無法有效定位問題。

設計上能採用這樣的設計︰

// 核心設計原則示意
group("用戶旅程層", () => {
  group("模組層", () => {
      http.get("/api/login");
  });
});

或者有些設計原則供參考︰

• 業務流程映射原則，每個group對應一個可識別的業務環節。

• 性能隔離原則，例如單獨vs批量、讀取密集vs寫入密集。

• 固定定位友好原則，依照包含外部依賴的關鍵點獨立分組

• 指標分級監控原則，為不同層級設定差異化SLO，關鍵路徑設置更嚴格標準。

graph TD
    A[需要監控獨立SLO?] -->|Yes| B[建立獨立group]
    A -->|No| C[屬於同一業務上下文?]
    C -->|Yes| D[合併到現有group]
    C -->|No| E[建立新group]
    B --> F[是否包含多個原子操作?]
    F -->|Yes| G[拆分子group]
    F -->|No| H[保持 group]

k6 輸出結果至 OpenTelemetry Collector

k6 支援蠻多種輸出方式的，但如果要即時的進行分析與探索。那最常見的應該是輸出至 InfluxDB、Promethues。但身為喜愛 OTel 的我，肯定要分享怎輸出至 OTel collector。如果輸出至 OTel collector，那麼後端長期儲存是什麼服務就不那麼重要了。

恰巧 Prometheus 在 v2.47 版本開始支援 OTel 協議，我們就能讓 Collector 寫入到 Prometheus 中。但 v3 版本的設定方式有點不太一樣，所以這次演示會以最新版本 v3.2.1 為主。

至於 k6 啟動時當然也要給參數，才知道怎寫入資料至 Collector。

k6 官網有提供文件。 這裡我以 OTel gRPC 演示。

因為是 local 的環境，所以需要K6_OTEL_GRPC_EXPORTER_INSECURE 與 K6_OTEL_GRPC_EXPORTER_ENDPOINT 來設定 OTel gRPC 的端點部份。K6_OTEL_METRIC_PREFIX 這是設定 k6 指標的前綴，方便我們快速搜尋 k6 指標。

K6_OTEL_GRPC_EXPORTER_INSECURE=true K6_OTEL_GRPC_EXPORTER_ENDPOINT=localhost:4317 K6_OTEL_METRIC_PREFIX=k6_ k6 run -o experimental-opentelemetry main.js

也能設定多個輸出，例如輸出成 JSON 檔案。有時在 CI pipeline 測試時，把結果作為 artifact 會很方便。

k6 run -o experimental-opentelemetry -o json=result.json main.js

很快就能再 Prometheus localhost:9090 中看到 k6 指標的資料了

我們取任意一個 data point 來簡單了解，這是一個 Histogram 類型的資料，OTel 會把我們設定的 group name 和 scenario name 都設定為 label 。

所以就能像這樣搜尋 k6_group_duration_milliseconds_bucket{scenario="checkout_flow", group="::[Home Page] Flood homepage requests"}。

otel-col    | HistogramDataPoints #5
otel-col    | Data point attributes:
otel-col    |      -> group: Str(::[Home Page] Flood homepage requests)
otel-col    |      -> scenario: Str(checkout_flow)
otel-col    | StartTimestamp: 2025-03-06 10:14:15.832513783 +0000 UTC
otel-col    | Timestamp: 2025-03-06 10:14:33.110112959 +0000 UTC
otel-col    | Count: 5
otel-col    | Sum: 7765.793092
otel-col    | Min: 1251.524456
otel-col    | Max: 1977.759433
otel-col    | ExplicitBounds #0: 0.000000
otel-col    | ExplicitBounds #1: 5.000000
otel-col    | ExplicitBounds #2: 10.000000
otel-col    | ExplicitBounds #3: 25.000000
otel-col    | ExplicitBounds #4: 50.000000
otel-col    | ExplicitBounds #5: 75.000000
otel-col    | ExplicitBounds #6: 100.000000
otel-col    | ExplicitBounds #7: 250.000000
otel-col    | ExplicitBounds #8: 500.000000
otel-col    | ExplicitBounds #9: 750.000000
otel-col    | ExplicitBounds #10: 1000.000000
otel-col    | Buckets #0, Count: 0
otel-col    | Buckets #1, Count: 0
otel-col    | Buckets #2, Count: 0
otel-col    | Buckets #3, Count: 0
otel-col    | Buckets #4, Count: 0
otel-col    | Buckets #5, Count: 0
otel-col    | Buckets #6, Count: 0
otel-col    | Buckets #7, Count: 0
otel-col    | Buckets #8, Count: 4
otel-col    | Buckets #9, Count: 0
otel-col    | Buckets #10, Count: 1

整合 GitLab CI Pipeline

其實本來想介紹 GitHub Action，但蠻多大大都分享過。剛好小弟公司是 GitLab 就改用 GitLab 演示。

首先呢，因為我們的測試有用到瀏覽器，所以需要多些設定才能使用。首先就是 k6 image 的選擇，雖然上一篇講過 k6 主幹已經支援 browser 測試。但 image 確有區分，要選擇-with-browser 結尾的 k6 image 才是有支援 Browser 測試功能的。

其次是我們是在容器內測試的，也不會有 UI 畫面，所以需要設定K6_BROWSER_HEADLESS。以及這樣測試需要足夠的權限所以要關閉 sandbox K6_BROWSER_ARGS: 'no-sandbox'。

因為有瀏覽器的測試我們會 screenshot 截圖做驗證，搭配剛剛提到的測試結果 result.json，就能搭配 artifacts 來保存下來。

以下是基本版本的 .gitlab-ci.yml。

stages:
  - test

k6:
  stage: test
  image: 
    name: grafana/k6:latest-with-browser
    entrypoint: ['']
  before_script:
    - mkdir -p screenshots
  variables:
    K6_BROWSER_ARGS: 'no-sandbox'
    K6_BROWSER_HEADLESS: true
  script:
    - k6 run main.js --out json=result.json
  artifacts:
    paths:
      - result.json
      - screenshots/
    access: all
    expire_in: "1 days"

因為測試其實可能是排程啟動，或是有新的版本要發佈會啟動測試，所以能搭配 rules 來使用。

客製化 xk6 整合 GitLab

還記得之前那篇透過 Grafana xk6 編譯出客製化的 k6 工具，真正使用時絕大部分都會使用客製化的 k6 image。

這裡我們除了能安裝自己寫的套件外，也能安裝 k6 官方認可的擴充套件。這裡我安裝 xk6-faker 與 xk6-kv。順便也把 XK6_HEADLESS 與 K6_BROWSER_ARGS 都設定進去，以及我們需要進行瀏覽器測試的 chromium。

這是小弟我這次演示用的客製化 k6 image。

FROM golang:1.24-bullseye AS builder
WORKDIR $GOPATH/src/go.k6.io/k6
ADD . .
RUN go install -trimpath  go.k6.io/xk6/cmd/xk6@latest

RUN xk6 build \
    --with github.com/grafana/xk6-faker@latest \
    --with github.com/oleiade/xk6-kv \
    --output "/tmp/k6"

FROM debian:bullseye

RUN apt-get update && \
    apt-get install -y chromium

WORKDIR /scripts
COPY --from=builder /tmp/k6 /usr/bin/k6
COPY . .

ENV XK6_HEADLESS=true
# no-sandbox chrome arg is required to run chrome browser in
# alpine and avoids the usage of SYS_ADMIN Docker capability
ENV K6_BROWSER_ARGS=no-sandbox

ENTRYPOINT ["k6"]
CMD []

我們也能每次執行測試都先編譯出 k6 image，然後透過這 image run k6 script。但！其實絕大部分有異動的是測試腳本，而不是 k6 image 本身，且 xk6 build 也頗費時完全沒必要每次執行測試前都編譯一次 image。剛好 GitLab 與 GitHub 其實都有提供 image registry。

所以 .gitlab-ci.yml 我們改一下。加入了 build stage。在 GitLab 網站上有教學怎麼使用 GitLab image registry。所以 build 就是透過 docker in docker 將我們客製化 xk6 編譯完成並上傳至 registry。

而 test stage 則是更改 image name 的位置就好。variables 也可以不用了，因為 dockerfile 已經是這設定了，除非我們有需求要覆蓋。

stages:
  - build
  - test

build:
  stage: build
  image: docker:28.0.1-dind
  services:
    - docker:28.0.1-dind
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  script:
    - echo "$CI_JOB_TOKEN" | docker login $CI_REGISTRY -u $CI_REGISTRY_USER --password-stdin
    - docker build -t registry.gitlab.com/demogroup7643309/k6demo .
    - docker push registry.gitlab.com/demogroup7643309/k6demo
  rules:
    - if: $CI_COMMIT_TAG =~ /^build_/
      when: always

k6:
  stage: test
  image: 
    name: registry.gitlab.com/demogroup7643309/k6demo
    entrypoint: ['']
  before_script:
    - mkdir -p screenshots
  script:
    - k6 run main.js --out json=result.json
  artifacts:
    paths:
      - result.json
      - screenshots/
    access: all
    expire_in: "1 days"
  rules:
    - if: $CI_COMMIT_TAG =~ /^test_/
      when: always

透過 Git tag 決定要執行哪個 stage。

動態更換測試腳本的變數

最常用到的應該是 DOMAIN 的變更或是一些 secret value。都能透過 GitLab CI variables 去替換。或者跟上面一樣在 .gitlab-ci.yml 的 variables 中做替換。

下圖是 k6 套用 options 的順序（相同欄位碰撞則後蓋前）︰

而 options 其實也有可能要依據環境或需求做更換。我們寫在程式中的是屬於 Defaults ，這種方式通常可以透過 k6 執行時給 flag 決定要套用對應的 options config。

├── config/
│   ├── dev.js
│   ├── pre.js
│   └── prod.js

或是採用根本很難管理跟閱讀的 CLI flag 或環境變數（小弟個人不推薦，難以版控）。

# CLI flag
k6 run \
  --thresholds "group_duration{group:::[Checkout Flow] Perform checkout}=p(95)<800" \
  --stage "0s:2,5s:5" \
  --duration 30s \
  main.js

# Environment variables
export K6_SCENARIOS_CHECKOUT_FLOW_STAGES='[{ "duration": "10s", "target": 10 }]'
export K6_THRESHOLDS_GROUP_DURATION='{"group:::[Checkout Flow] Perform checkout": ["p(95)<800"]}'
k6 run your_script.js

除了第一種方式，也能透過 --config 替換

k6 run --config k6.config.json main.js

或是以下方式讀檔案替換。

新增k6.config.json

{
"thresholds": {
    "http_req_failed": ["rate<0.01"], 
    "browser_web_vital_lcp": ["p(90) < 1500"],
    "browser_web_vital_inp": ["p(90) < 1500"],
    "browser_http_req_failed": ["rate < 0.3"],
    "group_duration{group:::[Recommendations] Get product recommendations}": ["p(95)<1000"], 
    "group_duration{group:::[Home Page] Flood homepage requests}": ["p(95)<1000"], 
    "group_duration{group:::[Browse Products] Fetch product details}": ["p(95)<500"], 
    "group_duration{group:::[Advertisements] Fetch advertisements}": ["p(95)<500"],
    "group_duration{group:::[Cart Operations] Add to cart and view cart}": ["p(95)<500"],
    "group_duration{group:::[Checkout Flow] Perform checkout}": ["p(95)<1000"],
    "checks": ["rate>0.95"]
  },
  "scenarios": {
    "checkout_flow": {
      "executor": "ramping-vus",
      "exec": "apiFlow",
      "stages": [
        { "duration": "5s", "target": 2 },
        { "duration": "5s", "target": 5 }
      ]
    },
    "ui_flow_test": {
      "executor": "shared-iterations",
      "exec": "ui",
      "options": {
        "browser": {
          "type": "chromium"
        }
      }
    }
  }
}

修改 main.js

const externalConfig = JSON.parse(open('./k6.config.json'));

export const options = {
  ...externalConfig,
};

總結

其實這是我最近工作算學習跟使用上的筆記。只是透過 OpenTelemetry Demo 演示一次。因為台灣國內蠻少有人討論實務上怎設計的，我就邊根據真實情況邊紀錄下來。

k6 難的不是怎使用這些語法與 API，而是怎設計有價值的測試場景與檢驗標準，要能和產品的 SLO 相戶呼應。

推薦文章

1. How to visualize k6 results: guidelines for choosing the right metrics

   https://grafana.com/blog/2023/04/11/how-to-visualize-load-testing-results/#key-metrics-to-visualizehttpsk6ioblogways-to-visualize-k6-resultskey-metrics-to-visualize

2. Understanding Grafana k6: A simple guide to the load testing tool

   https://grafana.com/blog/2023/08/10/understanding-grafana-k6-a-simple-guide-to-the-load-testing-tool/