11 posts tagged with "Tips & Tricks"

在写 TiDB Cloud 的 Python SDK 时，我需要实现一个等待集群创建成功的函数 wait_for_available() ，以便用户等待集群变为可用状态。这个函数需要计算集群创建所花费的时间，并在超过用户设置的超时时间时抛出异常。这是一个常见的计算时间差的场景。可以通过记录集群创建的开始时间、获取当前时间，然后计算与开始时间开始时间的差值来实现该功能。伪代码如下：

start_time = GetCurrentTime()
while True:
    current_time = GetCurrentTime()
    duration = current_time - start_time
    if duration > timeout:
        raise TimeoutError
    if cluster.status == "AVAILABLE":
        break

Scenario​

When maintaining a Git repository, particularly one for documentation, it is common to have files that haven't been updated in a while. To address this issue, you can link a section or sentence in the document to the corresponding code or an existing issue, so that the document can be updated when the code changes or the issue is resolved. This is useful when starting a new project, but it can be difficult to maintain these links for an existing large project.

To identify potential outdated files, you can use the git log command to retrieve the last commit log of each file and find long-term inactive files. The following sections describe how to generate a last commit report for a repository and how to write the script step by step.

This document outlines how to utilize code to enhance the accuracy of configuration documentation, particularly for technical writers who might not be well-versed in coding but are responsible for creating or reviewing configuration files. By following the examples in this document, writers can ensure that their configuration files are correct, efficient, and easy to understand, while also reducing the risk of errors or inconsistencies.

This document introduces how to configure Algolia DocSearch for your Docusaurus website and crawl your website automatically using GitHub Actions.

前段时间在写 tidbcloudy（一个 TiDB Cloud Python SDK）的时候用到了 Poetry 来管理依赖以及发布到 PyPI。经历了五次小版本的迭代，感觉 Poetry 时常让我有一种“写 SDK 也不难，发版也可以很简单”的错觉，所以决定记录一下我是如何使用 Poetry 来完成我的第一个 SDK 的。

Scenario​

I use webpack to bundle some JS scripts today, see openapi-scripts/pull/16. The following lists details of the project and reasons for bundling:

• The openapi-scripts repository is used to post-process the OpenAPI specification file generated by buf.build to be compatible with Redocly.
• To use JS scripts, I build a Docker image ghcr.io/oreoxmt/openapi-scripts.
• The Docker image is under rapid development, and it takes a long time to pull the latest image due to 3901 node modules.

Step 1. Install webpack-cli​

To install webpack-cli, run the following command. For more information about the directory structure, refer to Basic setup.

yarn add webpack webpack-cli --dev

npm install webpack webpack-cli --save-dev

Step 2. Configure a webpack.config.js​

For more details about configuration in webpack, see Configuration.

• Create a webpack.config.js file in the root directory of the repository.

• Then, configure the webpack.config.js file.

  info

  The entry is ./src/main.js in the repository.

  const path = require('path');
  
  module.exports = {
  entry: './src/main.js',
  output: {
      filename: 'main.js',
      path: path.resolve(__dirname, 'dist'),
  },
  };
  

• Use the preceding configuration file to generate a dist/main.js file.

  • yarn
  • npm

  npx webpack --config webpack.config.js
  

• Oops! An error occurs after executing the preceding command.

  BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
  This is no longer the case. Verify if you need this module and configure a polyfill for it.
  
  If you want to include a polyfill, you need to:
      - add a fallback 'resolve.fallback: { "http": require.resolve("stream-http") }'
      - install 'stream-http'
  If you don't want to include a polyfill, you can use an empty module like this:
      resolve.fallback: { "http": false }
  @ ./node_modules/@apidevtools/json-schema-ref-parser/lib/options.js 9:21-48
  @ ./node_modules/@apidevtools/json-schema-ref-parser/lib/normalize-args.js 3:16-36
  @ ./node_modules/@apidevtools/json-schema-ref-parser/lib/index.js 6:22-49
  @ ./src/deref.js 2:19-65
  @ ./src/main.js 3:14-35
  

• Then, I Google the error message and find the answer in Stack Overflow and Targets in webpack. To use node webpack to compile for usage, add the target: 'node' into the webpack.config.js file.

  const path = require('path');
  
  module.exports = {
  target: 'node',
  entry: './src/main.js',
  output: {
      filename: 'main.js',
      path: path.resolve(__dirname, 'dist'),
  }
  };
  

• Try to execute npx webpack --config webpack.config.js again and a warning message shows:

  $ npx webpack --config webpack.config.js
  asset main.js 484 KiB [emitted] [minimized] (name: main) 1 related asset
  ...
  WARNING in configuration
  The 'mode' option has not been set, webpack will fallback to 'production' for this value.
  Set 'mode' option to 'development' or 'production' to enable defaults for each environment.
  You can also set it to 'none' to disable any default behavior. Learn more: https://webpack.js.org/configuration/mode/
  
  webpack 5.74.0 compiled with 1 warning in 1990 ms
  

• To enables deterministic mangled names for modules and chunks, add mode: 'production' into the webpack.config.js file.

  const path = require('path');
  
  module.exports = {
  target: 'node',
  entry: './src/main.js',
  output: {
      filename: 'main.js',
      path: path.resolve(__dirname, 'dist'),
  },
  mode: 'production',
  };
  

• Try to execute npx webpack --config webpack.config.js again and it works to generate a dist/main.js and dist/main.js.LICENSE.txt.

  $ npx webpack --config webpack.config.js
  ...
  json modules 155 KiB
  modules by path ./node_modules/har-schema/lib/ 6.93 KiB 18 modules
  modules by path ./node_modules/ajv/lib/ 5.58 KiB
      ./node_modules/ajv/lib/refs/json-schema-draft-06.json 2.46 KiB
      + 2 modules
  ./node_modules/mime-db/db.json 143 KiB
  webpack 5.74.0 compiled successfully in 2524 ms
  

• To test the generated dist/main.js file, run the following command.

  $ node dist/main.js --help
  Usage: postprocess [options] [command]
  
  Postprocess an OpenAPI document for ReDoc
  
  Options:
  -V, --version                                output the version number
  -h, --help                                   display help for command
  
  Commands:
  deref <in-filename> [out-filename]           Use $RefParser to dereference a JSON schema
  importmd <in-filename> <md-folder> <gen-md>  Merge markdown files in <md-folder> to a markdown <gen-file>, and import
                                              it as $ref to JSON info.description.
  addlogo <in-filename> <url> <alt> <href>     Add a logo to JSON info.x-logo
  gencode <in-filename>                        Generate sample code to JSON as x-code-samples
  devtier <in-filenam>                         Add sample for creating a dev tier cluster
  help [command]                               display help for command
  

Step 3. Modify the Dockerfile​

Modify the Dockerfile as follows:

FROM node:16
WORKDIR /app
COPY package.json yarn.lock ./src/ .
RUN yarn install
ENTRYPOINT ["node", "/app/main.js"]

FROM node:16 AS build
WORKDIR /build
COPY . .
RUN yarn install && yarn run build

FROM node:16
COPY --from=build /build/dist/main.js ./
ENTRYPOINT ["node", "main.js"]

• The new Dockerfile contains two stages. For more information about multi-stage builds, refer to Use multi-stage builds.

• The first stage is a build stage, which tells Docker to:

  • build an image starting with the node:16 image and name it build

  • copy all files in the repository to the /build folder

    tip

    To ignore some files, you can create a soft link from .gitignore to .dockerignore using ln:

    ln -s .gitignore .dockerignore
    

  • run yarn install and yarn run build

    The yarn run build command is equivalent to npx webpack --config webpack.config.js, which is the scripts set in package.json. Besides, move dependencies into devDependencies as follows:

    • original package.json
    • modified package.json

      {
      "name": "openapi-scripts",
      ...
      "devDependencies": {
        "webpack": "^5.74.0",
        "webpack-cli": "^4.10.0"
      },
      "scripts": {},
      ...
      "dependencies": {
        "@apidevtools/json-schema-ref-parser": "^9.0.9",
        "commander": "^9.3.0",
        "openapi-snippet": "^0.14.0"
      }
    }
    

• The second stage is a result stage, which builds an image for production.

Step 4. Build the Docker image​

To build an image for testing, run the following command to build a test image:

$ docker build -t test .
[1/2] STEP 1/4: FROM node:16 AS build
[1/2] STEP 2/4: WORKDIR /build
--> 73bbc8c7552
[1/2] STEP 3/4: COPY . .
--> 53e13c02e1a
[1/2] STEP 4/4: RUN yarn install && yarn run build
yarn install v1.22.19
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
Done in 22.27s.
yarn run v1.22.19
$ webpack --config webpack.config.js
...
Done in 6.36s.
--> 5aa7f288822
[2/2] STEP 1/3: FROM node:16
[2/2] STEP 2/3: COPY --from=build /build/dist/main.js ./
--> 0c7bf448572
[2/2] STEP 3/3: ENTRYPOINT ["node", "main.js"]
[2/2] COMMIT test
--> 9dd21695569
Successfully tagged localhost/test:latest

Start a container with the test image and run the ls command to see files in container:

$ docker run -i -t --entrypoint bash 9dd2
root@007:/# ls -la
total 484
dr-xr-xr-x.   1 root   root        28 Aug  6 06:43 .
dr-xr-xr-x.   1 root   root        28 Aug  6 06:43 ..
drwxr-xr-x.   1 root   root       179 Aug  2 01:26 bin
drwxr-xr-x.   2 root   root         6 Mar 19 13:44 boot
drwxr-xr-x.   5 root   root       360 Aug  6 06:43 dev
drwxr-xr-x.   1 root   root        31 Aug  6 06:43 etc
drwxr-xr-x.   1 root   root        18 Aug  2 04:21 home
drwxr-xr-x.   1 root   root        42 Aug  2 01:26 lib
-rw-r--r--.   1 root   root    495164 Aug  7  2022 main.js
drwxr-xr-x.   2 root   root         6 Aug  1 00:00 media
drwxr-xr-x.   2 root   root         6 Aug  1 00:00 mnt
drwxr-xr-x.   1 root   root        27 Aug  2 04:24 opt
dr-xr-xr-x. 161 nobody nogroup      0 Aug  6 06:43 proc
drwx------.   1 root   root        20 Aug  2 04:24 root
drwxr-xr-x.   1 root   root        42 Aug  6 06:43 run
drwxr-xr-x.   1 root   root        20 Aug  2 01:26 sbin
drwxr-xr-x.   2 root   root         6 Aug  1 00:00 srv
dr-xr-xr-x.  12 nobody nogroup      0 Aug  5 00:14 sys
drwxrwxrwt.   1 root   root        32 Aug  2 04:24 tmp
drwxr-xr-x.   1 root   root        19 Aug  1 00:00 usr
drwxr-xr-x.   1 root   root        17 Aug  1 00:00 var

It works!

Scenario​

1. There is an API definition file open-api-swagger.json in a private repository production/api and I have no write access to it.
2. I want to deploy a preview website of the open-api-swagger.json:
   • test some Redoc configuration in my repository Oreoxmt/preview-api
   • build the open-api-swagger.json specification file into an HTML file index.html using redoc-cli build
   • deploy it on Netlify or other services

My Reading List from July 16, 2022, to July 31, 2022.

Lists​

新一期的生活小技巧分享终于更新啦，这次是社畜奥奥的打工小技巧分享。

如果你需要同时使用多个终端，又不希望两个键盘、两个鼠标来回切换，或者说你只有一个键盘、一个鼠标却想要控制多个终端的话，那这个超级低成本的小技巧就非常适合你啦！

最近在学习 Hadoop 的相关内容，厦门大学数据库实验室提供了相关内容的讲义，但是下载的时候要点击 17 次，文件默认的名称也较为繁琐，改名又需要操作 17 次。所以懒懒的奥利奥同学就开始研究有没有一些不太手动的方法，虽然这个过程花费的时间好像会更多（

如果你感兴趣的话，那就继续看下去吧。

应用场景：实现这个页面所有 PPT 的一键下载，以及后续文件名的修改

代码获取：github

Oreo 同学最近开始记笔记和写心情了，但是就是不走寻常路偏偏想整一个自己的网站来写博客。正巧之前有一个 oreo.life 的域名，很快就定下来了 Hexo + GitHub Pages + Cloudflare CDN 的方案，Oreo 自己折腾了一段时间，成功部署了现在你看到的博客并在坚持更新，可喜可贺可喜可贺。

但是有个问题，GitHub Pages 对应的 Repo 存的是渲染过后的 HTML 文件，我们总得有个地方来存原始的 Markdown 文档。于是更新博客首先需要在 _source 目录写 Markdown，然后在根目录执行 hexo generate 和 hexo deploy 更新网站。命令行操作就先不说了，还要研究什么 Node.js、npm 是不是太过分了，在 Windows 配置环境超麻烦的好吗。世界如此美好，我却还要存 node_modules，这样不好，不好。

成，好歹咱也是做过 CI 的，GitHub Actions 也出来一段时间了看起来反响不错，那肯定也可以用 Actions 完成从博客源码到 Pages 部署的完整流程。Google 一搜发现已经有很多很多人做了类似的事情了，但是总有一些小地方不够好看或者是不满足需求，那就只能自己重新造一遍轮子了。